Repository: oppia/oppia

Issue:
Add 'test' running mode to the server
(This issue is related to the dynamic feature gating project: [design doc](https://docs.google.com/document/d/1FjbG2Cb0OnDVuE36jzux2Sn0sybMQozkk3hjkfeMoC0/edit?usp=sharing))

Currently Oppia runs in either of the two modes: dev or prod, which is indicated by the `DEV_MODE` field in `assets/constants.ts`. To better support the need of QA testing, and to have a more fine-grained control of enabling features base on the mode of the server, we should add another mode 'test'.

To achieve this, we need to:
- replace boolean `DEV_MODE` with `SERVER_MODE` of enum value
- change existing references to `DEV_MODE`
- modify launching script of the server to support the new mode
- modify `_get_server_mode` method in `core/domain/platform_feature_services.py` to return `SERVER_MODES.test` when the server is running in test mode.
- add tests in `core/domain/platform_feature_services_test.py` to test the evaluation process in test environment
  - currently there are tests like `test_evaluate_dev_feature_for_prod_server_returns_false`, new tests such as `test_evaluate_dev_feature_for_test_server_returns_false`, `test_evaluate_test_feature_for_test_server_returns_true` should be added.


/cc @BenHenning @vinitamurthi 
Add 'test' running mode to the server
(This issue is related to the dynamic feature gating project: [design doc](https://docs.google.com/document/d/1FjbG2Cb0OnDVuE36jzux2Sn0sybMQozkk3hjkfeMoC0/edit?usp=sharing))

Currently Oppia runs in either of the two modes: dev or prod, which is indicated by the `DEV_MODE` field in `assets/constants.ts`. To better support the need of QA testing, and to have a more fine-grained control of enabling features base on the mode of the server, we should add another mode 'test'.

To achieve this, we need to:
- replace boolean `DEV_MODE` with `SERVER_MODE` of enum value
- change existing references to `DEV_MODE`
- modify launching script of the server to support the new mode
- modify `_get_server_mode` method in `core/domain/platform_feature_services.py` to return `SERVER_MODES.test` when the server is running in test mode.
- add tests in `core/domain/platform_feature_services_test.py` to test the evaluation process in test environment
  - currently there are tests like `test_evaluate_dev_feature_for_prod_server_returns_false`, new tests such as `test_evaluate_dev_feature_for_test_server_returns_false`, `test_evaluate_test_feature_for_test_server_returns_true` should be added.


/cc @BenHenning @vinitamurthi 


Discussion:
@MegrezZhu please update your instructions list to be comprehensive (e.g. any changes that need to happen to platform parameter files to support the new server mode).
> @MegrezZhu please update your instructions list to be comprehensive (e.g. any changes that need to happen to platform parameter files to support the new server mode).

@BenHenning Done, added descriptions for todos related to the platform parameter files.
Hi @oppia/core-maintainers, this issue is not assigned to any project. Can you please update the same? Thanks!
Closing since the feature gating project is over, so the proposed mode doesn't appear to be necessary
This is actually not complete. See [this comment thread](https://github.com/oppia/oppia/pull/15916#discussion_r953475337) -- thanks @nithinrdy for raising this issue.

Also @U8NWXD based on this I am a bit worried that you might have closed issues that are actually not complete. Would you mind taking a look again at the ones you closed and seeing if there are any that you are not 100% sure about? I can help verify if needed.

Thanks!
Also, one thought @U8NWXD: I think we should perhaps require either proof that an issue is resolved, or a link to an associated merged PR, before closing an issue as fixed -- what do you think?
@seanlip OK thanks for noting that this hasn't been completed. However, since it's been abandoned for 2 years (last update was Aug 17, 2020), was it still appropriate to close this based on our new issue-closing policy? (Just want to make sure I understand the policy.)

It's not clear to me how this issue would fix @nithinrdy's problem, but that might just be because I don't understand what's supposed to happen in the proposed "test" mode. The issue documents how to add the new mode, but not what that mode is supposed to do.

I've also updated my comment since I didn't mean to assert that this task had been successfully completed. I just meant that since the feature gating project has been effectively over for years with no work on this issue, the proposed mode didn't appear to be needed.
> Also @U8NWXD based on this I am a bit worried that you might have closed issues that are actually not complete. Would you mind taking a look again at the ones you closed and seeing if there are any that you are not 100% sure about? I can help verify if needed.

Sure. However, note that not all issues were closed because they were done. I also closed issues that I didn't expect to get to within 2 years, in line with our new triage policy. Here are the issues I closed that I'm not 100% sure about:

* [ ] https://github.com/oppia/oppia/issues/12662
* [ ] https://github.com/oppia/oppia/issues/12466
* [ ] https://github.com/oppia/oppia/issues/15299
* [ ] https://github.com/oppia/oppia/issues/12190
* [ ] https://github.com/oppia/oppia/issues/12117
* [ ] https://github.com/oppia/oppia/issues/9458
* [ ] https://github.com/oppia/oppia/issues/9251
* [ ] https://github.com/oppia/oppia/issues/9180
* [ ] https://github.com/oppia/oppia/issues/8988
* [ ] https://github.com/oppia/oppia/issues/8547
* [ ] https://github.com/oppia/oppia/issues/7806
* [ ] https://github.com/oppia/oppia/issues/7487
* [ ] https://github.com/oppia/oppia/issues/7321
* [ ] https://github.com/oppia/oppia/issues/6615
* [ ] https://github.com/oppia/oppia/issues/4122
Thanks @U8NWXD! Of the items in the list, the only two I'm not sure about are #12117 and  #12662. Let's discuss them at the dev workflow leads meeting on Friday.

For the other two points (whether it's appropriate to close the issue, and whether it fixes the problem @nithinrdy reported), let me get back to you after discussing with Ben. I suspect that we might need to sharpen the 2-year criterion a bit more and this is a good test case to think through, but Ben has more context on this issue than I do.
@BenHenning PTAL
I think this still makes sense to keep open from our new close policy, but we probably want to hand it specially. We definitely want to introduce a test mode to the server as part of implementing robust feature staging, but we haven't prioritized utilizing feature staging due to needing to dedicate resources to more important user-facing efforts. For that reason, it doesn't make sense to close since we still need to do this work, but it may be worth adding this to a team's backlog/icebox until we can dedicate time to finishing it.
@MegrezZhu please update your instructions list to be comprehensive (e.g. any changes that need to happen to platform parameter files to support the new server mode).
> @MegrezZhu please update your instructions list to be comprehensive (e.g. any changes that need to happen to platform parameter files to support the new server mode).

@BenHenning Done, added descriptions for todos related to the platform parameter files.
Hi @oppia/core-maintainers, this issue is not assigned to any project. Can you please update the same? Thanks!
Closing since the feature gating project is over, so the proposed mode doesn't appear to be necessary
This is actually not complete. See [this comment thread](https://github.com/oppia/oppia/pull/15916#discussion_r953475337) -- thanks @nithinrdy for raising this issue.

Also @U8NWXD based on this I am a bit worried that you might have closed issues that are actually not complete. Would you mind taking a look again at the ones you closed and seeing if there are any that you are not 100% sure about? I can help verify if needed.

Thanks!
Also, one thought @U8NWXD: I think we should perhaps require either proof that an issue is resolved, or a link to an associated merged PR, before closing an issue as fixed -- what do you think?
@seanlip OK thanks for noting that this hasn't been completed. However, since it's been abandoned for 2 years (last update was Aug 17, 2020), was it still appropriate to close this based on our new issue-closing policy? (Just want to make sure I understand the policy.)

It's not clear to me how this issue would fix @nithinrdy's problem, but that might just be because I don't understand what's supposed to happen in the proposed "test" mode. The issue documents how to add the new mode, but not what that mode is supposed to do.

I've also updated my comment since I didn't mean to assert that this task had been successfully completed. I just meant that since the feature gating project has been effectively over for years with no work on this issue, the proposed mode didn't appear to be needed.
> Also @U8NWXD based on this I am a bit worried that you might have closed issues that are actually not complete. Would you mind taking a look again at the ones you closed and seeing if there are any that you are not 100% sure about? I can help verify if needed.

Sure. However, note that not all issues were closed because they were done. I also closed issues that I didn't expect to get to within 2 years, in line with our new triage policy. Here are the issues I closed that I'm not 100% sure about:

* [ ] https://github.com/oppia/oppia/issues/12662
* [ ] https://github.com/oppia/oppia/issues/12466
* [ ] https://github.com/oppia/oppia/issues/15299
* [ ] https://github.com/oppia/oppia/issues/12190
* [ ] https://github.com/oppia/oppia/issues/12117
* [ ] https://github.com/oppia/oppia/issues/9458
* [ ] https://github.com/oppia/oppia/issues/9251
* [ ] https://github.com/oppia/oppia/issues/9180
* [ ] https://github.com/oppia/oppia/issues/8988
* [ ] https://github.com/oppia/oppia/issues/8547
* [ ] https://github.com/oppia/oppia/issues/7806
* [ ] https://github.com/oppia/oppia/issues/7487
* [ ] https://github.com/oppia/oppia/issues/7321
* [ ] https://github.com/oppia/oppia/issues/6615
* [ ] https://github.com/oppia/oppia/issues/4122
Thanks @U8NWXD! Of the items in the list, the only two I'm not sure about are #12117 and  #12662. Let's discuss them at the dev workflow leads meeting on Friday.

For the other two points (whether it's appropriate to close the issue, and whether it fixes the problem @nithinrdy reported), let me get back to you after discussing with Ben. I suspect that we might need to sharpen the 2-year criterion a bit more and this is a good test case to think through, but Ben has more context on this issue than I do.
@BenHenning PTAL
I think this still makes sense to keep open from our new close policy, but we probably want to hand it specially. We definitely want to introduce a test mode to the server as part of implementing robust feature staging, but we haven't prioritized utilizing feature staging due to needing to dedicate resources to more important user-facing efforts. For that reason, it doesn't make sense to close since we still need to do this work, but it may be worth adding this to a team's backlog/icebox until we can dedicate time to finishing it.

Resolving patch:
diff --git a/core/domain/platform_feature_services.py b/core/domain/platform_feature_services.py
--- a/core/domain/platform_feature_services.py
+++ b/core/domain/platform_feature_services.py
@@ -32,6 +32,7 @@
 
 from __future__ import annotations
 
+from core import feconf
 from core import platform_feature_list
 from core.constants import constants
 from core.domain import platform_parameter_domain
@@ -149,21 +150,21 @@ def update_feature_flag_rules(
         feature_name, committer_id, commit_message, new_rules)
 
 
-# TODO(#10211): Currently Oppia runs in either of the two modes:
-# dev or prod. There should be another mode 'test' added for QA testing,
-# once it is added, this function needs to be updated to take that into
-# consideration.
 def _get_server_mode() -> platform_parameter_domain.ServerMode:
     """Returns the running mode of Oppia.
 
     Returns:
-        Enum(SERVER_MODES). The server mode of Oppia, dev if Oppia is running
-        in development mode, prod if in production mode.
+        Enum(SERVER_MODES). The server mode of Oppia. This is "dev" if Oppia is
+        running in development mode, "test" if Oppia is running in production
+        mode but not on the main website, and "prod" if Oppia is running in
+        full production mode on the main website.
     """
     return (
         platform_parameter_domain.ServerMode.DEV
         if constants.DEV_MODE
         else platform_parameter_domain.ServerMode.PROD
+        if feconf.ENV_IS_OPPIA_ORG_PRODUCTION_SERVER
+        else platform_parameter_domain.ServerMode.TEST
     )
 
 
diff --git a/core/feconf.py b/core/feconf.py
--- a/core/feconf.py
+++ b/core/feconf.py
@@ -556,6 +556,10 @@ def get_empty_ratings() -> Dict[str, int]:
 OPPIA_PROJECT_ID = 'dev-project-id'
 GOOGLE_APP_ENGINE_REGION = 'us-central1'
 
+# This flag is used for distinguishing the prod/test environments for feature
+# gating.
+ENV_IS_OPPIA_ORG_PRODUCTION_SERVER = bool(OPPIA_PROJECT_ID == 'oppiaserver')
+
 # NOTE TO RELEASE COORDINATORS: Replace these GCS bucket paths with real prod
 # buckets. It's OK for them to be the same.
 DATAFLOW_TEMP_LOCATION = 'gs://todo/todo'


Test patch:
diff --git a/core/controllers/platform_feature_test.py b/core/controllers/platform_feature_test.py
--- a/core/controllers/platform_feature_test.py
+++ b/core/controllers/platform_feature_test.py
@@ -18,6 +18,7 @@
 
 import enum
 
+from core import feconf
 from core.constants import constants
 from core.domain import caching_services
 from core.domain import platform_feature_services as feature_services
@@ -273,12 +274,47 @@ def test_get_with_dummy_feature_disabled_in_dev_raises_404(self) -> None:
                 expected_status_int=404
             )
 
+    def test_get_with_dummy_feature_enabled_in_test_returns_ok(self) -> None:
+        dev_mode_ctx = self.swap(constants, 'DEV_MODE', False)
+        server_ctx = self.swap(
+            feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False)
+        dummy_feature_prod_stage_context = self._mock_dummy_feature_stage(
+            param_domain.FeatureStages.TEST)
+
+        with dev_mode_ctx, server_ctx, dummy_feature_prod_stage_context:
+            self._set_dummy_feature_status_for_mode(
+                True, param_domain.ServerMode.TEST
+            )
+
+            result = self.get_json(
+                '/platform_feature_dummy_handler',
+            )
+            self.assertEqual(result, {'msg': 'ok'})
+
+    def test_get_with_dummy_feature_disabled_in_test_raises_404(self) -> None:
+        dev_mode_ctx = self.swap(constants, 'DEV_MODE', False)
+        server_ctx = self.swap(
+            feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False)
+        dummy_feature_prod_stage_context = self._mock_dummy_feature_stage(
+            param_domain.FeatureStages.TEST)
+
+        with dev_mode_ctx, server_ctx, dummy_feature_prod_stage_context:
+            self._set_dummy_feature_status_for_mode(
+                False, param_domain.ServerMode.TEST
+            )
+            self.get_json(
+                '/platform_feature_dummy_handler',
+                expected_status_int=404
+            )
+
     def test_get_with_dummy_feature_enabled_in_prod_returns_ok(self) -> None:
         dev_mode_ctx = self.swap(constants, 'DEV_MODE', False)
+        server_ctx = self.swap(
+            feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True)
         dummy_feature_prod_stage_context = self._mock_dummy_feature_stage(
             param_domain.FeatureStages.PROD)
 
-        with dev_mode_ctx, dummy_feature_prod_stage_context:
+        with dev_mode_ctx, server_ctx, dummy_feature_prod_stage_context:
             self._set_dummy_feature_status_for_mode(
                 True, param_domain.ServerMode.PROD
             )
@@ -290,10 +326,12 @@ def test_get_with_dummy_feature_enabled_in_prod_returns_ok(self) -> None:
 
     def test_get_with_dummy_feature_disabled_in_prod_raises_404(self) -> None:
         dev_mode_ctx = self.swap(constants, 'DEV_MODE', False)
+        server_ctx = self.swap(
+            feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True)
         dummy_feature_prod_stage_context = self._mock_dummy_feature_stage(
             param_domain.FeatureStages.PROD)
 
-        with dev_mode_ctx, dummy_feature_prod_stage_context:
+        with dev_mode_ctx, server_ctx, dummy_feature_prod_stage_context:
             self._set_dummy_feature_status_for_mode(
                 False, param_domain.ServerMode.PROD
             )
diff --git a/core/domain/platform_feature_services_test.py b/core/domain/platform_feature_services_test.py
--- a/core/domain/platform_feature_services_test.py
+++ b/core/domain/platform_feature_services_test.py
@@ -20,6 +20,7 @@
 
 import enum
 
+from core import feconf
 from core import utils
 from core.constants import constants
 from core.domain import caching_services
@@ -34,6 +35,7 @@ class ParamNames(enum.Enum):
 
     FEATURE_A = 'feature_a'
     FEATURE_B = 'feature_b'
+    FEATURE_C = 'feature_c'
 
 
 ServerMode = platform_parameter_domain.ServerMode
@@ -51,8 +53,9 @@ def setUp(self) -> None:
 
         registry.Registry.parameter_registry.clear()
         # Parameter names that might be used in following tests.
-        param_names = ['feature_a', 'feature_b']
-        param_name_enums = [ParamNames.FEATURE_A, ParamNames.FEATURE_B]
+        param_names = ['feature_a', 'feature_b', 'feature_c']
+        param_name_enums = [
+            ParamNames.FEATURE_A, ParamNames.FEATURE_B, ParamNames.FEATURE_C]
         caching_services.delete_multi(
             caching_services.CACHE_NAMESPACE_PLATFORM_PARAMETER, None,
             param_names)
@@ -60,8 +63,11 @@ def setUp(self) -> None:
         self.dev_feature = registry.Registry.create_feature_flag(
             ParamNames.FEATURE_A, 'a feature in dev stage',
             FeatureStages.DEV)
+        self.test_feature = registry.Registry.create_feature_flag(
+            ParamNames.FEATURE_B, 'a feature in test stage',
+            FeatureStages.TEST)
         self.prod_feature = registry.Registry.create_feature_flag(
-            ParamNames.FEATURE_B, 'a feature in prod stage',
+            ParamNames.FEATURE_C, 'a feature in prod stage',
             FeatureStages.PROD)
         registry.Registry.update_platform_parameter(
             self.dev_feature.name, self.user_id, 'edit rules',
@@ -80,6 +86,24 @@ def setUp(self) -> None:
             ]
         )
 
+        registry.Registry.update_platform_parameter(
+            self.test_feature.name, self.user_id, 'edit rules',
+            [
+                platform_parameter_domain.PlatformParameterRule.from_dict({
+                    'filters': [
+                        {
+                            'type': 'server_mode',
+                            'conditions': [
+                                ['=', ServerMode.DEV.value],
+                                ['=', ServerMode.TEST.value],
+                            ]
+                        }
+                    ],
+                    'value_when_matched': True
+                })
+            ]
+        )
+
         registry.Registry.update_platform_parameter(
             self.prod_feature.name, self.user_id, 'edit rules',
             [
@@ -139,6 +163,7 @@ def test_create_evaluation_context_for_client_returns_correct_context(
     def test_get_all_feature_flag_dicts_returns_correct_dicts(self) -> None:
         expected_dicts = [
             self.dev_feature.to_dict(),
+            self.test_feature.to_dict(),
             self.prod_feature.to_dict(),
         ]
         self.assertEqual(
@@ -159,13 +184,37 @@ def test_get_all_feature_flag_values_in_dev_returns_correct_values(
                     context),
                 {
                     self.dev_feature.name: True,
+                    self.test_feature.name: True,
+                    self.prod_feature.name: True,
+                })
+
+    def test_get_all_feature_flag_values_in_test_returns_correct_values(
+        self
+    ) -> None:
+        constants_swap = self.swap(constants, 'DEV_MODE', False)
+        env_swap = self.swap(
+            feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False)
+        with constants_swap, env_swap:
+            context = feature_services.create_evaluation_context_for_client({
+                'platform_type': 'Android',
+                'browser_type': None,
+                'app_version': '1.0.0',
+            })
+            self.assertEqual(
+                feature_services.evaluate_all_feature_flag_values_for_client(
+                    context),
+                {
+                    self.dev_feature.name: False,
+                    self.test_feature.name: True,
                     self.prod_feature.name: True,
                 })
 
     def test_get_all_feature_flag_values_in_prod_returns_correct_values(
         self
     ) -> None:
-        with self.swap(constants, 'DEV_MODE', False):
+        constants_swap = self.swap(constants, 'DEV_MODE', False)
+        env_swap = self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True)
+        with constants_swap, env_swap:
             context = feature_services.create_evaluation_context_for_client({
                 'platform_type': 'Android',
                 'browser_type': None,
@@ -176,6 +225,7 @@ def test_get_all_feature_flag_values_in_prod_returns_correct_values(
                     context),
                 {
                     self.dev_feature.name: False,
+                    self.test_feature.name: False,
                     self.prod_feature.name: True,
                 })
 
@@ -184,20 +234,51 @@ def test_evaluate_dev_feature_for_dev_server_returns_true(self) -> None:
             self.assertTrue(
                 feature_services.is_feature_enabled(self.dev_feature.name))
 
+    def test_evaluate_test_feature_for_dev_server_returns_true(self) -> None:
+        with self.swap(constants, 'DEV_MODE', True):
+            self.assertTrue(
+                feature_services.is_feature_enabled(self.test_feature.name))
+
     def test_evaluate_prod_feature_for_dev_server_returns_true(self) -> None:
         with self.swap(constants, 'DEV_MODE', True):
             self.assertTrue(
                 feature_services.is_feature_enabled(self.prod_feature.name))
 
+    def test_evaluate_dev_feature_for_test_server_returns_false(self) -> None:
+        with self.swap(constants, 'DEV_MODE', False):
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False):
+                self.assertFalse(
+                    feature_services.is_feature_enabled(self.dev_feature.name))
+
+    def test_evaluate_test_feature_for_test_server_returns_true(self) -> None:
+        with self.swap(constants, 'DEV_MODE', False):
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False):
+                self.assertTrue(
+                    feature_services.is_feature_enabled(self.test_feature.name))
+
+    def test_evaluate_prod_feature_for_test_server_returns_true(self) -> None:
+        with self.swap(constants, 'DEV_MODE', False):
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', False):
+                self.assertTrue(
+                    feature_services.is_feature_enabled(self.prod_feature.name))
+
     def test_evaluate_dev_feature_for_prod_server_returns_false(self) -> None:
         with self.swap(constants, 'DEV_MODE', False):
-            self.assertFalse(
-                feature_services.is_feature_enabled(self.dev_feature.name))
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True):
+                self.assertFalse(
+                    feature_services.is_feature_enabled(self.dev_feature.name))
+
+    def test_evaluate_test_feature_for_prod_server_returns_false(self) -> None:
+        with self.swap(constants, 'DEV_MODE', False):
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True):
+                self.assertFalse(
+                    feature_services.is_feature_enabled(self.test_feature.name))
 
     def test_evaluate_prod_feature_for_prod_server_returns_true(self) -> None:
         with self.swap(constants, 'DEV_MODE', False):
-            self.assertTrue(
-                feature_services.is_feature_enabled(self.prod_feature.name))
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True):
+                self.assertTrue(
+                    feature_services.is_feature_enabled(self.prod_feature.name))
 
     def test_evaluate_feature_for_prod_server_matches_to_backend_filter(
         self
@@ -225,8 +306,9 @@ def test_evaluate_feature_for_prod_server_matches_to_backend_filter(
             ]
         )
         with self.swap(constants, 'DEV_MODE', False):
-            self.assertTrue(
-                feature_services.is_feature_enabled(self.prod_feature.name))
+            with self.swap(feconf, 'ENV_IS_OPPIA_ORG_PRODUCTION_SERVER', True):
+                self.assertTrue(
+                    feature_services.is_feature_enabled(self.prod_feature.name))
 
     def test_get_feature_flag_values_with_unknown_name_raises_error(
         self
diff --git a/core/tests/webdriverio/accessibility.js b/core/tests/webdriverio/accessibility.js
--- a/core/tests/webdriverio/accessibility.js
+++ b/core/tests/webdriverio/accessibility.js
@@ -23,6 +23,8 @@ var waitFor = require('../webdriverio_utils/waitFor.js');
 var users = require('../webdriverio_utils/users.js');
 var workflow = require('../webdriverio_utils/workflow.js');
 
+var ExplorationPlayerPage = require(
+  '../webdriverio_utils/ExplorationPlayerPage.js');
 var LibraryPage = require('../webdriverio_utils/LibraryPage.js');
 
 var EXPLORATION = {
@@ -53,6 +55,7 @@ describe('screenreader and keyboard user accessibility features', function() {
     await users.createAndLoginSuperAdminUser(
       'user11@accessibility.com', 'user11accessibility');
     libraryPage = new LibraryPage.LibraryPage();
+    explorationPlayerPage = new ExplorationPlayerPage.ExplorationPlayerPage();
   });
 
   afterAll(async function() {
@@ -492,6 +495,9 @@ describe('screenreader and keyboard user accessibility features', function() {
       await libraryPage.findExploration('A new exploration');
       await libraryPage.playExploration('A new exploration');
 
+      // This forces a wait until the exploration card fully loads on the page.
+      await explorationPlayerPage.expectExplorationToNotBeOver();
+
       // Should test the skip to main content shortcut.
       var skipLink = $('.e2e-test-skip-link');
       await checkActionShortcuts('s', skipLink);
diff --git a/core/tests/webdriverio/subscriptionsFlow.js b/core/tests/webdriverio/subscriptionsFlow.js
--- a/core/tests/webdriverio/subscriptionsFlow.js
+++ b/core/tests/webdriverio/subscriptionsFlow.js
@@ -51,10 +51,10 @@ describe('Subscriptions functionality', function() {
     await users.login('learner1@subscriptions.com');
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creator1Idsubscriptions');
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creator2Idsubscriptions');
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await preferencesPage.get();
     await preferencesPage.expectDisplayedFirstSubscriptionToBe('creator...');
     await preferencesPage.expectDisplayedLastSubscriptionToBe('creator...');
@@ -67,13 +67,13 @@ describe('Subscriptions functionality', function() {
     await users.login('learner2@subscriptions.com');
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creator1Idsubscriptions');
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creator2Idsubscriptions');
 
     // Subscribe and then unsubscribe from the same user.
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await preferencesPage.get();
     await preferencesPage.expectSubscriptionCountToEqual(1);
     await preferencesPage.expectDisplayedFirstSubscriptionToBe('creator...');
diff --git a/core/tests/webdriverio_desktop/checkpointFeatures.js b/core/tests/webdriverio_desktop/checkpointFeatures.js
--- a/core/tests/webdriverio_desktop/checkpointFeatures.js
+++ b/core/tests/webdriverio_desktop/checkpointFeatures.js
@@ -95,7 +95,7 @@ describe('Checkpoints functionality', function() {
     await adminPage.getFeaturesTab();
     var checkpointCelebrationFlag = (
       await adminPage.getCheckpointCelebrationFeatureElement());
-    await adminPage.enableFeatureForProd(checkpointCelebrationFlag);
+    await adminPage.enableFeatureForTest(checkpointCelebrationFlag);
 
     await createDummyExploration();
     var handle = await browser.getWindowHandle();
@@ -174,6 +174,8 @@ describe('Checkpoints functionality', function() {
     async function() {
       await topicAndStoryViewerPage.get(
         'math', 'topic-cf-one', 'checkpointfeaturesstory');
+      await topicAndStoryViewerPage.expectCompletedLessonCountToBe(0);
+
       await topicAndStoryViewerPage.goToChapterIndex(0);
       await explorationPlayerPage.submitAnswer('Continue', null);
       await explorationPlayerPage.dismissLessonInfoTooltip();
diff --git a/core/tests/webdriverio_desktop/creatorDashboard.js b/core/tests/webdriverio_desktop/creatorDashboard.js
--- a/core/tests/webdriverio_desktop/creatorDashboard.js
+++ b/core/tests/webdriverio_desktop/creatorDashboard.js
@@ -83,7 +83,7 @@ describe('Creator dashboard functionality', function() {
     await users.login('user2@creatorDashboard.com');
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creatorDashboardOwner');
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await libraryPage.get();
     await libraryPage.findExploration(EXPLORATION_TITLE_1);
     await libraryPage.playExploration(EXPLORATION_TITLE_1);
@@ -109,7 +109,7 @@ describe('Creator dashboard functionality', function() {
     await users.login('user4@creatorDashboard.com');
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(
       'creatorDashboardOwner');
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await libraryPage.get();
     await libraryPage.findExploration(EXPLORATION_TITLE_2);
     await libraryPage.playExploration(EXPLORATION_TITLE_2);
diff --git a/core/tests/webdriverio_desktop/learnerDashboard.js b/core/tests/webdriverio_desktop/learnerDashboard.js
--- a/core/tests/webdriverio_desktop/learnerDashboard.js
+++ b/core/tests/webdriverio_desktop/learnerDashboard.js
@@ -113,9 +113,9 @@ describe('Learner dashboard functionality', function() {
     await users.login('learner1@learnerDashboard.com');
     // Subscribe to both the creators.
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(creator1Id);
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
     await subscriptionDashboardPage.navigateToUserSubscriptionPage(creator2Id);
-    await subscriptionDashboardPage.navigateToSubscriptionButton();
+    await subscriptionDashboardPage.clickSubscribeButton();
 
     // Completing exploration 'Activations' to activate /learner_dashboard.
     await libraryPage.get();
diff --git a/core/tests/webdriverio_desktop/navigation.js b/core/tests/webdriverio_desktop/navigation.js
--- a/core/tests/webdriverio_desktop/navigation.js
+++ b/core/tests/webdriverio_desktop/navigation.js
@@ -167,28 +167,29 @@ describe('Static Pages Tour', function() {
   it('should visit the Teach page', async function() {
     await browser.url('/teach');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-teach-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-teach-page'), 'Teach page takes too long to appear');
   });
 
   it('should visit the Home page', async function() {
     await browser.url('/');
     await waitFor.pageToFullyLoad();
-    var splashPage = $('.e2e-test-splash-page');
     await waitFor.visibilityOf(
-      splashPage, 'Splash page takes too long to appear');
-    expect(await splashPage.isExisting()).toBe(true);
+      $('.e2e-test-splash-page'), 'Splash page takes too long to appear');
   });
 
   it('should visit the About page', async function() {
     await browser.url('/about');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-about-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-about-page'), 'About page takes too long to appear');
   });
 
   it('should visit the Contact page', async function() {
     await browser.url('/contact');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-contact-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-contact-page'), 'Contact page takes too long to appear');
   });
 
   it('should visit the Donate page', async function() {
@@ -204,31 +205,38 @@ describe('Static Pages Tour', function() {
   it('should visit the Partnerships page', async function() {
     await browser.url('/partnerships');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-partnerships-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-partnerships-page'),
+      'Partnerships page takes too long to appear');
   });
 
   it('should visit the About the Oppia Foundation page', async function() {
     await browser.url('/about-foundation');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-about-foundation-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-about-foundation-page'),
+      'About Foundation page takes too long to appear');
   });
 
   it('should visit the Privacy page', async function() {
     await browser.url('/privacy-policy');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-privacy-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-privacy-page'), 'Privacy page takes too long to appear');
   });
 
   it('should visit the Terms page', async function() {
     await browser.url('/terms');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-terms-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-terms-page'), 'Terms page takes too long to appear');
   });
 
   it('should visit the Thanks page', async function() {
     await browser.url('/thanks');
     await waitFor.pageToFullyLoad();
-    expect(await $('.e2e-test-thanks-page').isExisting()).toBe(true);
+    await waitFor.visibilityOf(
+      $('.e2e-test-thanks-page'), 'Terms page takes too long to appear');
   });
 
   it('should visit the Volunteer page', async function() {
@@ -253,6 +261,12 @@ describe('Error reporting', function() {
     await users.createUser('lorem@preferences.com', 'loremPreferences');
     await users.login('lorem@preferences.com');
     await preferencesPage.get();
+    // This delay is needed so that the page gets a chance to fully load before
+    // cookies are deleted. The page makes a backend request for feature flags
+    // and the additional pause avoids that being recorded as an error due to
+    // the user no longer being logged in.
+    // eslint-disable-next-line oppia/e2e-practices
+    await browser.pause(2000);
     // Deleting the cookies simulates an expired session error.
     await browser.deleteCookies();
     await browser.setupInterceptor();
diff --git a/core/tests/webdriverio_desktop/topicAndStoryViewer.js b/core/tests/webdriverio_desktop/topicAndStoryViewer.js
--- a/core/tests/webdriverio_desktop/topicAndStoryViewer.js
+++ b/core/tests/webdriverio_desktop/topicAndStoryViewer.js
@@ -95,7 +95,7 @@ describe('Topic and Story viewer functionality', function() {
     await adminPage.getFeaturesTab();
     var endChapterFlag = (
       await adminPage.getEndChapterCelebrationFeatureElement());
-    await adminPage.enableFeatureForProd(endChapterFlag);
+    await adminPage.enableFeatureForTest(endChapterFlag);
 
     await createDummyExplorations();
     var handle = await browser.getWindowHandle();
diff --git a/core/tests/webdriverio_utils/AdminPage.js b/core/tests/webdriverio_utils/AdminPage.js
--- a/core/tests/webdriverio_utils/AdminPage.js
+++ b/core/tests/webdriverio_utils/AdminPage.js
@@ -232,9 +232,9 @@ var AdminPage = function() {
   };
 
   // This function is meant to be used to enable a feature gated behind
-  // a feature flag in prod mode, which is the server environment the E2E
+  // a feature flag in test mode, which is the server environment the E2E
   // tests are run in.
-  this.enableFeatureForProd = async function(featureElement) {
+  this.enableFeatureForTest = async function(featureElement) {
     await this.removeAllRulesOfFeature(featureElement);
 
     await action.click(
@@ -261,7 +261,7 @@ var AdminPage = function() {
       'Value Selector takes too long to appear'
     );
     await (featureElement.$(serverModeSelectorLocator)).selectByVisibleText(
-      'prod');
+      'test');
 
     await this.saveChangeOfFeature(featureElement);
   };
diff --git a/core/tests/webdriverio_utils/ClassroomPage.js b/core/tests/webdriverio_utils/ClassroomPage.js
--- a/core/tests/webdriverio_utils/ClassroomPage.js
+++ b/core/tests/webdriverio_utils/ClassroomPage.js
@@ -23,23 +23,21 @@ var action = require('./action.js');
 var ClassroomPage = function() {
   var topicSummaryTile = $('.e2e-test-topic-summary-tile');
   var launchDiagnosticTestPageButton = $('.e2e-test-take-diagnostic-test');
-  var topicSummaryTilesSelector = function() {
-    return $$('.e2e-test-topic-summary-tile');
-  };
 
   this.get = async function(classroomName) {
     await browser.url('/learn/' + classroomName);
     await waitFor.pageToFullyLoad();
   };
 
-  this.expectNumberOfTopicsToBe = async function(count) {
-    var topicSummaryTiles = await topicSummaryTilesSelector();
-    if (count > 0) {
+  this.expectNumberOfTopicsToBe = async function(expectedCount) {
+    if (expectedCount > 0) {
       await waitFor.visibilityOf(
         topicSummaryTile, 'Topic summary tile is not visible');
-      expect(topicSummaryTiles.length).toEqual(count);
+      let actualCount = await $$('.e2e-test-topic-summary-tile').length;
+      expect(actualCount).toEqual(expectedCount);
     } else {
-      expect(topicSummaryTiles.length).toEqual(0);
+      let actualCount = await $$('.e2e-test-topic-summary-tile').length;
+      expect(actualCount).toEqual(0);
     }
   };
 
diff --git a/core/tests/webdriverio_utils/LibraryPage.js b/core/tests/webdriverio_utils/LibraryPage.js
--- a/core/tests/webdriverio_utils/LibraryPage.js
+++ b/core/tests/webdriverio_utils/LibraryPage.js
@@ -101,6 +101,8 @@ var LibraryPage = function() {
   this.get = async function() {
     await browser.url(LIBRARY_URL_SUFFIX);
     await waitFor.pageToFullyLoad();
+    // Click on a neutral element.
+    await action.click('Main header', mainHeader);
   };
 
   this.getHomePage = async function() {
diff --git a/core/tests/webdriverio_utils/PreferencesPage.js b/core/tests/webdriverio_utils/PreferencesPage.js
--- a/core/tests/webdriverio_utils/PreferencesPage.js
+++ b/core/tests/webdriverio_utils/PreferencesPage.js
@@ -189,9 +189,13 @@ var PreferencesPage = function() {
     expect(await audioLanguageSelector.getText()).not.toEqual(language);
   };
 
-  this.expectSubscriptionCountToEqual = async function(value) {
-    var subscriptions = await subscriptionsSelector();
-    expect(await subscriptions.length).toEqual(value);
+  this.expectSubscriptionCountToEqual = async function(expectedCount) {
+    if (expectedCount > 0) {
+      await waitFor.visibilityOf(
+        subscription, 'Subscription tile is not visible');
+    }
+    let actualCount = await subscriptionsSelector().length;
+    expect(actualCount).toEqual(expectedCount);
   };
 
   this.expectUserBioToBe = async function(bio) {
diff --git a/core/tests/webdriverio_utils/StoryEditorPage.js b/core/tests/webdriverio_utils/StoryEditorPage.js
--- a/core/tests/webdriverio_utils/StoryEditorPage.js
+++ b/core/tests/webdriverio_utils/StoryEditorPage.js
@@ -152,6 +152,9 @@ var StoryEditorPage = function() {
     } else {
       await action.click('Publish Story Button', publishStoryButton);
     }
+
+    await waitFor.visibilityOf(
+      unpublishStoryButton, 'Story is taking too long to get published.');
   };
 
   this.unpublishStory = async function() {
diff --git a/core/tests/webdriverio_utils/SubscriptionDashboardPage.js b/core/tests/webdriverio_utils/SubscriptionDashboardPage.js
--- a/core/tests/webdriverio_utils/SubscriptionDashboardPage.js
+++ b/core/tests/webdriverio_utils/SubscriptionDashboardPage.js
@@ -58,7 +58,7 @@ var SubscriptionDashboardPage = function() {
     expect(subscriptionName.length).toEqual(value);
   };
 
-  this.navigateToSubscriptionButton = async function() {
+  this.clickSubscribeButton = async function() {
     await waitFor.elementToBeClickable(
       subscriptionButton, 'Subscription button is not clickable');
     var subscribeButtonStatusBeforeClick = await subscriptionButton.getText();
diff --git a/core/tests/webdriverio_utils/waitFor.js b/core/tests/webdriverio_utils/waitFor.js
--- a/core/tests/webdriverio_utils/waitFor.js
+++ b/core/tests/webdriverio_utils/waitFor.js
@@ -78,14 +78,20 @@ var invisibilityOf = async function(element, errorMessage) {
 
 /**
  * Consider adding this method after each browser.url() call.
+ * However, note that it does not guarantee that the page is fully loaded. It
+ * is advised to add additional page-specific checks to each page's get()
+ * method that verify the visibility of elements that are guaranteed to be
+ * on that page.
  */
 var pageToFullyLoad = async function() {
   var loadingMessage = await $('.e2e-test-loading-fullpage');
+  // Wait for the message to disappear.
   await loadingMessage.waitForDisplayed({
     timeout: 15000,
     reverse: true,
-    timeoutMsg: 'Pages takes more than 15 sec to load\n' +
-    new Error().stack + '\n'
+    timeoutMsg: (
+      'Loading message takes more than 15 sec to disappear\n' +
+      new Error().stack + '\n')
   });
 };