Repository: oppia/oppia

Issue:
[Backend Flake] json.decoder.JSONDecodeError
## Example 1:

Log:

```
 Tasks still running:
  setup_test (started 15:30:51)
  scripts.start_test (started 15:30:51)
----------------------------------------
Coverage data for scripts.start_test is in .coverage.fv-az993-701.7670.xsemhdrebgrgwhzl
15:30:55 FINISHED setup_test: 3.3 secs
----------------------------------------
Coverage data for scripts.run_lighthouse_tests_test is in .coverage.fv-az993-701.7670.xkztrnceoumokplu
15:31:04 FINISHED scripts.run_lighthouse_tests_test: 8.8 secs
Coverage data for scripts.run_backend_tests_test is in .coverage.fv-az993-701.7670.qrjtmerwhysxfuub
...
Traceback (most recent call last):
  File "/home/runner/work/oppia/oppia/core/tests/gae_suite.py", line 36, in <module>
    from scripts import common # isort:skip  pylint: disable=wrong-import-position, wrong-import-order
  File "/home/runner/work/oppia/oppia/scripts/common.py", line 36, in <module>
    from core import constants
  File "/home/runner/work/oppia/oppia/core/constants.py", line 146, in <module>
    constants = Constants(parse_json_from_ts(  # pylint:disable=invalid-name
  File "/home/runner/work/oppia/oppia/core/constants.py", line 49, in parse_json_from_ts
    json.loads(text_without_comments[json_start:json_end]))
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 5960 column 4 (char 136941)

15:31:04 ERROR scripts.run_backend_tests_test: 0.2 secs
...
 Tasks still running:
  scripts.start_test (started 15:30:51)
  extensions.issues.CyclicStateTransitions.CyclicStateTransitions_test (started 15:31:10)
```

The `scripts/start_test.py` test is running both before and after the `scripts/run_backend_tests_test.py` test fails, and it modifies the `assets/constants.ts` file that the `JSONDecodeError` fails on:

https://github.com/oppia/oppia/blob/80b51354b48da67fdf58813b37d188c97fe7dd24/scripts/start.py#L162

https://github.com/oppia/oppia/blob/80b51354b48da67fdf58813b37d188c97fe7dd24/scripts/build.py#L263-L265

This looks like a possible race condition between `start.py` modifying `constants.ts` and `run_backend_test_test.py` importing `scripts/common.py`, which requires loading `assets/constants.ts`.

## Example 2:

Log:

```
 Tasks still running:
  scripts.run_e2e_tests_test (started 16:27:09)
  core.domain.question_fetchers_test (started 16:31:40)
...
 Coverage data for core.domain.learner_group_services_test is in .coverage.fv-az491-144.7677.tulgsbykuthdzjdx
Error 1
Traceback (most recent call last):
  File "/home/runner/work/oppia/oppia/core/tests/gae_suite.py", line 36, in <module>
    from scripts import common # isort:skip  pylint: disable=wrong-import-position, wrong-import-order
  File "/home/runner/work/oppia/oppia/scripts/common.py", line 36, in <module>
    from core import constants
  File "/home/runner/work/oppia/oppia/core/constants.py", line 146, in <module>
    constants = Constants(parse_json_from_ts(  # pylint:disable=invalid-name
  File "/home/runner/work/oppia/oppia/core/constants.py", line 49, in parse_json_from_ts
    json.loads(text_without_comments[json_start:json_end]))
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/hostedtoolcache/Python/3.8.15/x64/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 280 column 17 (char 12110)

16:31:53 ERROR core.domain.learner_group_services_test: 0.2 secs
```

The `scripts.run_e2e_tests.py` script modifies `assets/constants.ts` too:

https://github.com/oppia/oppia/blob/80b51354b48da67fdf58813b37d188c97fe7dd24/scripts/run_e2e_tests.py#L200

Therefore this case can also be explained by a race condition between modifying `assets/constants.ts` and reading it.

## More Examples

* 2023-03-11: https://github.com/oppia/oppia/actions/runs/4392342798/jobs/7692058695#step:8:1526
* 2023-03-19: https://github.com/oppia/oppia/actions/runs/4462691657/jobs/7837411133#step:8:1267

## Solution

Any backend test whose code under test modifies files like `assets/constants.ts` must mock those modification calls to ensure that the file doesn't actually get modified. This is essential for any files that get used widely because backend tests are run in parallel, so the backend test may be running concurrently with any other backend test.

We can update `GenericTestBase.setUp()` to mock `build.modify_constants()` and throw an error whenever the mock function gets called. Not sure whether this will work though since it's unclear whether you can mock a function twice.

https://github.com/oppia/oppia/blob/80b51354b48da67fdf58813b37d188c97fe7dd24/core/tests/test_utils.py#L2319


Discussion:
I can take this.

Resolving patch:
diff --git a/scripts/build.py b/scripts/build.py
--- a/scripts/build.py
+++ b/scripts/build.py
@@ -220,51 +220,6 @@ def generate_app_yaml(deploy_mode: bool = False) -> None:
         prod_yaml_file.write(content)
 
 
-def modify_constants(
-    prod_env: bool = False,
-    emulator_mode: bool = True,
-    maintenance_mode: bool = False
-) -> None:
-    """Modify constants.ts and feconf.py.
-
-    Args:
-        prod_env: bool. Whether the server is started in prod mode.
-        emulator_mode: bool. Whether the server is started in emulator mode.
-        maintenance_mode: bool. Whether the site should be put into
-            the maintenance mode.
-    """
-    dev_mode_variable = (
-        '"DEV_MODE": false' if prod_env else '"DEV_MODE": true')
-    common.inplace_replace_file(
-        common.CONSTANTS_FILE_PATH,
-        r'"DEV_MODE": (true|false)',
-        dev_mode_variable,
-        expected_number_of_replacements=1
-    )
-    emulator_mode_variable = (
-        '"EMULATOR_MODE": true' if emulator_mode else '"EMULATOR_MODE": false')
-    common.inplace_replace_file(
-        common.CONSTANTS_FILE_PATH,
-        r'"EMULATOR_MODE": (true|false)',
-        emulator_mode_variable,
-        expected_number_of_replacements=1
-    )
-
-    enable_maintenance_mode_variable = (
-        'ENABLE_MAINTENANCE_MODE = %s' % str(maintenance_mode))
-    common.inplace_replace_file(
-        common.FECONF_PATH,
-        r'ENABLE_MAINTENANCE_MODE = (True|False)',
-        enable_maintenance_mode_variable,
-        expected_number_of_replacements=1
-    )
-
-
-def set_constants_to_default() -> None:
-    """Set variables in constants.ts and feconf.py to default values."""
-    modify_constants(prod_env=False, emulator_mode=True, maintenance_mode=False)
-
-
 def _minify_css(source_path: str, target_path: str) -> None:
     """Runs the given file through a minifier and outputs it to target_path.
 
@@ -1425,7 +1380,7 @@ def main(args: Optional[Sequence[str]] = None) -> None:
                 'minify_third_party_libs_only should not be '
                 'set in non-prod env.')
 
-    modify_constants(
+    common.modify_constants(
         prod_env=options.prod_env,
         emulator_mode=not options.deploy_mode,
         maintenance_mode=options.maintenance_mode)
diff --git a/scripts/common.py b/scripts/common.py
--- a/scripts/common.py
+++ b/scripts/common.py
@@ -912,3 +912,48 @@ def run_ng_compilation() -> None:
     if not os.path.isdir(ng_bundles_dir_name):
         print('Failed to complete ng build compilation, exiting...')
         sys.exit(1)
+
+
+def set_constants_to_default() -> None:
+    """Set variables in constants.ts and feconf.py to default values."""
+    modify_constants(prod_env=False, emulator_mode=True, maintenance_mode=False)
+
+
+def modify_constants(
+    prod_env: bool = False,
+    emulator_mode: bool = True,
+    maintenance_mode: bool = False
+) -> None:
+    """Modify constants.ts and feconf.py.
+
+    Args:
+        prod_env: bool. Whether the server is started in prod mode.
+        emulator_mode: bool. Whether the server is started in emulator mode.
+        maintenance_mode: bool. Whether the site should be put into
+            the maintenance mode.
+    """
+    dev_mode_variable = (
+        '"DEV_MODE": false' if prod_env else '"DEV_MODE": true')
+    inplace_replace_file(
+        CONSTANTS_FILE_PATH,
+        r'"DEV_MODE": (true|false)',
+        dev_mode_variable,
+        expected_number_of_replacements=1
+    )
+    emulator_mode_variable = (
+        '"EMULATOR_MODE": true' if emulator_mode else '"EMULATOR_MODE": false')
+    inplace_replace_file(
+        CONSTANTS_FILE_PATH,
+        r'"EMULATOR_MODE": (true|false)',
+        emulator_mode_variable,
+        expected_number_of_replacements=1
+    )
+
+    enable_maintenance_mode_variable = (
+        'ENABLE_MAINTENANCE_MODE = %s' % str(maintenance_mode))
+    inplace_replace_file(
+        FECONF_PATH,
+        r'ENABLE_MAINTENANCE_MODE = (True|False)',
+        enable_maintenance_mode_variable,
+        expected_number_of_replacements=1
+    )
diff --git a/scripts/start.py b/scripts/start.py
--- a/scripts/start.py
+++ b/scripts/start.py
@@ -153,7 +153,7 @@ def main(args: Optional[Sequence[str]] = None) -> None:
         if parsed_args.source_maps:
             build_args.append('--source_maps')
         build.main(args=build_args)
-        stack.callback(build.set_constants_to_default)
+        stack.callback(common.set_constants_to_default)
 
         stack.enter_context(servers.managed_redis_server())
         stack.enter_context(servers.managed_elasticsearch_dev_server())


Test patch:
diff --git a/core/tests/test_utils.py b/core/tests/test_utils.py
--- a/core/tests/test_utils.py
+++ b/core/tests/test_utils.py
@@ -74,6 +74,7 @@
 from core.platform.taskqueue import cloud_tasks_emulator
 import main
 from proto_files import text_classifier_pb2
+from scripts import common
 
 import elasticsearch
 import requests_mock
@@ -1848,11 +1849,20 @@ def setUp(self) -> None:
         storage_services.CLIENT.namespace = self.id()
         # Set up apps for testing.
         self.testapp = webtest.TestApp(main.app_without_context)
+        # Mock set_constans_to_default method to throw an exception.
+        # Don't directly change constants file in the test.
+        # Mock this method again in your test.
+        self.contextManager = self.swap(
+            common, 'set_constants_to_default',
+            self.mock_set_constants_to_default)
+        self.contextManager.__enter__()
 
     def tearDown(self) -> None:
         datastore_services.delete_multi(
             list(datastore_services.query_everything().iter(keys_only=True)))
         storage_services.CLIENT.reset()
+        if hasattr(self, 'contextManager'):
+            self.contextManager.__exit__(None, None, None)
         super().tearDown()
 
     def run(self, result: Optional[unittest.TestResult] = None) -> None:
@@ -1922,6 +1932,13 @@ def get_pending_tasks(
         return self._platform_taskqueue_services_stub.get_pending_tasks(
             queue_name=queue_name)
 
+    def mock_set_constants_to_default(self) -> None:
+        """Change constants file in the test could lead to other
+        tests fail. Mock set_constants_to_default method in your test
+        will suppress this exception.
+        """
+        raise Exception('Please mock this method in the test.')
+
 
 class GenericTestBase(AppEngineTestBase):
     """Base test class with common/generic helper methods.
diff --git a/scripts/build_test.py b/scripts/build_test.py
--- a/scripts/build_test.py
+++ b/scripts/build_test.py
@@ -775,103 +775,6 @@ def test_generate_app_yaml_with_deploy_mode_with_nonexistent_var_raises(
         app_yaml_temp_file.close()
         app_dev_yaml_temp_file.close()
 
-    def test_modify_constants(self) -> None:
-        mock_constants_path = 'mock_app_dev.yaml'
-        mock_feconf_path = 'mock_app.yaml'
-        constants_path_swap = self.swap(
-            common, 'CONSTANTS_FILE_PATH', mock_constants_path)
-        feconf_path_swap = self.swap(common, 'FECONF_PATH', mock_feconf_path)
-
-        constants_temp_file = tempfile.NamedTemporaryFile()
-        # Here MyPy assumes that the 'name' attribute is read-only. In order to
-        # silence the MyPy complaints `setattr` is used to set the attribute.
-        setattr(
-            constants_temp_file, 'name', mock_constants_path)
-        with utils.open_file(mock_constants_path, 'w') as tmp:
-            tmp.write('export = {\n')
-            tmp.write('  "DEV_MODE": true,\n')
-            tmp.write('  "EMULATOR_MODE": false,\n')
-            tmp.write('};')
-
-        feconf_temp_file = tempfile.NamedTemporaryFile()
-        # Here MyPy assumes that the 'name' attribute is read-only. In order to
-        # silence the MyPy complaints `setattr` is used to set the attribute.
-        setattr(feconf_temp_file, 'name', mock_feconf_path)
-        with utils.open_file(mock_feconf_path, 'w') as tmp:
-            tmp.write(u'ENABLE_MAINTENANCE_MODE = False')
-
-        with constants_path_swap, feconf_path_swap:
-            build.modify_constants(prod_env=True, maintenance_mode=False)
-            with utils.open_file(
-                mock_constants_path, 'r') as constants_file:
-                self.assertEqual(
-                    constants_file.read(),
-                    'export = {\n'
-                    '  "DEV_MODE": false,\n'
-                    '  "EMULATOR_MODE": true,\n'
-                    '};')
-            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
-                self.assertEqual(
-                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = False')
-
-            build.modify_constants(prod_env=False, maintenance_mode=True)
-            with utils.open_file(
-                mock_constants_path, 'r') as constants_file:
-                self.assertEqual(
-                    constants_file.read(),
-                    'export = {\n'
-                    '  "DEV_MODE": true,\n'
-                    '  "EMULATOR_MODE": true,\n'
-                    '};')
-            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
-                self.assertEqual(
-                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = True')
-
-        constants_temp_file.close()
-        feconf_temp_file.close()
-
-    def test_set_constants_to_default(self) -> None:
-        mock_constants_path = 'mock_app_dev.yaml'
-        mock_feconf_path = 'mock_app.yaml'
-        constants_path_swap = self.swap(
-            common, 'CONSTANTS_FILE_PATH', mock_constants_path)
-        feconf_path_swap = self.swap(common, 'FECONF_PATH', mock_feconf_path)
-
-        constants_temp_file = tempfile.NamedTemporaryFile()
-        # Here MyPy assumes that the 'name' attribute is read-only. In order to
-        # silence the MyPy complaints `setattr` is used to set the attribute.
-        setattr(
-            constants_temp_file, 'name', mock_constants_path)
-        with utils.open_file(mock_constants_path, 'w') as tmp:
-            tmp.write('export = {\n')
-            tmp.write('  "DEV_MODE": false,\n')
-            tmp.write('  "EMULATOR_MODE": false,\n')
-            tmp.write('};')
-
-        feconf_temp_file = tempfile.NamedTemporaryFile()
-        # Here MyPy assumes that the 'name' attribute is read-only. In order to
-        # silence the MyPy complaints `setattr` is used to set the attribute.
-        setattr(feconf_temp_file, 'name', mock_feconf_path)
-        with utils.open_file(mock_feconf_path, 'w') as tmp:
-            tmp.write(u'ENABLE_MAINTENANCE_MODE = True')
-
-        with constants_path_swap, feconf_path_swap:
-            build.set_constants_to_default()
-            with utils.open_file(
-                mock_constants_path, 'r') as constants_file:
-                self.assertEqual(
-                    constants_file.read(),
-                    'export = {\n'
-                    '  "DEV_MODE": true,\n'
-                    '  "EMULATOR_MODE": true,\n'
-                    '};')
-            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
-                self.assertEqual(
-                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = False')
-
-        constants_temp_file.close()
-        feconf_temp_file.close()
-
     def test_safe_delete_file(self) -> None:
         """Test safe_delete_file with both existent and non-existent
         filepath.
@@ -1007,7 +910,7 @@ def mock_clean() -> None:
         build_using_webpack_swap = self.swap(
             build, 'build_using_webpack', mock_build_using_webpack)
         modify_constants_swap = self.swap(
-            build, 'modify_constants', mock_modify_constants)
+            common, 'modify_constants', mock_modify_constants)
         compare_file_count_swap = self.swap(
             build, '_compare_file_count', mock_compare_file_count)
         generate_python_package_swap = self.swap(
@@ -1066,7 +969,7 @@ def mock_clean() -> None:
         build_using_webpack_swap = self.swap(
             build, 'build_using_webpack', mock_build_using_webpack)
         modify_constants_swap = self.swap(
-            build, 'modify_constants', mock_modify_constants)
+            common, 'modify_constants', mock_modify_constants)
         compare_file_count_swap = self.swap(
             build, '_compare_file_count', mock_compare_file_count)
         clean_swap = self.swap(build, 'clean', mock_clean)
@@ -1112,7 +1015,7 @@ def mock_clean() -> None:
         ensure_files_exist_swap = self.swap(
             build, '_ensure_files_exist', mock_ensure_files_exist)
         modify_constants_swap = self.swap(
-            build, 'modify_constants', mock_modify_constants)
+            common, 'modify_constants', mock_modify_constants)
         clean_swap = self.swap(build, 'clean', mock_clean)
 
         with ensure_files_exist_swap, modify_constants_swap, clean_swap:
@@ -1182,7 +1085,7 @@ def mock_clean() -> None:
         ensure_files_exist_swap = self.swap(
             build, '_ensure_files_exist', mock_ensure_files_exist)
         modify_constants_swap = self.swap(
-            build, 'modify_constants', mock_modify_constants)
+            common, 'modify_constants', mock_modify_constants)
         clean_swap = self.swap(build, 'clean', mock_clean)
         with ensure_files_exist_swap, modify_constants_swap, clean_swap:
             build.main(args=['--prod_env', '--minify_third_party_libs_only'])
diff --git a/scripts/common_test.py b/scripts/common_test.py
--- a/scripts/common_test.py
+++ b/scripts/common_test.py
@@ -1280,3 +1280,99 @@ def mock_print(msg: str) -> None:
         ):
             common.setup_chrome_bin_env_variable()
         self.assertIn('Chrome is not found, stopping...', print_arr)
+
+    def test_modify_constants(self) -> None:
+        mock_constants_path = 'mock_app_dev.yaml'
+        mock_feconf_path = 'mock_app.yaml'
+        constants_path_swap = self.swap(
+            common, 'CONSTANTS_FILE_PATH', mock_constants_path)
+        feconf_path_swap = self.swap(common, 'FECONF_PATH', mock_feconf_path)
+
+        constants_temp_file = tempfile.NamedTemporaryFile()
+        # Here MyPy assumes that the 'name' attribute is read-only. In order to
+        # silence the MyPy complaints `setattr` is used to set the attribute.
+        setattr(
+            constants_temp_file, 'name', mock_constants_path)
+        with utils.open_file(mock_constants_path, 'w') as tmp:
+            tmp.write('export = {\n')
+            tmp.write('  "DEV_MODE": true,\n')
+            tmp.write('  "EMULATOR_MODE": false,\n')
+            tmp.write('};')
+
+        feconf_temp_file = tempfile.NamedTemporaryFile()
+        # Here MyPy assumes that the 'name' attribute is read-only. In order to
+        # silence the MyPy complaints `setattr` is used to set the attribute.
+        setattr(feconf_temp_file, 'name', mock_feconf_path)
+        with utils.open_file(mock_feconf_path, 'w') as tmp:
+            tmp.write(u'ENABLE_MAINTENANCE_MODE = False')
+
+        with constants_path_swap, feconf_path_swap:
+            common.modify_constants(prod_env=True, maintenance_mode=False)
+            with utils.open_file(
+                mock_constants_path, 'r') as constants_file:
+                self.assertEqual(
+                    constants_file.read(),
+                    'export = {\n'
+                    '  "DEV_MODE": false,\n'
+                    '  "EMULATOR_MODE": true,\n'
+                    '};')
+            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
+                self.assertEqual(
+                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = False')
+
+            common.modify_constants(prod_env=False, maintenance_mode=True)
+            with utils.open_file(
+                mock_constants_path, 'r') as constants_file:
+                self.assertEqual(
+                    constants_file.read(),
+                    'export = {\n'
+                    '  "DEV_MODE": true,\n'
+                    '  "EMULATOR_MODE": true,\n'
+                    '};')
+            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
+                self.assertEqual(
+                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = True')
+
+        constants_temp_file.close()
+        feconf_temp_file.close()
+
+    def test_set_constants_to_default(self) -> None:
+        mock_constants_path = 'mock_app_dev.yaml'
+        mock_feconf_path = 'mock_app.yaml'
+        constants_path_swap = self.swap(
+            common, 'CONSTANTS_FILE_PATH', mock_constants_path)
+        feconf_path_swap = self.swap(common, 'FECONF_PATH', mock_feconf_path)
+
+        constants_temp_file = tempfile.NamedTemporaryFile()
+        # Here MyPy assumes that the 'name' attribute is read-only. In order to
+        # silence the MyPy complaints `setattr` is used to set the attribute.
+        setattr(
+            constants_temp_file, 'name', mock_constants_path)
+        with utils.open_file(mock_constants_path, 'w') as tmp:
+            tmp.write('export = {\n')
+            tmp.write('  "DEV_MODE": false,\n')
+            tmp.write('  "EMULATOR_MODE": false,\n')
+            tmp.write('};')
+
+        feconf_temp_file = tempfile.NamedTemporaryFile()
+        # Here MyPy assumes that the 'name' attribute is read-only. In order to
+        # silence the MyPy complaints `setattr` is used to set the attribute.
+        setattr(feconf_temp_file, 'name', mock_feconf_path)
+        with utils.open_file(mock_feconf_path, 'w') as tmp:
+            tmp.write(u'ENABLE_MAINTENANCE_MODE = True')
+        self.contextManager.__exit__(None, None, None)
+        with constants_path_swap, feconf_path_swap:
+            common.set_constants_to_default()
+            with utils.open_file(
+                mock_constants_path, 'r') as constants_file:
+                self.assertEqual(
+                    constants_file.read(),
+                    'export = {\n'
+                    '  "DEV_MODE": true,\n'
+                    '  "EMULATOR_MODE": true,\n'
+                    '};')
+            with utils.open_file(mock_feconf_path, 'r') as feconf_file:
+                self.assertEqual(
+                    feconf_file.read(), 'ENABLE_MAINTENANCE_MODE = False')
+        constants_temp_file.close()
+        feconf_temp_file.close()
diff --git a/scripts/run_e2e_tests.py b/scripts/run_e2e_tests.py
--- a/scripts/run_e2e_tests.py
+++ b/scripts/run_e2e_tests.py
@@ -197,10 +197,10 @@ def run_tests(args: argparse.Namespace) -> Tuple[List[bytes], int]:
         dev_mode = not args.prod_env
 
         if args.skip_build:
-            build.modify_constants(prod_env=args.prod_env)
+            common.modify_constants(prod_env=args.prod_env)
         else:
             build_js_files(dev_mode, source_maps=args.source_maps)
-        stack.callback(build.set_constants_to_default)
+        stack.callback(common.set_constants_to_default)
 
         stack.enter_context(servers.managed_redis_server())
         stack.enter_context(servers.managed_elasticsearch_dev_server())
diff --git a/scripts/run_e2e_tests_test.py b/scripts/run_e2e_tests_test.py
--- a/scripts/run_e2e_tests_test.py
+++ b/scripts/run_e2e_tests_test.py
@@ -56,6 +56,11 @@ def setUp(self) -> None:
         super().setUp()
         self.exit_stack = contextlib.ExitStack()
 
+        def mock_constants() -> None:
+            print('mock_set_constants_to_default')
+        self.swap_mock_set_constants_to_default = self.swap(
+            common, 'set_constants_to_default', mock_constants)
+
     def tearDown(self) -> None:
         try:
             self.exit_stack.close()
@@ -265,8 +270,8 @@ def test_start_tests_when_no_other_instance_running(self) -> None:
             ]))
         self.exit_stack.enter_context(self.swap_with_checks(
             sys, 'exit', lambda _: None, expected_args=[(0,)]))
-
-        run_e2e_tests.main(args=[])
+        with self.swap_mock_set_constants_to_default:
+            run_e2e_tests.main(args=[])
 
     def test_work_with_non_ascii_chars(self) -> None:
         def mock_managed_webdriverio_server(
@@ -311,7 +316,8 @@ def mock_managed_webdriverio_server(
             ]))
         args = run_e2e_tests._PARSER.parse_args(args=[])  # pylint: disable=protected-access
 
-        lines, _ = run_e2e_tests.run_tests(args)
+        with self.swap_mock_set_constants_to_default:
+            lines, _ = run_e2e_tests.run_tests(args)
 
         self.assertEqual(
             [line.decode('utf-8') for line in lines],
@@ -390,10 +396,10 @@ def test_start_tests_skip_build(self) -> None:
             run_e2e_tests, 'install_third_party_libraries', lambda _: None,
             expected_args=[(True,)]))
         self.exit_stack.enter_context(self.swap_with_checks(
-            build, 'modify_constants', lambda *_, **__: None,
+            common, 'modify_constants', lambda *_, **__: None,
             expected_kwargs=[{'prod_env': False}]))
         self.exit_stack.enter_context(self.swap_with_checks(
-            build, 'set_constants_to_default', lambda: None))
+            common, 'set_constants_to_default', lambda: None))
         self.exit_stack.enter_context(self.swap_with_checks(
             servers, 'managed_elasticsearch_dev_server', mock_managed_process))
         self.exit_stack.enter_context(self.swap_with_checks(
@@ -464,7 +470,8 @@ def test_start_tests_in_debug_mode(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
             sys, 'exit', lambda _: None, expected_args=[(0,)]))
 
-        run_e2e_tests.main(args=['--debug_mode'])
+        with self.swap_mock_set_constants_to_default:
+            run_e2e_tests.main(args=['--debug_mode'])
 
     def test_start_tests_in_with_chromedriver_flag(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
@@ -503,8 +510,9 @@ def test_start_tests_in_with_chromedriver_flag(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
             sys, 'exit', lambda _: None, expected_args=[(0,)]))
 
-        run_e2e_tests.main(
-            args=['--chrome_driver_version', CHROME_DRIVER_VERSION])
+        with self.swap_mock_set_constants_to_default:
+            run_e2e_tests.main(
+                args=['--chrome_driver_version', CHROME_DRIVER_VERSION])
 
     def test_start_tests_in_webdriverio(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
@@ -543,8 +551,9 @@ def test_start_tests_in_webdriverio(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
             sys, 'exit', lambda _: None, expected_args=[(0,)]))
 
-        run_e2e_tests.main(
-            args=['--suite', 'collections'])
+        with self.swap_mock_set_constants_to_default:
+            run_e2e_tests.main(
+                args=['--suite', 'collections'])
 
     def test_do_not_run_with_test_non_mobile_suite_in_mobile_mode(self) -> None:
         self.exit_stack.enter_context(self.swap_with_checks(
@@ -569,4 +578,5 @@ def test_do_not_run_with_test_non_mobile_suite_in_mobile_mode(self) -> None:
             servers, 'managed_cloud_datastore_emulator', mock_managed_process))
 
         with self.assertRaisesRegex(SystemExit, '^1$'):
-            run_e2e_tests.main(args=['--mobile', '--suite', 'collections'])
+            with self.swap_mock_set_constants_to_default:
+                run_e2e_tests.main(args=['--mobile', '--suite', 'collections'])
diff --git a/scripts/start_test.py b/scripts/start_test.py
--- a/scripts/start_test.py
+++ b/scripts/start_test.py
@@ -57,7 +57,8 @@ def mock_context_manager() -> MockCompilerContextManager:
             return MockCompilerContextManager()
         self.swap_print = self.swap(
             common, 'print_each_string_after_two_new_lines', mock_print)
-
+        def mock_constants() -> None:
+            print('mock_set_constants_to_default')
         # We need to create a swap for install_third_party_libs because
         # scripts/start.py installs third party libraries whenever it is
         # imported.
@@ -107,6 +108,8 @@ def mock_context_manager() -> MockCompilerContextManager:
         self.swap_create_managed_web_browser = self.swap_to_always_raise(
             servers, 'create_managed_web_browser',
             Exception(MANAGED_WEB_BROWSER_ERROR))
+        self.swap_mock_set_constants_to_default = self.swap(
+            common, 'set_constants_to_default', mock_constants)
 
     def test_start_servers_successfully(self) -> None:
         with self.swap_install_third_party_libs:
@@ -119,7 +122,8 @@ def test_start_servers_successfully(self) -> None:
                 with self.swap_create_server, self.swap_webpack_compiler:
                     with self.swap_extend_index_yaml, self.swap_dev_appserver:
                         with self.swap_firebase_auth_emulator, self.swap_print:
-                            start.main(args=[])
+                            with self.swap_mock_set_constants_to_default:
+                                start.main(args=[])
 
         self.assertIn(
             [
@@ -143,7 +147,8 @@ def test_start_servers_successfully_in_production_mode(self) -> None:
                 with self.swap_firebase_auth_emulator, self.swap_dev_appserver:
                     with self.swap_extend_index_yaml, swap_build:
                         with self.swap_print:
-                            start.main(args=['--prod_env'])
+                            with self.swap_mock_set_constants_to_default:
+                                start.main(args=['--prod_env'])
 
         self.assertIn(
             [
@@ -169,7 +174,8 @@ def test_start_servers_successfully_in_maintenance_mode(self) -> None:
                 with self.swap_create_server, self.swap_webpack_compiler:
                     with self.swap_extend_index_yaml, self.swap_dev_appserver:
                         with self.swap_firebase_auth_emulator, self.swap_print:
-                            start.main(args=['--maintenance_mode'])
+                            with self.swap_mock_set_constants_to_default:
+                                start.main(args=['--maintenance_mode'])
 
         self.assertIn(
             [
@@ -196,7 +202,8 @@ def test_could_not_start_new_server_when_port_is_in_use(self) -> None:
                 with self.swap_firebase_auth_emulator, self.swap_dev_appserver:
                     with self.swap_extend_index_yaml, swap_check_port_in_use:
                         with self.swap_print, swap_build, self.swap_ng_build:
-                            start.main(args=['--no_browser'])
+                            with self.swap_mock_set_constants_to_default:
+                                start.main(args=['--no_browser'])
 
         self.assertIn(
             [
@@ -239,7 +246,8 @@ def test_source_maps_are_compiled_by_webpack(self) -> None:
                 with swap_emulator_mode, self.swap_dev_appserver:
                     with self.swap_extend_index_yaml, swap_build:
                         with self.swap_print, self.swap_ng_build:
-                            start.main(args=['--source_maps'])
+                            with self.swap_mock_set_constants_to_default:
+                                start.main(args=['--source_maps'])
 
         self.assertIn(
             [
@@ -265,7 +273,8 @@ def test_could_not_auto_launch_web_browser(self) -> None:
                     with self.swap_webpack_compiler, self.swap_dev_appserver:
                         with self.swap_extend_index_yaml, self.swap_print:
                             with self.swap_firebase_auth_emulator:
-                                start.main(args=[])
+                                with self.swap_mock_set_constants_to_default:
+                                    start.main(args=[])
 
         self.assertIn(
             [
@@ -287,3 +296,20 @@ def test_could_not_auto_launch_web_browser(self) -> None:
                 )
             ],
             self.print_arr)
+
+    def test_not_mock_set_constants_to_default_error(self) -> None:
+        with self.swap_install_third_party_libs:
+            from scripts import start
+        swap_build = self.swap_with_checks(
+            build, 'main', lambda **unused_kwargs: None,
+            expected_kwargs=[{'args': []}])
+        assert_raises_regexp = self.assertRaisesRegex(
+            Exception, 'Please mock this method in the test.')
+
+        with self.swap_cloud_datastore_emulator, self.swap_ng_build, swap_build:
+            with self.swap_elasticsearch_dev_server, self.swap_redis_server:
+                with self.swap_create_server, self.swap_webpack_compiler:
+                    with self.swap_extend_index_yaml, self.swap_dev_appserver:
+                        with self.swap_firebase_auth_emulator, self.swap_print:
+                            with assert_raises_regexp:
+                                start.main(args=[])