Repository: oppia/oppia

Issue:
[Feature Request]: Find a way to prevent package build errors on Google Dataflow
### Is your feature request related to a problem? Please describe.

The package build using https://github.com/oppia/oppia/blob/develop/setup.py started failing on Google Dataflow after some commits that did not modify the build directly.

We did not have `psutil` in _requirements.in_ (we have it in _requirements_dev.in_) and _setup.py_ calls _common.py_ which calls servers.py which imports `psutil`. Locally, we could build the package as _requirements_dev.in_is installed but on Apache Beam, we only install _requirements.in_ (this is correct behaviour as on prod, we only install _requirements.in_ there). This only surfaced because #16073 introduced the import of _servers.py_ into _common.py_.

![image](https://user-images.githubusercontent.com/10142938/209716216-76e33830-4f14-4faa-948e-0aab690fced1.png)


### Describe the solution you'd like

The local package build should be isolated so that if there is a possible impro error like this in the future, it is surfaced locally before the PR introducing the change is merged.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_
[Feature Request]: Find a way to prevent package build errors on Google Dataflow
### Is your feature request related to a problem? Please describe.

The package build using https://github.com/oppia/oppia/blob/develop/setup.py started failing on Google Dataflow after some commits that did not modify the build directly.

We did not have `psutil` in _requirements.in_ (we have it in _requirements_dev.in_) and _setup.py_ calls _common.py_ which calls servers.py which imports `psutil`. Locally, we could build the package as _requirements_dev.in_is installed but on Apache Beam, we only install _requirements.in_ (this is correct behaviour as on prod, we only install _requirements.in_ there). This only surfaced because #16073 introduced the import of _servers.py_ into _common.py_.

![image](https://user-images.githubusercontent.com/10142938/209716216-76e33830-4f14-4faa-948e-0aab690fced1.png)


### Describe the solution you'd like

The local package build should be isolated so that if there is a possible impro error like this in the future, it is surfaced locally before the PR introducing the change is merged.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_


Discussion:
Hi if anyone is not working on this I'd like to try this

@vanshdeepsingh-coder Hey, this is a bit harder issue. Please follow instructions here https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#setting-things-up and pick some good first issue to work on.
Hi if anyone is not working on this I'd like to try this

@vanshdeepsingh-coder Hey, this is a bit harder issue. Please follow instructions here https://github.com/oppia/oppia/wiki/Contributing-code-to-Oppia#setting-things-up and pick some good first issue to work on.

Resolving patch:
diff --git a/core/utils.py b/core/utils.py
--- a/core/utils.py
+++ b/core/utils.py
@@ -56,10 +56,6 @@
 TextModeTypes = Literal['r', 'w', 'a', 'x', 'r+', 'w+', 'a+']
 BinaryModeTypes = Literal['rb', 'wb', 'ab', 'xb', 'r+b', 'w+b', 'a+b', 'x+b']
 
-# TODO(#13059): We will be ignoring no-untyped-call and no-any-return here
-# These will be removed after python3 migration and adding stubs for new python3
-# libraries.
-
 
 class InvalidInputException(Exception):
     """Error class for invalid input."""
diff --git a/scripts/build.py b/scripts/build.py
--- a/scripts/build.py
+++ b/scripts/build.py
@@ -27,16 +27,14 @@
 import subprocess
 import threading
 
-import rcssmin
-
-# TODO(#15567): The order can be fixed after Literal in utils.py is loaded
-# from typing instead of typing_extensions, this will be possible after
-# we migrate to Python 3.8.
-from scripts import common # isort:skip pylint: disable=wrong-import-position
-from core import utils # isort:skip pylint: disable=wrong-import-position
-from scripts import servers # isort:skip pylint: disable=wrong-import-position
+from core import utils
+from scripts import common
+from scripts import install_python_dev_dependencies
+from scripts import install_third_party_libs
+from scripts import servers
 
-from typing import ( # isort:skip pylint: disable=wrong-import-position
+import rcssmin
+from typing import (
     Deque, Dict, List, Optional, Sequence, TextIO, Tuple, TypedDict)
 
 ASSETS_DEV_DIR = os.path.join('assets', '')
@@ -1378,9 +1376,20 @@ def generate_build_directory(hashes: Dict[str, str]) -> None:
 
 def generate_python_package() -> None:
     """Generates Python package using setup.py."""
-    print('Building Oppia package...')
-    subprocess.check_call('python setup.py -q sdist -d build', shell=True)
-    print('Oppia package build completed.')
+
+    # We first remove this dev dependencies because they should not be needed
+    # for the package build and we need to verify that they are actually not
+    # needed.
+    try:
+        print('Remove dev dependencies')
+        install_python_dev_dependencies.main(['--uninstall'])
+
+        print('Building Oppia package...')
+        subprocess.check_call('python setup.py -q sdist -d build', shell=True)
+        print('Oppia package build completed.')
+    finally:
+        install_third_party_libs.main()
+        print('Dev dependencies reinstalled')
 
 
 def clean() -> None:
diff --git a/scripts/install_python_dev_dependencies.py b/scripts/install_python_dev_dependencies.py
--- a/scripts/install_python_dev_dependencies.py
+++ b/scripts/install_python_dev_dependencies.py
@@ -39,6 +39,9 @@
 _PARSER.add_argument(
     '--assert_compiled', action='store_true',
     help='Assert that the dev requirements file is already compiled.')
+_PARSER.add_argument(
+    '--uninstall', action='store_true',
+    help='Uninstall all dev requirements.')
 
 
 def check_python_env_is_suitable() -> None:
@@ -82,9 +85,7 @@ def install_installation_tools() -> None:
 
 
 def install_dev_dependencies() -> None:
-    """Install dev dependencies from
-    COMPILED_REQUIREMENTS_DEV_FILE_PATH.
-    """
+    """Install dev dependencies from COMPILED_REQUIREMENTS_DEV_FILE_PATH."""
     subprocess.run(
         ['pip-sync', COMPILED_REQUIREMENTS_DEV_FILE_PATH],
         check=True,
@@ -92,6 +93,15 @@ def install_dev_dependencies() -> None:
     )
 
 
+def uninstall_dev_dependencies() -> None:
+    """Uninstall dev dependencies from COMPILED_REQUIREMENTS_DEV_FILE_PATH."""
+    subprocess.run(
+        ['pip', 'uninstall', '-r', COMPILED_REQUIREMENTS_DEV_FILE_PATH, '-y'],
+        check=True,
+        encoding='utf-8',
+    )
+
+
 def compile_pip_requirements(
     requirements_path: str, compiled_path: str
 ) -> bool:
@@ -127,14 +137,17 @@ def main(cli_args: Optional[List[str]] = None) -> None:
     install_installation_tools()
     not_compiled = compile_pip_requirements(
         REQUIREMENTS_DEV_FILE_PATH, COMPILED_REQUIREMENTS_DEV_FILE_PATH)
-    install_dev_dependencies()
-    if args.assert_compiled and not_compiled:
-        raise RuntimeError(
-            'The Python development requirements file '
-            f'{COMPILED_REQUIREMENTS_DEV_FILE_PATH} was changed by the '
-            'installation script. Please commit the changes. '
-            'You can get the changes again by running this command: '
-            'python -m scripts.install_python_dev_dependencies')
+    if args.uninstall:
+        uninstall_dev_dependencies()
+    else:
+        install_dev_dependencies()
+        if args.assert_compiled and not_compiled:
+            raise RuntimeError(
+                'The Python development requirements file '
+                f'{COMPILED_REQUIREMENTS_DEV_FILE_PATH} was changed by the '
+                'installation script. Please commit the changes. '
+                'You can get the changes again by running this command: '
+                'python -m scripts.install_python_dev_dependencies')
 
 
 # This code cannot be covered by tests since it only runs when this file


Test patch:
diff --git a/scripts/build_test.py b/scripts/build_test.py
--- a/scripts/build_test.py
+++ b/scripts/build_test.py
@@ -35,6 +35,7 @@
 
 from . import build
 from . import common
+from . import install_python_dev_dependencies
 from . import scripts_test_utils
 from . import servers
 
@@ -1053,8 +1054,7 @@ def mock_modify_constants(
             check_function_calls['modify_constants_gets_called'] = True
 
         def mock_compare_file_count(
-            unused_first_dir: str,
-            unused_second_dir: str
+            unused_first_dir: str, unused_second_dir: str
         ) -> None:
             check_function_calls['compare_file_count_gets_called'] = True
 
@@ -1070,10 +1070,16 @@ def mock_clean() -> None:
         compare_file_count_swap = self.swap(
             build, '_compare_file_count', mock_compare_file_count)
         clean_swap = self.swap(build, 'clean', mock_clean)
+        install_python_dev_dependencies_swap = self.swap_with_checks(
+            install_python_dev_dependencies,
+            'main',
+            lambda _: None,
+            expected_args=[(['--uninstall'],)]
+        )
 
         with ensure_files_exist_swap, build_using_webpack_swap:
             with modify_constants_swap, compare_file_count_swap:
-                with clean_swap:
+                with clean_swap, install_python_dev_dependencies_swap:
                     build.main(args=['--prod_env', '--source_maps'])
 
         self.assertEqual(check_function_calls, expected_check_function_calls)
diff --git a/scripts/install_python_dev_dependencies_test.py b/scripts/install_python_dev_dependencies_test.py
--- a/scripts/install_python_dev_dependencies_test.py
+++ b/scripts/install_python_dev_dependencies_test.py
@@ -157,6 +157,23 @@ def mock_run(*_args: str, **_kwargs: str) -> None:  # pylint: disable=unused-arg
         with run_swap:
             install_python_dev_dependencies.install_dev_dependencies()
 
+    def test_uninstall_dev_dependencies(self) -> None:
+
+        def mock_run(*_args: str, **_kwargs: str) -> None:  # pylint: disable=unused-argument
+            pass
+
+        run_swap = self.swap_with_checks(
+            subprocess, 'run', mock_run, expected_args=[
+                (['pip', 'uninstall', '-r', 'requirements_dev.txt', '-y'],),
+            ],
+            expected_kwargs=[
+                {'check': True, 'encoding': 'utf-8'},
+            ]
+        )
+
+        with run_swap:
+            install_python_dev_dependencies.uninstall_dev_dependencies()
+
     def test_compile_pip_requirements_no_change(self) -> None:
 
         def mock_run(*_args: str, **_kwargs: str) -> None:  # pylint: disable=unused-argument
@@ -256,6 +273,27 @@ def mock_compile(*_args: str) -> bool:
             with install_dependencies_swap:
                 install_python_dev_dependencies.main([])
 
+    def test_main_passes_with_uninstall(self) -> None:
+        def mock_compile(*_args: str) -> bool:
+            return False
+
+        assert_swap = self.swap_with_checks(
+            install_python_dev_dependencies, 'check_python_env_is_suitable',
+            lambda: None)
+        install_tools_swap = self.swap_with_checks(
+            install_python_dev_dependencies,
+            'install_installation_tools', lambda: None)
+        compile_swap = self.swap_with_checks(
+            install_python_dev_dependencies,
+            'compile_pip_requirements', mock_compile)
+        uninstall_dependencies_swap = self.swap_with_checks(
+            install_python_dev_dependencies,
+            'uninstall_dev_dependencies', lambda: None)
+
+        with assert_swap, install_tools_swap, compile_swap:
+            with uninstall_dependencies_swap:
+                install_python_dev_dependencies.main(['--uninstall'])
+
     def test_main_passes_with_assert_and_no_change(self) -> None:
         def mock_func() -> None:
             pass