Repository: oppia/oppia

Issue:
Cyclic Imports
The linter found cyclic imports when run locally:

```
----------------------------------------
Please fix the errors below:
----------------------------------------
************* Module core.domain.topic_fetchers_test
core/domain/topic_fetchers_test.py:1:0: R0401: Cyclic import (core.domain.exp_services -> core.domain.opportunity_services -> core.domain.suggestion_services -> core.domain.suggestion_registry) 
```

In this case:

* `core.domain.exp_services` imports `core.domain.opportunity_services`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/exp_services.py#L52
* `core.domain.opportunity_services` imports `core.domain.suggestion_services`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/opportunity_services.py#L29
* `core.domain.suggestion_services` imports `core.domain.suggestion_registry`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/suggestion_services.py#L33
* `core.domain.suggestion_registry
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/suggestion_registry.py#L29


This creates a cycle of imports. How has this gone undetected until now? Our linter should have caught this before the changes were merged, and this cycle of imports should have caused problems before now.
Cyclic Imports
The linter found cyclic imports when run locally:

```
----------------------------------------
Please fix the errors below:
----------------------------------------
************* Module core.domain.topic_fetchers_test
core/domain/topic_fetchers_test.py:1:0: R0401: Cyclic import (core.domain.exp_services -> core.domain.opportunity_services -> core.domain.suggestion_services -> core.domain.suggestion_registry) 
```

In this case:

* `core.domain.exp_services` imports `core.domain.opportunity_services`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/exp_services.py#L52
* `core.domain.opportunity_services` imports `core.domain.suggestion_services`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/opportunity_services.py#L29
* `core.domain.suggestion_services` imports `core.domain.suggestion_registry`:
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/suggestion_services.py#L33
* `core.domain.suggestion_registry
  https://github.com/oppia/oppia/blob/d010f1abfa257eac4be09b34ba4f5986cfb6f4ae/core/domain/suggestion_registry.py#L29


This creates a cycle of imports. How has this gone undetected until now? Our linter should have caught this before the changes were merged, and this cycle of imports should have caused problems before now.


Discussion:
Just a note -- @Hudda has kindly offered to take a quick look at this. Thanks @Hudda! 
@U8NWXD @seanlip This is what I discovered: To check for cyclic import all the files causing cyclic import must be loaded in the linter, only then linter will identify the cyclic error. Right now, I think we run linter in shards, so there's a possibility that all the files causing cyclic import error might not be in that shard's filegroup and therefore our linter does not detect any error.
@Hudda thanks, that makes sense. I started digging into this because of cyclic import errors I'm seeing on #15219, which just changes the codeowners file. Do you have any thoughts as to why changes to the codeowners file would cause these errors to surface?
@Hudda Actually, I don't think the shards can explain this. I also see cyclic imports intermittently when running the linter on all files. Further, all of `core/domain` is in one shard, so sharding shouldn't have affected the cycle at the top of this issue.

I think this error might be arising despite our code running correctly because pylint considers imports to be cyclic even if some imports are not at the top level of the file. For example, we sometimes put imports in functions specifically to avoid cyclic imports, but pylint still considers these to be cyclic imports. To see this, create a folder `tmp` and inside, create `a.py` and `b.py`. Fill `a.py` with this:

```python
CONSTANT_A = 'constant a'

def func_a():
    from b import func_b
    func_b()
    print('a')


if __name__ == '__main__':
    func_a()
```

Fill `b.py` with this:

```python
from a import CONSTANT_A

def func_b():
    print('b')
    print(CONSTANT_A)
```

Then if you `cd` into `tmp`, you should be able to execute `a.py` just fine:

```console
$ python a.py 
b
constant a
a
```

However, pylint will find errors:

```
$ pylint *
************* Module a
a.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
a.py:4:4: C0415: Import outside toplevel (b.func_b) (import-outside-toplevel)
************* Module b
b.py:1:0: C0114: Missing module docstring (missing-module-docstring)
b.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
b.py:1:0: R0401: Cyclic import (a -> b) (cyclic-import)

-----------------------------------
Your code has been rated at 4.55/10
```

(To get pylint to successfully import your modules, you may need to set your `PYTHONPATH` to the path to `tmp`.)

Notice that even though Python doesn't raise a cyclic import error and `a.py` runs just fine, pylint finds a cyclic import because it doesn't account for the fact that the `from b import func_b` import is inside a function.

To solve this, I think we'll need to either disable the cyclic import check globally (and trust our other tests to catch them) or disable it for particular lines where cyclic errors appear. Long-term it'd be great to stop putting imports inside functions, but that's a big undertaking.
Just a note -- @Hudda has kindly offered to take a quick look at this. Thanks @Hudda! 
@U8NWXD @seanlip This is what I discovered: To check for cyclic import all the files causing cyclic import must be loaded in the linter, only then linter will identify the cyclic error. Right now, I think we run linter in shards, so there's a possibility that all the files causing cyclic import error might not be in that shard's filegroup and therefore our linter does not detect any error.
@Hudda thanks, that makes sense. I started digging into this because of cyclic import errors I'm seeing on #15219, which just changes the codeowners file. Do you have any thoughts as to why changes to the codeowners file would cause these errors to surface?
@Hudda Actually, I don't think the shards can explain this. I also see cyclic imports intermittently when running the linter on all files. Further, all of `core/domain` is in one shard, so sharding shouldn't have affected the cycle at the top of this issue.

I think this error might be arising despite our code running correctly because pylint considers imports to be cyclic even if some imports are not at the top level of the file. For example, we sometimes put imports in functions specifically to avoid cyclic imports, but pylint still considers these to be cyclic imports. To see this, create a folder `tmp` and inside, create `a.py` and `b.py`. Fill `a.py` with this:

```python
CONSTANT_A = 'constant a'

def func_a():
    from b import func_b
    func_b()
    print('a')


if __name__ == '__main__':
    func_a()
```

Fill `b.py` with this:

```python
from a import CONSTANT_A

def func_b():
    print('b')
    print(CONSTANT_A)
```

Then if you `cd` into `tmp`, you should be able to execute `a.py` just fine:

```console
$ python a.py 
b
constant a
a
```

However, pylint will find errors:

```
$ pylint *
************* Module a
a.py:1:0: C0114: Missing module docstring (missing-module-docstring)
a.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
a.py:4:4: C0415: Import outside toplevel (b.func_b) (import-outside-toplevel)
************* Module b
b.py:1:0: C0114: Missing module docstring (missing-module-docstring)
b.py:3:0: C0116: Missing function or method docstring (missing-function-docstring)
b.py:1:0: R0401: Cyclic import (a -> b) (cyclic-import)

-----------------------------------
Your code has been rated at 4.55/10
```

(To get pylint to successfully import your modules, you may need to set your `PYTHONPATH` to the path to `tmp`.)

Notice that even though Python doesn't raise a cyclic import error and `a.py` runs just fine, pylint finds a cyclic import because it doesn't account for the fact that the `from b import func_b` import is inside a function.

To solve this, I think we'll need to either disable the cyclic import check globally (and trust our other tests to catch them) or disable it for particular lines where cyclic errors appear. Long-term it'd be great to stop putting imports inside functions, but that's a big undertaking.

Resolving patch:
diff --git a/scripts/linters/pre_commit_linter.py b/scripts/linters/pre_commit_linter.py
--- a/scripts/linters/pre_commit_linter.py
+++ b/scripts/linters/pre_commit_linter.py
@@ -80,16 +80,13 @@
 
 OTHER_SHARD_NAME = 'other'
 
+# Shards are specified by a mapping from shard name to a list of the
+# paths in that shard. For exaple, `'1': ['core/domain/']` will create a
+# shard named `'1'` that contains all the files under core/domain/. A
+# shard name matching OTHER_SHARD_NAME includes all files not under
+# another shard.  Currently we are not sharding the lint checks, so the
+# only shard is the `other` shard that contains all files.
 SHARDS = {
-    '1': [
-        'core/templates/',
-        'extensions/',
-        'core/tests/',
-        'core/storage/',
-        'core/controllers/',
-        'core/platform',
-        'core/jobs/',
-    ],
     'other': None,
 }
 


Test patch:
diff --git a/scripts/linters/pre_commit_linter_test.py b/scripts/linters/pre_commit_linter_test.py
--- a/scripts/linters/pre_commit_linter_test.py
+++ b/scripts/linters/pre_commit_linter_test.py
@@ -120,19 +120,29 @@ def mock_get_changed_filepaths():
             ['No files to check'], self.linter_stdout)
 
     def test_main_with_non_other_shard(self):
+        mock_shards = {
+            '1': [
+                'a/',
+                'b/',
+            ],
+        }
+
         def mock_get_filepaths_from_path(path, namespace):  # pylint: disable=unused-argument
-            if path == pre_commit_linter.SHARDS['1'][0]:
+            if path == mock_shards['1'][0]:
                 return [VALID_PY_FILEPATH]
             return []
 
+        shards_swap = self.swap(
+            pre_commit_linter, 'SHARDS', mock_shards)
+
         get_filenames_from_path_swap = self.swap_with_checks(
             pre_commit_linter, '_get_filepaths_from_path',
             mock_get_filepaths_from_path, expected_args=[
                 (prefix,)
-                for prefix in pre_commit_linter.SHARDS['1']
+                for prefix in mock_shards['1']
             ])
 
-        with self.print_swap, self.sys_swap:
+        with self.print_swap, self.sys_swap, shards_swap:
             with self.install_swap:
                 with get_filenames_from_path_swap:
                     pre_commit_linter.main(args=['--shard', '1'])
@@ -146,17 +156,27 @@ def mock_install_third_party_main():
             raise AssertionError(
                 'Third party libs should not be installed.')
 
+        mock_shards = {
+            '1': [
+                'a/',
+            ],
+        }
+
+        shards_swap = self.swap(
+            pre_commit_linter, 'SHARDS', mock_shards)
+
         get_filenames_from_path_swap = self.swap_with_checks(
             pre_commit_linter, '_get_filepaths_from_path',
-            mock_get_filepaths_from_path, expected_args=[
+            mock_get_filepaths_from_path,
+            expected_args=[
                 (prefix,)
-                for prefix in pre_commit_linter.SHARDS['1']
+                for prefix in mock_shards['1']
             ])
         install_swap = self.swap(
             install_third_party_libs, 'main',
             mock_install_third_party_main)
 
-        with self.print_swap, self.sys_swap, install_swap:
+        with self.print_swap, self.sys_swap, install_swap, shards_swap:
             with get_filenames_from_path_swap:
                 with self.assertRaisesRegex(
                     RuntimeError, 'mock_file in multiple shards'
@@ -172,9 +192,21 @@ def mock_get_filepaths_from_path(path, namespace):  # pylint: disable=unused-arg
             else:
                 return []
 
+        mock_shards = {
+            '1': [
+                'a/',
+            ],
+            'other': [
+                'b/',
+            ],
+        }
+
+        shards_swap = self.swap(
+            pre_commit_linter, 'SHARDS', mock_shards)
+
         filenames_from_path_expected_args = [(os.getcwd(),)] + [
             (prefix,)
-            for prefix in pre_commit_linter.SHARDS['1']
+            for prefix in mock_shards['1']
         ]
 
         get_filenames_from_path_swap = self.swap_with_checks(
@@ -182,7 +214,7 @@ def mock_get_filepaths_from_path(path, namespace):  # pylint: disable=unused-arg
             mock_get_filepaths_from_path,
             expected_args=filenames_from_path_expected_args)
 
-        with self.print_swap, self.sys_swap:
+        with self.print_swap, self.sys_swap, shards_swap:
             with self.install_swap:
                 with get_filenames_from_path_swap:
                     pre_commit_linter.main(