Repository: apache/airflow

Issue:
tests/core/test_providers_manager.py::TestProviderManager::test_hooks broken on 3.6
### Apache Airflow version

main (development)

### Operating System

Any

### Versions of Apache Airflow Providers

_No response_

### Deployment

Other

### Deployment details

_No response_

### What happened

`tests/core/test_providers_manager.py::TestProviderManager::test_hooks` is broken on 3.6 by #18209 since newer `importlib-resources` versions uses a different implementation and the mocks no longer works. This was actually visible before merging; all and only 3.6 checks in the PR failed. Let’s be more careful identifying CI failure patterns in the future 🙂 

Not exactly sure how to fix yet. I believe the breaking changes were introduced in [importlib-resources 5.2](https://github.com/python/importlib_resources/blob/v5.2.2/CHANGES.rst#v520), but restricting `<5.2` is not a long-term fix since the same version is also in the Python 3.10 stdlib and will bite us again very soon.

### What you expected to happen

_No response_

### How to reproduce

```
$ ./breeze tests -- tests/core/test_providers_manager.py::TestProviderManager::test_hooks
...
tests/core/test_providers_manager.py F                                                                  [100%]

================================================== FAILURES ==================================================
_______________________________________ TestProviderManager.test_hooks _______________________________________

self = <test_providers_manager.TestProviderManager testMethod=test_hooks>
mock_import_module = <MagicMock name='import_module' id='139679504564520'>

    @patch('airflow.providers_manager.importlib.import_module')
    def test_hooks(self, mock_import_module):
        # Compat with importlib_resources
        mock_import_module.return_value.__spec__ = Mock()
        with pytest.warns(expected_warning=None) as warning_records:
            with self._caplog.at_level(logging.WARNING):
>               provider_manager = ProvidersManager()

tests/core/test_providers_manager.py:123:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/providers_manager.py:242: in __init__
    self._provider_schema_validator = _create_provider_info_schema_validator()
airflow/providers_manager.py:92: in _create_provider_info_schema_validator
    schema = json.loads(importlib_resources.read_text('airflow', 'provider_info.schema.json'))
/usr/local/lib/python3.6/site-packages/importlib_resources/_legacy.py:46: in read_text
    with open_text(package, resource, encoding, errors) as fp:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

package = 'airflow', resource = 'provider_info.schema.json', encoding = 'utf-8', errors = 'strict'

    def open_text(
        package: Package,
        resource: Resource,
        encoding: str = 'utf-8',
        errors: str = 'strict',
    ) -> TextIO:
        """Return a file-like object opened for text reading of the resource."""
>       return (_common.files(package) / _common.normalize_path(resource)).open(
            'r', encoding=encoding, errors=errors
        )
E       TypeError: unsupported operand type(s) for /: 'Mock' and 'str'
```

### Anything else

_No response_

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -507,12 +507,11 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version
     'freezegun',
     'github3.py',
     'gitpython',
-    'importlib-resources~=1.4',
     'ipdb',
     'jira',
     'jsondiff',
     'mongomock',
-    'moto~=2.2, >=2.2.1.dev9',
+    'moto~=2.2, >=2.2.1',
     'mypy==0.770',
     'parameterized',
     'paramiko',


Test patch:
diff --git a/tests/core/test_providers_manager.py b/tests/core/test_providers_manager.py
--- a/tests/core/test_providers_manager.py
+++ b/tests/core/test_providers_manager.py
@@ -17,11 +17,13 @@
 # under the License.
 import logging
 import re
+import sys
 import unittest
-from unittest.mock import patch
+from unittest.mock import call, patch
 
 import pytest
 
+import airflow
 from airflow.providers_manager import ProviderInfo, ProvidersManager
 
 
@@ -116,6 +118,12 @@ def test_warning_logs_not_generated(self):
 
     @patch('airflow.providers_manager.importlib.import_module')
     def test_hooks(self, mock_import_module):
+        # importlib-resources>=5.2 relies on importing the containing module to
+        # read its content. The provider manager needs to read the validation
+        # schema 'provider_info.schema.json'. Not sure why this needs to be
+        # called twice. Also see comment on the assert at the end of the function.
+        mock_import_module.side_effect = [airflow, airflow]
+
         with pytest.warns(expected_warning=None) as warning_records:
             with self._caplog.at_level(logging.WARNING):
                 provider_manager = ProvidersManager()
@@ -123,7 +131,16 @@ def test_hooks(self, mock_import_module):
                 assert len(connections_list) > 60
         assert [] == [w.message for w in warning_records.list if "hook-class-names" in str(w.message)]
         assert len(self._caplog.records) == 0
-        mock_import_module.assert_not_called()
+
+        # The stdlib importlib.resources implementation in 3.7 through 3.9 does
+        # not rely on importlib.import_module, so the function should not be
+        # called. The implementation for 3.10+ and the backport to 3.6 uses it
+        # to import the top-level 'airflow' for 'provider_info.schema.json'.
+        # Also see comment on mocking at the beginning of the function.
+        if (3, 7) <= sys.version_info < (3, 10):
+            assert mock_import_module.mock_calls == []
+        else:
+            assert mock_import_module.mock_calls == [call("airflow"), call("airflow")]
 
     def test_hook_values(self):
         with pytest.warns(expected_warning=None) as warning_records: