Repository: huggingface/transformers

Issue:
pkg_resources' working_set caching breaks transformers import on google colab
## Environment info

- `transformers` version: 4.4.2
- Platform: google colab
- Python version: 3.7.10
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?):  n/a
- Using GPU in script?:  n/a
- Using distributed or parallel set-up in script?:  n/a

### Who can help

CC @stas00 as this was implemented in #8645

## To reproduce

You can find a complete example in [this google colab](https://colab.research.google.com/drive/1WT7GSd4uk9TLle9q9ftRNFy4BLZWCvZa?usp=sharing), also exported to [this gist](https://gist.github.com/konstin/e42d8f428fa11ba389e31be69cdc5646).

To reproduce, first [create a new google colab notebook](https://colab.research.google.com/#create=true). Let's install recent transformers and tqdm versions in it:

```
!pip install -U pip
!pip install -U "transformers<5.0.0,>=4.0.0" "tqdm<5.0.0,>=4.45.0"
```

This currently install transformers 4.4.2 and tqdm 4.59.0.

Surprisingly, now running `import transformers` fails. We get an error in pkg_resources, which is looking for the .dist-info of tqdm 4.41.1, when the installed version is 4.59.0:

```
[...]
/usr/local/lib/python3.7/dist-packages/pkg_resources/__init__.py in _get(self, path)
   1609 
   1610     def _get(self, path):
-> 1611         with open(path, 'rb') as stream:
   1612             return stream.read()
   1613 

FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.7/dist-packages/tqdm-4.41.1.dist-info/METADATA'
```

The cause is that pkg_resources uses the cached [WorkingSet](https://setuptools.readthedocs.io/en/latest/pkg_resources.html#workingset-objects), which contains the state before the pip install. We can confirm this by recreating pkg_resources' cache manually:

```python
import pkg_resources

pkg_resources.working_set = pkg_resources.WorkingSet()
```

Afterwards, importing transformers works.

The above example is the minimized version of our real [notebooks examples](https://github.com/sacdallago/bio_embeddings/tree/develop/notebooks):

```python
!pip install -U pip
!pip install -U bio_embeddings[all]

from bio_embeddings.embed import SeqVecEmbedder # This line fails with the tqdm .dist-info not found error
```

## Expected behavior

transformers should use the actual installed versions for checking compatibility instead of pkg_resources cache. This could be achieved e.g. by using [importlib_metadata](https://github.com/python/importlib_metadata) instead of pkg_resources [here](https://github.com/huggingface/transformers/blob/1c06240e1b3477728129bb58e7b6c7734bb5074e/src/transformers/utils/versions.py#L80) or by recreating pkg_resources` cache with `pkg_resources.working_set = pkg_resources.WorkingSet()` before checking versions.

I've used the following snippet to check that importlib_metadata works, which prints `4.41.1` and `4.59.0`:

```python
import pkg_resources
import importlib_metadata

print(pkg_resources.get_distribution("tqdm").version)
print(importlib_metadata.version("tqdm"))
```

I can prepare a pull request for either solution.


Discussion:
Thank you for this awesome report, @konstin, and identifying the cause of the problem and the solution.

This looks like a bug in `pkg_resources`. It should update its cache after install. 

The other workaround that seems to work is to just do:
```
!pip install -U "transformers<5.0.0,>=4.0.0" "tqdm<5.0.0,>=4.45.0"
!pip install -U "transformers<5.0.0,>=4.0.0" "tqdm<5.0.0,>=4.45.0"
```
So the 2nd one updates the cache.

You may want to report this bug to `pkg_resources`.

Since `transformers` has started using `importlib_metadata` extensively as of recent I think your proposed solution sounds great, so yes please - the proposed PR sounds perfect.

Thank you.

Resolving patch:
diff --git a/src/transformers/file_utils.py b/src/transformers/file_utils.py
--- a/src/transformers/file_utils.py
+++ b/src/transformers/file_utils.py
@@ -46,19 +46,13 @@
 
 import requests
 from filelock import FileLock
+from transformers.utils.versions import importlib_metadata
 
 from . import __version__
 from .hf_api import HfFolder
 from .utils import logging
 
 
-# The package importlib_metadata is in a different place, depending on the python version.
-if sys.version_info < (3, 8):
-    import importlib_metadata
-else:
-    import importlib.metadata as importlib_metadata
-
-
 logger = logging.get_logger(__name__)  # pylint: disable=invalid-name
 
 ENV_VARS_TRUE_VALUES = {"1", "ON", "YES", "TRUE"}
diff --git a/src/transformers/utils/versions.py b/src/transformers/utils/versions.py
--- a/src/transformers/utils/versions.py
+++ b/src/transformers/utils/versions.py
@@ -22,7 +22,12 @@
 
 from packaging import version
 
-import pkg_resources
+
+# The package importlib_metadata is in a different place, depending on the python version.
+if sys.version_info < (3, 8):
+    import importlib_metadata
+else:
+    import importlib.metadata as importlib_metadata
 
 
 ops = {
@@ -39,7 +44,7 @@ def require_version(requirement: str, hint: Optional[str] = None) -> None:
     """
     Perform a runtime check of the dependency versions, using the exact same syntax used by pip.
 
-    The installed module version comes from the `site-packages` dir via `pkg_resources`.
+    The installed module version comes from the `site-packages` dir via `importlib_metadata`.
 
     Args:
         requirement (:obj:`str`): pip style definition, e.g.,  "tokenizers==0.9.4", "tqdm>=4.27", "numpy"
@@ -70,20 +75,22 @@ def require_version(requirement: str, hint: Optional[str] = None) -> None:
     if pkg == "python":
         got_ver = ".".join([str(x) for x in sys.version_info[:3]])
         if not ops[op](version.parse(got_ver), version.parse(want_ver)):
-            raise pkg_resources.VersionConflict(
+            raise ImportError(
                 f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}."
             )
         return
 
     # check if any version is installed
     try:
-        got_ver = pkg_resources.get_distribution(pkg).version
-    except pkg_resources.DistributionNotFound:
-        raise pkg_resources.DistributionNotFound(requirement, ["this application", hint])
+        got_ver = importlib_metadata.version(pkg)
+    except importlib_metadata.PackageNotFoundError:
+        raise importlib_metadata.PackageNotFoundError(
+            f"The '{requirement}' distribution was not found and is required by this application. {hint}"
+        )
 
     # check that the right version is installed if version number was provided
     if want_ver is not None and not ops[op](version.parse(got_ver), version.parse(want_ver)):
-        raise pkg_resources.VersionConflict(
+        raise ImportError(
             f"{requirement} is required for a normal functioning of this module, but found {pkg}=={got_ver}.{hint}"
         )
 


Test patch:
diff --git a/tests/test_versions_utils.py b/tests/test_versions_utils.py
--- a/tests/test_versions_utils.py
+++ b/tests/test_versions_utils.py
@@ -16,9 +16,13 @@
 
 import numpy
 
-import pkg_resources
 from transformers.testing_utils import TestCasePlus
-from transformers.utils.versions import require_version, require_version_core, require_version_examples
+from transformers.utils.versions import (
+    importlib_metadata,
+    require_version,
+    require_version_core,
+    require_version_examples,
+)
 
 
 numpy_ver = numpy.__version__
@@ -57,7 +61,7 @@ def test_core(self):
         for req in ["numpy==1.0.0", "numpy>=1000.0.0", f"numpy<{numpy_ver}"]:
             try:
                 require_version_core(req)
-            except pkg_resources.VersionConflict as e:
+            except ImportError as e:
                 self.assertIn(f"{req} is required", str(e))
                 self.assertIn("but found", str(e))
 
@@ -65,7 +69,7 @@ def test_core(self):
         for req in ["numpipypie>1", "numpipypie2"]:
             try:
                 require_version_core(req)
-            except pkg_resources.DistributionNotFound as e:
+            except importlib_metadata.PackageNotFoundError as e:
                 self.assertIn(f"The '{req}' distribution was not found and is required by this application", str(e))
                 self.assertIn("Try: pip install transformers -U", str(e))
 
@@ -87,7 +91,7 @@ def test_examples(self):
         # the main functionality is tested in `test_core`, this is just the hint check
         try:
             require_version_examples("numpy>1000.4.5")
-        except pkg_resources.VersionConflict as e:
+        except ImportError as e:
             self.assertIn("is required", str(e))
             self.assertIn("pip install -r examples/requirements.txt", str(e))
 
@@ -100,6 +104,6 @@ def test_python(self):
         for req in ["python>9.9.9", "python<3.0.0"]:
             try:
                 require_version_core(req)
-            except pkg_resources.VersionConflict as e:
+            except ImportError as e:
                 self.assertIn(f"{req} is required", str(e))
                 self.assertIn(f"but found python=={python_ver}", str(e))