Importing hides underlying error
Author: aphedges
Pull request: no

## Environment info

(Couldn't run due to bug this issue is about but did my best to fill it in)

- `transformers` version: 4.9.1
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.11
- PyTorch version (GPU?): 1.7.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

No person for general `transformers` issues is listed

## To reproduce

Steps to reproduce the behavior:

1. Run `pip install torch-scatter==2.0.6 -f https://pytorch-geometric.com/whl/torch-1.7.0+cpu.html` (install version that has a bug)
2. Use a machine with CUDA support.
3. Run `python -c 'from transformers import AutoModelForCausalLM'` to see the following output:

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'AutoModelForCausalLM' from 'transformers' (venv/lib/python3.7/site-packages/transformers/__init__.py)
```

4. Run `transformers-cli env` to see the following output:

```python
Traceback (most recent call last):
  File "venv/lib/python3.7/site-packages/torch_scatter/__init__.py", line 14, in <module>
    f'{library}_{suffix}', [osp.dirname(__file__)]).origin)
AttributeError: 'NoneType' object has no attribute 'origin'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "venv/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "venv/lib/python3.7/site-packages/transformers/commands/transformers_cli.py", line 23, in <module>
    from .run import RunCommand
  File "venv/lib/python3.7/site-packages/transformers/commands/run.py", line 17, in <module>
    from ..pipelines import SUPPORTED_TASKS, TASK_ALIASES, Pipeline, PipelineDataFormat, pipeline
  File "venv/lib/python3.7/site-packages/transformers/pipelines/__init__.py", line 30, in <module>
    from .automatic_speech_recognition import AutomaticSpeechRecognitionPipeline
  File "/venv/lib/python3.7/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 20, in <module>
    from .base import Pipeline
  File "venv/lib/python3.7/site-packages/transformers/pipelines/base.py", line 43, in <module>
    from ..models.auto.modeling_auto import AutoModel
  File "venv/lib/python3.7/site-packages/transformers/models/auto/modeling_auto.py", line 271, in <module>
    from ..tapas.modeling_tapas import (
  File "venv/lib/python3.7/site-packages/transformers/models/tapas/modeling_tapas.py", line 51, in <module>
    from torch_scatter import scatter
  File "venv/lib/python3.7/site-packages/torch_scatter/__init__.py", line 17, in <module>
    raise AttributeError(e)
AttributeError: 'NoneType' object has no attribute 'origin'
```


(I edited the stack trace to remove the parts of the path outside the virtual environment for improved readability.)

I was fortunate to find this easy-to-create stack trace when writing up this issue. It was actually difficult to find out what was the cause. I had to find the failing line in `transformers` (`from torch_scatter import scatter`) in a much more annoying manner instead.

## Expected behavior

Regular importing of the form `from transformers import ...` should display the full stack trace of the underlying error to provide a usable error message for debugging.

I obviously do not expect this underlying error to be fixed, given that it's not part of `transformers`. However, given that a comment near the failure considers `torch_scatter` a `soft dependency`, it might be a good idea to emit a warning when the package fails to import instead of causing the entire `transformers` library to fail. I'm not using a model that uses `torch_scatter` in the first place, so it shouldn't be required this way.

Comments:
I cannot reproduce your exact issue, but installing a version of torch-scatter with an incompatible CUDA version will indeed kill the runtime with an `OSError`. We could catch that error when defining the `_scatter_available` variable, which currently only looks at the package installation.

WDYT @sgugger @NielsRogge ?

Yes, this should be checked as easy as possible to yield an easier error message.

As for the initial issue, there is no try/except that are ignored on our side, so I'm afraid this is a more general problem with the Python import system. It's not the first time I see it hiding the underlying issues. If you have any idea of what we could do on our side to display those error messages, I'm all ears.

It's understandable that you can't get my specific problem reproduced. Therefore, I created a much simpler proof-of-concept:

1. Run `pip install transformers==4.9.1 torch==1.9.0`.
2. Run `python -c 'from transformers.file_utils import is_scatter_available; print(is_scatter_available())'` to confirm that `torch_scatter` is not found by `transformers`.
3. Create `setup.py`:

```python
from setuptools import setup
setup(name='torch_scatter')
```

3. Create `torch_scatter.py`:

```python
scatter = None
```

4. Run `pip install .`
5. Run `python -c 'from transformers.file_utils import is_scatter_available; print(is_scatter_available())'` to confirm that `torch_scatter` is being found by `transformers`.
6. Run `python -c 'from transformers import AutoModelForCausalLM'` to see there is no error thrown.
7. Add `None.origin` to a new line at the top of `torch_scatter.py`.
8. Run `python -c 'from transformers import AutoModelForCausalLM'` again to see the following error

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'AutoModelForCausalLM' from 'transformers' (/Users/ahedges/.pyenv/versions/scatter/lib/python3.7/site-packages/transformers/__init__.py)
```

Given that I could see complete errors when using the wrong CUDA version of `torch_scatter`, I decided to try multiple exception-triggering statements on the first line of `torch_scatter.py`:

- `None.origin`: `ImportError: cannot import name 'AutoModelForCausalLM' from 'transformers'`
- `foo`: `NameError: name 'foo' is not defined`
- `raise RuntimeError("This is a message")`: `RuntimeError: This is a message`
- `[0][2]`: `IndexError: list index out of range`
- `0 / 0`: `ZeroDivisionError: division by zero`

All but the first had properly displayed errors and stack traces. This led me to believe that the specific issue isn't exceptions getting ignored but `AttributeError`s in particular.

I do not have a strong knowledge of `transformers`'s import system or the Python import system in general, but I used [`transformers/__init__.py`](https://github.com/huggingface/transformers/blob/v4.9.1/src/transformers/__init__.py) and [`transformers/file_utils.py`](https://github.com/huggingface/transformers/blob/v4.9.1/src/transformers/file_utils.py) to create a very simplified script that reproduces the problem. I made two files:

- `test.py`:

```python
import importlib
import os
import sys
from types import ModuleType

class _LazyModule(ModuleType):
    def __init__(self, name, module_file, import_structure, extra_objects=None):
        super().__init__(name)
        self._modules = set(import_structure.keys())
        self._class_to_module = {}
        for key, values in import_structure.items():
            for value in values:
                self._class_to_module[value] = key
        self.__all__ = list(import_structure.keys()) + sum(import_structure.values(), [])
        self.__file__ = module_file
        self.__path__ = [os.path.dirname(module_file)]
        self._objects = {} if extra_objects is None else extra_objects
        self._name = name
        self._import_structure = import_structure

    def __getattr__(self, name: str):
        if name in self._objects:
            return self._objects[name]
        if name in self._modules:
            value = self._get_module(name)
        elif name in self._class_to_module.keys():
            module = self._get_module(self._class_to_module[name])
            value = getattr(module, name)
        else:
            raise AttributeError(f"module {self.__name__} has no attribute {name}")

        setattr(self, name, value)
        return value

    def _get_module(self, module_name: str):
        return importlib.import_module("." + module_name, self.__name__)

_import_structure = {"auto": ["AutoModel"]}

sys.modules[__name__] = _LazyModule(__name__, globals()["__file__"], _import_structure)
```

- `auto.py`:

```python
from torch_scatter import scatter

AutoModel = None
```

I could trigger the same kinds of errors that I got with `transformers` with `python -c 'from test import AutoModel'`. I then modified `_LazyModule.__getattr__()` to always `raise AttributeError()`, and I end up getting `ImportError: cannot import name 'AutoModel' from 'test'` with the following stack trace:

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'AutoModel' from 'test' (/Users/ahedges/Downloads/scatter_test/test.py)
```

Replacing the `AttributeError` with `RuntimeError` gets a more detailed stack trace:

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1032, in _handle_fromlist
  File "/Users/ahedges/Downloads/scatter_test/test.py", line 24, in __getattr__
    raise RuntimeError()
RuntimeError
```

If I replace `test.py` with the line `from auto import AutoModel`, then the `AttributeError` stack trace is displayed properly. This lends evidence to the fact that this bug is related to how `transformers` implements importing.

From these experiments, I think the problem is that the `transformers` importing machinery is specifically ignoring any `AttributeError`s while allowing others to propagate freely. Annoyingly, I can't find any mention of such behavior in the Python docs, so I can't tell if this is part of any official interface.

I'm unsure of how to better resolve this issue, though. It might make sense to modify `_LazyModule._get_module()` (the only part of the class that should be able to throw such an error without messing with the import machinery itself) to have better handling of `AttributeError`s. Maybe printing stack traces for them, but that could get annoying. Maybe embed it in a more general type, but I have no clue how that will interfere with the importing system.

I apologize for the very long read, but I hope this helps.


The reason is `pip` can't get you the right version of `torch_scatter`. I add a `try-catch` to prevent instant kill and throw an elegant message instead. #13040

@JetRunner, your PR is for a different but related issue than the one I reported here.

Plus, I'd like to point out that `pip` can get you the right version of `torch_scatter`. You just need to installing using the appropriate wheel page with the `-f` option.

Thank you for the detailed analysis @aphedges 
First, could you confirm if the problem appears again on master? It should not and you have an env setup for debugging so you should be able to see that quickly. I'll follow the steps of your reproducer later today if you can't confirm.

Then, from your deep analysis, it looks like there is a problem with the `AttributeError` in the import machinery, somewhere. We don't ignore them in the `_LazyModule` part, or any part of the Transformers library dedicated to imports, so my first thought is that it comes from Python itself, but I'll need to investigate more to be sure.

@sgugger, I can confirm that your commit 9870093 prevents this issue for me. The unclear error message will still be a problem for anyone importing from `modeling_tapas.py`, such as the following:

```python
$ python -c 'from transformers import TapasModel'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'TapasModel' from 'transformers' (/Users/ahedges/.pyenv/versions/scatter/lib/python3.7/site-packages/transformers/__init__.py)
```

However, this is much more limited in scope than before, which is very good. I don't use TAPAS, so I should be fine now.

I agree with you that the error getting lost seems to come from Python itself. I could not find `AttributeError`s being caught by importing in `transformers` during my investigation, but I unfortunately couldn't find any documentation of similar behavior in the official Python documentation, either. Part of the reason that this debugging was so difficult was because large portions of the stack were in Python internals that PyCharm's debugger couldn't reach. I'm not sure what `transformers` should (or can) do anything to deal with this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.