Unnessasry library imports?
Author: sweetcocoa
Pull request: no

### System Info

I don't think it has anything to do with version or running environment. But if needed, I will add example system environments.

### Who can help?

@Narsil @sgugger 

It's hard to choose who is relevant, but I tag the people who seems most likely to be relevant.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/huggingface/transformers/blob/4d40109c3a93c9b8bbca204cb046ed510f1c72e8/src/transformers/utils/generic.py#L74

For many deep learning frameworks (e.g. TensorFlow, Pytoch, JAX, etc.), importing them causes significant overhead. 

Also, in some rare cases, just doing an import can cause interactions with other libraries that lead to errors. 
- ([Link](https://github.com/apache/tvm/issues/12326): an error caused by overlapping symbols between dynamic libraries, which is fortunately fixable).
- ([Link](https://github.com/pytorch/pytorch/issues/99637) : In some cases, If Tensorflow is imported before Torch, the script will hang and need to be forced kill).
    
  ```python
  >>> from transformers.utils import is_tensor
  >>> import tensorflow as tf
  2023-08-25 13:07:44.801120: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
  2023-08-25 13:07:44.802180: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
  2023-08-25 13:07:44.822067: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
  2023-08-25 13:07:44.822321: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
  To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
  2023-08-25 13:07:45.104908: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
  >>> x = tf.zeros(1)
  >>> is_tensor(x)
  (Hang at this point.)
  ```
  above example executed with 
  ```
  Python 3.9.16
  torch==2.0.1+cpu
  tensorflow==2.13.0
  ```

This leads me to believe that imports in deep learning frameworks should be minimized and avoided if possible. 

In my opinion, the `is_tensor` function in `transformers.utils` is meant to check if the argument is a tensor class used by various frameworks, but the problem with running this function is that it imports unnecessary frameworks in the Python execution environment. For example, it will force Python to import torch even if your code doesn't use torch. 

```python
>>> from transformers.utils import is_tensor
>>> import sys
>>> is_tensor(1)
2023-08-25 12:58:25.681444: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-25 12:58:25.682571: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-08-25 12:58:25.702710: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2023-08-25 12:58:25.703004: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-25 12:58:26.013967: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
False
>>> "torch" in sys.modules.keys()
True
```

To avoid this, there is another way. It's to compare the string of the object's class name to the target class. This method is simple, but works well in most cases. 
```python
>> str(type(x)) == "<class 'torch.Tensor'>"
True
```

However, there is one case where this doesn't work, and that is when something is an object that inherits from the tensor class. In this case, the object's class name will change, and we won't be able to tell if it's a tensor by comparing strings. 

```python
>>> class NewTensor(torch.Tensor): pass
...
>>> x = NewTensor()
>>> str(type(x)) == "<class 'torch.Tensor'>"
False
>>> isinstance(x, torch.Tensor)
True
```

So my suggestion is to change the logic of is_tensor to roughly the following.

1. compare the class name of the object to the name of the target class to see if it is a tensor.
2. if it's False for tensors in all frameworks, then import the library and compare it using the isinstance method.

With this code change, the performance drop should be negligible since string comparisons are not that big of an overhead, and in some cases, our code may even see a performance gain since we won't be importing unnecessary libraries.

What do you think?

### Expected behavior

If `is_tensor` uses string comparison first, It works fine in most cases without importing unused libraries. In certain situations, we may not even encounter hangs.

Comments:
Hi. I understand the situation you described. Using string to compare against isn't a super good way in general (as we don't know if there are some exceptional cases), but this approach does avoid significant overhead of loading several frameworks.

WDYT, @amyeroberts @ArthurZucker ?

We can't allow this as it is very brittle, but we can for sure change the order of the checks. @ydshieh would you like to open a PR? 

Not sure what order we prefer. Here the torch is imported first which is good IIRC. Unless a user imports TensorFlow manually before calling this method - which might cause hang.

Yes, importing torch first is preferable. Given the brittleness of checking against string and that most usage is with pytorch, I don't think we should include this change as it stands. 

Looking in the library, it seems that this function is only used in one place [here](https://github.com/huggingface/transformers/blob/a35f889acc91cb40bd8c6659691aeb27581a69b1/src/transformers/utils/generic.py#L278) in the `ModelOutput` class. As each ModelOutput will already have a predefined framework there'll be a solution which won't require us to dynamically check and import different libraries. For example, we could have a `TFModelOutput` class, and `ModelOutput` and `TFModelOutput` each define their own `is_tensor` method to use when `__post_init__` is called. WDYT?

There are 3 other usage, one is from

https://github.com/huggingface/transformers/blob/245dcc49ef9862a7165aec7be9c4a3299b8d06a1/src/transformers/tokenization_utils_base.py#L735

Regarding the output classes, we currently have things like
```
class TFBaseModelOutput(ModelOutput):
```
So it is not very motivating to have yet `TFModelOutput`, `FlaxModelOutput` etc. and make changes to use them. I would suggest to keep as what we have for now, and revisit this situation in the future if things get significant.

@sweetcocoa: Regarding `Tensorflow is imported before Torch`, you could try `TF_FORCE_GPU_ALLOW_GROWTH=true` as env. variable.

> There are 3 other usage, one is from

I definitely might have missed some references. The one in tokenization_base isn't using the function in `utils/generic.py`. Instead, `is_tensor` is defined above directly e.g. [here](https://github.com/huggingface/transformers/blob/245dcc49ef9862a7165aec7be9c4a3299b8d06a1/src/transformers/tokenization_utils_base.py#L703C3-L703C3). And it's the same in [feature_extraction_utils.py](https://github.com/huggingface/transformers/blob/245dcc49ef9862a7165aec7be9c4a3299b8d06a1/src/transformers/feature_extraction_utils.py#L151) and [tokenization_jukebox.py](https://github.com/huggingface/transformers/blob/245dcc49ef9862a7165aec7be9c4a3299b8d06a1/src/transformers/models/jukebox/tokenization_jukebox.py#L312).

> I would suggest to keep as what we have for now, and revisit this situation in the future if things get significant.

Agreed! Let's revisit if there's more activity here or we see similar things arising in other issues.


Sorry, yeah, I messed up the definition of `is_tensor` 😭 !

@amyeroberts @ydshieh @ArthurZucker Thanks to everyone who has discussed this issue! I understand the reluctance to use a string representation to check that a variable is an instance of a particular class (I don't think this is `Pythonic`, so I don't think my solution is clean either!). 

However, I don't think there can be any 'execptional' cases for my proposed approach of `using a string representation and then importing the library`. Am I missing something? (unless, maliciously, someone decides to name a non-tensor class as `torch.Tensor`).

My personal favorite solution is to set up a separate executable with only the frameworks I need, and as @ydshieh pointed out, control via environment variables is a good option. 

However, huggingface's transformers is a very popular and widely used framework, so it's also widely used by a wide range of beginner developers who may not be comfortable applying such a solution. I suggested this issue because I was helping someone troubleshoot an unexplained hang and found the reason in `is_tensor` (I had to explain to him why he needed to import torch in code that didn't use torch).

Thanks again for discussing the issues I raised. 



@sweetcocoa The question you raised is a very good and valid one, thank you again 🤗 . It's just we don't have a clean way to handle it and it's not significant (at least not yet). And indeed, having a clean and separate (virtual) working environment is the good practice, which is stated [here](https://huggingface.co/docs/transformers/installation#install-with-pip).

Note that we can also be smart and avoid importing any framework by making some checks first (like numpy arrays or if the object is a list/tuple, we can return `False` before trying to import torch/tf/flax). I'm not sure what was the initial use case here and if it would fully solve the issue at hand, but it can make the situation a bit better.

Oh and while the pattern matching is a brittle approach to do the check, it can inform the order in which we want to do those framework imports (e.g. if we see the repr looks like a torch Tensor, we try torch first, but if we see it looks like a tf tensor, we try tf first). Happy to make a PoC in a PR if that helps.

Yes, sound super great, and would love to review your PR :-)