GPU Usage?
Author: AdityaSoni19031997
Pull request: no
Labels: name: wontfix

**Question**
> Note the query/issue might not have anything to do with the library as such, just looking for info as to why it will happen. Thanks for understanding.

- Why would the GPU show a usage[verified using `nvidia-smi`] of 420MB/32GB when i import `transformers`?

Note this only happens when i have `tensorflow-gpu 2.0` version in the same enviornment, otherwise it just works normally.

Pytorch v1.3.0
Transformers v2.1.1
tf-gpu v2.0.0
apex 0.1
GPU Tesla V100-SXM2-32GB
NVIDIA-SMI 410.79       
Driver Version: 410.79 
CUDA Version: 10.0
Python 3.7.3

Thanks.

(Sorry for a vague title and a quey)


Extra Outputs when i import transformers

```
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
2019-10-12 18:33:00.840558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-10-12 18:33:00.881658: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: Tesla V100-SXM2-32GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:b5:00.0
2019-10-12 18:33:00.881762: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881816: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881854: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881892: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881931: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.881967: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882003: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2019-10-12 18:33:00.882015: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2019-10-12 18:33:00.882328: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-10-12 18:33:00.909789: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2400000000 Hz
2019-10-12 18:33:00.916559: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec5e0b90 executing computations on platform Host. Devices:
2019-10-12 18:33:00.916618: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-10-12 18:33:01.606656: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b9ec644220 executing computations on platform CUDA. Devices:
2019-10-12 18:33:01.606754: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Tesla V100-SXM2-32GB, Compute Capability 7.0
2019-10-12 18:33:01.607078: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-10-12 18:33:01.607115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      
>>> 
```


Comments:
the same here.

installing last apex code from repository: git clone https://github.com/NVIDIA/apex

it says it's apex-0.1 version, but i think it should say apex-1.0

I run into this problem while trying to create virtual GPU devices:

```python
import tensorflow as tf
import transformers

devices =  tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_virtual_device_configuration(devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)]) 
```
that ends up with error:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-d51e3d242817> in <module>
----> 1 tf.config.experimental.set_virtual_device_configuration(devices[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])

~/miniconda3/envs/transformers/lib/python3.7/site-packages/tensorflow_core/python/framework/config.py in set_virtual_device_configuration(device, virtual_devices)
    554     virtual_devices: (optional) Need to update
    555   """
--> 556   context.context().set_virtual_device_configuration(device, virtual_devices)

~/miniconda3/envs/transformers/lib/python3.7/site-packages/tensorflow_core/python/eager/context.py in set_virtual_device_configuration(self, dev, virtual_devices)
   1269     if self._context_handle is not None:
   1270       raise RuntimeError(
-> 1271           "Virtual devices cannot be modified after being initialized")
   1272 
   1273     self._virtual_device_map[dev] = virtual_devices

RuntimeError: Virtual devices cannot be modified after being initialized
```
versions:
* Platform Linux-5.0.9-050009-generic-x86_64-with-debian-buster-sid
* Python 3.7.5 (default, Oct 25 2019, 15:51:11) 
* [GCC 7.3.0]
* PyTorch 1.2.0
* Tensorflow 2.0.0

The reason is the use of class variable `dummy_inputs` in `transformers/modeling_tf_utils.py:54` where tensorflow is initialized (and starts using GPU) at import time. I created a [PR](https://github.com/huggingface/transformers/pull/1735) that should fix this.

Thanks for the PR and figuring it where the issue lied.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.