codecarbon plugin issues
Author: stas00
Pull request: no

https://github.com/huggingface/transformers/pull/12304 added the `codecarbon` plugin and there are multiple issues with it:


1. it needs to be documented in 

https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/training_args.py#L316-L319

along all the other plugins

2. It doesn't respect user's logging level. It needs to read the set for the current rank log-level and pass it explicitly to the object instance here: 

https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/integrations.py#L782

via the `log_level` argument (but which expects a string like "warning" and not the real `logging.WARNING` which is normally used, so one needs to remap from real `logging` level to the string CC expects.

3. same logs are logged more than once in different formats:
```
[codecarbon INFO @ 19:33:14] Tracking Nvidia GPU via pynvml
[codecarbon INFO @ 19:33:14] Tracking Intel CPU via RAPL interface
08/23/2021 19:33:14 - INFO - codecarbon - Tracking Nvidia GPU via pynvml
08/23/2021 19:33:14 - INFO - codecarbon - Tracking Intel CPU via RAPL interface
```

4. it breaks the training as it can't find some server.

```
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='get.geojs.io', port=443): Max retries exceeded with url: /v1/ip/geo.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcd95312700>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/core/util.py", line 10, in suppress
    yield
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 348, in stop
    emissions_data = self._prepare_emissions_data()
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 367, in _prepare_emissions_data
    geo: GeoMetadata = self._get_geo_metadata()
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/emissions_tracker.py", line 612, in _get_geo_metadata
    return GeoMetadata.from_geo_js(self._data_source.geo_js_url)
  File "/mnt/nvme1/code/huggingface/codecarbon/codecarbon/external/geography.py", line 83, in from_geo_js
    response: Dict = requests.get(url, timeout=0.5).json()
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='get.geojs.io', port=443): Max retries exceeded with url: /v1/ip/geo.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fcd95312700>: Failed to establish a new connection: [Errno 111] Connection refused'))
08/23/2021 19:33:20 - WARNING - codecarbon - stopping.
```

This part of CC never worked for me.  It always fails here for me and not just in this integration. Only Offline version of the tracker works w/o this failure. Could we use the offline tracker by default instead?

----------------------

To reproduce - I run:
```
python examples/pytorch/translation/run_translation.py --model_name_or_path google/mt5-small --do_train --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config_name ro-en --output_dir output_dir --per_device_train_batch_size=4 --logging_step 2 --save_steps 0 --fp16 --max_train_samples 10 --save_total_limit 0 --overwrite_output_dir --save_strategy no
```
Thank you.

@JetRunner, @sgugger 

Comments:
Thanks for the feedback @stas00!

I don't think it runs automatically - it needs to be activated by setting a flag, just like other integrations. I will double check what's going on there. For other issues, I'll investigate as well.

Investigating it some more it's like all the other plugins. So it runs automatically unless the user explicitly turns it off. Not my favorite Trainer feature.

I had to add `--report_to none` to disable it and everything else. 

https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/training_args.py#L316-L319

That doc is outdated.

I will update the Issue to remove the incorrect part of the report, as I can turn it off explicitly. 

Now that it can be disabled there is absolutely no rush with the rest.

@stas00 I 100% agree with you that the default value of `--report_to` should be `none` instead of `all`. Detecting what's installed then enabling everything is a weird and aggressive behavior.
cc @sgugger 

This is planned to be so in v5. But for back-compat reasons it remains "all" - I don't remember all the history, but it probably needs to be on by default for `tensorboard` and none of the other plugins (or perhaps several plugins).

i.e. most likely the default shouldn't be 'all' but only the list of plugins that are required for back-compat.

Sounds good.

@stas00 For the online tracker,  could you `ping get.geojs.io` so I can determine what's wrong there? Since it works fine on my side.

```
$ ping get.geojs.io
PING get.geojs.io (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.036 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.038 ms
^C
--- get.geojs.io ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2041ms
rtt min/avg/max/mdev = 0.036/0.045/0.061/0.011 ms
```

but it doesn't respond to 80 or 443 ports:
```
$ HEAD https://get.geojs.io
500 Can't connect to get.geojs.io:443 (Connection refused)
Content-Type: text/plain
Client-Date: Tue, 24 Aug 2021 06:05:27 GMT
Client-Warning: Internal response

$ HEAD http://get.geojs.io
500 Can't connect to get.geojs.io:80 (Connection refused)
Content-Type: text/plain
Client-Date: Tue, 24 Aug 2021 06:05:36 GMT
Client-Warning: Internal response
```

As I mentioned in OP this is not a new problem, has been like this for at least one month.

Got it. My guess is the `geojs` service is not available in some countries. Let me see how we can add an offline option.

It will also break on instances without internet, like JZ.

pinging @JetRunner 

> pinging @JetRunner

will add when I'm back from the leave

Unfortunately it doesn't look like this is ever going to be fixed.

Could we please disable this plugin from loading by default if it has no maintainer that takes care of making it work?

@sgugger, @LysandreJik 

Switching to `codecarbon.OfflineEmissionsTracker` should at least solve this particular issue.

I'm happy to review any PR @stas00 :-)

The problem is that this feature was added without tests. So it's far from just changing its main class.

Moreover we dropped `codecarbon` from the BigScience since it has multiple issues and was causing more issues than it was solving. After investing so much time into trying to make this module work, I'm not keen on giving it any more of my energy. So my vote is to disable it for now until it becomes more mature.

To keep it, at the very least @JetRunner (who added it) or someone else needs to write a test for it.

Hey @stas00
As you know I have already left HF for a full-time PhD and I don't have any bandwidth to do it. I don't want to comment on the quality of codecarbon but feel free to remove it if you think it's buggy. For BigScience, our WG will switch to another way for carbon emission estimation.

Thank you for letting us know that you have no resources to work on this, @JetRunner 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.