Repository: huggingface/transformers

Issue:
`Trainer.evaluate()` crashes when using only tensorboardX
## Environment info

- `transformers` version: 4.9.1
- Platform: Linux-3.10.0-1160.31.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.9
- PyTorch version (GPU?): 1.8.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes, but not relevant
- Using distributed or parallel set-up in script?: no

### Who can help

This might be a one-line fix, and I will be submitting a PR shortly. However, it might be a sign of a bigger problem, so I'm still tagging the person listed for the trainer, @sgugger.

## Information

Model I am using: `gpt2` (not model-specific issue, though)

The problem arises when using:

- [x] the official example scripts: (give details below)

The tasks I am working on is the one given in the example script.

## To reproduce

Steps to reproduce the behavior:

1. Create an environment with [`requirements.txt`](https://github.com/huggingface/transformers/blob/v4.9.1/examples/pytorch/language-modeling/requirements.txt) and `tensorboardX==2.4` installed but without tensorboard itself installed.
2. Run [`run_clm.py`](https://github.com/huggingface/transformers/blob/v4.9.1/examples/pytorch/language-modeling/run_clm.py) with the following script (based on [the example in the README](https://github.com/huggingface/transformers/blob/v4.9.1/examples/pytorch/language-modeling/README.md#gpt-2gpt-and-causal-language-modeling)):

```bash
time python run_clm.py \
  --model_name_or_path gpt2 \
  --dataset_name wikitext \
  --dataset_config_name wikitext-2-raw-v1 \
  --do_train \
  --do_eval \
  --output_dir output_dir \
  --logging_dir output_dir/logs \
  --logging_strategy epoch \
  --num_train_epochs 3 \
  --per_device_train_batch_size 4 \
  --gradient_accumulation_steps 2 \
  --max_train_samples 16 \
  --max_eval_samples 8 \
  --report_to tensorboard
```

3. See the stack trace that was output:

```python
Traceback (most recent call last):
  File "run_clm.py", line 515, in <module>
    main()
  File "run_clm.py", line 483, in main
    metrics = trainer.evaluate()
  File "venv/lib/python3.7/site-packages/transformers/trainer.py", line 2055, in evaluate
    self.log(output.metrics)
  File "venv/lib/python3.7/site-packages/transformers/trainer.py", line 1720, in log
    self.control = self.callback_handler.on_log(self.args, self.state, self.control, logs)
  File "venv/lib/python3.7/site-packages/transformers/trainer_callback.py", line 371, in on_log
    return self.call_event("on_log", args, state, control, logs=logs)
  File "venv/lib/python3.7/site-packages/transformers/trainer_callback.py", line 388, in call_event
    **kwargs,
  File "venv/lib/python3.7/site-packages/transformers/integrations.py", line 391, in on_log
    self.tb_writer.add_scalar(k, v, state.global_step)
  File "venv/lib/python3.7/site-packages/tensorboardX/writer.py", line 453, in add_scalar
    self.comet_logger.log_metric(tag, display_name, scalar_value, global_step)
AttributeError: 'NoneType' object has no attribute 'log_metric'
```

(I edited the stack trace to remove the parts of the path outside the virtual environment for improved readability.)

## Expected behavior

The script should not crash.

## Notes

I figured out what is causing the crash. When training ends, `TensorBoardCallback.on_train_end()` is called, which runs `self.tb_writer.close()`, which sets `self.tb_writer.comet_logger` to `None`. When `TensorBoardCallback.on_log()` is called again during evaluation, `self.comet_logger` is called again, even though it's `None`. The bug appears to essentially be a use-after-free bug. This specific exception only happens when tensorboard is not installed because only tensorboardX uses `comet_logger`.

The solution is simple: set `self.tb_writer` to `None` immediately after the call to `self.tb_writer.close()`. When `TensorBoardCallback.on_log()` is called again during evaluation, the method detects that `self.tb_writer is None` and re-initializes it, which makes everything work, at least finishing without crashing. I will be releasing a PR with this fix very soon.

However, given that more of these logging callbacks can be called during evaluation and some of them also have `on_train_end()` functions that close resources, there might be a bigger problem here involving the calling of logging integrations during evaluation. I don't know enough about them to determine that for myself, though.



Discussion:

Resolving patch:
diff --git a/src/transformers/integrations.py b/src/transformers/integrations.py
--- a/src/transformers/integrations.py
+++ b/src/transformers/integrations.py
@@ -401,6 +401,7 @@ def on_log(self, args, state, control, logs=None, **kwargs):
     def on_train_end(self, args, state, control, **kwargs):
         if self.tb_writer:
             self.tb_writer.close()
+            self.tb_writer = None
 
 
 class WandbCallback(TrainerCallback):


Test patch: