Repository: huggingface/transformers

Issue:
Some unintended things happen in Seq2SeqTrainer example
I posted this report in the HuggingFace Forum at first, but @BramVanroy kindly told me to post the report here instead of the forum.
The link to the post in the forum: https://discuss.huggingface.co/t/some-unintended-things-happen-in-seq2seqtrainer-example/2361

## Environment info
     
- `transformers` version: 4.0.0-rc-1
    - The latest commit: commit 5ced23dc845c76d5851e534234b47a5aa9180d40
- Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

 Trainer: @sgugger
 examples/seq2seq: @patil-suraj

## Information

Model I am using (Bert, XLNet ...): facebook/bart-large

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

I used the XSum dataset following the README of `examples/seq2seq`.

## To reproduce

### What seems strange

- The number of data pairs is not correctly recognized.
- MLflow cannot treat the params (too long).

I wasn’t sure if I should divide these into two issues, but in the end, I decided on one.
If it is better to divide them into two, I will modify it.

I first noticed this strangeness when I use a different dataset than the those in the example.
I again follow the README of `examples/seq2seq` to check if my modification causes the problem or not.

Having checked https://github.com/huggingface/transformers/issues/8792, I used `--evaluation_strategy epoch` instead of `--evaluate_during_training`.

### Run official example scripts

```
$ CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
    --data_dir $XSUM_DIR \
    --learning_rate=3e-5 \
    --fp16 \
    --do_train --do_eval --do_predict \
    --evaluation_strategy epoch \
    --predict_with_generate \
    --n_val 1000 \
    --model_name_or_path facebook/bart-large \
    --output_dir ./xsum_bart-large/ \
    --save_total_limit 5 \
    2>&1 | tee tmp.log
```

## Expected behavior

### Log

```
[INFO|trainer.py:667] 2020-11-30 08:10:43,836 >> ***** Running training *****
[INFO|trainer.py:668] 2020-11-30 08:10:43,836 >>   Num examples = 204016
[INFO|trainer.py:669] 2020-11-30 08:10:43,836 >>   Num Epochs = 3
[INFO|trainer.py:670] 2020-11-30 08:10:43,836 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:671] 2020-11-30 08:10:43,836 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:672] 2020-11-30 08:10:43,836 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:673] 2020-11-30 08:10:43,836 >>   Total optimization steps = 76506

...

mlflow.exceptions.MlflowException: Param value '{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_leng' had length 293, which exceeded length limit of 250
```

### (Reference) Dataset length

```sh
$ cd $XSUM_DIR/
$ wc -l *
    11333 test.source
    11333 test.target
   204017 train.source
   204017 train.target
    11327 val.source
    11327 val.target
   453354 total
```

### Details 

#### The number of examples shown

At first, I tried to use the dataset with 40,000 pairs for training, but it was shown that `Num examples = 39999`.
I don't know why, so I've checked the example with the XSum dataset.

Checking the number of lengths, it seems the XSum train set used in the example has 204017 pairs, but it is shown `Num examples = 204016` as above.

I thought the dataset was supposed to start with the first line, but am I mistaken? For example, is the first line treated as a header?

#### MLflow can not treat params in this case

As shown above, the length of `param value` exceeds the limit that MLflow can handle.
Do I just need to change the settings of MLflow? Or, should I add some modifications to `param value` to be used in MLflow?

Thank you in advance.


Discussion:
I have never looked at the `finetune_trainer.py` script so I can't reply for the number of examples part.
For the MLFlow problem, I don't understand how the value of this parameter could be longer than 250 (if interpreted has a string) could you print it out for debugging?
Thank you for your quick response!

First, here is more detail of the error message about MLFlow problem. 
I apologize that I didn't give the information in the first of this issue.

```python
Traceback (most recent call last):
  File "finetune_trainer.py", line 310, in <module>
    main()
  File "finetune_trainer.py", line 254, in main
    trainer.train(
  File "/path/to/transformers/src/transformers/trainer.py", line 713, in train
    self.control = self.callback_handler.on_train_begin(self.args, self.state, self.control)
  File "/path/to/transformers/src/transformers/trainer_callback.py", line 336, in on_train_begin
    return self.call_event("on_train_begin", args, state, control)
  File "/path/to/transformers/src/transformers/trainer_callback.py", line 374, in call_event
    result = getattr(callback, event)(
  File "/path/to/transformers/src/transformers/integrations.py", line 502, in on_train_begin
    self.setup(args, state, model)
  File "/path/to/transformers/src/transformers/integrations.py", line 497, in setup
    mlflow.log_params(dict(combined_dict_items[i : i + MLflowCallback.MAX_LOG_SIZE]))
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/tracking/fluent.py", line 470, in log_params
    MlflowClient().log_batch(run_id=run_id, metrics=[], params=params_arr, tags=[])
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/tracking/client.py", line 830, in log_batch
    self._tracking_client.log_batch(run_id, metrics, params, tags)
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/tracking/_tracking_service/client.py", line 246, in log_batch
    self.store.log_batch(run_id=run_id, metrics=metrics, params=params, tags=tags)
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/store/tracking/file_store.py", line 852, in log_batch
    _validate_batch_log_data(metrics, params, tags)
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/utils/validation.py", line 221, in _validate_batch_log_data
    _validate_param(param.key, param.value)
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/utils/validation.py", line 101, in _validate_param
    _validate_length_limit("Param value", MAX_PARAM_VAL_LENGTH, value)
  File "$HOME/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/mlflow/utils/validation.py", line 169, in _validate_length_limit
    raise MlflowException(
mlflow.exceptions.MlflowException: Param value '{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_leng' had length 293, which exceeded length limit of 250
```
The error message says the error is caused in line 497 of `integrations.py`.  

https://github.com/huggingface/transformers/blob/5ced23dc845c76d5851e534234b47a5aa9180d40/src/transformers/integrations.py#L497

I added logger.info before that.

```python
                # debug
                logger.info("--- dict --- %s", dict(combined_dict_items[i : i + MLflowCallback.MAX_LOG_SIZE]))
```

Then, the output is as below:

```python
[INFO|integrations.py:499] 2020-11-30 16:39:51,612 >> --- dict --- {'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'use_bfloat16': False, 'pruned_heads': {}, 'tie_word_embeddings': True, 'is_encoder_decoder': True, 'is_decoder': False, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 128, 'min_length': 12, 'do_sample': False, 'early_stopping': True, 'num_beams': 4, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 3, 'bad_words_ids': None, 'num_return_sequences': 1, 'chunk_size_feed_forward': 0, 'architectures': ['BartModel', 'BartForConditionalGeneration', 'BartForSequenceClassification'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1, 'LABEL_2': 2}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 0, 'pad_token_id': 1, 'eos_token_id': 2, 'sep_token_id': None, 'decoder_start_token_id': 2, 'task_specific_params': {'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_length': 62, 'min_length': 11, 'num_beams': 6}}, 'xla_device': None, '_name_or_path': 'facebook/bart-large', 'classif_dropout': 0.1, 'model_type': 'bart', 'num_hidden_layers': 12, 'vocab_size': 50265, 'd_model': 1024, 'encoder_ffn_dim': 4096, 'encoder_layers': 12, 'encoder_attention_heads': 16, 'encoder_layerdrop': None, 'decoder_layerdrop': None, 'decoder_ffn_dim': 4096, 'decoder_layers': 12, 'decoder_attention_heads': 16, 'max_position_embeddings': 1024, 'init_std': 0.02, 'activation_function': 'gelu', 'scale_embedding': False, 'normalize_embedding': True, 'normalize_before': False, 'add_final_layer_norm': False, 'add_bias_logits': False, 'static_position_embeddings': False, 'attention_dropout': None, 'activation_dropout': 0.1, 'dropout': None, 'classifier_dropout': 0.0, 'extra_pos_embeddings': 2, 'force_bos_token_to_be_generated': False, 'do_blenderbot_90_layernorm': False, 'use_cache': True, 'output_dir': './xsum_bart-large_no_cuda/', 'overwrite_output_dir': False, 'do_train': True, 'do_eval': True, 'do_predict': True, 'model_parallel': False, 'evaluation_strategy': 'epoch', 'prediction_loss_only': False, 'per_device_train_batch_size': 8, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 1, 'eval_accumulation_steps': None, 'learning_rate': 3e-05, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 3.0, 'max_steps': -1, 'warmup_steps': 0, 'logging_dir': 'runs/Nov30_16-39-34_hamo', 'logging_first_step': False, 'logging_steps': 500, 'save_steps': 500, 'save_total_limit': 5, 'no_cuda': True, 'seed': 42, 'fp16': False}
```

The error message seems to indicate the `'task_specific_params'`, so I've checked the length of it.

```
>>> str = "{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_length': 62, 'min_length': 11, 'num_beams': 6}}"
>>> len(str)
293
```

Should I have added some processing to `task_specific_params`?

Thank you.

Mmm, that's weird. Pining @noise-field as this was the user that added integration with MLFlow.
Hi, @forest1988 
Thank you for the detailed bug description. MLFlow does limit the parameter length (see: mlflow/mlflow#1976). 

I think we probably need to stop sending arbitrarily nested parameters as string literals because they are:
- not actually single parameters
- can easily overflow the 250 symbols limit

Another idea would be to skip long parameters and produce a warning like in case of invalid metrics values. 

@sgugger what would you suggest would be a better option? I could fix it this week.


Maybe we could just skip the args we are trying to send to MLFlow when they get over the limit?
Hi, I have the same issue. I'm using more or less the standard `run_glue.py` script for finetuning. Most models worked but BART threw the same error as above.
Fortunately, this error happened at the start. But, I wrote my own trainer callback handler which failed only after 1-22 hours in the training process and interrupted the training, because some backend API failed to respond.

I'm not sure whether it might make some sense to just wrap all the callback calls into try-catch blocks so the training will continue in any case?
Here is my solution https://github.com/huggingface/transformers/issues/8967#issue-758695096
There was an error where the callback tried to log a value that is too long for MLflow. It was fixed in this PR: #8875
Hi, 
It seems PR #8875 will solve this issue, are there any problems that block the PR from merging?

(Added: I'm sorry for the duplicate comments.)

Currently, I am dealing with this issue temporarily as follows.
(Trainer works without MLflow integration)

```
# remove MLflowCalback temporarily
from transformers.integrations import MLflowCallback
trainer.callback_handler.remove_callback(MLflowCallback)
```
Oh sorry @noise-field it seems like your PR slipped through the cracks of our review process. In general, don't hesitate to ping the person that reviewed your PR if there is no activity in a week and you believe you addressed every comment.

Resolving patch:
diff --git a/src/transformers/integrations.py b/src/transformers/integrations.py
--- a/src/transformers/integrations.py
+++ b/src/transformers/integrations.py
@@ -707,12 +707,13 @@ class MLflowCallback(TrainerCallback):
     A :class:`~transformers.TrainerCallback` that sends the logs to `MLflow <https://www.mlflow.org/>`__.
     """
 
-    MAX_LOG_SIZE = 100
-
     def __init__(self):
         assert is_mlflow_available(), "MLflowCallback requires mlflow to be installed. Run `pip install mlflow`."
         import mlflow
 
+        self._MAX_PARAM_VAL_LENGTH = mlflow.utils.validation.MAX_PARAM_VAL_LENGTH
+        self._MAX_PARAMS_TAGS_PER_BATCH = mlflow.utils.validation.MAX_PARAMS_TAGS_PER_BATCH
+
         self._initialized = False
         self._log_artifacts = False
         self._ml_flow = mlflow
@@ -738,10 +739,21 @@ def setup(self, args, state, model):
             if hasattr(model, "config") and model.config is not None:
                 model_config = model.config.to_dict()
                 combined_dict = {**model_config, **combined_dict}
+            # remove params that are too long for MLflow
+            for name, value in list(combined_dict.items()):
+                # internally, all values are converted to str in MLflow
+                if len(str(value)) > self._MAX_PARAM_VAL_LENGTH:
+                    logger.warning(
+                        f"Trainer is attempting to log a value of "
+                        f'"{value}" for key "{name}" as a parameter. '
+                        f"MLflow's log_param() only accepts values no longer than "
+                        f"250 characters so we dropped this attribute."
+                    )
+                    del combined_dict[name]
             # MLflow cannot log more than 100 values in one go, so we have to split it
             combined_dict_items = list(combined_dict.items())
-            for i in range(0, len(combined_dict_items), MLflowCallback.MAX_LOG_SIZE):
-                self._ml_flow.log_params(dict(combined_dict_items[i : i + MLflowCallback.MAX_LOG_SIZE]))
+            for i in range(0, len(combined_dict_items), self._MAX_PARAMS_TAGS_PER_BATCH):
+                self._ml_flow.log_params(dict(combined_dict_items[i : i + self._MAX_PARAMS_TAGS_PER_BATCH]))
         self._initialized = True
 
     def on_train_begin(self, args, state, control, model=None, **kwargs):
@@ -757,13 +769,10 @@ def on_log(self, args, state, control, logs, model=None, **kwargs):
                     self._ml_flow.log_metric(k, v, step=state.global_step)
                 else:
                     logger.warning(
-                        "Trainer is attempting to log a value of "
-                        '"%s" of type %s for key "%s" as a metric. '
-                        "MLflow's log_metric() only accepts float and "
-                        "int types so we dropped this attribute.",
-                        v,
-                        type(v),
-                        k,
+                        f"Trainer is attempting to log a value of "
+                        f'"{v}" of type {type(v)} for key "{k}" as a metric. '
+                        f"MLflow's log_metric() only accepts float and "
+                        f"int types so we dropped this attribute."
                     )
 
     def on_train_end(self, args, state, control, **kwargs):
@@ -771,13 +780,12 @@ def on_train_end(self, args, state, control, **kwargs):
             if self._log_artifacts:
                 logger.info("Logging artifacts. This may take time.")
                 self._ml_flow.log_artifacts(args.output_dir)
-            self._ml_flow.end_run()
 
     def __del__(self):
         # if the previous run is not terminated correctly, the fluent API will
         # not let you start a new run before the previous one is killed
         if self._ml_flow.active_run is not None:
-            self._ml_flow.end_run(status="KILLED")
+            self._ml_flow.end_run()
 
 
 INTEGRATION_TO_CALLBACK = {


Test patch: