Issue 1:
I posted this report in the HuggingFace Forum at first, but @BramVanroy kindly told me to post the report here instead of the forum.

The link to the post in the forum: https://discuss.huggingface.co/t/some-unintended-things-happen-in-seq2seqtrainer-example/2361

Environment info

transformers version: 4.0.0-rc-1

The latest commit: commit 5ced23d

Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10

Python version: 3.8.3

PyTorch version (GPU?): 1.7.0 (True)

Tensorflow version (GPU?): 2.3.1 (True)

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: No

Who can help

Trainer: @sgugger

examples/seq2seq: @patil-suraj

Information

Model I am using (Bert, XLNet ...): facebook/bart-large

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below)

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below)

I used the XSum dataset following the README of examples/seq2seq.

To reproduce

What seems strange

The number of data pairs is not correctly recognized.

MLflow cannot treat the params (too long).

I wasn’t sure if I should divide these into two issues, but in the end, I decided on one.

If it is better to divide them into two, I will modify it.

I first noticed this strangeness when I use a different dataset than the those in the example.

I again follow the README of examples/seq2seq to check if my modification causes the problem or not.

Having checked #8792, I used --evaluation_strategy epoch instead of --evaluate_during_training.

Run official example scripts

$ CUDA_VISIBLE_DEVICES=0 python finetune_trainer.py \
--data_dir $XSUM_DIR \
--learning_rate=3e-5 \
--fp16 \
--do_train --do_eval --do_predict \
--evaluation_strategy epoch \
--predict_with_generate \
--n_val 1000 \
--model_name_or_path facebook/bart-large \
--output_dir ./xsum_bart-large/ \
--save_total_limit 5 \
2>&1 | tee tmp.log

Expected behavior

Log

[INFO|trainer.py:667] 2020-11-30 08:10:43,836 >> ***** Running training *****
[INFO|trainer.py:668] 2020-11-30 08:10:43,836 >> Num examples = 204016
[INFO|trainer.py:669] 2020-11-30 08:10:43,836 >> Num Epochs = 3
[INFO|trainer.py:670] 2020-11-30 08:10:43,836 >> Instantaneous batch size per device = 8
[INFO|trainer.py:671] 2020-11-30 08:10:43,836 >> Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:672] 2020-11-30 08:10:43,836 >> Gradient Accumulation steps = 1
[INFO|trainer.py:673] 2020-11-30 08:10:43,836 >> Total optimization steps = 76506

...

mlflow.exceptions.MlflowException: Param value '{'summarization': {'length_penalty': 1.0, 'max_length': 128, 'min_length': 12, 'num_beams': 4}, 'summarization_cnn': {'length_penalty': 2.0, 'max_length': 142, 'min_length': 56, 'num_beams': 4}, 'summarization_xsum': {'length_penalty': 1.0, 'max_leng' had length 293, which exceeded length limit of 250

(Reference) Dataset length

$ cd $XSUM_DIR/
$ wc -l *
11333 test.source
11333 test.target
204017 train.source
204017 train.target
11327 val.source
11327 val.target
453354 total

Details

The number of examples shown

At first, I tried to use the dataset with 40,000 pairs for training, but it was shown that Num examples = 39999.

I don't know why, so I've checked the example with the XSum dataset.

Checking the number of lengths, it seems the XSum train set used in the example has 204017 pairs, but it is shown Num examples = 204016 as above.

I thought the dataset was supposed to start with the first line, but am I mistaken? For example, is the first line treated as a header?

MLflow can not treat params in this case

As shown above, the length of param value exceeds the limit that MLflow can handle.

Do I just need to change the settings of MLflow? Or, should I add some modifications to param value to be used in MLflow?

Thank you in advance.

Issue 2:
System Info

transformers version: 4.26.0.dev0

Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.28

Python version: 3.9.16

Huggingface_hub version: 0.12.0

PyTorch version (GPU?): 1.13.1 (False)

Tensorflow version (GPU?): 2.11.0 (False)

Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)

Jax version: 0.3.6

JaxLib version: 0.3.5

Using GPU in script?: False

Using distributed or parallel set-up in script?: False

Who can help?

@sgugger

Information

The official example scripts

My own modified scripts

Tasks

An officially supported task in the examples folder (such as GLUE/SQuAD, ...)

My own task or dataset (give details below)

Reproduction

When I try to add a new model as per the tutorial here, I get the following error with the given set of inputs:

$ transformers-cli add-new-model-like
What is the model you would like to duplicate? Please provide the lowercase `model_type` (e.g. roberta): roberta
What is the name (with no special casing) for your new model in the paper (e.g. RoBERTa)? NewTransformer
What identifier would you like to use for the `model_type` of this model? [newtransformer]
What lowercase name would you like to use for the module (folder) of this model? [newtransformer]
What prefix (camel-cased) would you like to use for the model classes of this model (e.g. Roberta)? [NewTransformer]
What prefix (upper-cased) would you like to use for the constants relative to this model? [NEWTRANSFORMER]
What will be the name of the config class for this model? [NewTransformerConfig]
Please give a checkpoint identifier (on the model Hub) for this new model (e.g. facebook/roberta-base):
Will your new model use the same processing class as roberta (RobertaTokenizer) (yes/no)? no
What will be the name of the tokenizer class for this model? [NewTransformerTokenizer]
Traceback (most recent call last):
File "/home/stuli/.conda/envs/bin/transformers-cli", line 8, in <module>
sys.exit(main())
File "/scratch/gpfs/stuli/transformers/src/transformers/commands/transformers_cli.py", line 54, in main
service = args.func(args)
File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1351, in add_new_model_like_command_factory
return AddNewModelLikeCommand(config_file=args.config_file, path_to_repo=args.path_to_repo)
File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1382, in __init__
) = get_user_input()
File "/scratch/gpfs/stuli/transformers/src/transformers/commands/add_new_model_like.py", line 1583, in get_user_input
image_processor_class=image_processor_class,
UnboundLocalError: local variable 'image_processor_class' referenced before assignment

Expected behavior

There should be no error with the given sequence of inputs when creating a new model.