Issue 1:
Dear @stas00 and whoever else is willing to help!

So far I have only checked pegasus' rouge scores on 2/12 datasets for which we have checkpoints.

For the other 10 datasets I either haven't tried or have tried briefly and gotten stuck.

The full scope of the project is that:

for each dataset:

There is an automated way to download the data, either from S3 or source. (To the extent possible, much of the logic in this script should eventually live in the datasets package).

we know our pegasus implementation's rouge score

2b) if our score is very different than the authors', we know whether that difference is due to data preprocessing, and if it is, we can preprocess the dataset similarly to the pegasus authors.

Our rouge score is within 0.3 Rouge2 of the reported. (Authors) column below.

Steps

Getting Data

By far the most difficult part of each project is getting the dataset. And giving up quickly if you can't and writing a github issue somewhere.

I tried 1 approach to getting data: this script

It worked for gigaword, I just haven't done the evaluation, but it failed for aeslc and then I gave up.

Another complementary approach would be to try to directly use the pegasus dataset code

This will likely push preprocessing issues towards the back of the project. (when we try to send PRs to the datasets repo), but might be better than using my script.

After you get data

When you have gotten a dataset you can sanity check

python -m torch.distributed.launch --nproc_per_node=2 run_distributed_eval.py \
--model_name google/pegasus-large $\ # see note 1
--save_dir xsum_generations \
--data_dir xsum \
--prefix test \
--n_obs 100 \

Note 1: you can just keep running pegasus-large and expect a high single digits or better rouge2 score,to avoid downloading all the checkpoints or, you can change this to the relevant checkpoint.

Note 2: I am happy to run all the evals on newer hardware, very easy for me.

Note 3: We can do data sharing by getting you aws creds, or some other solution. Key is that I can download from command line, e.g. Google Drive + gdown.

Misc thoughts:

arxiv and pubmed are listed under scientific_papers in the datasets package.

This is really 10 projects (1 each dataset, 2 of which I've started). If I were you I would ignore the started 2 and start on a few other ones.

If a dataset only has train/test or train/val or some other splits, see how the pegasus authors did the split.

Partial credit is valuable!

this could easily have been an issue for the datasets project rather than the transformers project.

There is no reason to merge PRs quickly for this project, but eventually we want a (much better) download_summ_dataset.py script or instructions for using other libs to accomplish the same outcome.

Will be good for both of us to learn the datasets internals.

Raw Billsum has multiple line articles, which breaks everything :( , (we could try to support raw nlp datasets in our DataLoader)

Here is a copy of the table we are trying to fill out in #6844 : (I made a new issue to avoid spamming that one)

dataset
Authors
This Repo

xsum
47.60/24.83/39.64
46.87/24.46/39.15

cnn_dailymail
44.16/21.56/41.30
see 1

newsroom
45.07/33.39/41.28
have .tar file

multi_news
47.65/18.75/24.95

gigaword
39.65/20.47/36.76
39.79/20.56/36.80

wikihow
46.39/22.12/38.41 *
Asked Authors

reddit_tifu
27.99/9.81/22.94
32.75/11.68/24.97

big_patent
52.29/33.08/41.66 *

arxiv
44.21/16.95/25.67

pubmed
45.97/20.15/28.25

aeslc
37.68/21.25/36.51
37.1/21.4/35.94

billsum
59.67/41.58/47.59
54.99/37.43/43.07

Originally from mixed & stochastic column of this table

This was really long, and probably disorganized, so feel free to ask clarifying questions here or on slack!

cc @stas00

I got similar scores on cnn-dailymail by finetuning the authors' model on our dataset for a bit.

reddit_tifu: added --min_length 32

Issue 2:
Environment info

transformers version: 4.17.0

Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9

Python version: 3.6.13

PyTorch version (GPU?): 1.7.1 (True)

Tensorflow version (GPU?): not installed (NA)

Flax version (CPU?/GPU?/TPU?): not installed (NA)

Jax version: not installed

JaxLib version: not installed

Using GPU in script?: Yes

Using distributed or parallel set-up in script?: Using accelerate but there's only 1 GPU currently

Who can help

@LysandreJik, @SaulLu, @sgugger

-->

Information

Model I am using (Bert, XLNet ...): YituTech/conv-bert-base

The problem arises when using:

the official example scripts: (give details below)

my own modified scripts: (give details below) . Modified scripts but save_model method which is causing issues is exact same as this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460

The tasks I am working on is:

an official GLUE/SQUaD task: (give the name)

my own task or dataset: (give details below) - Similar GLUE like paired sentence classification task

To reproduce

Steps to reproduce the behavior:

For some models such as the one mentioned above, I've seen some times the script abruptly fails with a weird PanicException while trying to save the tokenizer. I don't think this is model specific issue, because other 3p models have saved successfully for me always, so looks like something to do with Transformers/tokenizers lib. See code that I'm using to save the model below, along with detailed logs as well.

I've no idea about this issue, and how to fix it. For other standard models like bert, roberta etc the saving has worked fine always, along with some other 3p models that I've tried as well. Please let me know if you any need more information from me on this?

I'm saving both the model and tokenizer only in main_process, I hope that's right? I find this slightly weird because clearly if you see from the logs below model is getting saved to dir successfully, but tokenizer specifically is failing. Not sure what could be the peculiarity here causing this? Any suggestions that you might have to fix this would be really helpful!

Relevant code to save model, which is causing the issue - same as this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460 . I've also included a couple of lines at the top in the code blurb below to show how I'm initializing the config/tokenizer/model if that might be of any relevance:

config = AutoConfig.from_pretrained(args.model_name_or_path, num_labels=num_labels, finetuning_task=None)
tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, use_fast=not args.use_slow_tokenizer)
model = AutoModelForSequenceClassification.from_pretrained(
args.model_name_or_path,
from_tf=bool(".ckpt" in args.model_name_or_path),
config=config,
cache_dir=args.output_dir
)

def save_model(model, tokenizer, args):
if args.output_dir is not None and accelerator.is_main_process:
# Handle the repository creation
if args.output_dir is not None:
os.makedirs(args.output_dir, exist_ok=True)
accelerator.wait_for_everyone()
unwrapped_model = accelerator.unwrap_model(model)
unwrapped_model.save_pretrained(args.output_dir, save_function=accelerator.save)
tokenizer.save_pretrained(args.output_dir)

Logs:

03/18/2022 11:51:47 - INFO - __main__ - ***** Running training *****
03/18/2022 11:51:47 - INFO - __main__ - Num examples = 25227
03/18/2022 11:51:47 - INFO - __main__ - Num Epochs = 10
03/18/2022 11:51:47 - INFO - __main__ - Instantaneous batch size per device = 32
03/18/2022 11:51:47 - INFO - __main__ - Total train batch size (w. parallel, distributed & accumulation) = 32
03/18/2022 11:51:47 - INFO - __main__ - Gradient Accumulation steps = 1
03/18/2022 11:51:47 - INFO - __main__ - Total optimization steps = 7890
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [28:11<00:00, 6.46it/s]03/18/2022 12:20:32 - INFO - __main__ - *** Predict ***
Configuration saved in conv-bert-base/config.json
Model weights saved in conv-bert-base/pytorch_model.bin
tokenizer config file saved in conv-bert-base/tokenizer_config.json
Special tokens file saved in conv-bert-base/special_tokens_map.json
thread '<unnamed>' panicked at 'no entry found for key', /__w/tokenizers/tokenizers/tokenizers/src/models/mod.rs:36:66
Traceback (most recent call last):
File "run.py", line 566, in <module>
main()
File "run.py", line 563, in main
tokenizer.save_pretrained(args.output_dir)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 2108, in save_pretrained
filename_prefix=filename_prefix,
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 599, in _save_pretrained
self.backend_tokenizer.save(tokenizer_file)
pyo3_runtime.PanicException: no entry found for key
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [29:26<00:00, 4.47it/s]
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/accelerate_cli.py", line 41, in main
args.func(args)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 384, in launch_command
simple_launcher(args)
File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 142, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/python', 'run.py', '--model_name_or_path', 'YituTech/conv-bert-base', '--train_file', '../data/train.csv', '--validation_file', '../data/val.csv', '--test_file', '../data/test.csv', '--max_length', '128', '--per_device_train_batch_size', '32', '--learning_rate', '2e-5', '--num_train_epochs', '10', '--output_dir', 'conv-bert-base/']' returned non-zero exit status 1.

Expected behavior

Tokenizer save should be successful.