text: 
  
  TITLE
  Project: Gather summarization datasets and try to replicate pegasus results on them
  
  COMMENTS
  37
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Dear @stas00 and whoever else is willing to help!
  
  So far I have only checked pegasus' rouge scores on 2/12 datasets for which we have checkpoints.
  For the other 10 datasets I either haven't tried or have tried briefly and gotten stuck.
  
  The full scope of the project is that:
  for each dataset:
  
  
  1) There is an automated way to download the data, either from S3 or source. (To the extent possible, much of the logic in this script should eventually live in the `datasets` package). 
  2) we know our pegasus implementation's rouge score
  2b) if our score is very different than the authors', we know whether that difference is due to data preprocessing, and if it is, we can preprocess the dataset similarly to the pegasus authors.
  3) Our rouge score is within 0.3 Rouge2 of the reported. (Authors) column below.
  
  
  
  ### Steps
  
  #### Getting Data
  
  By far the most difficult part of each project is getting the dataset. And giving up quickly if you can't and writing a github issue somewhere.
  I tried 1 approach to getting data: [this script](https://gist.github.com/sshleifer/c4aed7bf4418b50caee731e94be05d9f)
  It worked for gigaword, I just haven't done the evaluation, but it failed for `aeslc` and then I gave up. 
  
  
  Another complementary approach would be to try to directly use the [pegasus dataset code](https://github.com/google-research/pegasus/blob/master/pegasus/data/public_supervised_datasets.py)
  
  This will likely push preprocessing issues towards the back of the project. (when we try to send PRs to the datasets repo), but might be better than using my script. 
  
  
  #### After you get data
  
  When you have gotten a dataset you can sanity check 
  
  ```bash
  python -m torch.distributed.launch --nproc_per_node=2  run_distributed_eval.py \
  	--model_name google/pegasus-large $\  # see note 1
      --save_dir xsum_generations \
      --data_dir xsum \
  	--prefix test \
  	--n_obs 100 \
  ```
  Note 1: you can just keep running pegasus-large and expect a high single digits or better rouge2 score,to avoid downloading all the checkpoints or, you can change this to the relevant checkpoint.
  Note 2: I am happy to run all the evals on newer hardware, very easy for me.
  Note 3: We can do data sharing by getting you aws creds, or some other solution. Key is that I can download from command line, e.g. Google Drive + gdown. 
  
  
  ### Misc thoughts:
  
  + arxiv and pubmed are listed under `scientific_papers` in the datasets package.
  + This is really 10 projects (1 each dataset, 2 of which I've started). If I were you I would ignore the started 2 and start on a few other ones. 
  + If a dataset only has train/test or train/val or some other splits, see how the pegasus authors did the split.
  + Partial credit is valuable!
  + this could easily have been an issue for the datasets project rather than the transformers project.
  + There is no reason to merge PRs quickly for this project, but eventually we want a (much better) download_summ_dataset.py script or instructions for using other libs to accomplish the same outcome.
  + Will be good for both of us to learn the datasets internals.
  + Raw Billsum has multiple line articles, which breaks everything :( , (we could try to support raw nlp datasets in our `DataLoader`)
  
  Here is a copy of the table we are trying to fill out in #6844 : (I made a new issue to avoid spamming that one)
  
  | dataset | Authors| This Repo|
  | ---- | ----|----|
  | xsum | 47.60/24.83/39.64|  46.87/24.46/39.15|
  | cnn_dailymail | 44.16/21.56/41.30| see 1|
  | newsroom | 45.07/33.39/41.28 | have `.tar` file|
  | multi_news | 47.65/18.75/24.95|
  | gigaword | 39.65/20.47/36.76| 39.79/20.56/36.80|
  | wikihow | 46.39/22.12/38.41 *| Asked Authors |
  | reddit_tifu | 27.99/9.81/22.94|32.75/11.68/24.97|
  | big_patent |52.29/33.08/41.66 *| |
  | arxiv | 44.21/16.95/25.67| |
  | pubmed | 45.97/20.15/28.25| |
  | aeslc | 37.68/21.25/36.51|37.1/21.4/35.94| 
  | billsum | 59.67/41.58/47.59|54.99/37.43/43.07|
  
  Originally from mixed & stochastic column of this [table](https://github.com/google-research/pegasus#results-update)
  
  
  This was really long, and probably disorganized, so feel free to ask clarifying questions here or on slack!
  cc @stas00 
  
  
  1) I got similar scores on cnn-dailymail by finetuning the authors' model on our dataset for a bit.
  2) reddit_tifu: added `--min_length 32`