Strange hyperparameter warning
Author: avacaondata
Pull request: no

## Environment info

- `transformers` version: Master Branch
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.7 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@amogkam 

## Information

Model I am using (Bert, XLNet ...):

BART

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Go to https://docs.ray.io/en/master/tune/examples/pbt_transformers.html and copy that code.
2. Execute that code.
3. Wait and when some iterations have been made, you'll see that a constant warning persists: 

```{python}
2021-01-28 17:33:56,863	WARNING trial_runner.py:420 -- Trial Runner checkpointing failed: Checkpoint must not be in-memory.
```
Although it seems a ray-related problem, from reading https://github.com/huggingface/transformers/pull/6747 I have arrived to the conclusion that maybe as the checkpointing integration has been removed from Transformers, PBT is no longer working. 
When I look into the logs, I see that effectively no perturbation is being made, and it should because perturbation_interval is set to 

## Expected behavior

It's expected that if I set perturbation_interval to 1, perturbations are made every 1 training iteration, but PBT is not doing any perturbation at all and I think it's because of some problem in the integration for checkpointing between Transformers and Ray Tune. 

Comments:
Hey @alexvaca0 thanks for bringing this up. Indeed our example is outdated. I updated it here https://github.com/amogkam/ray/blob/hf-pbt/python/ray/tune/examples/pbt_transformers/pbt_transformers.py and when running on the latest Ray wheels and the latest transformer release (4.2.2), I am seeing perturbations happening as expected
![image](https://user-images.githubusercontent.com/8068268/106226121-5e02d080-619b-11eb-9d32-2f5ac39446c9.png)

I'll also try this out on transformers master branch and see if it's working on that as well, and will update here again. Thanks!

I just tried on transformers master and am seeing it work as well. Please let me know if this updated example works for you @alexvaca0! 

Thank you so much for your quick and very helpful answer @amogkam I'm using also the latest ray wheels and master version of transformers, therefore that example should work for me too! I'm going to try it as soon as I can so that I can tell you if it works for me or not! Thank you :) 

I just checked your code and I don't find any change with respect to the official example except for the evaluation strategy, which was steps and now it's epochs, but maybe I'm missing something. I'll try first that example and then I'll try to apply it to my dataset. I have one more question regarding PBT with transformers: I'm observing that from time to time the models "re-start" from the beggining (that is, a model that had trained for 1.53 epochs suddenly returns to step 0 and starts from there). In my configuration I set number of epochs to 10, expecting that each of the 4 models in the population trains for 10 epochs, but mutating their configurations in the process. However, they'd never reach that number of epochs if they continue restarting from the beginning... Is there something I'm missing here? Do you think this issue will be also solved with the new training script? @amogkam  Thank you !! :) 

@amogkam 

2021-01-29 15:01:01,078	WARNING trial_runner.py:370 -- Trial Runner checkpointing failed: Checkpoint must not be in-memory.

It still throws this error, and I've checked that another anomaly still persists: models not always re-start training from the point where they left, but start from the beginning again... Any clues why this may be happening? It's strange that this doesn't occur always, as sometimes models do re-start training from the point where they left...

Hey @alexvaca0, could you share what your stdout looks like please? Also is this with any modifications to the example, and can you share the full code that you are using? Thanks!

Could you give me your email so that I can share it with you that way? :) @amogkam 

Hey yes you can send it to amog@anyscale.com

Great! Already sent :) @amogkam 

I am having the same problem ```Trial Runner checkpointing failed: Checkpoint must not be in-memory.```, but it does some times manage to create checkpoints, as I have ```PopulationBasedTraining: 4 checkpoints, 2 perturbs```
Models seem to start training from the beginning again most of the time. I'm guessing it happens when the checkpoiting fails.

I also notice that I am getting some errors and warnings:

>WARNING function_runner.py:541 -- Function checkpointing is disabled. This may result in unexpected behavior when using checkpointing features or certain schedulers. To enable, set the train function arguments to be `func(config, checkpoint_dir=None)

>ERROR syncer.py:72 -- Log sync requires rsync to be installed.

Did you find a solution? 

Also, I don't really understand how the ```perturbation_interval``` and ```time_attr``` arguments work together. It seems to consider the models for perturbations at the number of ```logging_steps``` I set in ```TrainingArguments```, but as I understand it, it is supposed to do so after every training step (so after every minibatch?) when ```time_attr=training_iteration``` and ```perturbation_interval=1.``` Since that's how it seems to work, I set ```checkpoint_freq``` to the same value as ```logging_steps``` in ```TrainingArguments```


Here's what I think is the relevant part of my code:

``` python
class UCCTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        outputs = model(
            input_ids=inputs['input_ids'],
            attention_mask=inputs['attention_mask'],
            token_type_ids=inputs['token_type_ids']
        )
        loss = th.nn.BCEWithLogitsLoss()(outputs['logits'], inputs['labels'])
        return (loss, outputs) if return_outputs else loss


def model_init():
    return BertForSequenceClassification.from_pretrained(
        config.MODEL_NAME, return_dict=True
    )


def objective(metrics):
    try:
        return metrics[config.COMPUTE_OBJECTIVE]
    except KeyError:
        return metrics[f'eval_{config.COMPUTE_OBJECTIVE}']


def hp_space(trial):
    return {
        'learning_rate': tune.uniform(1e-5, 5e-5),
        'num_train_epochs': tune.choice([2, 3, 4, 5]),
        'seed': tune.choice(range(1, 50)),
        'weight_decay': tune.uniform(0.0, 0.3),
        'per_device_train_batch_size': tune.choice([10, 15, 20])
    }


def compute_metrics(eval_pred: EvalPrediction):
    scores = eval_pred.predictions  # np.array 4427x2
    labels = eval_pred.label_ids  # np.array 4427x2
    pred = np.argmax(scores, axis=1)
    labels_flat = np.argmax(labels, axis=1)
    return get_binary_metrics(pred, labels_flat)

if __name__ == '__main__':
    os.environ['WANDB_WATCH'] = 'all'
    tokenizer = BertTokenizer.from_pretrained(
        config.MODEL_NAME,
        do_lower_case=config.DO_LOWER_CASE
    )
    train_df = dataframe_from_json('data/train_balanced.json')
    train_binary = make_binary_df(train_df)
    train_data = UCCDataset(train_binary, tokenizer, config.MAX_LEN)
    total_steps = len(train_data)/config.TRAIN_BATCH_SIZE
    warmup_steps = round(0.1*total_steps)
    training_args = TrainingArguments(
        output_dir=config.OUTPUT_DIR,
        do_train=True,
        do_eval=True,
        evaluation_strategy='steps',
        learning_rate=config.LEARNING_RATE,
        weight_decay=0.1,
        logging_steps=config.LOG_INTERVAL,
        seed=1,
        disable_tqdm=True,
        report_to=['wandb'],
        run_name=config.RUN_NAME,
        load_best_model_at_end=config.LOAD_BEST_LAST,
        metric_for_best_model=config.COMPUTE_OBJECTIVE,
        logging_first_step=True,
        lr_scheduler_type='linear',
        warmup_steps=warmup_steps
    )
    val_df = get_clean_df(pd.read_csv('data/val.csv'))
    val_binary = make_binary_df(val_df)
    val_data = UCCDataset(val_df, tokenizer, config.MAX_LEN)
    model_config = BertConfig(
        vocab_size=tokenizer.vocab_size,
        pretrained_model_name_or_path=config.MODEL_NAME,
        num_labels=config.N_LABELS,
        return_dict=True
    )

    trainer = UCCTrainer(
        args=training_args,
        train_dataset=train_data,
        eval_dataset=val_data,
        tokenizer=tokenizer,
        model_init=model_init,
        compute_metrics=compute_metrics
    )

    ray_scheduler = PopulationBasedTraining(
        time_attr='training_iteration',
        metric=f'eval_{config.COMPUTE_OBJECTIVE}',
        mode='max',
        perturbation_interval=1,
        hyperparam_mutations={
            'learning_rate': tune.uniform(1e-5, 5e-5),
            'num_train_epochs': tune.choice([2, 3, 4, 5]),
            'seed': tune.choice(range(1, 50)),
            'weight_decay': tune.uniform(0.0, 0.3),
            'per_device_train_batch_size': tune.choice([10, 15, 20])
        }
    )
    best_model = trainer.hyperparameter_search(
        hp_space=hp_space,
        compute_objective=objective,
        n_trials=3,
        direction='maximize',
        backend='ray',
        # the following arguments are kwargs for tune.run
        scheduler=ray_scheduler,
        name='testmars5',
        resources_per_trial={'cpu': 1, 'gpu': 1},
        keep_checkpoints_num=3,
        checkpoint_score_attr="training_iteration",
        checkpoint_freq=config.LOG_INTERVAL
    )
```

Hey folks, is this still an issue?

cc @jwa018 @khrystynaFaryna