Inreproducible loss when train same setting bert model >=2 times

Hi there,

I am using my customized bert script to train a model. However, everything even I keep the same setting for lr, AdamW weight decay and epoch, and run on the same platform (cuda on SageMaker) with same torch (1.5.0) and transformers (2.11.0) versions,  the results still change a lot in terms of the loss. This make my different experiments not comparable.

Can someone who has experienced this before or have any ideas please advice me on what should I do? I really want to solve this inreproducible issue so that I can continue on my experiments. Super appreciated for your help!

Details as below:

For example, if I set epoch = 4, lr = 1e-5, decay for AdamW as 0.01.
For one run I got this result for the first epoch only showing the last complete 100 batches result:
```
2020-10-19 03:45:29,032 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 2.261e-05 | loss 0.267 | Elapsed 0:12:29
2020-10-19 03:45:40,550 - utils - INFO -   Training epoch took: 0:12:41
2020-10-19 03:45:40,550 - utils - INFO - Validating...
2020-10-19 03:46:14,588 - utils - INFO - | loss 0.019 | Elapsed 0:00:34
              precision    recall  f1-score      support
False          0.906472  0.979875  0.941745  2087.000000
True           0.475000  0.152610  0.231003   249.000000
accuracy       0.891695  0.891695  0.891695     0.891695
macro avg      0.690736  0.566243  0.586374  2336.000000
weighted avg   0.860480  0.891695  0.865986  2336.000000
2020-10-19 03:46:15,403 - utils - INFO - Testing...
2020-10-19 03:46:55,182 - utils - INFO - use model: 1 batch / 1319 step
              precision  recall  f1-score   support
False             0.906   0.984     0.944  2344.000
True              0.413   0.098     0.159   265.000
accuracy          0.894   0.894     0.894     0.894
macro avg         0.659   0.541     0.551  2609.000
weighted avg      0.856   0.894     0.864  2609.000
2020-10-19 03:46:55,188 - utils - INFO - best test F1 score: 0.8638224640164368
```

And for the second attempt I got this for the first epoch:
```
2020-11-07 17:08:08,821 - utils - INFO - | epoch   1 |  1300/ 1320 batches | lr 2.261e-05 | loss 0.286 | Elapsed 0:12:25
2020-11-07 17:08:20,487 - utils - INFO -   Training epoch took: 0:12:37
2020-11-07 17:08:20,487 - utils - INFO - Validating...
2020-11-07 17:08:54,609 - utils - INFO - | loss 0.018 | Elapsed 0:00:34
              precision    recall  f1-score      support
False          0.893408  1.000000  0.943703  2087.000000
True           0.000000  0.000000  0.000000   249.000000
accuracy       0.893408  0.893408  0.893408     0.893408
macro avg      0.446704  0.500000  0.471852  2336.000000
weighted avg   0.798177  0.893408  0.843112  2336.000000
2020-11-07 17:08:55,313 - utils - INFO - Testing...
2020-11-07 17:09:34,934 - utils - INFO - use model: 1 batch / 1319 step
              precision  recall  f1-score   support
False             0.898   1.000     0.946  2344.000
True              0.000   0.000     0.000   265.000
accuracy          0.898   0.898     0.898     0.898
macro avg         0.449   0.500     0.473  2609.000
weighted avg      0.807   0.898     0.850  2609.000
2020-11-07 17:09:34,938 - utils - INFO - best test F1 score: 0.8503599608647853
```

Note that, the last used lr rate per 100 batches are the same, while the average loss per 100 batches are slightly different. But this result in the predictions for the validation and testing data set very different.

I already set the seed during my model with this function below:
```
def set_seed(seed):
    """ Set all seeds to make results reproducible (deterministic mode).
        When seed is a false-y value or not supplied, disables deterministic mode. """
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
```

And my model script is like this:
```
class ReviewClassification(BertPreTrainedModel):
    def __init__(self, config,
                 add_agent_text, agent_text_heads):
        """
        :param config: Bert configuration, can set up some parameters, like  output_attention, output_hidden_states
        :param add_agent_text: whether to use the non text feature, and how.
                It can have three options: None, "concat" and "attention"
        :param agent_text_heads: number of the heads in agent attention mechanism. Only useful if add_agent_text are set to
                "attention"
        """
        super().__init__(config)
        # self.num_labels = 2
        self.add_agent_text = add_agent_text

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)

        embedding_size = config.hidden_size

        if self.add_agent_text == "concat":
            embedding_size = 2 * embedding_size
        elif self.add_agent_text == "attention":
            self.agent_attention = nn.MultiheadAttention(embedding_size, num_heads=agent_text_heads)
        else:
            # don't use the information in Agent text
            pass

        self.classifier = nn.Linear(embedding_size, 1) # self.classifier = nn.Linear(embedding_size, len(LABEL_NAME)) # bias: If set to False, the layer will not learn an additive bias
        self.init_weights()

        print(
            """
            add agent text         :{}
            agent text multi-head  :{}
            """.format(self.add_agent_text, agent_text_heads)
        )

    def forward(
            self,
            review_input_ids=None,
            review_attention_mask=None,
            review_token_type_ids=None,
            agent_input_ids=None,
            agent_attention_mask=None,
            agent_token_type_ids=None,
            labels=None,
    ):

        review_outputs = self.bert(
            review_input_ids,
            attention_mask=review_attention_mask,
            token_type_ids=review_token_type_ids,
            position_ids=None,
            head_mask=None,
            inputs_embeds=None,
        )
        if self.add_agent_text is not None:
            # means that self.add_agent_text is "concat" or "attention"
            # TODO: we can try that agent_outputs do not share the same parameter
            agent_outputs = self.bert(
                agent_input_ids,
                attention_mask=agent_attention_mask,
                token_type_ids=agent_token_type_ids,
                position_ids=None,
                head_mask=None,
                inputs_embeds=None,
            )

        if self.add_agent_text == "attention":
            review_hidden_states = review_outputs[0].transpose(0, 1)  # before trans: (bs, seq_len, hidden_size)

            # want to take it as query, we need the it has the shape (#target_seq_len, batch_size, embedding_size)
            agent_hidden_states = agent_outputs[0].mean(axis=1).unsqueeze(dim=0)  # (1, batch_size, hidden_size)

            attn_output, _ = self.agent_attention(agent_hidden_states, review_hidden_states, review_hidden_states)
            feature = attn_output.squeeze()  # (batch_size, seq_len)
        else:
            feature = review_outputs[1]  # (batch_size, seq_len) -? Should it be (batch_size, hidden_size)

        if self.add_agent_text == "concat":
            feature = torch.cat([feature, agent_outputs[1]], axis=1)

        logits = self.classifier(feature).squeeze()

        outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            loss_fct = nn.BCEWithLogitsLoss().cuda() #pos_weight=pos_weight
            loss = loss_fct(logits, labels)
            outputs = (loss,) + outputs

        return outputs  # (loss, logits, hidden_states, attentions)

```
The loss is calculated using BCEWithLogitsLoss() from torch.nn.

The train, validation and test part script is as below:
```
import time
import pickle
from path import Path
import numpy as np
import pandas as pd

from sklearn.metrics import precision_recall_fscore_support, classification_report, confusion_matrix
import torch
import torch.nn as nn

from utils import LABEL_NAME, isnotebook, set_seed, format_time

if isnotebook():
    from tqdm.notebook import tqdm
else:
    from tqdm import tqdm

def model_train(model, train_data_loader, valid_data_loader, test_data_loader,
                logger, optimizer, scheduler, num_epochs, seed, out_dir):
    # move model to gpu
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    num_gpus = torch.cuda.device_count()
    logger.info("Let's use {} GPUs!".format(num_gpus))

    # Set the seed value all over the place to make this reproducible.
    set_seed(seed=seed)

    # We'll store a number of quantities such as training and validation loss,
    # validation accuracy, and timings.
    training_stats = []
    print_interval = 100

    # Measure the total training time for the whole run.
    total_t0 = time.time()
    batch_size = train_data_loader.batch_size
    num_batch = len(train_data_loader)
    best_f1_score = {
        "weighted": 0,
        "averaged": 0
    }
    best_test_f1_score = 0

    # For each epoch...
    for epoch_i in range(0, num_epochs):

        # ========================================
        #               Training
        # ========================================

        # Perform one full pass over the training set.
        logger.info("")
        logger.info('======== Epoch {:} / {:} ========'.format(epoch_i + 1, num_epochs))
        logger.info('Training...')

        # Reset the total loss for this epoch.
        total_train_loss = 0

        # Measure how long the training epoch takes.
        t_train = time.time()

        model.train()

        # For each batch of training data...
        for step, batch in tqdm(enumerate(train_data_loader), desc="Training Iteration", total=num_batch):
            # Progress update every 100 batches.
            if step % print_interval == 0 and not step == 0:
                # Calculate elapsed time in minutes.
                elapsed = format_time(time.time() - t_train)
                avg_train_loss = total_train_loss / print_interval

                # Report progress.
                logger.info('| epoch {:3d} | {:5d}/{:5d} batches | lr {:.3e} | loss {:5.3f} | Elapsed {:s}'.format(
                    epoch_i+1, step, num_batch, scheduler.get_last_lr()[0], avg_train_loss, elapsed)
                )
                total_train_loss = 0
                training_stats.append(
                    {
                        'epoch': epoch_i + 1,
                        'step': step,
                        'train loss': avg_train_loss,
                    }
                )

            # Unpack this training batch from our dataloader.
            #
            # As we unpack the batch, we'll also copy each tensor to the GPU using the
            # `to` method.
            #
            # `batch` contains four pytorch tensors:
            #   "input_ids"
            #   "attention_mask"
            #   "token_type_ids"
            #   "binarized_labels"

            b_review_input_ids = batch["review_input_ids"].to(device)
            b_review_attention_mask = batch["review_attention_mask"].to(device)
            b_review_token_type_ids = batch["review_token_type_ids"].to(device)
            b_agent_input_ids = batch["agent_input_ids"].to(device)
            b_agent_attention_mask = batch["agent_attention_mask"].to(device)
            b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

            b_binarized_label = batch["binarized_label"].to(device)

            model.zero_grad()
            (loss, _) = model(review_input_ids=b_review_input_ids,
                              review_attention_mask=b_review_attention_mask,
                              review_token_type_ids=b_review_token_type_ids,
                              agent_input_ids=b_agent_input_ids,
                              agent_attention_mask=b_agent_attention_mask,
                              agent_token_type_ids=b_agent_token_type_ids,

                              labels=b_binarized_label
                              )

            # Accumulate the training loss over all of the batches so that we can
            # calculate the average loss at the end. `loss` is a Tensor containing a
            # single value; the `.item()` function just returns the Python value
            # from the tensor.

            if num_gpus > 1:
                total_train_loss += loss.mean().item()
                loss.mean().backward()  # use loss.mean().backward() instead of loss.backward() for multiple gpu trainings
            else:
                total_train_loss += loss.item()
                loss.backward()

            # Clip the norm of the gradients to 1.0.
            # This is to help prevent the "exploding gradients" problem.
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and take a step using the computed gradient.
            # The optimizer dictates the "update rule"--how the parameters are
            # modified based on their gradients, the learning rate, etc.
            optimizer.step()
            scheduler.step()
        # End of training epoch

        # Measure how long this epoch took.
        training_time = format_time(time.time() - t_train)

        logger.info("")
        logger.info("  Training epoch took: {:s}".format(training_time))

        # evaluate the model after one epoch.

        # ========================================
        #               Validation
        # ========================================
        # After the completion of each training epoch, measure our performance on
        # our validation set.

        logger.info("")
        logger.info("Validating...")

        t_valid = time.time()
        model.eval()
        ave_valid_loss, valid_f1_table, cm_table, f1_score = model_validate(model=model, data_loader=valid_data_loader)
        # Measure how long this epoch took.
        validation_time = format_time(time.time() - t_valid)

        logger.info("")
        logger.info('| loss {:5.3f} | Elapsed {:s}'.format(ave_valid_loss, validation_time))
        logger.info("  \n{:s}".format(valid_f1_table.to_string()))
        logger.info("")
        logger.info("  \n{:s}".format(cm_table.to_string()))

        # need to store the best model
        for key in best_f1_score.keys():
            if best_f1_score[key] < f1_score[key]:
                # remove the old model:
                file_list = [f for f in out_dir.files() if f.name.endswith(".pt") and f.name.startswith(key)]
                for f in file_list:
                    Path.remove(f)
                model_file = out_dir.joinpath('{:s}_epoch_{:02d}-f1_{:.3f}.pt'.format(
                    key, epoch_i + 1, f1_score[key])
                )
                best_f1_score[key] = f1_score[key]
                if num_gpus > 1:
                    torch.save(model.module.state_dict(), model_file)
                else:
                    torch.save(model.state_dict(), model_file)

        # ========================================
        #               Test
        # ========================================
        logger.info("")
        logger.info("Testing...")

        result_df = model_test(model=model, data_loader=test_data_loader)

        y_true = np.array(result_df["review_label"], dtype=np.bool) # This part may need double check
        y_pred = result_df["Probability"] > 0.5

        report = classification_report(y_true, y_pred, output_dict=True)
        metrics_df = pd.DataFrame(report).transpose()

        metrics_df = metrics_df.sort_index()

        weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
        averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

        best_test_f1_score = metrics_df.loc['weighted avg', 'f1-score'] \
            if best_test_f1_score < metrics_df.loc['weighted avg', 'f1-score'] else best_test_f1_score

        metrics_df = metrics_df.astype(float).round(3)

        # Calculate confusion matrix
        tn, fp, fn, tp  = confusion_matrix(y_true, y_pred).ravel()
        cm_df = pd.DataFrame(columns = ['Predicted No', 'Predicted Yes'],
                       index = ['Actual No', 'Actual Yes'])
        # adding rows to an empty
        # dataframe at existing index
        cm_df.loc['Actual No'] = [tn,fp]
        cm_df.loc['Actual Yes'] = [fn,tp]

        logger.info("use model: {} batch / {} step".format(epoch_i + 1, step))
        logger.info("\n" + "=" * 50)
        logger.info("\n" + metrics_df.to_string())
        logger.info("\n" + "=" * 50)
        logger.info("\n" + cm_df.to_string())
        logger.info("best test F1 score: {}".format(best_test_f1_score))
        logger.info("\n" + "=" * 50)
        # Below is to save the result files
        result_filename = "result_df_epoch_" + str(epoch_i + 1) + ".xlsx"
        result_df.to_excel(out_dir.joinpath(result_filename), index=False)

    logger.info("")
    logger.info("Training complete!")
    logger.info("Total training took {:} (h:mm:ss)".format(format_time(time.time() - total_t0)))

    # Save training_stats to csv file
    pd.DataFrame(training_stats).to_csv(out_dir.joinpath("model_train.log"), index=False)
    return model, optimizer, scheduler

def model_validate(model, data_loader):
    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    model.eval()
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    label_prop = data_loader.dataset.dataset.label_prop()

    total_valid_loss = 0

    batch_size = data_loader.batch_size
    num_batch = len(data_loader)

    y_pred, y_true = [], []

    # Evaluate data
    for step, batch in tqdm(enumerate(data_loader), desc="Validation...", total=num_batch):
        b_review_input_ids = batch["review_input_ids"].to(device)
        b_review_attention_mask = batch["review_attention_mask"].to(device)
        b_review_token_type_ids = batch["review_token_type_ids"].to(device)
        b_agent_input_ids = batch["agent_input_ids"].to(device)
        b_agent_attention_mask = batch["agent_attention_mask"].to(device)
        b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

        b_binarized_label = batch["binarized_label"].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():
            (loss, logits,) = model(review_input_ids=b_review_input_ids,
                                    review_attention_mask=b_review_attention_mask,
                                    review_token_type_ids=b_review_token_type_ids,
                                    agent_input_ids=b_agent_input_ids,
                                    agent_attention_mask=b_agent_attention_mask,
                                    agent_token_type_ids=b_agent_token_type_ids,

                                    labels=b_binarized_label)

        total_valid_loss += loss.item()
        ### The sigmoid function is used for the two-class logistic regression,
        ### whereas the softmax function is used for the multiclass logistic regression

        # Version 1
        # numpy_probas = logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # Version 2
        # transfored_logits = F.log_softmax(logits,dim=1)
        # numpy_probas = transfored_logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # Version 3
        # transfored_logits = torch.sigmoid(logits)
        # numpy_probas = transfored_logits.detach().cpu().numpy()
        # y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
        # y_true.extend(b_binarized_label.cpu().numpy())

        # New version - for num_label = 1
        transfored_logits = torch.sigmoid(logits)
        numpy_probas = transfored_logits.detach().cpu().numpy()
        y_pred.extend(numpy_probas)
        y_true.extend(b_binarized_label.cpu().numpy())

    # End of an epoch of validation

    # put model to train mode again.
    model.train()

    ave_loss = total_valid_loss / (num_batch * batch_size)

    y_pred = np.array(y_pred)
    y_pred[y_pred < 0.5] = 0
    y_pred[y_pred >= 0.5] = 1

    # Below is in case the input and target are not the same data format
    y_pred = np.array(y_pred, dtype=np.bool)
    y_true = np.array(y_true, dtype=np.bool)

    # compute the various f1 score for each label
    report = classification_report(y_true, y_pred, output_dict=True)
    metrics_df = pd.DataFrame(report).transpose()
    # metrics_df = pd.DataFrame(0, index=LABEL_NAME, columns=["Precision", "Recall", "F1","support"])
    # metrics_df.Precision = precision_recall_fscore_support(y_true, y_pred)[0]
    # metrics_df.Recall = precision_recall_fscore_support(y_true, y_pred)[1]
    # metrics_df.F1 = precision_recall_fscore_support(y_true, y_pred)[2]
    # metrics_df.support = precision_recall_fscore_support(y_true, y_pred)[3]

    # y_pred = np.array(y_pred)
    # y_pred[y_pred < 0] = 0
    # y_pred[y_pred > 0] = 1
    # y_pred = np.array(y_pred, dtype=np.bool)
    # y_true = np.array(y_true, dtype=np.bool)

    # metrics_df = pd.DataFrame(0, index=LABEL_NAME, columns=["Precision", "Recall", "F1"], dtype=np.float)
    # # or_y_pred = np.zeros(y_pred.shape[0], dtype=np.bool)
    # # or_y_true = np.zeros(y_true.shape[0], dtype=np.bool)
    # for i in range(len(LABEL_NAME)):
    #     metrics_df.iloc[i] = precision_recall_fscore_support(
    #         y_true=y_true[:, i], y_pred=y_pred[:, i], average='binary', zero_division=0)[0:3]

        # or_y_pred = or_y_pred | y_pred[:, i]
        # or_y_true = or_y_true | y_true[:, i]

    metrics_df = metrics_df.sort_index()
    # metrics_df.loc['Weighted Average'] = metrics_df.transpose().dot(label_prop)
    # metrics_df.loc['Average'] = metrics_df.mean()

    # metrics_df.loc['Weighted Average', 'F1'] = 2 / (1/metrics_df.loc['Weighted Average', "Recall"] +
    #                                                 1/metrics_df.loc['Weighted Average', "Precision"])
    # metrics_df.loc['Average', 'F1'] = 2 / (1/metrics_df.loc['Average', "Recall"] +
    #                                        1/metrics_df.loc['Average', "Precision"])

    weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
    averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

    # Calculate confusion matrix
    tn, fp, fn, tp  = confusion_matrix(y_true, y_pred).ravel()
    cm_df = pd.DataFrame(columns = ['Predicted No', 'Predicted Yes'],
                   index = ['Actual No', 'Actual Yes'])
    # adding rows to an empty
    # dataframe at existing index
    cm_df.loc['Actual No'] = [tn,fp]
    cm_df.loc['Actual Yes'] = [fn,tp]

    # pooled_f1_score = f1_score(y_pred=or_y_pred, y_true=or_y_true)

    return ave_loss, metrics_df, cm_df,{
        "weighted": weighted_f1_score,
        "averaged": averaged_f1_score,
    }

def model_test(model, data_loader):
    # Put the model in evaluation mode--the dropout layers behave differently
    # during evaluation.
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model.eval()
    model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    num_batch = len(data_loader)
    # Below need to modify if change the input
    review_id, review_label, hmd_text, head_cust_text = [], [], [], []
    agent = []
    pred_logits = []

    # Evaluate data
    for step, batch in tqdm(enumerate(data_loader), desc="Inference...", total=num_batch):
        if "anecdote_lead_final" in batch.keys():
            review_label.extend(batch["anecdote_lead_final"])
        review_id.extend(batch["_id"].tolist())
        hmd_text.extend(batch["hmd_comments"])
        head_cust_text.extend(batch["head_cust"])
        agent.extend(batch["new_transcript_agent"])

        b_review_input_ids = batch["review_input_ids"].to(device)
        b_review_attention_mask = batch["review_attention_mask"].to(device)
        b_review_token_type_ids = batch["review_token_type_ids"].to(device)
        b_agent_input_ids = batch["agent_input_ids"].to(device)
        b_agent_attention_mask = batch["agent_attention_mask"].to(device)
        b_agent_token_type_ids = batch["agent_token_type_ids"].to(device)

        # Tell pytorch not to bother with constructing the compute graph during
        # the forward pass, since this is only needed for backprop (training).
        with torch.no_grad():
            (logits,) = model(review_input_ids=b_review_input_ids,
                              review_token_type_ids=b_review_token_type_ids,
                              review_attention_mask=b_review_attention_mask,
                              agent_input_ids=b_agent_input_ids,
                              agent_token_type_ids=b_agent_token_type_ids,
                              agent_attention_mask=b_agent_attention_mask
                              )

        if logits.detach().cpu().numpy().size == 1:
            pred_logits.extend(logits.detach().cpu().numpy().reshape(1,))
        else:
            pred_logits.extend(logits.detach().cpu().numpy())

    # End of an epoch of validation
    # put model to train mode again.
    model.train()
    pred_logits = np.array(pred_logits)
    pred_prob = np.exp(pred_logits)
    pred_prob = pred_prob / (1 + pred_prob)
    pred_label = pred_prob.copy()
    pred_label[pred_label < 0.5] = 0
    pred_label[pred_label >= 0.5] = 1
    # compute the f1 score for each tags
    d = {'Probability':pred_prob,'Anecdotes Prediction':pred_label}
    pred_df = pd.DataFrame(d, columns=['Probability','Anecdotes Prediction'])
    result_df = pd.DataFrame(
        {
            "review_id": review_id,
            "hmd_text": hmd_text,
            "head_cust_text": head_cust_text,
            "agent": agent
        }
    )
    if len(review_label) != 0:
        result_df["review_label"] =  [x.item() for x in review_label]
    return pd.concat([result_df, pred_df], axis=1).set_index("review_id")

```

optimizer and scheduler part are defined as below:
```
if args.full_finetuning:
    param_optimizer = list(model.named_parameters())
    no_decay = ['bias', 'gamma', 'beta']
    optimizer_grouped_parameters = [
        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
         'weight_decay_rate': args.decay},
        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
         'weight_decay_rate': 0.0}
    ]
else:
    param_optimizer = list(model.classifier.named_parameters())
    optimizer_grouped_parameters = [
        {"params": [p for n, p in param_optimizer]}
    ]

# Note: AdamW is a class from the huggingface library (as opposed to pytorch)
optimizer = AdamW(optimizer_grouped_parameters,  # or param_optimizer
                  lr=args.lr,  # args.learning_rate - default is 5e-5, our notebook had 1e-5
                  eps=1e-8)  # args.adam_epsilon  - default is 1e-8.

# Create the learning rate scheduler.
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=len(data_loader["train"]) * args.num_epochs
)
```
And to run the model, I use below script:
```
model_train(model=model, train_data_loader=data_loader["train"], valid_data_loader=data_loader["valid"],
            test_data_loader=data_loader["test"], optimizer=optimizer, scheduler=scheduler,
            num_epochs=args.num_epochs, seed=args.seed, logger=logger, out_dir=out_dir)
```


[model card] fix md table

model card

Related to pretrained model cards

Just noticed, I had a borked md table that wasn't rendering as a table on the site in a few cards, this PR fixes that.

Thank you!

@julien-c


Fix run_mlm_wwm example

# What does this PR do?

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Add barthez model

# What does this PR do?

Add BARThez models, tokenizer and docs. BARThez is a french seq2seq model that uses BART objective and architecture (https://arxiv.org/abs/2010.12321)

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@patrickvonplaten @sshleifer


Training script "run_mlm.py" doesn't work for certain datasets

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:3.4.0
- Python version:3.8.3
- PyTorch version (GPU?):3.6.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

  @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@LysandreJik

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below) MLM

## To reproduce

Steps to reproduce the behavior:
Basically run the training script in examples/run_mlm.py using wikipedia dataset
```
python run_mlm.py \
--model_name_or_path roberta-base \
--dataset_name wikipedia \
--dataset_config_name 20200501.en \
--do_train \
--output_dir /tmp/test-mlm \
```

## Erorr Message
```
Traceback (most recent call last):
  File "run_mlm2.py", line 388, in <module>
    main()
  File "run_mlm2.py", line 333, in main
    tokenized_datasets = tokenized_datasets.map(
  File "/home/zeyuy/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 283, in map
    {
  File "/home/zeyuy/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 284, in <dictcomp>
    k: dataset.map(
  File "/home/zeyuy/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1236, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/zeyuy/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1207, in does_function_return_dict    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "run_mlm2.py", line 315, in group_texts
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
  File "run_mlm2.py", line 315, in <dictcomp>
    concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
TypeError: can only concatenate list (not "str") to list
```

## Expected behavior

The training script should work for all datasets in huggingface datasets. The problem is that the feature other than 'text' (or the feature we train on) interferes when we try to concatenate the tokenized 'text' ('input_ids', 'mask_ids' ...) from each instances.

A quick fix would be on line [295](https://github.com/huggingface/transformers/blob/77a257fc210a56f1fd0d75166ecd654cf58111f3/examples/language-modeling/run_mlm.py#L295), change

`            remove_columns=[text_column_name],`
to
`            remove_columns=column_names,`

Should I open a PR or someone want to do a quick fix?

<!-- A clear and concise description of what you would expect to happen. -->


Bug fix for apply_chunking_to_forward chunking dimension check

Chunking should be in the chunking dimension, an exception was raised if the complete shape of the inputs was not the same rather than only the chunking dimension

# What does this PR do?
Fixes #8349

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten Incorporating the fix from the issue at https://github.com/huggingface/transformers/issues/8349
Let me know if you want a unit test or something! Ran all the tests too:
<img width="936" alt="Screen Shot 2020-11-07 at 3 29 26 PM" src="https://user-images.githubusercontent.com/5157240/98450932-1e35cb00-210f-11eb-855a-2457c44e243b.png">


Trainer QA Model Label Names

In the Trainer Class, the default label names for a QA model is ["start_positions, end_positions"]. You can see this defined on the line I linked here. Is this supposed to be the case or should it be a list with 2 strings? The tutorials for using Trainer on the transformers documentation show setting up a custom dataset with encoding having two separate keys for the start and end positions. Should we be setting the label_names parameter in our training args or is this a bug? If this is not a bug, could the documentation for setting up a custom QA dataset be updated so others don't get confused as well.
https://github.com/huggingface/transformers/blob/77a257fc210a56f1fd0d75166ecd654cf58111f3/src/transformers/trainer.py#L331


[fsmt tokenizer] support lowercase tokenizer

This PR:

* [x] implements support for `do_lower_case` in the fsmt tokenizer - requested [here](https://github.com/huggingface/transformers/pull/8374#issuecomment-723419221)
* [x] adds test to validate the new feature
* [x] adds case detector in the converter script

@LysandreJik


DataCollatorForWholeWordMask error persists after fix

After quick fix #8379
Tried same flow of #8378
**Gives error:**
```
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in __call__(self, examples)
    321         for e in examples:
    322             ref_tokens = []
--> 323             for id in e["input_ids"].tolist():
    324                 token = self.tokenizer._convert_id_to_token(id)
    325                 ref_tokens.append(token)

AttributeError: 'list' object has no attribute 'tolist'
```

When I tried with _DataCollatorForLanguageModeling_, trainer works as usual.


Add LMHeadModel similar to BertLMHeadModel to modeling_distilbert.py

# What does this PR do?

Similar to the `BertLMHeadModel` this PR aims to add a `DistilBertForCausalLM` model in `modeling_distilbert.py`.

Fixes #7397

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


- `transformers` version: 3.4.0
- Platform: macOS-10.13.6-x86_64-i386-64bit
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

The problem arises when using trying to import:
from transformers import AutoTokenizer, AutoModel

Running on Jupyter notebook in conda virtual env
full error message below:
`---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-50-13d0bae188dc> in <module>
----> 1 from transformers import AutoTokenizer, AutoModel
      2
      3 tokenizer = AutoTokenizer.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")
      4 model = AutoModel.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/__init__.py in <module>
     20 # Integrations: this needs to come before other ml imports
     21 # in order to allow any 3rd-party code to initialize properly
---> 22 from .integrations import (  # isort:skip
     23     is_comet_available,
     24     is_optuna_available,

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/integrations.py in <module>
     57
     58 from .file_utils import is_torch_tpu_available
---> 59 from .trainer_callback import TrainerCallback
     60 from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun
     61 from .utils import logging

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/trainer_callback.py in <module>
     24 from tqdm.auto import tqdm
     25
---> 26 from .trainer_utils import EvaluationStrategy
     27 from .training_args import TrainingArguments
     28 from .utils import logging

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/trainer_utils.py in <module>
     23
     24 from .file_utils import is_tf_available, is_torch_available
---> 25 from .tokenization_utils_base import ExplicitEnum
     26
     27

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/tokenization_utils_base.py in <module>
     30 import numpy as np
     31
---> 32 from .file_utils import (
     33     add_end_docstrings,
     34     cached_path,

ImportError: cannot import name 'is_flax_available' from 'transformers.file_utils' (/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/file_utils.py)`


TPU padding in seq2seq codes

Hi
In the readme of seq2seq models https://github.com/huggingface/transformers/tree/master/examples/seq2seq it is written:

All sequences should be padded to be of equal length otherwise it leads to extremely slow training. (finetune_trainer.py does this automatically when running on TPU.)

could you point me please where in the codes this is done? and what do you mean here, isn't it like normal case that in each batch, we pad the sentence to the equal length? could you tell me the differences from what is done normally?

thanks

Best
Rabeeh


comment correction in modeling_bart.py

Hello, in here
https://github.com/huggingface/transformers/blob/07708793f20ec3a949ccab32cc4fe0c7272dcc4c/src/transformers/modeling_bart.py#L588

I think the comment should be
```
# Convert to Bart output format: (BS, seq_len, model_dim) -> (seq_len, BS, model_dim)
```
Before transpose , shape of x and encoder_hidden_states are both (BS, seq_len, model_dim) to me.
Thanks.


finetune_trainer crashes in the beginning

Hi
I am trying to run

```
python finetune_trainer.py \
--model_name_or_path /idiap/temp/rkarimi/pretrained_transformers/t5-small/ \
--data_dir=/home/rabeeh/data/test_data/wmt_en_ro \
--learning_rate=3e-5 \
--output_dir=/home/rabeeh/temp \
--max_source_length=512 \
--max_target_length=56 \
--do_train --do_predict \
--overwrite_output_dir\
"$@"

```

I got the following issue, thanks for your help
(internship) rkarimi@vgnh001:/idiap/user/rkarimi/dev/internship/seq2seq$ bash run_idiap.sh
Traceback (most recent call last):
  File "finetune_trainer.py", line 7, in <module>
    from seq2seq_trainer import Seq2SeqTrainer
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/seq2seq_trainer.py", line 7, in <module>
    from transformers import PreTrainedModel, Trainer, logging
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/transformers/integrations.py", line 81, in <module>
    from .file_utils import is_torch_tpu_available  # noqa: E402
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/transformers/file_utils.py", line 87, in <module>
    import datasets  # noqa: F401
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/__init__.py", line 27, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 40, in <module>
    from .arrow_reader import ArrowReader
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/arrow_reader.py", line 31, in <module>
    from .utils import cached_path, logging
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/__init__.py", line 20, in <module>
    from .download_manager import DownloadManager, GenerateMode
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 25, in <module>
    from .file_utils import HF_DATASETS_CACHE, cached_path, get_from_cache, hash_url_to_filename
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 118, in <module>
    os.makedirs(HF_MODULES_CACHE, exist_ok=True)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/idiap/home/rkarimi/.cache/huggingface'


[s2s/distill] hparams.tokenizer_name = hparams.teacher

fix bug that initialization of student model without tokenizer name

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


initialize student's tokenizer name Update distillation.py

fix a bug, that student's tokenizer can be initialized without tokenizer_name


training T5 on  multiple datasets

Hi
I'd need to train finetune_trainer with T5 on multiple datasets. Could you give me some pointers, how I can form a dataloader with multiple datasets in the very simple form of it?
thanks


Fix DataCollatorForWholeWordMask

# What does this PR do?

This is a quick fix for #8378

There is another `_tensorize_batch` in other class, I think we can replace it too.
And there might be tests missing?
@sgugger #8308


DataCollatorForWholeWordMask is missing _tensorize_batch method

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): Yes
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 @sgugger
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):BERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

Not sure if it's considered official, but it's just finetuning for language modeling on a custom dataset.
## To reproduce

Steps to reproduce the behavior:
```python
from transformers import DataCollatorForWholeWordMask
from transformers import Trainer, TrainingArguments
from transformers import TextDataset
from transformers import BertTokenizer
from transformers import BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking')
tokenizer.add_tokens(["[new]"])
model = BertForMaskedLM.from_pretrained('bert-large-uncased-whole-word-masking')
model.resize_token_embeddings(len(tokenizer))

model.train()
dataset = TextDataset(
      tokenizer=tokenizer,
      file_path="./bert_train_set.txt",
      block_size=512
  )
data_collator = DataCollatorForWholeWordMask(
    tokenizer
)

training_args = TrainingArguments(
    output_dir="./BERT",
    overwrite_output_dir=True,
    num_train_epochs=10,
    per_gpu_train_batch_size=16,
    save_steps=500,
    save_total_limit=2,
    learning_rate = 2e-5
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)
trainer.train()
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
/usr/local/lib/python3.6/dist-packages/transformers/data/data_collator.py in __call__(self, examples)
    316             examples = [{"input_ids": e} for e in examples]
    317
--> 318         batch_input = self._tensorize_batch(input_ids)
    319
    320         mask_labels = []

AttributeError: 'DataCollatorForWholeWordMask' object has no attribute '_tensorize_batch'
```
## Expected behavior

Trainer should proceed to feed tensors to the model. I think someone just forgot to copypaste _tensorize_batch from DataCollatorForPermutationLanguageModeling, either that or inheritance is off.


[fsmt convert script] fairseq broke chkpt data - fixing that

This PR's main purpose:

* [x] adjusts for breaking fairseq [changes](https://github.com/pytorch/fairseq/commit/3b27ed7996b0315f471c795cf9b7dfcc18467cbe)

Plus:

* [x] adds support for older `bpecodes` filenames - specifically `code` in iwslt14
* [x] improves reporting
* [x] removes CDN note that will be shortly become irrelevant https://github.com/huggingface/transformers/pull/8324

The break was manifesting during conversion in:
```
Traceback (most recent call last):
  File "src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py", line 271, in <module>
    convert_fsmt_checkpoint_to_pytorch(args.fsmt_checkpoint_path, args.pytorch_dump_folder_path)
  File "src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py", line 118, in convert_fsmt_checkpoint_to_pytorch
    src_lang = args["source_lang"]
KeyError: 'source_lang'
```

@sshleifer, @LysandreJik


[s2s] distill t5-large -> t5-small

# What does this PR do?
@sshleifer : This PR addresses the feature I was requesting on [this thread](https://discuss.huggingface.co/t/distillation-create-student-model-from-a-different-base-model-than-teacher/1501/8). It add a way to specify a base model to initialize the student model with (in `distillation.py`), instead of necessarily starting from the same base model as the teacher and selectively copying layers.

Following are the main changes:
1. Accepts a new command line argument `--student` that can take a base model name, such as `t5-small`.
2. If this new argument is specified, simply creates a student model from scratch, if not creates the student model per the original code.
3. In the `_step` function, calls the teacher decoder with the teacher's encoder outputs instead of student encoder outputs.
4. When teacher and student base models are different, since they can be of different architectures, and hence have different hidden sizes, does not calculate hidden loss.

Sample command to run this:
```
python distillation.py --teacher t5-large --data_dir $NQOPEN_DIR \
--student t5-small --tokenizer_name t5-small \
--teacher_tokenizer_name t5-large \
--learning_rate=3e-4 --freeze_encoder --freeze_embeds \
--do_train --train_batch_size 32 \
--do_predict --n_train 3 \
--model_name_or_path t5-large --eval_beams 2 --eval_max_gen_length 142 \
--val_check_interval 0.25 --n_val 1 \
--output_dir distilled-t5 --gpus 1 --logger_name wandb
```

I was able to get this training to run. I am not sure what tests I need to run, and also noticed some automated test failure notifications to my email that I didn't quite follow.

This is the first PR I am making here, so hope you can point me in the right direction as to what needs to be done if anything is missing.

[x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
https://discuss.huggingface.co/t/distillation-create-student-model-from-a-different-base-model-than-teacher/1501/8

## Who can review?
examples/seq2seq: @sshleifer


RobertaTokenizerFast is around 10 times slower than BertTokenizerFast #510

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-5.4.38-t2.el7.twitter.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.7.9
- PyTorch version (GPU?): 1.4.0+cu100 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information
Also opened this issue on the tokenizers repo: https://github.com/huggingface/tokenizers/issues/510

`RobertaTokenizerFast` with 60k vocab size is around 50 times slower  than the `BertTokenizerFast` for `transformers==3.3.1`.

This is really slowing down the processing time for training the language model. Is there a suggested fix for this?

Model I am using (Bert, XLNet ...): RobertaTokenizerFast

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python

In [37]: from transformers import BertTokenizerFast, RobertaTokenizerFast
    ...:

In [38]: bert_tokenizer = BertTokenizerFast.from_pretrained(bert_tokenizer_dir, max_len=512, truncation=True)
    ...: roberta_tokenizer = RobertaTokenizerFast.from_pretrained(roberta_tokenizer_dir, max_len=512, truncation=True)
    ...:

In [39]: line = "I am trying out this code for testing tokenizers and it is super good. Huge victory. But the difference in speed between BERT tokenizer and Robert
    ...: a tokenizers is quite slow."

In [40]: bert_tokenizer.encode_plus(line, add_special_tokens=True, truncation=True)
Out[40]: {'input_ids': [2, 51, 1881, 21557, 3212, 2937, 3590, 1945, 40605, 53576, 23981, 1013, 1985, 2179, 1943, 3863, 3841, 20, 38092, 65353, 20, 3180, 1931, 48800, 1822, 12679, 26777, 18732, 53576, 23981, 1985, 20839, 1016, 53576, 23981, 1013, 1943, 66009, 16390, 20, 3], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

In [41]: roberta_tokenizer.encode_plus(line, add_special_tokens=True, truncation=True)
Out[41]: {'input_ids': [0, 47, 979, 28120, 3145, 2924, 6628, 1046, 6747, 458, 618, 2063, 781, 719, 994, 1527, 891, 6074, 6464, 20, 550, 11852, 21228, 3085, 20, 16339, 898, 20420, 40977, 494, 24738, 34187, 444, 13841, 618, 2063, 27250, 994, 12086, 16449, 618, 2063, 781, 719, 891, 517, 1492, 23819, 20, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

In [42]: %%timeit -n 10000
    ...: tokens = roberta_tokenizer.encode_plus(line, add_special_tokens=True, truncation=True)
    ...:
10000 loops, best of 5: 9.09 ms per loop

In [43]: %%timeit -n 10000
    ...: tokens = bert_tokenizer.encode_plus(line, add_special_tokens=True, truncation=True)
    ...:
10000 loops, best of 5: 181 µs per loop

In [44]: roberta_tokenizer.vocab_size
Out[44]: 60000

In [45]: bert_tokenizer.vocab_size
Out[45]: 100000

In [47]: import transformers

In [48]: import tokenizers

In [49]: transformers.__version__
Out[49]: '3.3.1'

In [50]: tokenizers.__version__
Out[50]: '0.8.1.rc2'

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expect to not be much difference here. Why is this the case?

<!-- A clear and concise description of what you would expect to happen. -->


[wip] [fsmt] possible support of iwslt14 in fsmt

This is an attempt to see whether fsmt can support older fairseq archs, based on this request: https://github.com/huggingface/transformers/issues/8233

Currently it's just changing the conversion code directly to see if it can be converted.
```
python src/transformers/convert_fsmt_original_pytorch_checkpoint_to_pytorch.py --fsmt_checkpoint_path ./fairseq-en-el-model/checkpoint_best.pt --pytorch_dump_folder_path ./model-data
```

@lighteternal, please have a look.

I made the model configuration `args` based on the arch configuration. The only issue at the moment is sizes of encoder decoder - for some reason the vocab size seems to be reversed?

```
$wc -l fairseq-en-el-model/*txt
 12892 fairseq-en-el-model/dict.el.txt
  9932 fairseq-en-el-model/dict.en.txt
```
So this is English to Greek, correct?  And not the other way around, correct? So the source is `9932` and target is `12892`-long.

Your issue mentioned "Greek<->English", but this model must be one way - which is it?

When I run the script:
```
        size mismatch for model.encoder.embed_tokens.weight: copying a param with shape torch.Size([12896, 512]) from checkpoint, the shape in current model is torch.Size([9936, 512]).
        size mismatch for model.decoder.output_projection.weight: copying a param with shape torch.Size([9936, 512]) from checkpoint, the shape in current model is torch.Size([12896, 512]).
```

So it suggests that the encoder is `12896`- long, which should be the other way around, no? Unless it was trained on Greek to English.

Well, you can also experiment with the conversion.


removing runs folders

Hi
Do you know where in the codes when training finetune_trainer.py a folder "runs" created? could it be moved to output_dir? so this is always created in the directory running codes than output_dir
thanks


Fine-tuning for QA: how to prepare custom dataset?

I have csv dataset article,name
I want to use fine-tuning(Albert), don't undestand how to prepare my dataset to suite for fine-tuning tutorial (https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0)
Can't understand the reason why we add start and end idx of answer in context to 'answer' part of dataset.
Could I use just text in answer, without start and end ?
If context doesn't contain all answer words in a row, what indices should be selected then?


[make] rewrite modified_py_files in python to be cross-platform

As it can be seen here https://github.com/huggingface/transformers/pull/8359 some windows setups don't do well with the few lines of unix code that derive which files were modified, when called from `Makefile` so it looked like the best solution was to rewrite that logic in python which this PR does.

**status update:** If I understand correctly this PR works on windows @sgugger uses, but not for @jplu, but it should be merged regardless as it removes the hurdle reported in https://github.com/huggingface/transformers/pull/8359, which was preventing @jplu from using any `make` targets. Since `fixup` is an optional target and isn't a show stopper I recommend we merge this and then move the remaining issue into a separate issue and then find a windows expert to help make this target work for all windows users.

The rest of the post is the original request to validate that the PR works on windows.

----------------------------

@jplu  - please kindly validate that this works on windows. test it with:
1. no changes to any files under src utils tests examples

```
make fixup
```
this one should not fail but not do much either, my output is:
```
No library .py files were modified
python utils/check_copies.py
python utils/check_dummies.py
python utils/check_repo.py
Checking all models are properly tested.
Checking all models are properly documented.
python utils/style_doc.py src/transformers docs/source --max_len 119
```

2. with 1 change in some .py file under 2 of the above dirs, by hand or via `echo` if it works on windows:
```
echo -n "#" >> tests/conftest.py
echo -n "#" >> examples/README.md
echo -n "#" >> src/transformers/testing_utils.py
make fixup
```
this time I get:
```
Checking/fixing src/transformers/testing_utils.py tests/conftest.py
reformatted /mnt/nvme1/code/huggingface/transformers-fixup-python/tests/conftest.py
reformatted /mnt/nvme1/code/huggingface/transformers-fixup-python/src/transformers/testing_utils.py
All done! ✨ 🍰 ✨
2 files reformatted
python utils/check_copies.py
python utils/check_dummies.py
python utils/check_repo.py
Checking all models are properly tested.
Checking all models are properly documented.
python utils/style_doc.py src/transformers docs/source --max_len 119
```

If all is good you should get the same output as mine, sans different base path.

The key is to see:
```
Checking/fixing src/transformers/testing_utils.py tests/conftest.py
```
it may include `utils/get_modified_files.py` if you're checking out this PR as it's under git too, it doesn't matter. What we want to ensure is that out of 3 intentionally modified files 2 gets looked at as they are ending with `.py`.

Just in case you don't know how to checkout a pr, you can for example use `gh` https://github.com/cli/cli:
```
gh pr checkout 8371
```
but there are many other ways to do it.

Thank you!

@LysandreJik, @sgugger


BertTokenizerFast object has no attribute 'ids_to_tokens'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSTM: @stas00
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@mfuntowicz

## Information

BertTokenizer has an attribute dictionary `ids_to_tokens`. However, BertTokenizerFast does not have this dictionary. I would expect them to have the same interfaces.

Model I am using (Bert, XLNet ...): BERT, BertTokenizerFast

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
print(type(tokenizer).__name__) # BertTokenizer
print(tokenizer.ids_to_tokens[0]) # [PAD]

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
print(type(tokenizer).__name__) # BertTokenizerFast
print(tokenizer.ids_to_tokens[0]) # 'BertTokenizerFast' object has no attribute 'ids_to_tokens'
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

BertTokenizerFast should convert ID 0 to '[PAD]'


Model card: T5-base fine-tuned on QuaRTz

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


[TF generate] Cut encoder outptus to just last hidden states for now

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #8361

As shown in #8361, the current TF generate can throw errros if `config.output_attentions` is set to `True`. Because we cannot outptut `attentions` and `hidden_states` at the moment anyways in TF, this PR just cuts the encoder outputs to just the `last_hidden_states`

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Which model to choose for seq2seq(generating headers for articles)?

Hi
Task is generating headers for articles. I have dataset with articles and correct names.
I think about using T5 for summarization or GPT-2 for seq2seq modeling. I'm not sure what way to choose
Can you please give me advise which direction is more relevant for this task?


Some added tests for TokenClassificationArgumentHandler


TFTrainerArguments: ImportError: Method `device` requires PyTorch.

from transformers import TFTrainer, TFTrainingArguments
training_args = TFTrainingArguments(output_dir='models',
                                    num_train_epochs=1,
                                    per_device_train_batch_size=32,
                                    per_device_eval_batch_size=32,
                                    warmup_steps=500,
                                    weight_decay=0.05,
                                    logging_dir='logs'
                                   )

Above code produces the following error:

ImportError                               Traceback (most recent call last)
<ipython-input-3-017a860e7c76> in <module>
      5                                     warmup_steps=500,
      6                                     weight_decay=0.05,
----> 7                                     logging_dir='logs'
      8                                    )

<string> in __init__(self, output_dir, overwrite_output_dir, do_train, do_eval, do_predict, evaluate_during_training, evaluation_strategy, prediction_loss_only, per_device_train_batch_size, per_device_eval_batch_size, per_gpu_train_batch_size, per_gpu_eval_batch_size, gradient_accumulation_steps, eval_accumulation_steps, learning_rate, weight_decay, adam_beta1, adam_beta2, adam_epsilon, max_grad_norm, num_train_epochs, max_steps, warmup_steps, logging_dir, logging_first_step, logging_steps, save_steps, save_total_limit, no_cuda, seed, fp16, fp16_opt_level, local_rank, tpu_num_cores, tpu_metrics_debug, debug, dataloader_drop_last, eval_steps, dataloader_num_workers, past_index, run_name, disable_tqdm, remove_unused_columns, label_names, load_best_model_at_end, metric_for_best_model, greater_is_better, tpu_name, poly_power, xla)

/usr/local/lib/python3.6/dist-packages/transformers/training_args.py in __post_init__(self)
    352             self.run_name = self.output_dir
    353
--> 354         if self.device.type != "cuda" and self.fp16:
    355             raise ValueError("AMP (`--fp16`) can only be used on CUDA devices.")
    356

/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in wrapper(*args, **kwargs)
   1171             return func(*args, **kwargs)
   1172         else:
-> 1173             raise ImportError(f"Method `{func.__name__}` requires PyTorch.")
   1174
   1175     return wrapper

ImportError: Method `device` requires PyTorch.

Seems like a bug?


Patch token classification pipeline

This PR patches issues found with the `TokenClassificationPipeline` since the merge of https://github.com/huggingface/transformers/pull/5970, namely not being able to load a slow tokenizer in the pipeline.

It also sets the `ignore_subwords` to `False` by default, as this does not work with the slow tokenizers. No release have been done since the introduction of that argument, so it is not a breaking change.


Create README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Create README.md

Related to pretrained model cards


TF generate() function is incompatible with output_attention and output_hidden_states

Should Fix

This has been identified as a bug and should be fixed.

## Environment info

- `transformers` version: 3.4.0
- Platform: Mac OS Catalina (10.15.6)
- Python version: 3.6.8
- PyTorch version (GPU?):  N/A
- Tensorflow version (GPU?): 2.3.1 (no)
- Using GPU in script?: No, but bug is persistent regardless of device.
- Using distributed or parallel set-up in script?: No

### Who can help
 @sshleifer  @TevenLeScao  @patrickvonplaten

## Information

The generate() function in modeling_tf_utils assumes that outputs from a model call have a static number of outputs. If either `output_attention` or `output_hidden_states` is set, the number of outputs is different, causing the function to fail. The fix should be pretty simple and only involve checking for the output size (or completely switching to dict/NamedTuple outputs from model modules since variable length returns are brittle :)).

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a TF model with one of the above mentioned flags set.
2. Call `.generate()` on the model.

```python
import transformers
model = transformers.TFT5ForConditionalGeneration.from_pretrained('t5-small', output_hidden_states=True, output_attentions=True)
tokenizer = transformers.T5Tokenizer.from_pretrained('t5-small')
input_ids = tokenizer.batch_encode_plus(['test 1', 'test 2', 'test 3'], return_tensors="tf", padding='longest')
output_ids = model.generate(input_ids['input_ids'], attention_mask=input_ids['attention_mask'])

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/transformers/generation_tf_utils.py", line 405, in generate
    use_cache=use_cache,
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/transformers/generation_tf_utils.py", line 445, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/transformers/modeling_tf_t5.py", line 1352, in call
    training=training,
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/transformers/modeling_tf_t5.py", line 759, in call
    training=training,
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 985, in __call__
    outputs = call_fn(inputs, *args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/transformers/modeling_tf_t5.py", line 450, in call
    assert len(past_key_value) == expected_num_past_key_values, error_message
AssertionError: There should be 4 past states. 2 (past / key) for self attention.2 (past / key) for cross attention Got 3 past key / value states
```

## Expected behavior

This snippet should not crash and have the same behavior as the one below. Though one may argue that the `generate()` function in this case should also return states/attentions which would complicate things. However, even ignoring the flags when generating is better than crashing.

```python
import transformers
model = transformers.TFT5ForConditionalGeneration.from_pretrained('t5-small', output_hidden_states=False, output_attentions=False)
tokenizer = transformers.T5Tokenizer.from_pretrained('t5-small')
input_ids = tokenizer.batch_encode_plus(['test 1', 'test 2', 'test 3'], return_tensors="tf", padding='longest')
output_ids = model.generate(input_ids['input_ids'], attention_mask=input_ids['attention_mask'])
print(output_ids)

tf.Tensor(
[[   0 2300  209    1    0]
 [   0 2300  794  204    1]
 [   0 2300  220    1    0]], shape=(3, 5), dtype=int32)
```


Update README.md

model card

Related to pretrained model cards

Fix websitr address

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->


Fix some tooling for windows

# What does this PR do?
This PR fixes a small issue on the `check_repo.py` script with the encodings. I have also updated the `Makefile` to move the creation of the variables `fork_point_sha`, `joined_dirs` and `modified_py_files` in the `modified_only_fixup` target because they are used only there.

The makefile raises an error on Windows environment when creating the `modified_py_files` variable:
```
'tests' is not recognized as an internal or external command, operable program or batch file
```

ping @stas00 and @sgugger


[WIP] Add performer in flax

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR is a first draft to add the Performer: https://github.com/google-research/google-research/tree/master/performer in Trax to the library.

As stated here: https://github.com/google-research/google-research/tree/master/performer/fast_self_attention#performers-fast-self-attention-module all one has to do is to replace the `attention_fn` in `flax.linen.nn.SelfAttention` with the output of `make_fast_softmax_attention` which is done in this PR.

The function seems to integrate seamlessly with our current `FlaxBertModel` architecture, but it is questionable if the model can easily be fine-tuned on existing `BertModel` weights and if our current `FlaxBertModel` architecture corresponds to the architecture used by the Performer team to train & evaluate their models as described in the Paper.

One can verify that a forward pass of the Performer model works by running:

```python
pytest tests/test_modeling_performer.py
```

Tagging @TevenLeScao @mfuntowicz @thomwolf  for information.
@mfuntowicz - it seems like the weights can be loaded and a forward pass does throw any errors -> does that seem to work for you after taking a first look?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSTM: @stas00
 -->

- TF code: https://github.com/google-research/google-research/tree/master/performer/fast_attention/tensorflow


Cannot Load roberta tokenizer

# ❓ Questions & Help

My `transformers`'s version is `transformers 3.4.0`

I download a Chinese RoBERTa model, where:

```
models
├── RoBERTa_zh_Large_Pytorch
│   ├── config.json
│   ├── pytorch_model.bin
│   └── vocab.txt
```

I want to load the tokenizer from local:

```python
tokenizer_zh = RobertaTokenizer.from_pretrained('./models/RoBERTa_zh_Large_Pytorch/')
```

but I get :

```
OSErrorTraceback (most recent call last)
<ipython-input-4-bdedc9589347> in <module>
----> 1 tokenizer_zh = RobertaTokenizer.from_pretrained('./models/RoBERTa_zh_Large_Pytorch/')

/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1647                     ", ".join(s3_models),
   1648                     pretrained_model_name_or_path,
-> 1649                     list(cls.vocab_files_names.values()),
   1650                 )
   1651             )

OSError: Model name './models/RoBERTa_zh_Large_Pytorch/' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed './models/RoBERTa_zh_Large_Pytorch/' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
```


assert tgt_line, f"empty tgt line for index {index}" with t5

Hi
I am trying to run fineutne_trainer with this command and I got assert error  thanks for your help

export TPU_IB_ADDRESS=10.160.244.2

start=`date +%s`
python xla_spawn.py --num_cores 8 \
    finetune_trainer.py \
    --tokenizer_name t5-small --model_name_or_path t5-small \
    --data_dir wmt_en_de \
    --output_dir /home/rabeeh/outputs/verify --overwrite_output_dir \
    --learning_rate=3e-4 \
    --warmup_steps 500 \
    --per_device_train_batch_size=128 --per_device_eval_batch_size=128\
    --num_train_epochs=1 \
    --save_steps 500 --eval_steps 500 \
    --logging_steps 200 \
    --max_source_length 128 --max_target_length 128 --val_max_target_length 128 --test_max_target_length 128 \
    --task translation --label_smoothing 0.1 \
     --freeze_encoder --freeze_embeds \
     --num_train_epochs=1 \
                 --logging_first_step --logging_steps 200 \
                         --do_train --do_eval --evaluate_during_training \
                             --prediction_loss_only \

    "$@"

end=`date +%s`
runtime=$((end-start))
echo running time $runtime

logs

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/parallel_loader.py", line 141, in _loader_worker
    _, data = next(data_iter)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/root/home/rabeeh/examples/seq2seq/utils.py", line 246, in __getitem__
    assert tgt_line, f"empty tgt line for index {index}"
AssertionError: empty tgt line for index 4549523