BertForTokenClassification can not from_pretrained the fine-tuned model?

## ❓ Questions & Help
Thanks the great work!
While, when I wrap the `run_ner.py `scripts in sklearn api style for non-specialists, I met some problem.
It 's ok for training and evaluating, but when predicting the F1-score is much lower than  that of evaluating. As shown in following:

Evaluating result: the F1-score of 0.8242
```
***** Eval results 500 *****
f1 = 0.8101377518505809
loss = 0.10396960769538525
precision = 0.8009887553315238
recall = 0.8194981652286026
***** Eval results 1000 *****
f1 = 0.8242496050552922
loss = 0.09259376796307388
precision = 0.8206035584390052
recall = 0.8279281959734206
```
Predicting Results, the F1-score 0.0934

```
 precision    recall  f1-score   support

      tim     0.0954    0.0943    0.0949      2014
      org     0.0743    0.0688    0.0714      2021
      geo     0.1004    0.1087    0.1044      3771
      per     0.0843    0.0864    0.0853      1644
      gpe     0.1022    0.1010    0.1016      1623
      nat     0.3333    0.0769    0.1250        13
      art     0.0000    0.0000    0.0000        51
      eve     0.0400    0.0476    0.0435        21

micro avg     0.0930    0.0938    0.0934     11158
macro avg     0.0924    0.0938    0.0929     11158
```

**? ?? why? I have check the code many times. Is that the bug in saving or loading the fine-tuned model?**

The fine-tuning and predicting script based on `transformers-sklearn`

```python
import pandas as pd
from sklearn.model_selection import train_test_split
from transformers_sklearn import BERTologyNERClassifer

if __name__ == '__main__':

    data_df = pd.read_csv('datasets/gmbner/ner_dataset.csv',encoding="utf8")
    data_df.fillna(method="ffill",inplace=True)
    value_counts = data_df['Tag'].value_counts()
    label_list = list(value_counts.to_dict().keys())

    # ## 1. preparing data
    X = []
    y = []
    for label, batch_df in data_df.groupby(by='Sentence #',sort=False):
        words = batch_df['Word'].tolist()
        labels = batch_df['Tag'].tolist()
        assert len(words) == len(labels)
        X.append(words)
        y.append(labels)

    X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.1,random_state=520)

    ## 2. customize model
    ner = BERTologyNERClassifer(
        labels=label_list,
        model_type='bert',
        model_name_or_path='bert-base-cased',
        data_dir='ts_data/gmbner',
        output_dir='results/gmbner',
        num_train_epochs=3,
        learning_rate=5e-5,
        logging_steps=500,
        save_steps=500,
        overwrite_output_dir=True
    )
    #
   ## 3. fit
    ner.fit(X_train, y_train)
    # # # #
    ## 4. score
    report = ner.score(X_test, y_test)
    with open('gmbner.txt', 'w', encoding='utf8') as f:
        f.write(report)
```

This is the two scripts in [transformers-sklearn](https://github.com/trueto/transformers_sklearn) for NER task.
`token_classification.py`
```python
import os
import torch
import random
import logging
import numpy as np
from tqdm import tqdm, trange
from torch.nn import CrossEntropyLoss
from torch.utils.data import random_split,TensorDataset,\
    DistributedSampler,RandomSampler,SequentialSampler,DataLoader
from tensorboardX import SummaryWriter
from transformers_sklearn.utils.token_classification_utils import get_labels,\
    read_examples_from_X_y,convert_examples_to_features
from transformers_sklearn.utils.data_utils import to_numpy
from sklearn.base import BaseEstimator,ClassifierMixin
from transformers_sklearn.utils.metrics_utils import f1_score,recall_score,precision_score,classification_report

from transformers import AdamW, get_linear_schedule_with_warmup
from transformers import BertConfig, BertForTokenClassification, BertTokenizer
from transformers import RobertaConfig, RobertaForTokenClassification, RobertaTokenizer
from transformers import DistilBertConfig, DistilBertForTokenClassification, DistilBertTokenizer
from transformers import CamembertConfig, CamembertForTokenClassification, CamembertTokenizer
# from transformers import AlbertConfig,AlbertTokenizer
from transformers_sklearn.model_albert import AlbertForTokenClassification,AlbertTokenizer,AlbertConfig

ALL_MODELS = sum(
    (tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, RobertaConfig, DistilBertConfig)),
    ())

MODEL_CLASSES = {
    "bert": (BertConfig, BertForTokenClassification, BertTokenizer),
    "roberta": (RobertaConfig, RobertaForTokenClassification, RobertaTokenizer),
    "distilbert": (DistilBertConfig, DistilBertForTokenClassification, DistilBertTokenizer),
    "camembert": (CamembertConfig, CamembertForTokenClassification, CamembertTokenizer),
    "albert":(AlbertConfig,AlbertForTokenClassification,AlbertTokenizer)
}

logger = logging.getLogger(__name__)

def set_seed(seed=520,n_gpu=1):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    if n_gpu > 0:
        torch.cuda.manual_seed_all(seed)

class BERTologyNERClassifer(BaseEstimator,ClassifierMixin):

    def __init__(self,labels,data_dir='ts_data',model_type='bert',
                 model_name_or_path='bert-base-chinese',
                 output_dir='ts_results',config_name='',
                 tokenizer_name='',cache_dir='model_cache',
                 max_seq_length=512,do_lower_case=False,
                 per_gpu_train_batch_size=8,per_gpu_eval_batch_size=8,
                 gradient_accumulation_steps=1,
                 learning_rate=5e-5,weight_decay=0.0,
                 adam_epsilon=1e-8,max_grad_norm=1.0,
                 num_train_epochs=3.0,max_steps=-1,
                 warmup_steps=0,logging_steps=50,
                 save_steps=50,evaluate_during_training=True,
                 no_cuda=False,overwrite_output_dir=False,
                 overwrite_cache=False,seed=520,
                 fp16=False,fp16_opt_level='01',
                 local_rank=-1,val_fraction=0.1):
        self.labels = labels
        self.data_dir = data_dir
        self.model_type = model_type
        self.model_name_or_path = model_name_or_path
        self.output_dir = output_dir
        self.config_name = config_name
        self.tokenizer_name = tokenizer_name
        self.max_seq_length = max_seq_length
        self.do_lower_case = do_lower_case
        self.cache_dir = cache_dir
        self.per_gpu_train_batch_size = per_gpu_train_batch_size
        self.per_gpu_eval_batch_size = per_gpu_eval_batch_size
        self.gradient_accumulation_steps = gradient_accumulation_steps
        self.learning_rate = learning_rate
        self.weight_decay = weight_decay
        self.adam_epsilon = adam_epsilon
        self.max_grad_norm = max_grad_norm
        self.num_train_epochs = num_train_epochs
        self.max_steps = max_steps
        self.warmup_steps = warmup_steps
        self.logging_steps = logging_steps
        self.save_steps = save_steps
        self.evaluate_during_training = evaluate_during_training
        self.no_cuda = no_cuda
        self.overwrite_output_dir = overwrite_output_dir
        self.overwrite_cache = overwrite_cache
        self.seed = seed
        self.fp16 = fp16
        self.fp16_opt_level = fp16_opt_level
        self.local_rank = local_rank
        self.val_fraction = val_fraction

        self.id2label = {i: label for i, label in enumerate(self.labels)}
        self.label_map = {label: i  for i, label in enumerate(self.labels)}

        # Setup CUDA, GPU & distributed training
        if self.local_rank == -1 or self.no_cuda:
            device = torch.device("cuda" if torch.cuda.is_available() and not self.no_cuda else "cpu")
            self.n_gpu = torch.cuda.device_count() if not self.no_cuda else 1
        else:  # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
            torch.cuda.set_device(self.local_rank)
            device = torch.device("cuda", self.local_rank)
            torch.distributed.init_process_group(backend="nccl")
            self.n_gpu = 1
        self.device = device

        # Setup logging
        logging.basicConfig(format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
                            datefmt="%m/%d/%Y %H:%M:%S",
                            level=logging.INFO if self.local_rank in [-1, 0] else logging.WARN)
        logger.warning("Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
                       self.local_rank, device, self.n_gpu, bool(self.local_rank != -1), self.fp16)

        # Set seed
        set_seed(seed=self.seed,n_gpu=self.n_gpu)

    def fit(self,X,y):
        if not os.path.exists(self.data_dir):
            os.mkdir(self.data_dir)

        if not os.path.exists(self.output_dir):
            os.mkdir(self.output_dir)

        if os.path.exists(self.output_dir) and os.listdir(
                self.output_dir) and not self.overwrite_output_dir:
            raise ValueError(
                "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
                    self.output_dir))

        num_labels = len(self.labels)
        # self.labels = labels
        # Use cross entropy ignore index as padding label id so that only real label ids contribute to the loss later
        pad_token_label_id = CrossEntropyLoss().ignore_index
        self.pad_token_label_id = pad_token_label_id
        # Load pretrained model and tokenizer
        if self.local_rank not in [-1, 0]:
            torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

        self.model_type = self.model_type.lower()
        config_class, model_class, tokenizer_class = MODEL_CLASSES[self.model_type]
        config = config_class.from_pretrained(self.config_name if self.config_name else self.model_name_or_path,
                                              num_labels=num_labels,
                                              cache_dir=self.cache_dir if self.cache_dir else None,
                                              share_type='all' if self.model_type=='albert' else None)
        tokenizer = tokenizer_class.from_pretrained(
            self.tokenizer_name if self.tokenizer_name else self.model_name_or_path,
            do_lower_case=self.do_lower_case,
            cache_dir=self.cache_dir if self.cache_dir else None)
        model = model_class.from_pretrained(self.model_name_or_path,
                                            from_tf=bool(".ckpt" in self.model_name_or_path),
                                            config=config,
                                            cache_dir=self.cache_dir if self.cache_dir else None)

        if self.local_rank == 0:
            torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

        model.to(self.device)

        logger.info("Training/evaluation parameters %s", self)

        train_dataset = load_and_cache_examples(self, tokenizer, pad_token_label_id, X,y, mode="train")
        global_step, tr_loss = train(self, train_dataset,model,tokenizer,pad_token_label_id)
        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)

        # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
        if self.local_rank == -1 or torch.distributed.get_rank() == 0:
            # Create output directory if needed
            if not os.path.exists(self.output_dir) and self.local_rank in [-1, 0]:
                os.makedirs(self.output_dir)

            logger.info("Saving model checkpoint to %s", self.output_dir)
            # Save a trained model, configuration and tokenizer using `save_pretrained()`.
            # They can then be reloaded using `from_pretrained()`
            model_to_save = model.module if hasattr(model,"module") else model  # Take care of distributed/parallel training
            model_to_save.save_pretrained(self.output_dir)
            tokenizer.save_pretrained(self.output_dir)

            # Good practice: save your training arguments together with the trained model
            torch.save(self, os.path.join(self.output_dir, "training_args.bin"))

        return self

    def predict(self,X):
        # args = torch.load(os.path.join(self.output_dir, "training_args.bin"))
        # Load a trained model and vocabulary that you have fine-tuned
        _, model_class, tokenizer_class = MODEL_CLASSES[self.model_type]
        model = model_class.from_pretrained(self.output_dir)
        tokenizer = tokenizer_class.from_pretrained(self.output_dir)

        model.to(self.device)

        pad_token_label_id = CrossEntropyLoss().ignore_index
        # get dataset
        test_dataset = load_and_cache_examples(self,tokenizer,pad_token_label_id,X,y=None,mode='test')

        _, preds_list = evaluate(self,test_dataset,model,pad_token_label_id,mode='test')

        return preds_list

    def score(self, X, y, sample_weight=None):
        y_pred = self.predict(X)
        return classification_report(y,y_pred,digits=4)

def load_and_cache_examples(args, tokenizer,pad_token_label_id, X,y,mode):
    if args.local_rank not in [-1, 0] and mode=='train':
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

    # Load data features from cache or dataset file
    cached_features_file = os.path.join(args.data_dir, "cached_{}_{}_{}".format(mode,
        args.model_type,
        str(args.max_seq_length)))
    if os.path.exists(cached_features_file) and not args.overwrite_cache and mode=='train':
        logger.info("Loading features from cached file %s", cached_features_file)
        features = torch.load(cached_features_file)
    else:
        logger.info("Creating features from dataset file at %s", args.data_dir)
        examples = read_examples_from_X_y(X,y, mode)
        features = convert_examples_to_features(examples, args.label_map, args.max_seq_length, tokenizer,
                                                cls_token_at_end=bool(args.model_type in ["xlnet"]),
                                                # xlnet has a cls token at the end
                                                cls_token=tokenizer.cls_token,
                                                cls_token_segment_id=2 if args.model_type in ["xlnet"] else 0,
                                                sep_token=tokenizer.sep_token,
                                                sep_token_extra=bool(args.model_type in ["roberta"]),
                                                # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805
                                                pad_on_left=bool(args.model_type in ["xlnet"]),
                                                # pad on the left for xlnet
                                                pad_token=tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],
                                                pad_token_segment_id=4 if args.model_type in ["xlnet"] else 0,
                                                pad_token_label_id=pad_token_label_id
                                                )
        if args.local_rank in [-1, 0] and mode == 'train':
            logger.info("Saving features into cached file %s", cached_features_file)
            torch.save(features, cached_features_file)

    if args.local_rank == 0 and mode =='train':
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

    # Convert to Tensors and build dataset
    all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
    all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)
    all_segment_ids = torch.tensor([f.segment_ids for f in features], dtype=torch.long)
    all_label_ids = torch.tensor([f.label_ids for f in features], dtype=torch.long)

    dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
    return dataset

def train(args, train_dataset, model, tokenizer, pad_token_label_id):
    """ Train the model """
    if args.local_rank in [-1, 0]:
        tb_writer = SummaryWriter()

    args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)

    val_len = int(len(train_dataset)*args.val_fraction)
    train_len = len(train_dataset) - val_len
    train_ds, val_ds = random_split(train_dataset,[train_len,val_len])
    train_sampler = RandomSampler(train_ds) if args.local_rank == -1 else DistributedSampler(train_ds)
    train_dataloader = DataLoader(train_ds, sampler=train_sampler, batch_size=args.train_batch_size)

    if args.max_steps > 0:
        t_total = args.max_steps
        args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1
    else:
        t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
         "weight_decay": args.weight_decay},
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0}
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total)

    # Check if saved optimizer or scheduler states exist
    if os.path.isfile(os.path.join(args.model_name_or_path, "optimizer.pt")) and os.path.isfile(
            os.path.join(args.model_name_or_path, "scheduler.pt")
    ):
        # Load in optimizer and scheduler states
        optimizer.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "optimizer.pt")))
        scheduler.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "scheduler.pt")))

    if args.fp16:
        try:
            from apex import amp
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # multi-gpu training (should be after apex fp16 initialization)
    if args.n_gpu > 1:
        model = torch.nn.DataParallel(model)

    # Distributed training (should be after apex fp16 initialization)
    if args.local_rank != -1:
        model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank],
                                                          output_device=args.local_rank,
                                                          find_unused_parameters=True)

    # Train!
    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_ds))
    logger.info("  Num Epochs = %d", args.num_train_epochs)
    logger.info("  Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size)
    logger.info("  Total train batch size (w. parallel, distributed & accumulation) = %d",
                args.train_batch_size * args.gradient_accumulation_steps * (
                    torch.distributed.get_world_size() if args.local_rank != -1 else 1))
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", t_total)

    global_step = 0
    epochs_trained = 0
    steps_trained_in_current_epoch = 0
    # Check if continuing training from a checkpoint
    if os.path.exists(args.model_name_or_path):
        # set global_step to gobal_step of last saved checkpoint from model path
        global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0])
        epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
        steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)

        logger.info("  Continuing training from checkpoint, will skip to saved global_step")
        logger.info("  Continuing training from epoch %d", epochs_trained)
        logger.info("  Continuing training from global step %d", global_step)
        logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)

    tr_loss, logging_loss = 0.0, 0.0
    model.zero_grad()
    train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0])
    set_seed(seed=args.seed,n_gpu=args.n_gpu)  # Added here for reproductibility (even between python 2 and 3)
    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):

            # Skip past any already trained steps if resuming training
            if steps_trained_in_current_epoch > 0:
                steps_trained_in_current_epoch -= 1
                continue

            model.train()
            batch = tuple(t.to(args.device) for t in batch)
            inputs = {"input_ids": batch[0],
                      "attention_mask": batch[1],
                      "labels": batch[3]}
            if args.model_type != "distilbert":
                inputs["token_type_ids"] = batch[2] if args.model_type in ["bert", "xlnet"] else None  # XLM and RoBERTa don"t use segment_ids

            outputs = model(**inputs)
            loss = outputs[0]  # model outputs are always tuple in pytorch-transformers (see doc)

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()

            tr_loss += loss.item()
            if (step + 1) % args.gradient_accumulation_steps == 0:
                if args.fp16:
                    torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                    torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)

                optimizer.step()
                scheduler.step()  # Update learning rate schedule
                model.zero_grad()
                global_step += 1

                if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
                    # Log metrics
                    if args.local_rank == -1 and args.evaluate_during_training:  # Only evaluate when single GPU otherwise metrics may not average well
                        results, _ = evaluate(args, val_ds, model,pad_token_label_id,prefix=global_step)
                        for key, value in results.items():
                            tb_writer.add_scalar("eval_{}".format(key), value, global_step)
                    tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
                    tb_writer.add_scalar("loss", (tr_loss - logging_loss) / args.logging_steps, global_step)
                    logging_loss = tr_loss

                if args.local_rank in [-1, 0] and args.save_steps > 0 and global_step % args.save_steps == 0:
                    # Save model checkpoint
                    output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
                    if not os.path.exists(output_dir):
                        os.makedirs(output_dir)
                    model_to_save = model.module if hasattr(model, "module") else model  # Take care of distributed/parallel training
                    model_to_save.save_pretrained(output_dir)
                    tokenizer.save_pretrained(output_dir)

                    torch.save(args, os.path.join(output_dir, "training_args.bin"))
                    logger.info("Saving model checkpoint to %s", output_dir)

                    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
                    torch.save(scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
                    logger.info("Saving optimizer and scheduler states to %s", output_dir)

            if args.max_steps > 0 and global_step > args.max_steps:
                epoch_iterator.close()
                break
        if args.max_steps > 0 and global_step > args.max_steps:
            train_iterator.close()
            break

    if args.local_rank in [-1, 0]:
        tb_writer.close()

    return global_step, tr_loss / global_step

def evaluate(args, eval_dataset, model, pad_token_label_id, mode='dev',prefix=0):
    # eval_dataset = load_and_cache_examples(args, tokenizer, labels, pad_token_label_id, mode=mode)

    args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
    # Note that DistributedSampler samples randomly
    eval_sampler = SequentialSampler(eval_dataset) if args.local_rank == -1 else DistributedSampler(eval_dataset)
    eval_dataloader = DataLoader(eval_dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)

    # multi-gpu evaluate
    if args.n_gpu > 1 and mode == 'test':
        model = torch.nn.DataParallel(model)

    # Eval!
    if mode == 'dev':
        logger.info("***** Running evaluation %s *****", prefix)
    else:
        logger.info("***** Running predict *****")
    logger.info("  Num examples = %d", len(eval_dataset))
    logger.info("  Batch size = %d", args.eval_batch_size)
    eval_loss = 0.0
    nb_eval_steps = 0
    preds = None
    out_label_ids = None
    model.eval()
    for batch in tqdm(eval_dataloader, desc="Evaluating" if mode=='dev' else "Predicting"):
        batch = tuple(t.to(args.device) for t in batch)

        with torch.no_grad():
            inputs = {"input_ids": batch[0],"attention_mask": batch[1],"labels": batch[3]}
            if args.model_type != "distilbert":
                inputs["token_type_ids"] = batch[2] if args.model_type in ["bert", "xlnet"] else None  # XLM and RoBERTa don"t use segment_ids
            outputs = model(**inputs)
            tmp_eval_loss, logits = outputs[:2]

            if args.n_gpu > 1:
                tmp_eval_loss = tmp_eval_loss.mean()  # mean() to average on multi-gpu parallel evaluating

            eval_loss += tmp_eval_loss.item()
        nb_eval_steps += 1
        if preds is None:
            preds = logits.detach().cpu().numpy()
            out_label_ids = inputs["labels"].detach().cpu().numpy()
        else:
            preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
            out_label_ids = np.append(out_label_ids, inputs["labels"].detach().cpu().numpy(), axis=0)

    eval_loss = eval_loss / nb_eval_steps
    preds = np.argmax(preds, axis=2)

    # label_map = {i: label for i, label in enumerate(labels)}

    out_label_list = [[] for _ in range(out_label_ids.shape[0])]
    preds_list = [[] for _ in range(out_label_ids.shape[0])]

    for i in range(out_label_ids.shape[0]):
        for j in range(out_label_ids.shape[1]):
            if out_label_ids[i, j] != pad_token_label_id:
                out_label_list[i].append(args.id2label[out_label_ids[i][j]])
                preds_list[i].append(args.id2label[preds[i][j]])

    results = {
        "loss": eval_loss,
        "precision": precision_score(out_label_list, preds_list),
        "recall": recall_score(out_label_list, preds_list),
        "f1": f1_score(out_label_list, preds_list)
    }

    if mode == 'dev':
        output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
        with open(output_eval_file, "a") as writer:
            logger.info("***** Eval results %d *****",prefix)
            writer.write("***** Eval results {} *****".format(prefix))
            for key in sorted(results.keys()):
                msg = "{} = {}".format(key, str(results[key]))
                logger.info(msg)
                writer.write(msg)
                writer.write('\n')
            writer.write('\n')

    return results, preds_list
```

<!-- A clear and concise description of the question. -->


Batch size affecting output.

## ❓ Questions & Help

When running evaluation, why am i getting slightly different output when running a batch size of 1 compared to batch size greater than 1?


fix #2399 an ImportError in official example


import Error from official example caused by fastprogress

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): BERT

Language I am using the model on (English, Chinese....): ALL

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

V0.2.1 of fastprogress released a couple of days ago seems to cause errors in run_tf_ner.py in the official example.

Traceback (most recent call last):
  File "run_tf_ner.py", line 12, in <module>
    from fastprogress import master_bar, progress_bar
ImportError: cannot import name 'master_bar' from 'fastprogress' (/usr/local/lib/python3.7/dist-packages/fastprogress/__init__.py)

users need to either downgrade: pip3 install fastprogress==0.1.22
or change the code:
`
from fastprogress.fastprogress import master_bar, progress_bar
`

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Distilbert predicting mask

Hi,

This is probably me doing something wrong, but I can't get distilbert to give me a sensible prediciton when I mask part of a sentence.

This setup for BERT (based on the examples):

```
import logging
logging.basicConfig(level=logging.INFO)

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

text = "Hello how are you doing?"
tokenized_text = tokenizer.tokenize(text)

masked_index = 2
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0]

tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    predictions = outputs[0]

predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token)
```
gives the correct answer _are_ for _How are you doing?_.

But when I try the same with distilbert:

```
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')

text = "Hello how are you doing?"
tokenized_text = tokenizer.tokenize(text)

masked_index = 2
tokenized_text[masked_index] = '[MASK]'

indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
tokens_tensor = torch.tensor([indexed_tokens])

model = DistilBertModel.from_pretrained('distilbert-base-uncased')
model.eval()

with torch.no_grad():
    # not adding/adding the segment tokens. when I give those to the model, it throws an error
    last_hidden_states = model(tokens_tensor)
    outputs = last_hidden_states[0]

predicted_index = torch.argmax(outputs[0], dim=1)[masked_index].item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
print(predicted_token)
```

I practically always get some _unusedxxx_ as a result. At first I thought this was because distilbert is a smaller model, but no matter what I try, I keep getting unused, so I am guessing it's something else.

Thanks in advance!


unable to use distilbert multilingual model

## ❓ Questions & Help
I'm trying to use the distilbert-base-multilingual-cased model but have been unable to do so.

I have checked and I am using transformers version 2.3.0. I have already tried these things:
1)     tokenizer = AutoTokenizer.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json")
    model = AutoModel.from_pretrained("https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-pytorch_model.bin")

Gives following error message:
OSError: Model name 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-multilingual-cased-config.json' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

2)     tokenizer = AutoTokenizer.from_pretrained("distilbert-base-multilingual-cased")
    model = AutoModel.from_pretrained("distilbert-base-multilingual-cased")

Gives following error message: OSError: Model name 'distilbert-base-multilingual-cased' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'distilbert-base-multilingual-cased' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

3) Same as (2) but with DistilBertTokenizer and DistilBertModel.

Can I please get some help in fixing this issue?


Model2Model quickstart attention_mask dimensionality problem

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

BERT-base-uncased

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [X] the official example scripts: (give details)
[model2model tutorial code](https://huggingface.co/transformers/quickstart.html#model2model-example)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. Copy the [model2model tutorial code](https://huggingface.co/transformers/quickstart.html#model2model-example) into a new file and run it.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->
```
Traceback (most recent call last):
  File "huggingface_m2m_example.py", line 47, in <module>
    outputs = model(question_tensor, answer_tensor, decoder_lm_labels=labels_tensor)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_encoder_decoder.py", line 234, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 997, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 819, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 433, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 394, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 334, in forward
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/lfs1/jwei/anaconda3/envs/low_mt/lib/python3.7/site-packages/transformers-2.3.0-py3.7.egg/transformers/modeling_bert.py", line 257, in forward
RuntimeError: The size of tensor a (8) must match the size of tensor b (768) at non-singleton dimension 3
```

I printed out the attention masks and the attention scores [right before](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L253) and got the following:
```
question_tensor.shape:  torch.Size([1, 7])
answer_tensor.shape:  torch.Size([1, 8])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 7, 7])
attention_masks:  torch.Size([1, 1, 1, 7])
I am a decoder
BertSelfAttention is being called
attention_scores:  torch.Size([1, 12, 8, 8])
attention_masks:  torch.Size([1, 1, 7, 768])
```

It looks like this is the first time that cross attention is being called. The `is_decoder` flag is being passed as `False` in the tutorial code and we changed it to `True` in the code ourselves. The error is the same irrespective of that change.

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

The code runs as given in the tutorial.

## Environment

* OS: Linux
* Python version: 3.7.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): Latest master branch
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:


ALBERT pretrained models uses wrong type of GELU activation

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): ALBERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] my own modified scripts: i'm fine-tuning albert for multilabel-classifcation problem and then converting model into tf-lite format

The tasks I am working on is:
* [x] my own task or dataset: multilable text classification on SemEval 2018 task 1:E-c

## To Reproduce

Steps to reproduce the behavior:

1. Open any link to pretrained configuration at [transformers/configuration_albert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_albert.py). For example: [albert-base-v2-config.json](https://s3.amazonaws.com/models.huggingface.co/bert/albert-base-v2-config.json). But problem applies to all the pretrained configs.
2. Check the value of property "hidden_act" (it will be gelu)
3. Realize, that gelu stands for bert-like implementation(see code [here](https://github.com/huggingface/transformers/blob/1ae63e4e097fe26c900783dd5c1710cf562e222e/src/transformers/modeling_bert.py#L152)), while original [code](https://github.com/google-research/ALBERT/blob/e350db671ae96b8345cd2c0ee1306713642b9810/modeling.py#L296) uses OpenAI-GPT - like gelu (it is defined in transformers as "gelu_new")

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

All the configuration files should contain "gelu_new" under "hidden_act" key.

## Environment

* OS: doesn't matter
* Python version: 3.7
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch): master
* Using GPU: doesn't matter
* Distributed of parallel setup: doesn't matter
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->
While it possibly doesn't significantly change the model performance, it makes conferting model into tf-lite format a lot more tricky, because BERT-like gelu implementation uses tf.math.erf function, which is not in a tflite-builtins set. So i have to use tflite-select ops, which leads to using more heavy tflite-interpreter on android side. Also, it makes impossible to check the converted model performance on python side because default tensorflow-lite python interpreter can't interpret select-ops models.


Pretrained model installation issue

I run the script for this repo "https://github.com/alexa/wqa_tanda" in which i need to run the run_glue.py file from Transformer Model, while running that script it gives an error-
Couldn't reach server at "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json to download pretrained model configuration file" as shown -
![Screenshot (214)](https://user-images.githubusercontent.com/48346579/71734743-950c9280-2e72-11ea-9507-c04b0ab5ed52.png)


run_squad_w_distillation update

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): DistilBERT and BERT

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: run_squad_w_distillation.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQuAD

## To Reproduce

Steps to reproduce the behavior:

1. Fine tuning on question answering from BERT to DistilBERT

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
The files  utils_squad and utils_squad_evaluate are missing right now, so to make this script working I had to restore the files from previous version of this repo. What I expected is to use the script without using tricks

## Environment

* OS: Ubuntu 16.04
* Python version: 3.5.6
* PyTorch version: 1.3.1
* Using GPU nvidia P100 (2x) or 1080Ti

## Additional context

<!-- Add any other context about the problem here. -->


Unable to download community models

## 🐛 Bug

Model I am using (Bert, XLNet....): `bert-base-cased-finetuned-conll03-english`

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: running a small snippet from docs (see below)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: just trying to load the model at this stage

## To Reproduce

Steps to reproduce the behavior:

I'm following the instructions at https://huggingface.co/bert-large-cased-finetuned-conll03-english but failing at the first hurdle. This is the snippet from the docs that I've run:

```python
tokenizer = AutoTokenizer.from_pretrained("bert-large-cased-finetuned-conll03-english")
model = AutoModel.from_pretrained("bert-large-cased-finetuned-conll03-english")
```

It fails with this message:

```
OSError: Model name 'bert-base-cased-finetuned-conll03-english' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-japanese, bert-base-japanese-whole-word-masking, bert-base-japanese-char, bert-base-japanese-char-whole-word-masking, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-conll03-english/config.json' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.
```

The message mentions looking at https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-conll03-english/config.json and finding nothing.

I also tried with the CLI: `transformers-cli download bert-base-cased-finetuned-conll03-english` but I'm afraid that failed with a similar message. However both methods work for the namespaced models, e.g. `dbmdz/bert-base-italian-cased`.

## Expected behavior

The community model should download. :)

## Environment

* OS: openSUSE Tumbleweed 20200101
* Python version: 3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0
* Using GPU ? n/a
* Distributed of parallel setup ? n/a
* Any other relevant information:

## Additional context

I browsed https://s3.amazonaws.com/models.huggingface.co/ and see that the model is there, but paths are like:

https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-conll03-english-config.json

rather than:

https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-finetuned-conll03-english/config.json

(note `-config.json` vs `/config.json`)

If I download the files manually and rename, the model loads. So it looks like just a naming problem.


What dataset was used for the NER results reported in the docs for bert/roberta-large-cased and distilbert-base-uncased models?

## ❓ Questions & Help

Regarding [this section in the docs](https://huggingface.co/transformers/examples.html#comparing-bert-large-cased-roberta-large-cased-and-distilbert-base-uncased) and the NER results using bert-large-cased, roberta-large-cased, and distillbert-base-uncased ...

**What dataset was used?**

When I try them with the GermanEval2014 dataset, the results are inferior to that of the multi-lingual models (which makes sense) ... so my question, upon what dataset(s) were these models trained on that go the most excellent F scores reported in the docs?


Pipelines support

Core: Pipeline

Internals of the library; Pipeline.

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....):

I'm using roberta-base model for feature extraction through pipeline functionality.

Language I am using the model on English texts.

The problem arise when using:
* [x ] my own modified scripts: (give details)

```
from transformers import pipeline
import torch
#torch.set_default_tensor_type('torch.cuda.FloatTensor')

nlp = pipeline('feature-extraction', model='roberta-base', tokenizer='roberta-base', device=0)

def encode(input):
    with nlp.device_placement():
        return np.array(nlp(input)).mean(axis=1)

results = encode(['cybersecurity', 'cyber security', 'agriculture', 'data'])
```

## To Reproduce

Steps to reproduce the behavior:

1. Just run code above.

Error details

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-f0b27f7cd838> in <module>
----> 1 encode(['cybersecurity', 'cyber security', 'agriculture', 'data']).shape

<ipython-input-5-a0628a1cb908> in encode(input)
     12 def encode(input):
     13     with nlp.device_placement():
---> 14         return np.array(nlp(input)).mean(axis=1)

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    442
    443     def __call__(self, *args, **kwargs):
--> 444         return super().__call__(*args, **kwargs).tolist()
    445
    446

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    402             # Filter out features not available on specific models
    403             inputs = self.inputs_for_model(inputs)
--> 404             return self._forward(inputs)
    405
    406     def _forward(self, inputs):

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in _forward(self, inputs)
    417         else:
    418             with torch.no_grad():
--> 419                 predictions = self.model(**inputs)[0].cpu()
    420
    421         return predictions.numpy()

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     68                                                       token_type_ids=token_type_ids,
     69                                                       position_ids=position_ids,
---> 70                                                       inputs_embeds=inputs_embeds)
     71
     72

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    184
    185         if inputs_embeds is None:
--> 186             inputs_embeds = self.word_embeddings(input_ids)
    187         position_embeddings = self.position_embeddings(position_ids)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select
```
## Expected behavior
Sentences encoded properly.
## Environment

* OS: Ubuntu 18.01
* Python version: Python 3.6.5 :: Anaconda, Inc.
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): 2.3.0 and master
* Using GPU ? yes
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context
`torch.set_default_tensor_type('torch.cuda.FloatTensor')`
Uncommenting such line solves issue partially. Issue with CUDA tensor disappears but those sentences could not be encoded properly

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-10-f0b27f7cd838> in <module>
----> 1 encode(['cybersecurity', 'cyber security', 'agriculture', 'data']).shape

<ipython-input-9-138f4526e218> in encode(input)
     12 def encode(input):
     13     with nlp.device_placement():
---> 14         return np.array(nlp(input)).mean(axis=1)

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    442
    443     def __call__(self, *args, **kwargs):
--> 444         return super().__call__(*args, **kwargs).tolist()
    445
    446

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    402             # Filter out features not available on specific models
    403             inputs = self.inputs_for_model(inputs)
--> 404             return self._forward(inputs)
    405
    406     def _forward(self, inputs):

~/anaconda3/lib/python3.6/site-packages/transformers/pipelines.py in _forward(self, inputs)
    417         else:
    418             with torch.no_grad():
--> 419                 predictions = self.model(**inputs)[0].cpu()
    420
    421         return predictions.numpy()

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    733             head_mask = [None] * self.config.num_hidden_layers
    734
--> 735         embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
    736         encoder_outputs = self.encoder(embedding_output,
    737                                        attention_mask=extended_attention_mask,

~/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     68                                                       token_type_ids=token_type_ids,
     69                                                       position_ids=position_ids,
---> 70                                                       inputs_embeds=inputs_embeds)
     71
     72

~/anaconda3/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    188         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    189
--> 190         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    191         embeddings = self.LayerNorm(embeddings)
    192         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered
```
even with
CUDA_LAUNCH_BLOCKING=1
If we try to encode sentence by sentence everything works.


update the config.is_decoder=True before initialize the decoder

Currently the `PreTrainedEncoderDecoder`  class fails to initialize the "cross-attention layer" since it updates  `decoder.config.is_decoder = True` after decoder initialization.


Can't load finetuned model properly.

I am making a model for joint bert. After trained my model, i try to eval before saving and it gives with %95 accuracy. But the problem is when i save this trained model and load it, i get the awful result. I hope you can help me about finding why i cant load properly.
Here is some part of my code

```
class JointBertClassification(BertPreTrainedModel):
    def __init__(self, model_name, config, num_intent_labels, num_slot_labels, args):
        super(JointBertClassification, self).__init__(config)
        self.num_intent_labels = num_intent_labels
        self.num_slot_labels = num_slot_labels

        dropout_rate = args[
            "dropout_rate"
        ]
        self.bert = BertModel.from_pretrained(
            model_name, config=self.bert_config
        )  # Load pretrained bert
        self.intent_classifier = IntentClassifier(
            config.hidden_size, num_intent_labels, dropout_rate
        )
        self.slot_classifier = SlotClassifier(
            config.hidden_size, num_slot_labels, dropout_rate
        )
        # self.init_weights()

    def forward(
        self,
        input_ids,
        attention_mask,
        token_type_ids,
        intent_label_ids,
        slot_label_ids,
    ):
        ...
```

```
class JointModel:
    def __init__(
        self, model_type, model_name, intents=None, slots=None, args=None, use_cuda=None
    ):
        """
        Initializes a Joint Model

        Args:
            model_type: The type of model
            model_name: Default Transformer model name or path to directory containing Transformer model file
            intents (optional): A list of all Intent labels. If not given ATIS intents will set as default.
            slots (optional): A list of all Slot labels. If not given ATIS slots will set as default.
            args (optional): Default args will be used if thi parameter is not provided. If provided, it should be a dict containing the args that should be changed in the default args.
            use_cuda (optional): Use GPU if available. Setting to False will force model to use CPU only.
        """

        MODEL_CLASSES = {"bert": (BertConfig, JointBertClassification, BertTokenizer)}

        self.config_class, self.model_class, tokenizer_class = MODEL_CLASSES[model_type]

        if intents:
            self.intent_labels = intents
        else:
            self.intent_labels = pd.read_csv(
                "jointbert/data/atis/vocab.intent", header=None, index_col=0
            ).index.tolist()
        self.num_intents = len(self.intent_labels)

        if slots:
            self.slot_labels = slots
        else:
            self.slot_labels = pd.read_csv(
                "jointbert/data/atis/vocab.slot", header=None, index_col=0
            ).index.tolist()
        self.num_slots = len(self.slot_labels)

        self.tokenizer = tokenizer_class.from_pretrained(model_name)
        self.bert_config = self.config_class.from_pretrained(model_name)

        self.model = self.model_class(
            model_name,
            self.bert_config,
            num_slot_labels=self.num_slots,
            num_intent_labels=self.num_intents,
            args={"dropout_rate": 0.2},
        )

        if use_cuda:
            if torch.cuda.is_available():
                self.device = torch.device("cuda")
            else:
                raise ValueError(
                    "'use_cuda' set to True when cuda is unavaiable. Make sure CUDA is avaiable or set use_cuda=False"
                )
        else:
            self.device = "cpu"

        self.results = {}
        self.args = {
            "output_dir": "outputs/",
            "cache_dir": "cache_dir/",
            "fp16": False,
            "fp16_opt_level": "O1",
            "max_seq_length": 128,
            "train_batch_size": 8,
            "gradient_accumulation_steps": 1,
            "eval_batch_size": 8,
            "num_train_epochs": 1,
            "weight_decay": 0,
            "learning_rate": 4e-5,
            "adam_epsilon": 1e-8,
            "warmup_ratio": 0.06,
            "warmup_steps": 0,
            "max_grad_norm": 1.0,
            "logging_steps": 50,
            "save_steps": 2000,
            "evaluate_during_training": False,
            "overwrite_output_dir": False,
            "reprocess_input_data": False,
            "process_count": 1,
            "n_gpu": 1,
            "silent": False,
        }

        if args:
            self.args.update(args)

        self.args["model_name"] = model_name
        self.args["model_type"] = model_type

        self.pad_token_label_id = CrossEntropyLoss().ignore_index
```

The Saving Part after training
```
    def train_model(
        self,
        train_data,
        output_dir=None,
        show_running_loss=True,
        args=None,
        eval_df=None,
    ):
        if args:
            self.args.update(args)

        if self.args["silent"]:
            show_running_loss = False

        if not output_dir:
            output_dir = self.args["output_dir"]

        if (
            os.path.exists(output_dir)
            and os.listdir(output_dir)
            and not self.args["overwrite_output_dir"]
        ):
            raise ValueError("--")

        self._move_model_to_device()
        train_dataset = self.load_and_cache_examples(train_data)

        global_set, tr_loss = self.train(
            train_dataset,
            output_dir,
            show_running_loss=show_running_loss,
            eval_df=eval_df,
        )

        if not os.path.exists(output_dir):
            os.makedirs(output_dir)

        model_to_save = (
            self.model.module if hasattr(self.model, "module") else self.model
        )

        model_to_save.save_pretrained(output_dir)
        self.tokenizer.save_pretrained(output_dir)
        torch.save(self.args, os.path.join(output_dir, "training_args.bin"))
        print(
            "Training of {} model complete. Saved to {}. Training Loss : {}".format(
                self.args["model_type"], output_dir, tr_loss
            )
        )

```


Pre-trained model returns different outputs(random outputs)

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hello,
I had recently play around Huggingface library, i wrote a simple script for question answering task. and for that,  i used TFXLNetForQuestionAnsweringSimple model (pre-trained model), but i had different outputs for the same inputs and model each time i run the program.
Did i miss something?
here is my script:

context = "Jim Henson was a puppeteer"
question = "Who was Jim Henson ?"

#XLNET
tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
model = TFXLNetForQuestionAnsweringSimple.from_pretrained("xlnet-base-cased")
en_plus = tokenizer.encode_plus(context, question, add_special_tokens=True)
en = en_plus['input_ids']
token_type_ids = en_plus['token_type_ids']
input_ids = tf.constant([en])

segments_tensors = tf.constant([token_type_ids])
outputs = model(input_ids)
start_scores, end_scores = outputs[:2]

ss = tf.argmax(start_scores.numpy()[0]).numpy()
es = tf.argmax(end_scores.numpy()[0]).numpy()

answer = tokenizer.decode(en[ss: es+1], clean_up_tokenization_spaces=True)

print(answer)

Thank you in advance for your help.


Different usage between BertModel and AlbertModel

## ❓ Questions & Help
Hi~
```
bert_path = 'D:/pretrain/pytorch/albert_base/'
tokenizer = BertTokenizer.from_pretrained(bert_path)
BERT = BertModel.from_pretrained(bert_path)
...
with torch.no_grad():
    last_hidden_states = BERT(input_ids)[0]
```
I found ```last_hidden_states``` was not fixed  when I reload ```BertModel.from_pretrained(bert_path)```.

```
bert_path = 'D:/pretrain/pytorch/albert_base/'
tokenizer = BertTokenizer.from_pretrained(bert_path)
BERT = AlbertModel.from_pretrained(bert_path)
...
with torch.no_grad():
    last_hidden_states = BERT(input_ids)[0]
```
I found ```last_hidden_states ``` was fixed.
But When I tried
```
bert_path = 'D:/pretrain/pytorch/chinese_roberta_wwm_ext/'
tokenizer = BertTokenizer.from_pretrained(bert_path)
BERT = RobertaModel.from_pretrained(bert_path)
...
with torch.no_grad():
    last_hidden_states = BERT(input_ids)[0]
```
I found ```last_hidden_states``` was still not fixed.

```
bert_path = 'D:/pretrain/pytorch/chinese_roberta_wwm_ext/'
tokenizer = BertTokenizer.from_pretrained(bert_path)
BERT = BertModel.from_pretrained(bert_path)
...
with torch.no_grad():
    last_hidden_states = BERT(input_ids)[0]
```
I found ```last_hidden_states``` was fixed.

Is there any difference in their usage between BertModel, AlbertModel and RobertaModel?

In my past projects, I used BERT(freeze)+LSTM. This is the first time to use ALBERT.

Thanks~

<!-- A clear and concise description of the question. -->


The method os.rename() in file_utils.py  make a permissionError

## ❓ Questions & Help
Is someone happen to this question?
<!-- A clear and concise description of the question. -->

![image](https://user-images.githubusercontent.com/22883367/71649688-af981c00-2d4b-11ea-811b-7fd9ebef4776.png)


Releasing file lock

`With` scope creates a file lock, which leads to the following error:

INFO:filelock:Lock 1408081097608 released on C:\Users\dimag\.cache\torch\transformers\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock
Traceback (most recent call last):
  File "C:\Users\dimag\Anaconda3\envs\pytorch\lib\site-packages\transformers\tokenization_utils.py", line 398, in _from_pretrained
    resume_download=resume_download,
  File "C:\Users\dimag\Anaconda3\envs\pytorch\lib\site-packages\transformers\file_utils.py", line 212, in cached_path
    user_agent=user_agent,
  File "C:\Users\dimag\Anaconda3\envs\pytorch\lib\site-packages\transformers\file_utils.py", line 392, in get_from_cache
    os.rename(temp_file.name, cache_path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\dimag\\.cache\\torch\\transformers\\tmpnhzxze8u' -> 'C:\\Users\\dimag\\.cache\\torch\\transformers\\26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084'


clarification on output

Hi,

On using bert_uncased on the following sentence:

`Hello this is my dog`

and to get attentions I use:

`last_hidden_states, pooler_outputs, hidden_states, attentions = outputs`

`attentions` gives:
a tuple of 12 tensors where each tensor is of size [1,12,5,5]

I was wondering if the 12 tensors in the tuple are for each attention head or hidden layer.

Thanks!


Proposition to include community models in readme


how to use distilledgpt2

Hi
I want to use distilledgpt2, I cannot see the config file and modeling files, could you please assist me how to use it
thanks


errors encountered with run_lm_finetuning.py

Hi
I am using run_lm_finetuning.py, I encountered the following issues:
- block_size value is by default = -1, which creates the following error, can be solved by setting the default value to 512:
```
  File "run_lm_finetuning.py", line 712, in <module>
    main()
  File "run_lm_finetuning.py", line 662, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 198, in train
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer36/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 64, in __init__
    "value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integeral value, but got num_samples=0
```
- global_step = int(args.model_name_or_path.split("-")[-1].split("/")[0]) can crash, let assume the "args.model_name_or_path=gpt2" then the result of the expression is int(""), which will crash, maybe setting it to 0?

- when running the script for bert model I got also the following error, I am using pytorch 1.2.
```
(transformer) rkarimi@italix17:/idiap/user/rkarimi/dev/lm_heads$ python run_lm_finetuning.py  --output_dir=/idiap/temp/rkarimi/lm_heads/distilbert  --model_type=distilbert --model_name_or_path=/idiap/temp/rkarimi/pretrained_transformers/bert_distil  --do_train  --train_data_file=/idiap/temp/rkarimi/resources/wikitext-2-raw/wiki.train.raw  --do_eval  --eval_data_file=/idiap/temp/rkarimi/resources/wikitext-2-raw/wiki.test.raw --mlm --block_size=511
To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html
01/02/2020 16:53:27 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0, distributed training: False, 16-bits training: False
01/02/2020 16:53:27 - INFO - transformers.configuration_utils -   loading configuration file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/config.json
01/02/2020 16:53:27 - INFO - transformers.configuration_utils -   Model config {
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "finetuning_task": null,
  "hidden_dim": 3072,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "max_position_embeddings": 512,
  "n_heads": 12,
  "n_layers": 6,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 30522
}

01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   Model name '/idiap/temp/rkarimi/pretrained_transformers/bert_distil' not found in model shortcut name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). Assuming '/idiap/temp/rkarimi/pretrained_transformers/bert_distil' is a path or url to a directory containing tokenizer files.
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   Didn't find file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/added_tokens.json. We won't load it.
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   Didn't find file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/special_tokens_map.json. We won't load it.
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   Didn't find file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/tokenizer_config.json. We won't load it.
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   loading file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/vocab.txt
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   loading file None
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   loading file None
01/02/2020 16:53:27 - INFO - transformers.tokenization_utils -   loading file None
01/02/2020 16:53:27 - INFO - transformers.modeling_utils -   loading weights file /idiap/temp/rkarimi/pretrained_transformers/bert_distil/pytorch_model.bin
01/02/2020 16:53:28 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=511, cache_dir='', config_name='', device=device(type='cpu'), do_eval=True, do_lower_case=False, do_train=True, eval_all_checkpoints=False, eval_data_file='/idiap/temp/rkarimi/resources/wikitext-2-raw/wiki.test.raw', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, local_rank=-1, logging_steps=50, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='/idiap/temp/rkarimi/pretrained_transformers/bert_distil', model_type='distilbert', n_gpu=0, no_cuda=False, num_train_epochs=1.0, output_dir='/idiap/temp/rkarimi/lm_heads/distilbert', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=50, save_total_limit=None, seed=42, server_ip='', server_port='', tokenizer_name='', train_data_file='/idiap/temp/rkarimi/resources/wikitext-2-raw/wiki.train.raw', warmup_steps=0, weight_decay=0.0)
01/02/2020 16:53:28 - INFO - __main__ -   Creating features from dataset file at /idiap/temp/rkarimi/resources/wikitext-2-raw
01/02/2020 16:53:53 - INFO - __main__ -   Saving features into cached file /idiap/temp/rkarimi/pretrained_transformers/bert_distil_cached_lm_511_wiki.train.raw
01/02/2020 16:53:53 - INFO - __main__ -   ***** Running training *****
01/02/2020 16:53:53 - INFO - __main__ -     Num examples = 4303
01/02/2020 16:53:53 - INFO - __main__ -     Num Epochs = 1
01/02/2020 16:53:53 - INFO - __main__ -     Instantaneous batch size per GPU = 4
01/02/2020 16:53:53 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
01/02/2020 16:53:53 - INFO - __main__ -     Gradient Accumulation steps = 1
01/02/2020 16:53:53 - INFO - __main__ -     Total optimization steps = 1076
01/02/2020 16:53:53 - INFO - __main__ -     Continuing training from checkpoint, will skip to saved global_step
01/02/2020 16:53:53 - INFO - __main__ -     Continuing training from epoch 0
01/02/2020 16:53:53 - INFO - __main__ -     Continuing training from global step 0
01/02/2020 16:53:53 - INFO - __main__ -     Will skip the first 0 steps in the first epoch
Epoch:   0%|                                                                                | 0/1 [00:00<?, ?it/sTraceback (most recent call last):                                                                                | 0/1076 [00:00<?, ?it/s]
  File "run_lm_finetuning.py", line 738, in <module>
    main()
  File "run_lm_finetuning.py", line 688, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 325, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/transformers/modeling_distilbert.py", line 540, in forward
    inputs_embeds=inputs_embeds)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/transformers/modeling_distilbert.py", line 477, in forward
    inputs_embeds = self.embeddings(input_ids)   # (bs, seq_length, dim)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/transformers/modeling_distilbert.py", line 96, in forward
    position_embeddings = self.position_embeddings(position_ids)        # (bs, max_seq_length, dim)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/transformer/lib/python3.6/site-packages/torch/nn/functional.py", line 1467, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at /opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237
```
The issue will resolve by setting smaller block_size <= 510, it would be very nice to document this in the codes that one needs to set the block_size <= 510 as a temporary solution. thanks

- In mask_tokens function, the following lines needs to be set to -1 not -100 which is the ignore_index used in the "BertForMaskedLM" model:
labels[~masked_indices] = -100 => -1

thanks.


finetune transformer

Hi
I greatly appreciate showing me how to pretrain a transformer model like BERT, I mean not finetuning but pretraining, Is there any code in your repo doing this? thanks a lot for your help.


added pad_to_max_length option to batch_encode_plus


Text generation on GPU: Moved the encoded_prompt to correct device

Moved the `encoded_prompt` to the correct device to solve the problem when using GPU.

Solves the problem mentioned in #227 #1414 #2360


Classification of sentence pair with two different languages

I have been working on multi-lingual sentence similarity (English-Hindi)

### for example:

> Sentence 1 (English)

> Sentence 2 (Translation in Hindi of Sentence 1)

> Sentence Similarity Score.

Any idea on how do I train for sentence similarity using `xlm-mlm-xnli15-1024`?


Is the position of the scheduler.step() correct?

## ❓ Questions & Help

In the [lm_finetuning_file](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py#L285-L319), the scheduler used is "get_linear_schedule_with_warmup" which in turn uses the underlying "LambdaLR" [ref](https://github.com/huggingface/transformers/blob/594ca6deadb6bb79451c3093641e3c9e5dcfa446/src/transformers/optimization.py#L47);

On a careful study of the file, the "scheduler" is being called for every batch, but that isn't what's as per the official PyTorch [docs](https://pytorch.org/docs/stable/optim.html#torch.optim.lr_scheduler.LambdaLR) to change LR since they are calling it at "epoch-level",

```
>>> lambda1 = lambda epoch: epoch // 30
>>> lambda2 = lambda epoch: 0.95 ** epoch
>>> scheduler = LambdaLR(optimizer, lr_lambda=[lambda1, lambda2])
>>> for epoch in range(100):
>>>     train(...)
>>>     validate(...)
>>>     scheduler.step()
```

I meant it makes sense to me to do it at batch level change the lr etc but not sure why the PyTorch has it differently?

Thanks.


Fine-tuning BertAbs on new dataset?

## 🚀 Feature

Hi,
I wonder if there can be a script for fine-tuning BertAbs on new dataset?
Or if you have some hint to provide about this task? Not sure how to access loss from ```modeling_bertabs.py```.

Thanks


RuntimeError: The size of tensor a (30524) must match the size of tensor b (30522) at non-singleton dimension 2 --- run_lm_finetuning.py

## 🐛 Bug

I am using ```run_lm_finetuning.py``` to fine-tune bert-base-uncased on my dataset and I am getting the following error:
I also truncated my dataset to have num examples dividable by the batch_size.
Note that fine-tuning gpt2 on the same dataset works fine.

```
12/30/2019 17:23:28 - INFO - __main__ -   ***** Running training *****
12/30/2019 17:23:28 - INFO - __main__ -     Num examples = 4048
12/30/2019 17:23:28 - INFO - __main__ -     Num Epochs = 3
12/30/2019 17:23:28 - INFO - __main__ -     Instantaneous batch size per GPU = 4
12/30/2019 17:23:28 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
12/30/2019 17:23:28 - INFO - __main__ -     Gradient Accumulation steps = 1
12/30/2019 17:23:28 - INFO - __main__ -     Total optimization steps = 3036
Epoch:   0%|                                                                                | 0/3 [00:00<?, ?it/sTraceback (most recent call last):                                                                                | 0/1012 [00:00<?, ?it/s]
  File "run_lm_finetuning.py", line 722, in <module>
    main()
  File "run_lm_finetuning.py", line 672, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 306, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_bert.py", line 990, in forward
    prediction_scores = self.cls(sequence_output)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_bert.py", line 496, in forward
    prediction_scores = self.predictions(sequence_output)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_bert.py", line 486, in forward
    hidden_states = self.decoder(hidden_states) + self.bias
RuntimeError: The size of tensor a (30524) must match the size of tensor b (30522) at non-singleton dimension 2
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu
* Python version: python3.6
* PyTorch version: 1.3.0
* PyTorch Transformers version (or branch):
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:


What is the "could not find answer" warning in squad.py

Hello,

I am trying to run run_squad.py for BERT (italian-cased) with an italian version of squad.

During the creation of features from dataset, I got some answer skipped like in the following:
<img width="478" alt="Screenshot 2019-12-30 at 23 30 19" src="https://user-images.githubusercontent.com/26765504/71603304-81081e80-2b5c-11ea-8333-73608e3141a7.png">

Can you tell why is this happening and if this influences the overall accuracy of the training?


Encounter an "index out of range problem"

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi!
I'm currently using BERT to obtain sentence embeddings for Chinese text inputs.
Things are fine for most cases. However when I am dealing with this text:
```python
weibo_content = "貌似还是没有完全修复。  http://As.international.anger.grows.over.reports.of.mass.carnage.at.the.hands.of.the.Syrian.regime.a.U.N.Security.Council.draft.resolution.condemning.Syria.failed.to.be.adopted.Saturday.after.vetowielding.members.Russia.and.China.voted.against.it.Ambassadors.from.the.other.permanent.members.of.the.council..the.United.States.France.and.the.United.Kingdom..said.they.were.furious.at.Russia.and.China.for.failing.to.halt.the.worsening.bloody.violence.that.has.consumed.the.Middle.Eastern.nation.Thirteen.Security.Council.members.voted.in.favor.of.the.resolution.The.vote.was.a.major.diplomatic.setback.for.countries.hoping.to.send.a.unified.message.to.embattled.Syrian.President.Bashar.alAssad.and.also.for.opposition.groups.that.look.toward.the.United.Nations.for.support.Those.that.have.blocked.potentially.the.last.effort.to.resolve.this.peacefully.will.have.any.future.blood.spill.on.their.hands.U.S.Ambassador.Susan.Rice.told.CNN.The.people.of.Syria.have.yet.again.been.abandoned.by.this.Council.and.by.the.international.community.Some.Syrians.have.cried.out.for.international.action.to.stop.attacks.on.civilians.more.so.after.opposition.groups.said.at.least.321.civilians.were.killed.and.hundreds.wounded.in.the.city.of.Homs.in.the.past.two.days.The.opposition.Syrian.National.Council.blamed.government.forces.for.the.attack.in.Homs.calling.it.one.of.the.most.horrific.massacres.since.the.start.of.the.Syrian.uprising.Residential.buildings.and.homes.were.randomly.and.heavily.bombed.the.group.said.The.Local.Coordination.Committees.LCC.a.Syrian.opposition.group.said.90.people.had.been.killed.in.Syria.on.Saturday.including.61.in.Homs.10.in.Idlib.and.19.in.a.Damascus.suburb.In.a.bid.to.pressure.the.government.the.group.called.for.a.twoday.civil.strike.to.start.on.Sunday.Another.opposition.group.the.Syrian.Observatory.for.Human.Rights.reported.that.48.people.were.killed.across.Syria.on.Saturday.including.six.army.defectors.and.18.members.of.the.Syrian.security.forces"
```
something goes wrong...

Okay, This text piece contains an url with many English words in it. But, I think using "bert-base-chinese" can still handle this situation.

So the following code goes like:
```python
import torch
from transformers import BertTokenizer, BertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
model = BertModel.from_pretrained('./bert-base-chinese')
model.eval()

weibo_content = "[cls]" + weibo_content + "[sep]"  # weibo_content is the target text to be extracted embeddings from
tokenized_weibo_content = tokenizer.tokenize(weibo_content)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_weibo_content)
segments_ids = [1] * len(tokenized_weibo_content)
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

with torch.no_grad():
    encoded_layers, _ = model(tokens_tensor, segments_tensors)
```
In the last step of the above code, I meet with the following problem:
```python
Traceback (most recent call last):
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-12-c7fe4edd73d7>", line 7, in <module>
    encoded_layers, _ = model(tokens_tensor, segments_tensors)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\transformers\modeling_bert.py", line 735, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\transformers\modeling_bert.py", line 187, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\torch\nn\modules\sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "C:\Users\xzzz\Anaconda3\envs\cfdstorch\lib\site-packages\torch\nn\functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at C:\w\1\s\tmp_conda_3.6_111945\conda\conda-bld\pytorch_1572952852006\work\aten\src\TH/generic/THTensorEvenMoreMath.cpp:418
```
Does anyone know why this happens?


Pipelines: add PoS support

Internals of the library; Pipeline.

## 🚀 Feature

As `Pipelines` were recently added for many tasks including NER, Sentiment Analysis, it'd be great to also enable Part-of-Speech tagging.

## Motivation

PoS tagging is a very useful task, and often used as an evaluating downstream task for new models.

## Additional context

Current available tasks for `Pipelines` are described [here](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L831).


few changes due to the torch version inconsistency in summarization example

This small change intends to fix the issue #2297.
it's generally a version inconsistent issue.
in ver 1.1.0, torch.gt outputs:

_torch.gt(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[ 0, 1],
[ 0, 0]], dtype=torch.uint8)_

while in ver 1.2.0, it outputs:

_torch.ge(torch.tensor([[1, 2], [3, 4]]), torch.tensor([[1, 1], [4, 4]]))
tensor([[True, True], [False, True]])_

Thus, I added a version checking function, and revise the tensor type in tensor.gt()


Clarification regarding past/layer_past in GPT-2

## ❓ Questions & Help

I'm hoping to get some clarification regarding how past/layer_past are meant to work in GPT-2. My prior impression was that the query/key/value at every layer (other than the first) should be influenced by all tokens the model is able to see. As such, it shouldn't make much sense to use pre-computed key/value activations from prior steps. Could you clarify this issue?

(Hope my question makes sense.)


Load the google bert model(ckpt) from TFBertForPreTraining error

I want use the google chinese bert ckpt model in transforms, and env use tf2.
the transforms can load the ckpt  into pytorch model
But I want load the ckpt into tf.keras model, How can I do this ?

![image](https://user-images.githubusercontent.com/12653212/71570221-ce5d9f00-2b0e-11ea-8595-7b8b22db87f2.png)

model = BertForPreTraining.from_pretrained(checkpoint_path, from_tf=True, config=config)

Success.

But:
![image](https://user-images.githubusercontent.com/12653212/71570227-f0572180-2b0e-11ea-9978-f9c679a3e37b.png)


How Can I load the google bert model(ckpt)?

TF2.0
## ❓ Questions & Help
import os
pretrained_path = 'chinese_L-12_H-768_A-12'
config_path = os.path.join(pretrained_path, 'bert_config.json')
checkpoint_path = os.path.join(pretrained_path, 'bert_model.ckpt')
vocab_path = os.path.join(pretrained_path, 'vocab.txt')

config = BertConfig.from_json_file(config_path)
model = TFBertForPreTraining.from_pretrained(checkpoint_path, config=config)

![image](https://user-images.githubusercontent.com/12653212/71568255-d3b3ed00-2b00-11ea-836b-5dc5ad7b9d86.png)


upgrading new transformer doesn't work

## ❓ Questions & Help

Hi,

I have pulled the repo again since a lot of stuff changed/added.
When I try to use ```pip install --upgrade .``` command, nothing changes and I am stuck in the following step forever:
```
(py36) pytorch-transformers$ pip install --upgrade .
Processing /home/pytorch-transformers

```

Plus, since a lot of folders are renamed and scripts moved to ```src``` folder, whenI try to do ```from transformers import BertTokenizer, BertModel, BertForMaskedLM```
I get following error:
```
ImportError                               Traceback (most recent call last)
<ipython-input-1-34ecfe73cb1a> in <module>
      1 import torch
----> 2 from transformers import BertTokenizer, BertModel, BertForMaskedLM, BertConfig, BertForPreTraining, BertConfig
      3
      4 # OPTIONAL: if you want to have more information on what's happening, activate the logger as follows
      5 import logging

ImportError: cannot import name 'BertTokenizer' from 'transformers' (unknown location)
```

Would you please help with this? What is the reason to move to src?


How to fine-tune PreTrainedEncoderDecoder on new dataset?

## ❓ Questions & Help

Hi,

Many thanks for your recent work implementing [this paper](https://arxiv.org/pdf/1907.12461.pdf).
I wonder if you have or can provide script and documentation for fine-tuning PreTrainedEncoderDecoder on new dataset?

Many thanks


Finetuning on several tasks

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hello,

Is it possible to finetune a Transformer on some dataset, and then finetune the model again on another dataset with a different number of output labels ?

I tried this and got the following error:
```
RuntimeError: Error(s) in loading state_dict for BertForSequenceClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([2, 768]) from checkpoint, the shape in current model is torch.Size([3, 768]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([2]) from checkpoint, the shape in current model is torch.Size([3]).
```
Thanks in advance


Why albert has a print statement during forward?

## ❓ Questions & Help

both AlbertTransformer and AlbertLayerGroup have print statements in the forward method, which messes up logging / printing during training


Improve logging message in feature conversion functions

This PR adds the total number of examples to process to the log message produced during the feature conversion step.


CTRL - RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

## 🐛 Bug

Hi,
The error
`RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'`

arise while running CTRL using examples/run_generation.py

Model I am using (Bert, XLNet....): **CTRL**

Language I am using the model on (English, Chinese....): **English**

The problem arise when using:
running CTRL using run_generation.py

`python examples/run_generation.py  --model_type ctrl --model_name ctrl --temperature 0.2 --repetition 1.2`

Full trace:

```
Traceback (most recent call last):
  File "examples/run_generation.py", line 236, in <module>
    main()
  File "examples/run_generation.py", line 222, in main
    repetition_penalty=args.repetition_penalty,
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad
    return func(*args, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_utils.py", line 744, in generate
    effective_batch_size,
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_utils.py", line 775, in _generate_no_beam_search
    outputs = self(**model_inputs)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_ctrl.py", line 520, in forward
    inputs_embeds=inputs_embeds,
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/transformers/src/transformers/modeling_ctrl.py", line 388, in forward
    inputs_embeds = self.w(input_ids)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 118, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/media/disk1/guytevet/venvs/py3/lib/python3.6/site-packages/torch/nn/functional.py", line 1454, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

```
## Environment

* OS: Ubuntu 18.04.2
* Python version: 3.6
* PyTorch version: 1.0.1.post2
* PyTorch Transformers version (or branch): master, installed from source
-e git+https://github.com/huggingface/transformers.git@f75bf05ce6a05ef316363de129c29f2e00cacd7b#egg=transformers
* Using GPU ? Yes
* Distributed of parallel setup ?


Confusion about the target_mapping parameter of the xlnet model

Why code at https://github.com/huggingface/transformers/blob/f75bf05ce6a05ef316363de129c29f2e00cacd7b/src/transformers/modeling_xlnet.py#L1029 is ` target_mapping[0, 0, -1] = 1.0`, i think it should be `  target_mapping[:, :, -1] = 1.0`

And I‘m confused about `taget_mapping` parameter, What is the difference between it and the `perm_mask` parameter?


Quickstart BERT Example: Assertion Error

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Bert

Language I am using the model on (English, Chinese....): English

The problem arises when:
* I run the official BERT Example in my local Jupyter Lab environment: Copy pasted the code and ran it in one cell.

## To Reproduce

Steps to reproduce the behavior:

1. Download and install Transformers from source
2. Start Jupyter lab
3. Copy paste and run the Quickstart BERT Example

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

Expected the assertion to pass.

## Environment

* OS: Oracle Linux Server 7.7
* Python version: 3.7.5
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): master
* Using GPU ? No
* Distributed of parallel setup ? No

## Additional context

<!-- Add any other context about the problem here. -->

![transformers_bug](https://user-images.githubusercontent.com/4649183/71553748-ff6e9e80-2a3e-11ea-813e-51d291d78870.png)


GLUE benchmark score for XLNet_base_cased?

## ❓ Questions & Help

Can someone provide the GLUE benchmark scores for different GLUE tasks. Or a script for preforming predictions on the test files will be really helpful.