Problem saving and/or loading fine-tuned model
Author: ant-louis
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
I have finetuned bert-base-cased for multi-class classification with BertForSequenceClassification and got great results (about 88% accuracy on my dev data). However, when I save the finetuned model, load it and run the evaluation on the exact same dev data, I got awful results (about 0.17 accuracy).

At first glance, it seems that either I am wrongly saving the fine-tuned model OR wrongly loading it after training. Would it be possible that save_pretrained only save the weights of the BERT model without the ones of the classifier above ?  @patrickvonplaten 

## Details
Here is how I save the fine-tuned model after training:
`model_to_save = model.module if hasattr(model, "module") else model # Take care of distributed/parallel training`
 `model_to_save.save_pretrained(args.output_dir)`
 `tokenizer.save_pretrained(args.output_dir)`

And here is how I load the fine-tuned model for running evaluation:
`model = BertForSequenceClassification.from_pretrained(
                         args.model_name_or_path, 
                         num_labels = args.num_labels, 
                         output_attentions = False, 
                         output_hidden_states = False, 
                         cache_dir = args.cache_dir)`
where `args.model_name_or_path` is the path of my .bin checkpoint, and args.num_labels stays unchanged during all process.

## Full code

```python
def train(args, model, tokenizer, dataset, tb_writer, categories):

    # Load training dataset.
    if args.do_eval and args.eval_filepath is None:
        print("No validation file given: splitting dataset to train/test datasets...\n")
        train_dataset, validation_dataset = split_data(dataset, args.test_percent, args.seed)
    else:
        train_dataset = dataset

    print("Creating training dataloader...\n")
    train_data, train_sampler, train_dataloader = create_dataloader(train_dataset, args.batch_size, training_data=True)
                                                       
    # Setting up Optimizer & Learning Rate Scheduler.
    optimizer = AdamW(model.parameters(),
                  lr = args.learning_rate,
                  eps = args.adam_epsilon
                )
    total_steps = len(train_dataloader) * args.num_epochs
    scheduler = get_linear_schedule_with_warmup(optimizer, 
                                                num_warmup_steps = 0, # Default value in run_glue.py
                                                num_training_steps = total_steps)
    # Init some useful variables.
    global_step = 0
    tr_loss, logging_loss = 0.0, 0.0

    # For each epoch...
    t = time.time()
    for epoch_i in range(0, args.num_epochs):
        
        # Perform one full pass over the training set.
        print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, args.num_epochs))
        print('Training...')

        # Measure how long the training epoch takes.
        t0 = time.time()

        # Put the model into training mode. Don't be mislead--the call to 
        # `train` just changes the *mode*, it doesn't *perform* the training.
        # `dropout` and `batchnorm` layers behave differently during training
        # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):

            # Unpack this training batch from our dataloader. 
            # As we unpack the batch, we'll also copy each tensor to the GPU using the `to` method.
            # `batch` contains three pytorch tensors:
            #   [0]: input ids 
            #   [1]: attention masks
            #   [2]: labels 
            b_input_ids = batch[0].to(args.device)
            b_input_mask = batch[1].to(args.device)
            b_labels = batch[2].to(args.device)

            # Always clear any previously calculated gradients before performing a backward pass. 
            # PyTorch doesn't do this automatically because accumulating the gradients is "convenient while training RNNs". 
            # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
            model.zero_grad()        

            # Perform a forward pass (evaluate the model on this training batch).
            # This will return the loss (rather than the model output) because we have provided the `labels`.
            # The documentation for this `model` function is here: 
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            outputs = model(b_input_ids, 
                        token_type_ids=None, 
                        attention_mask=b_input_mask, 
                        labels=b_labels)

            # The call to `model` always returns a tuple, so we need to pull the loss value out of the tuple.
            loss = outputs[0]
            
            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training

            # Accumulate the training loss over all of the batches so that we can calculate the average loss at the end. 
            # `loss` is a Tensor containing a single value; the `.item()` function just returns the Python value from the tensor.
            tr_loss += loss.item()

            # Perform a backward pass to calculate the gradients.
            loss.backward()

            # Clip the norm of the gradients to 1.0. This is to help prevent the "exploding gradients" problem.
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and take a step using the computed gradient.
            # The optimizer dictates the "update rule"--how the parameters are modified based on their gradients, the learning rate, etc.
            optimizer.step()

            # Update the learning rate.
            scheduler.step()
            
            # Update global step.
            global_step += 1
            
            # Progress update every 'logging_steps' batches.
            if args.logging_steps > 0 and step != 0 and step % args.logging_steps == 0:
                
                # Calculate elapsed time in minutes.
                elapsed = format_time(time.time() - t0)
                
                # Compute average training loss over the last 'logging_steps'. Write it to Tensorboard.
                loss_scalar = (tr_loss - logging_loss) / args.logging_steps
                tb_writer.add_scalar('Train/Loss', loss_scalar, global_step)
                logging_loss = tr_loss
                
                # Print the log.
                print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.    Training loss: {:.2f}'.format(step, len(train_dataloader), elapsed, loss_scalar))

        print("  Training epoch took: {:}\n".format(format_time(time.time() - t0)))
        
        if args.do_eval and args.eval_filepath is None:
            print("Running Validation...")
            # After the completion of each training epoch, measure our performance on our validation set.
            t0 = time.time()
            result, df_wrong, df_right = evaluate(args, model, validation_dataset, categories)
            
            # Write results to tensorboard.
            tb_writer.add_scalar('Test/Accuracy', result[0], epoch_i + 1)
            tb_writer.add_scalar('Test/Recall', result[1], epoch_i + 1)
            tb_writer.add_scalar('Test/Precision', result[2], epoch_i + 1)
            tb_writer.add_scalar('Test/F1 score', result[3], epoch_i + 1)
            tb_writer.add_scalar('Test/MCC', result[4], epoch_i + 1)
            
            # Plot confusion matrix.
            plot_confusion_matrix(result[5], categories, args.output_dir)
            
            # Save dataframes of wrong and right predictions for further analysis.
            df_wrong.to_csv(os.path.join(args.output_dir, 'preds_wrong.csv'))
            df_right.to_csv(os.path.join(args.output_dir, 'preds_right.csv'))
            
            print("  Validation took: {:}\n".format(format_time(time.time() - t0)))
            
    print("Training complete!  Took: {}\n".format(format_time(time.time() - t)))
        
    print("Saving model to {}...\n.".format(args.output_dir))
    model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
    model_to_save.save_pretrained(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    torch.save(args, os.path.join(args.output_dir, 'training_args.bin'))  # Good practice: save your training arguments together with the trained model
    return


def evaluate(args, model, validation_dataset, categories):

    # Creating validation dataloader.
    validation_data, validation_sampler, validation_dataloader = create_dataloader(validation_dataset, args.batch_size, training_data=False)
    
    # Get validation sentences.
    validation_sentences = validation_dataset[3]
    
    # Tracking variables
    nb_eval_steps = 0
    preds = None
    out_label_ids = None
    
    # Put the model in evaluation mode--the dropout layers behave differently during evaluation.
    model.eval()

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Add batch to GPU.
        b_input_ids, b_input_mask, b_labels = tuple(t.to(args.device) for t in batch)

        # Telling the model not to compute or store gradients, saving memory and speeding up validation
        with torch.no_grad():        

            # Forward pass, calculate logit predictions.
            # This will return the logits rather than the loss because we have not provided labels.
            # token_type_ids is the same as the "segment ids", which differentiates sentence 1 and 2 in 2-sentence tasks.
            outputs = model(b_input_ids, 
                            token_type_ids=None, 
                            attention_mask=b_input_mask)

        # Get the "logits" output by the model. The "logits" are the output values prior to applying an activation function like the softmax.
        logits = outputs[0]

        # Move logits and labels to CPU and store them.
        if preds is None:
            preds = logits.detach().cpu().numpy()
            out_label_ids = b_labels.detach().cpu().numpy()
        else:
            preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
            out_label_ids = np.append(out_label_ids, b_labels.detach().cpu().numpy(), axis=0)
                
        # Track the number of batches
        nb_eval_steps += 1
            
    # Take the max predicitions.
    preds = np.argmax(preds, axis=1)
    
    # Report results.
    result = compute_metrics(preds, out_label_ids, categories)
    print("  * Accuracy: {0:.4f}".format(result[0]))
    print("  * Recall: {0:.4f}".format(result[1]))
    print("  * Precision: {0:.4f}".format(result[2]))
    print("  * F1 score: {0:.4f}".format(result[3]))
    print("  * MCC: {0:.4f}".format(result[4]))

    # Get wrong and right predictions.
    df_wrong, df_right = analyze_predictions(preds, out_label_ids, validation_sentences)
    
    return result, df_wrong, df_right

def main(args):

    # Create tensorboard summarywriter.
    tb_writer = SummaryWriter()
    
    # Create output dir if none mentioned.
    if args.output_dir is None:
        model_name = os.path.splitext(os.path.basename(args.model_name_or_path))[0]
        args.output_dir = "./output/" + model_name + '/'
    if not os.path.exists(args.output_dir):
        os.makedirs(args.output_dir)
        
    # Set the seed value all over the place to make this reproducible.
    set_seed(args.seed)
    
    print("\n========================================")
    print('               Load model                 ')
    print("========================================\n")
    print("Loading BertForSequenceClassification model...\n")
    model = BertForSequenceClassification.from_pretrained(
        args.model_name_or_path, # Use the 12-layer BERT model, with an cased vocab.
        num_labels = args.num_labels, # The number of output labels
        output_attentions = False, # Whether the model returns attentions weights.
        output_hidden_states = False, # Whether the model returns all hidden-states.
        cache_dir = args.cache_dir,
    )
    #model = BertForSequenceClassification.from_pretrained(args.model_name_or_path)
    print('Loading BertTokenizer...\n')
    tokenizer = BertTokenizer.from_pretrained(args.model_name_or_path, do_lower_case=False)
    
    print("Setting up CUDA & GPU...")
    if torch.cuda.is_available():
        if args.gpu_id:
            torch.cuda.set_device(args.gpu_id)
            args.n_gpu = 1
            print("-> GPU training available! As '--gpu_id' was set, only GPU {} {} will be used (no parallel training).\n".format(torch.cuda.get_device_name(args.gpu_id), args.gpu_id))
        else:
            args.n_gpu = torch.cuda.device_count()
            gpu_ids = list(range(0, args.n_gpu))
            if args.n_gpu > 1:
                model = torch.nn.DataParallel(model, device_ids=gpu_ids, output_device=gpu_ids[-1])
            print("-> GPU training available! Training will use GPU(s) {}\n".format(gpu_ids))
        args.device = torch.device("cuda")
    else:
        args.device = torch.device("cpu")
        args.n_gpu = 0
        print("-> No GPU available, using the CPU instead.\n")
    model.to(args.device)  # Tell pytorch to run the model on the device.
    
    
    print("\n========================================")
    print('               Processing data            ')
    print("========================================\n")
    df, categories = load_data(args)
        
    print("Tokenizing sentences...")
    tokenized = tokenize_sentences(tokenizer, df)
    attention_masks = create_masks(tokenized)
    
    dataset = (tokenized, df.Class_id.values, attention_masks, df.Sentence.values)
    if args.do_train:
        print("\n========================================")
        print('            Launching training            ')
        print("========================================\n")
        train(args, model, tokenizer, dataset, tb_writer, categories)
    
    elif args.do_eval and args.eval_filepath is not None:
        print("\n========================================")
        print('            Launching validation          ')
        print("========================================\n")
        result, df_wrong, df_right = evaluate(args, model, dataset, categories)
        
        # Save dataframes of wrong and right predictions for further analysis.
        df_wrong.to_csv(os.path.join(args.output_dir, 'wrong_preds.csv'))
        df_right.to_csv(os.path.join(args.output_dir, 'right_preds.csv'))
```
  

Comments:
Hi @antoilouis,

Thanks for posting this. 
A quick wild guess might be that after training you save the model in `args.output_dir` but then load it from a different `args.model_name_or_path` (also since you said that if you evaluate the model right after training you get good results. 

My advice to solve this wolud be the following: 
Train for 1 epoch and a tiny part of the dataset. Print out / Save some model weights, then save load it again and check whether the weights are equal. They should be equal. 

Oh and also please post your environment information here. Make sure that you have the newest version of `transformers`. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I meet the same question . do you have any advice now? Thanks!

I am currently facing the same issue with the `BertForTokenClassification` model, the model behaves differently each time it is loaded. Are there solutions?