Loading Fine Tuned BERT Sequence Model after Training 
Author: tsivaguru
Pull request: no
Labels: name: wontfix

# 🐛 Bug
Loading Fine Tuned BERT Sequence Model after Training 
## Information

Hi All,

iam facing following issue while loading pretrained BERT Sequence model with my own data

RuntimeError: Error(s) in loading state_dict for DataParallel:
Missing key(s) in state_dict: "module.out.weight", "module.out.bias".
Unexpected key(s) in state_dict: "bert.embeddings.word_embeddings.weight", "bert.embeddings.position_embeddings.weight", "bert.embeddings.token_type_embeddings.weight", "bert.embeddings.LayerNorm.weight", "bert.embeddings.LayerNorm.bias", "bert.encoder.layer.0.attention.self.query.weight", "bert.encoder.layer.0.attention.self.query.bias", "bert.encoder.layer.0.attention.self.key.weight", "bert.encoder.layer.0.attention.self.key.bias", "bert.encoder.layer.0.attention.self.value.weight", "bert.encoder.layer.0.attention.self.value.bias", "bert.encoder.layer.0.attention.output.dense.weight", "bert.encoder.layer.0.attention.output.dense.bias", "bert.encoder.layer.0.attention.output.LayerNorm.weight", "bert.encoder.layer.0.attention.output.LayerNorm.bias", "bert.encoder.layer.0.intermediate.dense.weight", "bert.encoder.layer.0.intermediate.dense.bias", "bert.encoder.layer.0.output.dense.weight", "bert.encoder.layer.0.output.dense.bias", "bert.encoder.layer.0.output.LayerNorm.weight", "bert.encoder.layer.0.output.LayerNorm.bias", "bert.encoder.layer.1.attention.self.query.weight", "bert.encoder.layer.1.attention.self.query.bias", "bert.encoder.layer.1.attention.self.key.weight", "bert.encoder.layer.1.attention.self.key.bias", "bert.encoder.layer.1.attention.self.value.weight", "bert.encoder.layer.1.attention.self.value.bias", "bert.encoder.layer.1.attention.output.dense.weight", "bert.encoder.layer.1.attention.output.dense.bias", "bert.encoder.layer.1.attention.output.LayerNorm....

any idea about this error

Model I am using (Bert, XLNet ...):

BERT SequenceClassification

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Not able to load the Fined Tuned BERT Model from saved model directory

MODEL = BERTBaseUncased()
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    MODEL = nn.DataParallel(MODEL)
    MODEL.load_state_dict(torch.load(MODEL_PATH, map_location=device))

    MODEL.to(DEVICE)
    MODEL.eval()

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
my own task with my dataset

## To reproduce

Steps to reproduce the behavior:

1. Save the Fined Tuned BERT Sequence  Classification Model
2. Create  MODEL_PATH and config file Path from Saved Model Location
3. Run below code to test the Fined Model using Rest API to test the test data

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
import config
import torch
import flask
import time
from flask import Flask
from flask import request

#import functools
import torch.nn as nn
import joblib
from transformers import BertTokenizer,BertConfig,BertModel

app = Flask(__name__)

MODEL = None
MODEL_PATH = r'/content/model_save/pytorch_model.bin'
config_file = r'/content/model_save/config.json'
DEVICE = "cuda"
PREDICTION_DICT = dict()
memory = joblib.Memory("../input/", verbose=0)

class BERTBaseUncased(nn.Module):
    def __init__(self):
        super(BERTBaseUncased, self).__init__()
        #self.bert = transformers.BertModel.from_pretrained(config.BERT_PATH)
        #config = BertConfig.from_pretrained("bert-base-uncased", output_hidden_states=True)
        config = BertConfig(config_file)
        #self.bert = BertModel(config)
        self.bert = BertTokenizer.from_pretrained("bert-base-uncased", config=config)
        
        #self.bert  = BertTokenizer.from_pretrained(r"/content/model_save/pytorch_model.bin", do_lower_case=False,encoding="utf-8")
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768, 1)

    def forward(self, ids, mask, token_type_ids):
        _, o2 = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)
        bo = self.bert_drop(o2)
        output = self.out(bo)
        return output

def predict_from_cache(sentence):
    if sentence in PREDICTION_DICT:
        return PREDICTION_DICT[sentence]
    else:
        result = sentence_prediction(sentence)
        PREDICTION_DICT[sentence] = result
        return result


@memory.cache
def sentence_prediction(sentence):
    tokenizer = config.TOKENIZER
    max_len = config.MAX_LEN
    review = str(sentence)
    review = " ".join(review.split())

    inputs = tokenizer.encode_plus(
        review, None, add_special_tokens=True, max_length=max_len
    )

    ids = inputs["input_ids"]
    mask = inputs["attention_mask"]
    token_type_ids = inputs["token_type_ids"]

    padding_length = max_len - len(ids)
    ids = ids + ([0] * padding_length)
    mask = mask + ([0] * padding_length)
    token_type_ids = token_type_ids + ([0] * padding_length)

    ids = torch.tensor(ids, dtype=torch.long).unsqueeze(0)
    mask = torch.tensor(mask, dtype=torch.long).unsqueeze(0)
    token_type_ids = torch.tensor(token_type_ids, dtype=torch.long).unsqueeze(0)

    ids = ids.to(DEVICE, dtype=torch.long)
    token_type_ids = token_type_ids.to(DEVICE, dtype=torch.long)
    mask = mask.to(DEVICE, dtype=torch.long)

    outputs = MODEL(ids=ids, mask=mask, token_type_ids=token_type_ids)

    outputs = torch.sigmoid(outputs).cpu().detach().numpy()
    return outputs[0][0]


@app.route("/predict")
def predict():
    sentence = request.args.get("sentence")
    start_time = time.time()
    article_prediction = sentence_prediction(sentence)
    response = {}
    response["response"] = {
       
        "sentence": str(article_prediction),
        
        "time_taken": str(time.time() - start_time),
    }
    return flask.jsonify(response)


if __name__ == "__main__":
    MODEL = BERTBaseUncased()
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    MODEL = nn.DataParallel(MODEL)
    MODEL.load_state_dict(torch.load(MODEL_PATH, map_location=device))

    MODEL.to(DEVICE)
    MODEL.eval()
    app.run()
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 
- Platform: GPU 
- Python version: 3.6
- PyTorch version (GPU?):  1.5.0+cu101
- Tensorflow version (GPU?): 2.x
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?: Dont Know


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.