batch_encode_plus does not lead to the same predictions as encode_plus
Author: yhifny
Pull request: no
Labels: name: wontfix

I use batch_encode_plus to speed up the predictions but it leads to different results compared to "encode_plus"
```
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")
inputs = tokenizer.encode_plus(QA_input['question'], QA_input['context'], padding = True, add_special_tokens=True, 

```
and
```
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")
questions = [(q['question'],q['context']) for q in q_dict]
inputs = tokenizer.batch_encode_plus(questions, padding=True, add_special_tokens=True, return_tensors="pt")

```

Most of the time in the predictions are the same but sometimes they are different:


```
def predict_batch_using_model(model, model_name, q_dict):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    questions = [(q['question'],q['context']) for q in q_dict]
    inputs = tokenizer.batch_encode_plus(questions, padding=True, add_special_tokens=True, return_tensors="pt")
    logger.debug('inputs batch_encode_plus: %s\n',inputs)
    
    if torch.cuda.is_available():
        inputs.to('cuda')
    
    answer_start_scores, answer_end_scores = model(**inputs)
    
    # a list of (answer, probs_start)
    answer_probs_start_batch = []
    
    for i in range(len(q_dict)):
        input_ids = inputs["input_ids"].tolist()[i]

        text_tokens = tokenizer.convert_ids_to_tokens(input_ids)

        answer_start = torch.argmax(
            answer_start_scores[i]
        )  # Get the most likely beginning of answer with the argmax of the score
        answer_end = torch.argmax(answer_end_scores[i]) + 1  # Get the most likely end of answer with the argmax of the score
        logger.debug('answer_start, answer_end: %d %d %d\n',i, answer_start, answer_end)
        answer = tokenizer.convert_tokens_to_string(text_tokens[answer_start:answer_end])

        total_scores = answer_start_scores[i].add_(answer_end_scores[i])  # in place addition
        total_scores = total_scores.cpu().data.numpy()

        probs = _compute_softmax(total_scores)

        answer_probs_start_batch.append( ( answer, probs[answer_start]))
    
    return answer_probs_start_batch

def predict_using_model(model, model_name, QA_input):
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    inputs = tokenizer.encode_plus(QA_input['question'], QA_input['context'], padding = True, add_special_tokens=True, return_tensors="pt")
    logger.debug('inputs encode_plus: %s\n',inputs)
    
    if torch.cuda.is_available():
        inputs.to('cuda')

    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score
    logger.debug('answer_start, answer_end: %d %d %d\n',0, answer_start, answer_end)

    answer = tokenizer.convert_tokens_to_string(text_tokens[answer_start:answer_end])

    total_scores = answer_start_scores.add_(answer_end_scores)  # in place addition
    total_scores = total_scores.cpu().data.numpy()

    probs = _compute_softmax(total_scores)

    return answer, probs[answer_start]

```
the input dictionary
`
{"q": "what is color of thomas train", "gt_answer": "blue", "results": [{"system": "Google KG", "response": [], "latency": 350}, {"system": "Google CSE", "response": [{"source": "www.strasburgrailroad.com", "title": "15 Fun Facts About Thomas the Tank Engine - Strasburg Rail Road", "snippet": "Aug 15, 2017 ... Thomas' iconic blue color is also the official color of the North Western Railway. \nBefore Thomas was blue he was originally teal green with\u00a0..."}, {"source": "www.youtube.com", "title": "Learn Colors with My First Railways | Playing Around with Thomas ...", "snippet": "Oct 21, 2017 ... About Thomas & Friends: Based on a series of children's books, \"Thomas & \nFriends\" features Thomas the Tank Engine adventures with other\u00a0..."}, {"source": "en.wikipedia.org", "title": "Thomas the Tank Engine - Wikipedia", "snippet": "Thomas the Tank Engine is an anthropomorphised fictional steam locomotive in \nThe Railway ... In The Adventure Begins which is a retelling of Thomas's early \ndays on Sodor, he is a bluish-green colour when he first arrives on Sodor, his \ntanks\u00a0..."}, {"source": "play.thomasandfriends.com", "title": "Meet the Thomas & Friends Engines | Thomas & Friends", "snippet": "Discover all the engines from Sodor! Thomas & Friends fans can learn about all \ntheir favorite characters from the Thomas & Friends books, TV series and\u00a0..."}, {"source": "www.theguardian.com", "title": "Thomas the Tank Engine had to shut the hell up to save children ...", "snippet": "Jul 22, 2014 ... Thomas the Tank Engine had to shut the hell up to save children everywhere. \nThis article is more than 6 years old. Tracy Van Slyke. Classism\u00a0..."}, {"source": "www.amazon.com", "title": "RoomMates RMK1035SCS Thomas & Friends Peel ... - Amazon.com", "snippet": "RoomMates RMK1035SCS Thomas & Friends Peel and Stick Wall Decals ,Multi \ncolor. +. RoomMates RMK1831SCS Thomas The Tank Engine Peel and Stick\u00a0..."}, {"source": "ttte.fandom.com", "title": "Nia | Thomas the Tank Engine Wikia | Fandom", "snippet": "Nia is a Kenyan tank engine who befriended and accompanied Thomas on his \njourney ... Noticing how heavy his train was getting, she offered to help, but \nThomas ... of the Steam Team to have a snowplough that is not the same colour \nas her."}, {"source": "www.amazon.com", "title": "Thomas The Tank Engine Color Block Cotton Hand ... - Amazon.com", "snippet": "Buy Thomas The Tank Engine Color Block Cotton Hand Towel: Home & Kitchen - \nAmazon.com \u2713 FREE DELIVERY possible on eligible purchases."}, {"source": "ttte.fandom.com", "title": "Thomas the Tank Engine Colors", "snippet": "Thomas the Tank Engine: Colors is a book. Characters Thomas, Edward, Henry, \nJames, Percy, Bill..."}, {"source": "www.pinterest.com", "title": "Train cake, Thomas train cake, Thomas the train", "snippet": "Fondant Train Topper with Mini Train Cupcake Toppers. Each Topper is made to \norder and can be customized to suit your color scheme. Lot comes\u00a0..."}], "latency": 663}, {"system": "Bing entity", "response": [], "latency": 698}, {"system": "Bing web", "response": [{"source": "www.youtube.com", "title": "What Color Was Thomas the Tank Engine? | The Earl's Quiz ...", "snippet": "Based on a series of children's books, \"Thomas & Friends\" features Thomas the Tank Engine adventures with other locomotives on the island of Sodor. Thomas often gets into trouble, but never gives ..."}, {"source": "british-learning.com", "title": "Thomas The Train Color Pages To Print \u2013 Learning How to Read", "snippet": "Thomas and friends coloring pages 55 thomas and friends pictures to print and color. 55 thomas and friends printable coloring pages for kids. 30 free printable thomas the train coloring pages. For boys and girls kids and adults teenagers and toddlers preschoolers and older kids at school."}, {"source": "www.hometalk.com", "title": "Does anybody know what color blue is used for Thomas the ...", "snippet": "Here is a step by step YouTube guide to painting Thomas The Tank Engine and midway through, the blue used is referred to as a medium blue. Lighter than Navy, darker than Sky, maybe like a colonial blue? https://www.youtube.com/watch?v=MU8L6tIHk08"}], "latency": 879}], "dt": "2020-08-14T15:06:39.638346+00:00"}`


  we observe difference in the prediction of the tenth context. Any reason for that?


Comments:
Hi, are you sure your issue comes from the tokenizer? If you encode your text using `encode_plus` and `batch_encode_plus`, do you see a difference in the tokens generated?

I only use   encode_plus and batch_encode_plus and call model inference. I do not think the model inference is the problem as you see in the function calls. so I think it is coming from  encode_plus and batch_encode_plus. Regarding your question, I see that that batch_encode_plus add ones at the end of the list " 1,     1,     1,     1,     1,     1]".  and I thought this is this difference may be a reason for the problem.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.