How to generate prediction/answer from a custom model fined-tuned/trained for self-defined questions?
Author: ZhiliWang
Pull request: no
Labels: name: wontfix

I used run_squad.py to fine-tuned a pre-trained DistilledBERT, and I wonder how exactly to implement my model to answer a list of self-defined questions. It seems to me that the pipeline only works with one of the "regular" models (BERT DistBert, XLM, XLNET, etc.), or a model that has already been uploaded to the community. I spent a lot of time researching this but couldn't find a solution that suits the best for my case. Would anyone please explain, and if possibly, provide a demo? Here are all the files generated after my fine-tuning and evaluation:

config.json     
pytorch_model.bin   
training_args.bin 
nbest_predictions_.json 
special_tokens_map.json 
vocab.txt 
predictions_.json   
tokenizer_config.json

Thank you!

SO Link (no answer yet): https://stackoverflow.com/questions/62057333/how-exactly-to-generate-prediction-answer-from-a-custom-model-fined-tuned-traine

Comments:
Considering you saved the model(the files you mentioned above) in 'model_dir' here's how you can do it.

```python
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

tokenizer = AutoTokenizer.from_pretrained("model_dir")
model = AutoModelForQuestionAnswering.from_pretrained("model_dir")

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer.encode_plus(question, text, return_tensors="pt")

input_ids = encoding["input_ids"]
attention_mask = encoding["attention_mask"]

start_scores, end_scores = model(input_ids, attention_mask=attention_mask)
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())

answer_tokens = all_tokens[torch.argmax(start_scores) :torch.argmax(end_scores)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
```

Or better yet, use the `pipline`
```python

from transformers import pipeline

nlp = pipeline('question-answering', model='model_dir', tokenizer='model_dir')

nlp({
    'question': "Who was Jim Henson?"
    'context': "Jim Henson was a nice puppet"
})
```

> Considering you saved the model(the files you mentioned above) in 'model_dir' here's how you can do it.
> 
> ```python
> from transformers import AutoTokenizer, AutoModelForQuestionAnswering
> import torch
> 
> tokenizer = AutoTokenizer.from_pretrained("model_dir")
> model = AutoModelForQuestionAnswering.from_pretrained("model_dir")
> 
> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
> encoding = tokenizer.encode_plus(question, text, return_tensors="pt")
> 
> input_ids = encoding["input_ids"]
> attention_mask = encoding["attention_mask"]
> 
> start_scores, end_scores = model(input_ids, attention_mask=attention_mask)
> all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
> 
> answer_tokens = all_tokens[torch.argmax(start_scores) :torch.argmax(end_scores)+1]
> answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
> ```
> 
> Or better yet, use the `pipline`
> 
> ```python
> from transformers import pipeline
> 
> nlp = pipeline('question-answering', model='model_dir', tokenizer='model_dir')
> 
> nlp({
>     'question': "Who was Jim Henson?"
>     'context': "Jim Henson was a nice puppet"
> })
> ```

Hey Suraj thank you so much! My mistake was coding the pipeline incorrectly, but now it's doing fine. Thank you!

However, I have another question if you don't mind: 
When I used your method above by coding the system from scratch without using "pipeline", I would get this "index out of range" error, unless i limit my input context within about 2,103 characters (my full input text contains 57,373 characters). Nevertheless, this issue never occurred in pipeline. Do you have any idea why this happened? I think I fine-tuned a large-distilled-BERT-uncase so sequence length should not be an issue here. Is there a significant difference between the pipeline and coding the components from scratch?

Hi @ZhiliWang, the pipeline is a high level abstraction that takes care of several things so that you don't have to. Overflowing sequences is a good example, where the pipeline will automatically truncate it. You can specify `max_seq_length=n` to the pipeline if you want to manage that parameter yourself.

You can do the same with `encode_plus` by specifying the `max_length` argument.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.