How to use fine-tuned BERT to fill <mask>
Author: Odimmsun
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I have fine-tuned a BERT model by classification task, use transformers.BertForSequenceClassification. Now,  i  want use this model to fill mask when i give a input like: 'my dog \<mask\> beautiful'. 

Can this be implemented? 
I would really appreciate if someone can teach me how to implement.


Comments:
Hi @Odimmsun , `BertForSequenceClassification` is used for classification task and not for mak-filling. You can use the pre-trained BERT model as it is for make-filling. There is pipeline for this task which you can find here https://huggingface.co/transformers/usage.html#masked-language-modeling

Basically what you'll need to do is this

```python3
from transformers import pipeline

nlp = pipeline("fill-mask")
print(nlp(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks."))
```

you can also pass your own model to the pipeline using the `model` parameter.

> Hi @Odimmsun , `BertForSequenceClassification` is used for classification task and not for mak-filling. You can use the pre-trained BERT model as it is for make-filling. There is pipeline for this task which you can find here https://huggingface.co/transformers/usage.html#masked-language-modeling
> 
> Basically what you'll need to do is this
> 
> ```python
> from transformers import pipeline
> 
> nlp = pipeline("fill-mask")
> print(nlp(f"HuggingFace is creating a {nlp.tokenizer.mask_token} that the community uses to solve NLP tasks."))
> ```
> 
> you can also pass your own model to the pipeline using the `model` parameter.

Hi @patil-suraj ,thank you for your reply. Now, if i use "nlp = pipeline("fill-mask")", the model i used is not fine-tuned, but i  want to use a fine-tuned model  to  fill mask.  How should i do to implement this?



The pre-trained BERT model does mask-filling out of the box, but if you want to use your own fine-tuned model then just pass the model(path or url) to the `model` parameter .

```python3
nlp = pipeline("fill-mask", model="your_model_path")
```

\`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-d85876ca5c10> in <module>()
      1 my_model.to('cpu')
      2 nlp = pipeline(task='fill-mask', model=my_model, tokenizer=tokenizer)
----> 3 nlp('我是你<mask>爸爸')

/usr/local/lib/python3.6/dist-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    804                 values, predictions = topk.values.numpy(), topk.indices.numpy()
    805             else:
--> 806                 masked_index = (input_ids == self.tokenizer.mask_token_id).nonzero().item()
    807                 logits = outputs[i, masked_index, :]
    808                 probs = logits.softmax(dim=0)

ValueError: only one element tensors can be converted to Python `scalars`
\`
hi, my model is fine-tuned by BertForSequenceClassification. Then i use it to fill mask, the error raised as above.  Now， i am confused.

`BertForSequenceClassification` is meant for classification, it won't work for mask-filling task. If you want to fine-tune BERT for masked lm task then you should use BertForMaskedLM

Check this example for how to fine-tune bert for masked LM

https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling

Thank @patil-suraj, do you know how to use BART with in-filling scheme (where spans of text are replaced with a single mask token)? I have not seen this pipline 



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> `---------------------------------------------------------------------------
> ValueError Traceback (most recent call last)
> in ()
> 1 my_model.to('cpu')
> 2 nlp = pipeline(task='fill-mask', model=my_model, tokenizer=tokenizer)
> ----> 3 nlp('我是你爸爸')
> 
> /usr/local/lib/python3.6/dist-packages/transformers/pipelines.py in **call**(self, *args, **kwargs)
> 804 values, predictions = topk.values.numpy(), topk.indices.numpy()
> 805 else:
> --> 806 masked_index = (input_ids == self.tokenizer.mask_token_id).nonzero().item()
> 807 logits = outputs[i, masked_index, :]
> 808 probs = logits.softmax(dim=0)
> 
> ValueError: only one element tensors can be converted to Python `scalars`
> `
> hi, my model is fine-tuned by BertForSequenceClassification. Then i use it to fill mask, the error raised as above. Now， i am confused.

老哥稳！