Does Bart Model can fill <mask> with variable length?
Author: HuangZhenyang
Pull request: no

Hi, is it possible for bart in huggingface to fill <mask> with variable length?

For example, 
```python
from transformers import BartTokenizer, BartForConditionalGeneration

tokenizer = BartTokenizer.from_pretrained(config["bart_path"])
TXT = "My dog is so <mask>."
model = BartForConditionalGeneration.from_pretrained(config["bart_path"])
input_ids = tokenizer.encode_plus(TXT, return_tensors='pt')['input_ids']  # batch_encode_plus([TXT]
logits = model(input_ids, output_hidden_states=True)[0]
masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
probs = logits[0, masked_index].softmax(dim=0)
values, predictions = probs.topk(5)
print(tokenizer.decode(predictions).split())
```
This code will output `['.', 'cute', 'sweet', 'funny', 'awesome']`. Is Bart able to fill <mask> with more than one words like "cute and smart"? If so, what should I do? Is there an example?

Thank you.

Comments:
There is an example in the [docs](https://huggingface.co/transformers/model_doc/bart.html) of BART:

```
from transformers import BartForConditionalGeneration, BartTokenizer

model = BartForConditionalGeneration.from_pretrained("facebook/bart-large", force_bos_token_to_be_generated=True)
tok = BartTokenizer.from_pretrained("facebook/bart-large")

example_english_phrase = "UN Chief Says There Is No <mask> in Syria"
batch = tok(example_english_phrase, return_tensors='pt')
generated_ids = model.generate(batch['input_ids'])
assert tok.batch_decode(generated_ids, skip_special_tokens=True) == ['UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria']
```

As stated in the docs:
> The pretraining task involves randomly shuffling the order of the original sentences and a novel in-filling scheme, where spans of text are replaced with a single mask token.

In other words, it's possible as shown in the example above.

However, could you please ask such questions on the forum rather than here? We like to keep Github issues for bugs/feature requests. 

Thanks!

Just FYI, this code in the documentation doesn't work. I have latest (4.10.2) transformers.
The error is 
```
TypeError: __init__() got an unexpected keyword argument 'force_bos_token_to_be_generated'
```
And if I remove the keyword argument and run I get `['UNALSO SEE']` as result, not the one expected.

UPDATE: I see it was reported already https://github.com/huggingface/transformers/issues/12296. Please update the docs :) 


cc @patil-suraj @patrickvonplaten 

Hi @xsway , I reproduce the result that @NielsRogge shows. My code is as follows

```
from transformers import BartForConditionalGeneration, BartTokenizer
article_en = "UN Chief Says There Is No <mask> in Syria"
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large").cuda()
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")

tokenizer.src_lang = "en_XX"
encoded_en = tokenizer(article_en, return_tensors="pt")
for key, value in encoded_en.items():
    encoded_en[key] = value.cuda()
generated_tokens = model.generate(encoded_en['input_ids'], forced_bos_token_id=tokenizer.bos_token_id) # add forced_bos_token_id
result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
assert result == ['UN Chief Says There Is No Plan to Stop Chemical Weapons in Syria']
```
However, my transformers version is 4.9.2, I am not sure whether it works for the 4.10.2 transformers.
Hope this helps :)

https://github.com/huggingface/transformers/pull/14434 to fix the docs

Hi,

I realized while this approach works but the model doesn't just fill the `<mask>` span and it might also go beyond that and changes other part of the given text. For example for below code:

```
masked_text = "Police said in the first four months of the project, they laid more than 100 charges against 10 people, in connection with the illegal towing industry. “Once we started our investigation, we found that the people involved were not only breaking the law, but they were also  <mask> said Sgt. Sean Cassidy of the Toronto Police Service. “They were breaking the laws surrounding the storage of the vehicles, the fees that they were charging and the manner in which they were charging,” he added. "

max_length = 20
print(max_length)
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large").cuda()
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")

tokenizer.src_lang = "en_XX"
encoded_en = tokenizer(text, return_tensors="pt")
for key, value in encoded_en.items():
    encoded_en[key] = value.cuda()

max_length += encoded_en.input_ids.shape[1]
generated_tokens = model.generate(encoded_en['input_ids'], forced_bos_token_id=tokenizer.bos_token_id, max_length=max_length, do_sample=True, top_p=0.96, num_return_sequences=5)
result = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
```

And below is one of the generated sequences with **bold** showing where the model unnecessarily changed the text and _italic_ is the correctly filled masked span.:

"Police said in the first four months of the project, they laid more than 100 charges against 10 people, in connection with the illegal towing industry. “Once we started our investigation, we found that the people involved were not only breaking the law, but they were also _breaking a lot of other laws as well,_” said Sgt. Sean Cassidy of the Toronto Police Service.**Article Continued Below**“They were breaking the laws surrounding the storage of the vehicles, the fees that they were charging and the manner in which they are charging **for the services that they’re providing to the public,',** "


I wonder if this is what it is and BART cannot effectively be used for this use case? or should we fine-tune a separate model to do this infilling task only?

Just curious; does infilling have a function to explicitly reconnect to the suffix—i.e., the sentence continuation following the `<mask>`? Or is it just assumed that conditioning on the context before and after the `<mask>` will _predispose_ it to generate an infill that reconnects with the suffix? Of course, this would be a fair assumption, given that the original training objective is based on this kind of infilling.

From the behaviour I've observed I'm guessing it's the latter. I certainly haven't been able to find anything in the source that would force it to reconnect. (But if I missed it, maybe somebody can point me to the line?)

@fabrahman Hi, I am also interested in the issue you mentioned—can we force BART to infill texts just at the masked positions (spans)? Have you found any solutions to that? I'm looking into the codes and wonder if there is such a mechanism @jbmaxwell commented above. I will appreciate it if you can share some clues or ideas :) Thank you!

Hi @HenryCai11, unfortunately it is as I mentioned in my comment above; Bart _attempts_ to reconnect the original suffix, but there are no guarantees that it will. However, for an approach using GPT-2, see my post here:

https://discuss.huggingface.co/t/is-bart-guaranteed-to-not-mess-up-unmasked-tokens-during-text-infilling/21218/2

Thank you so much @jbmaxwell, the post really helps. I will read the paper shared in the post, and hopefully find a way to use BART in that way. Again, thanks a lot!!!