Add tokenizer kwargs to fill mask pipeline.
Author: nmcahill
Pull request: yes


This pr addresses #25994 by adding tokenizer_kwargs as an input preprocessing parameter to the fill mask pipeline. 

Attn: @BramVanroy @Narsil.



Comments:
Thanks. Okay I will look into that. First time contributing to HF. 

Try to install the styling package with `pip install -U "transformers[quality]"` 😉 

Anything else you can think of? I can't seem to get the setup and quality checks to pass...

Hey @nmcahill, the recommended tool to run here is `make fixup` which takes care of everything under the hood and does so quite fast.

I've taken the liberty to add code wrappers around your example and run `make fixup` on your PR directly so that we may merge this PR and include it in today's release. Thank you for your contribution!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26234). All of your documentation changes will be reflected on that endpoint.

Thanks @LysandreJik! 

The test `tests/pipelines/test_pipelines_fill_mask.py::FillMaskPipelineTests::test_large_model_pt` started to fail since this PR being merged. It is the new test block added in this PR

https://github.com/huggingface/transformers/blob/3e68944cc482e915b27b24eec1f87e40be522ec0/tests/pipelines/test_pipelines_fill_mask.py#L219-L229

The results we get is shown at the end. 

@nmcahill  Could you check if the `tokenizer_kwargs={"truncation": True},` does its job here? 

Thank you in advance.


```
(Pdb) nested_simplify(outputs, decimals=6),
([{'score': 0.281868, 'token': 6, 'token_str': ',', 'sequence': 'My name is,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipisc
ing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lo
rem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum d
olor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit am
et, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consect
etur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipis
cing elit,Lorem'}, {'score': 0.095431, 'token': 46686, 'token_str': ':,', 'sequence': 'My name is:,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit am
et, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consect
etur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipis
cing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,L
orem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum 
dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem ipsum dolor sit amet, consectetur adipiscing elit,Lorem'}],)

```

FYI: both `test_large_model_pt` and `test_large_model_tf` are failing

What's strange is that I expect to see: `"My name is <mask>Lorem ipsum dolor sit amet,..."` which would be the expected output of  ` "My name is <mask>" + "Lorem ipsum dolor sit amet, consectetur adipiscing elit," * 100, ` but instead i see "My name is,Lorem ipsum dolor sit amet,". 

I know these tests passed when my pr was approved. I will need to get some time at night to dig into this... 

It might simply be that the terminal output is not showing the word `<mask>` though. 

Other than that oddity, though, the fact that it is returning scores at all instead of failing with long input text is the true point of this unit test... I am not sure why the tokens and scores have changed since I tested this locally, but I'm tempted to change the unit test to check for results at all rather than checking for a particular set of tokens/scores. Would that work for everyone?

> I am not sure why the tokens and scores have changed

Hi @nmcahill Thank you for the response 🤗 

This might be the hardware difference, we use GPU T4. As long as `sequence` is the correct one (i.e. being truncated here), we can adjust the values in other fields.

My main concern here is that it looks we pass `tokenizer_kwargs={"truncation": True}` but it doesn't seem have the effect in this test.

Take your time on this, but if you could not allocate some time in the following weeks, let me know 🙏 

So the behavior if truncation is set to False and the input string is very
long would be that the model.forward will throw an error. The truncation
happens between the tokenizer and and the model so the bit that is actually
truncated is the vector called “input_ids” in model inputs, the input
string never gets truncated so no need to check that visually.

To prove it to yourself that the truncation=True works, try setting it to
false and seeing if the model.forward fails.

If it doesn’t fail with Truncation=False then I’ll definitely try fixing
it. But as far as I can tell, I think this is probably working as expected.



On Wed, Dec 6, 2023 at 1:53 AM Yih-Dar ***@***.***> wrote:

> I am not sure why the tokens and scores have changed
>
> Hi @nmcahill <https://github.com/nmcahill> Thank you for the response 🤗
>
> This might be the hardware difference, we use GPU T4. As long as sequence
> is the correct one (i.e. being truncated here), we can adjust the values in
> other fields.
>
> My main concern here is that it looks we pass tokenizer_kwargs={"truncation":
> True} but it doesn't seem have the effect in this test.
>
> Take your time on this, but if you could not allocate some time in the
> following weeks, let me know 🙏
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/pull/26234#issuecomment-1842451624>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AGI5ZZEMOKMT5EWHDKEVILTYIAXBLAVCNFSM6AAAAAA45GOORWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBSGQ2TCNRSGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


OK, got it. I was confused as I saw you added the expected output as `"sequence": "My name is grouped"` which led me to think the `sequence` is truncated. But this is not the case as you mentioned above.