`return_assistant_tokens_mask` argument is blocked in `ProcessorMixin.apply_chat_template`
Author: MilkClouds
Pull request: no
Labels: name: Chat Template

name: Processing

# Issue
After https://github.com/huggingface/transformers/pull/36149, for **all** classes inheriting ProcessorMixin, `return_assistant_tokens_mask=True` argument for `apply_chat_template` is blocked.

# How to recall

in transformers v4.49.0, run script at https://huggingface.co/HuggingFaceM4/idefics2-8b-chatty/discussions/9. It raises error.
```
# script output
Traceback (most recent call last):
  File "/mnt/home/claude/GitHub/open-world-agents/test.py", line 82, in <module>
    inputs = processor.apply_chat_template(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/processing_utils.py", line 1361, in apply_chat_template
    prompt = self.tokenizer.apply_chat_template(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/tokenization_utils_base.py", line 1624, in apply_chat_template
    raise ValueError("`return_assistant_tokens_mask=True` is incompatible with `return_dict=False`")
ValueError: `return_assistant_tokens_mask=True` is incompatible with `return_dict=False`
```

in transformers v4.48.0, which is before #36149, it works well.
```
# script output
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]
```

# Reason I've figured out

#36149 hard-code `tokenize=False` and `return_dict=False` when calling `self.tokenizer.apply_chat_template`.
```py
        prompt = self.tokenizer.apply_chat_template(
            conversations,
            chat_template=chat_template,
            tokenize=False,
            return_dict=False,
            **tokenizer_template_kwargs,
        )
```

But `return_assistant_tokens_mask` need `tokenize` to be True and `return_dict` to be True. So, `return_assistant_tokens_mask` option is impossible to enable after #36149.



Comments:
Good catch! The processor cannot work with tokenized ids though because the multimodal tokens usually add special image placeholders from within processor's call. And unfortunately we don;t support already tokenized text at input

I see a workaround here to allow `tokenizer` always return rendered prompt when some certain flag is on (`return_rendered_prompt?`) or we could do `decode(input_ids)`. But decoding tokens might lead to weird unexpected behavior when users set `padding/truncation` etc. @Rocketknight1  will you agree with passing a new flag or do you have any other preferences?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@zucchini-nlp I'm sorry, I missed this one until the actions bot flagged it! It seems quite difficult, though - we probably want change this inside the `processor` logic so that if `return_assistant_tokens_mask` is set then we:

1) Render the template in the tokenizer and create `input_ids` and `assistant_masks`
2) Expand image tokens in the processor
3) Add some logic to figure out which "run" of image tokens corresponds to single image tokens in the unexpanded input, then apply the `assistant_mask` value from that single image token to the entire "run"

Does that make sense? I feel like the goal is for `return_assistant_tokens_mask` on Processors to work transparently for users without any extra flags, but I'm okay if we need to add some internal logic or flag-passing.

@Rocketknight1 IIUC you mean the processor will accept pre-tokenized token ids and expand for images. IMO this won't not clean and easy to do, since some models have a huge code block which infers how to expand inputs. Support for token ids on top of that, might add more complexity thus higher maintenance cost

The only issue here is that `return_assistant_tokens_mask` cannot be handled after rendering, and requires special `render_for_assistant_mask` to be called. So we need a way to render correct jinja without tokenizing

Another way (without flags) can be to split out all jinja-related logic into `chat_template_utils.py` so it can be called by processors without relying on the tokenizer. This can clean up a bit tokenizer's `apply_chat_template` if more features and special flag handling is going to be added. Whichever way we go, works fine by me, lmk what you think

I think splitting out the Jinja logic into `chat_template_utils` to make it independent of the Tokenizer/Processor is good! Let me know if you want me to take any part of that.

Great! I can work on that later and will tag you when a draft is ready :)