Customize widget text-generation inference with prepended input
Author: borisdayma
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Some models for text generation make use of special tokens or additional initialization text.
It would be useful to add the possibility to pass additional text to the input string (before or after) and also let people strip input text.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

It is not clear how the widget API generates text.
For example with my little demo huggingtweets, I see 2 limitations:

1. I always strip input with `text.strip` and add a white space at the beginning. Otherwise we get wrong tokenization of `"Paris"` vs `" Paris"`
2. With the way my model is trained, I always want to add `
` at the beginning (training adds it between every tweet)

Input such as `"     this is my input   "` would become, after stripping, adding white space and special token `"
 this is my input"` and be then passed to tokenizer.

I can see other cases where people add their own special tokens for certain tasks.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I don't think there is any public access to the API so cannot contribute there.

Comments:
There is some documentation here https://huggingface.co/docs but we will add more later this week.

I don't think we want to mask (pun intended) too much in the inference widget what the model actually consumes, but did you see that:
- you can specify your model's widget's example input, by adding metadata to your model card, cf. https://huggingface.co/docs#how-can-i-control-my-models-widgets-example-inputs
- you can define a prefix in your config.json's `task_specific_params`, see T5 for instance: https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json (cc @patrickvonplaten)

> * you can define a prefix in your config.json's `task_specific_params`, see T5 for instance: https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json (cc @patrickvonplaten)

I had not noticed it! This looks comprehensive and exactly what is needed!

> * you can define a prefix in your config.json's `task_specific_params`, see T5 for instance: https://s3.amazonaws.com/models.huggingface.co/bert/t5-base-config.json (cc @patrickvonplaten)

Just a quick note that the prefix here is not for text generation pipeline but I used the same idea in PR #5885 with existing "padding_text" variable that I turned into a task specific configuration parameter.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This was implemented