Generating exactly 1 word (instead of token) using autoregressive LMs
Author: joey234
Pull request: no

### Feature request

I am not sure if there exists any way to make the autoregressive type of LMs (like GPT) generate exactly 1 next word (may consists of multiple tokens) instead of just 1 token.
e.g. `  Singing is a kind of ____ -> entertain (token), entertainment (word)`

### Motivation

I came across this problem when trying to evaluate LMs performance on the next word prediction task, especially cloze style prompting (like in the LAMA dataset). AFAIK, most existing solutions just generate 1 token, which could lead to incorrect evaluation.


### Your contribution

My current workaround is to overgenerate then split on whitespace to get the first word. Then to get the scores for the generated word, I would calculate the product of all the constituted tokens. This is in no way optimal, especially when we need to get the top k word predictions as we need to increase the number of beams and return sequences. Moreover, the first few generated tokens would likely to be the same across different beams.

Comments:
cc @gante 

Hi @joey234 👋 

By default, we don't directly support that functionality. While generating, the model is unaware of where a word starts and ends -- it exclusively works at a token level. The solution you described is IMO the simplest way to do it, and the engineering effort of a working solution probably exceeds the extra computing time... unless you intend to use it for a large number of models!

I also haven't seen other requests for this feature, so I won't devote resources to building it :) However, if you're interested in building it, I'd be happy to guide you.

P.S.: you mentioned "I would calculate the product" related to scores. Be mindful that our scores are UNORMALIZED LOG-PROBABILITIES, so the probability of a word is `exp(sum of the log_softmax(scores))` or `prod(softmax(scores))` ⚠️ 

_________________________________________

Here are my two cents on how to build a solution to this problem. You have two options, A and B. If it was me working on this problem, I'd go with B (A seems painful to build). Both option can be easily extended to force generate to stop after N words.

### Option A (most compute efficient)

In essence, you must mix two pieces of knowledge: 1) knowing which tokens correspond to the end of a word; 2) building a custom stopping mechanism.

Piece 1) depends from model to model, and you should refer to our tokenizer documentation to find the needed information. Here's an intro to how tokenizers work: https://huggingface.co/course/chapter2/4?fw=pt

As for 2), you can see related examples of stopping criteria [here](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/stopping_criteria.py). In essence, you want to return `True` when a token from 1) is generated. You'd have to build a new class and pass an instance to `generate` using the `stopping_criteria` argument.

### Option B (easiest to build)

Here you would only need to build a stopping criteria class. In essence, the class would decode the text with the tokenizer at each generation step and, if the word is complete (e.g. if a space is detected), return `True`. As described in option A, you would pass an instance to `generate` using the `stopping_criteria` argument. 

Thank you so much for the guidance 🤗. I will try building based on the `stopping_criteria` argument.

@joey234 Hi, any progress?