Streamable way in model.generate
Author: lucasjinreal
Pull request: no
Labels: name: WIP

### Feature request

As the llm becomes bigger and bigger, a quite slow speed in generate caused long time waiting for whole response.

would it be OK add a `model.stream_generate` or something like for users to use?

Actually some community guys implementated such a thing, but this could be a problem too: users don't which is reliable to use, it could be better for official introduce such as way.

I noticed there are some work form hf like text_generation, but what users might want is simple a tiny API along side model.geneate rather than a whole lib

### Motivation

For instance huge models inference.

### Your contribution

not yet

Comments:
cc @gante 

"streamer (BaseStreamer, optional) — Streamer object that will be used to stream the generated sequences. Generated tokens are passed through streamer.put(token_ids) and the streamer is responsible for any further processing. kwargs — Ad hoc parametrization of generate_config and/or additional model-specific kwargs that will be forwarded to the forward function of the model. If the model is an encoder-decoder model, encoder specific kwargs should not be prefixed and decoder specific kwargs should be prefixed with decoder_."
 https://huggingface.co/docs/transformers/v4.29.1/en/main_classes/text_generation#:~:text=be%20much%20smaller.-,streamer%20(BaseStreamer%2C%20optional)%20%E2%80%94%20Streamer%20object%20that%20will%20be%20used,prefixed%20and%20decoder%20specific%20kwargs%20should%20be%20prefixed%20with%20decoder_.,-Returns


There is already an argument in the generate function for that

https://github.com/flozi00/atra/blob/375bd740c37fb42d35048ae33ae414841f22938a/atra/text_utils/chat.py#LL98C7-L98C7

There is an working implementation with gradio for example

That's indeed already supported: https://huggingface.co/docs/transformers/main/en/generation_strategies#streaming

Thank u all.

From the official doc, seems it was printting to console by default:

![image](https://github.com/huggingface/transformers/assets/21303438/93a020fb-f6aa-4cb3-a228-cef02b3f2d08)

How can I yeild the decoded text one by one?

From the link Flozi post above, I notice they using a thread to start generate,

So in my case, what is the standared way to got it work as I needed?

Hey @lucasjinreal 👋 

As written above, atm there are two options (see [their docstrings](https://huggingface.co/docs/transformers/main/en/internal/generation_utils#transformers.TextStreamer) for examples):
1. Stream the text directly into the console
2. Spin up a parallel thread and use it to stream the text strings

This is quite inconvenient (threads 🤮) and feature-limited (streaming tokens would be nice!). The next step in the plan is to remove the threading part -- have a look at [this comment](https://github.com/huggingface/transformers/issues/23640#issuecomment-1585762715)

@gante hello, I made it by using thread, when this was removed, please pin me if you can ^.-

me too

@gante Holy moly, one hundres years later, I still need thread?

I am not famillar about transformers source code, otherwise I really want do a PR, please make it effiency, the world changes very fast!

@lucasjinreal, that is not a productive way of engaging with a maintainer of any open-source library. It will only get your requests deprioritized.

There are competing priorities within a library and few hands to complete them -- if you _need_ a feature within a certain timeframe, feel free to build it yourself. After all, the code is available for you to expand on.

@gante  I have already built myself, but we need a built-in way or a more elegant way, is there anything wrong with it? This request are at least 6 months ago, is this so low priority to add it? If so, just ignore it, I will close it. leave it un done forever.  BTW. @sgugger