Stop output to stdout in streamers.py methods
Author: wnm3
Pull request: no
Labels: name: bug

### System Info

transformers 4.48.3 on MacOS 15.4 or Ubuntu 6.8.0-1019-ibm using Python 3.11.11

### Who can help?

@gante et al:

I'm using the AsyncTextIteratorStreamer (and others) in streamers.py to receive tokens as they become available. However, the code is printing a status message to stdout for each iteration, showing:
Loop x secs: y device=y input_ids device=z

I could see this being enabled with an argument "show_progress" (defaulting to False to disable them), but I don't see a reason to interrupt applications' screen displays using these methods. I've tried printing what I've received, but it gets garbled with these other outputs intervening.

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Call .generate() in a thread using an AsyncTextIterationStreamer and iterate through the returned values as shown in the comments of streamers.py:

```
        >>> async def main():
        ...     # Important: AsyncTextIteratorStreamer must be initialized inside a coroutine!
        ...     streamer = AsyncTextIteratorStreamer(tok)
        ...     generation_kwargs = dict(inputs, streamer=streamer, max_new_tokens=20)
        ...     thread = Thread(target=model.generate, kwargs=generation_kwargs)
        ...     thread.start()
        ...     generated_text = ""
        ...     async for new_text in streamer:
        ...         generated_text += new_text
        >>>     print(generated_text)
        >>> asyncio.run(main())
        An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,
```


### Expected behavior

Unless requested via argument to enable debugging, no output should be written to stdout by library methods.

Comments:
Hi @wnm3 👋 

Reproducing the example in the docstring of `AsyncTextIteratorStreamer`, I don't get the issue you describe.

Would you be able to give me a stand-alone reproducer for your issue?

sure, may need to wait until Monday. However, you can just look at the code for [streamers.py](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/streamers.py) in the underlying TextStreamer class on_finalized_text method -- it uses print to output the "printable" text set up in the put method:

Ideally, the on_finalized_text could check a self.show_progress bool that defaults to False in the constructor but allows people to enable the printed output if desired.

```
in the main branch: line 117 and 131
        self.on_finalized_text(printable_text)
in the main branch: line 131:
        self.on_finalized_text(printable_text, stream_end=True)
in the main branch: lines 133-135
    def on_finalized_text(self, text: str, stream_end: bool = False):
        """Prints the new text to stdout. If the stream is ending, also prints a newline."""
        print(text, flush=True, end="" if not stream_end else None)
```

In this case, it is the TextIteratorStreamer that is producing the output:
```
            streamer = TextIteratorStreamer(self.stt_tokenizer, skip_prompt=True)

            model_outputs = self.stt_llm_model.generate(
                streamer=streamer,
                inputs_embeds=combined_embeds,
                max_new_tokens=200,
                num_beams=1,  # can't use 4 with streamer (yet)
                do_sample=False,
                min_length=1,
                top_p=1.0,
                repetition_penalty=3.0,
                length_penalty=1.0,
                temperature=1.0,
                attention_mask=None,
                bos_token_id=self.stt_tokenizer.bos_token_id,
                eos_token_id=self.stt_tokenizer.eos_token_id,
                pad_token_id=self.stt_tokenizer.pad_token_id,
            )
            print("\nServer heard: ", end="")
            response_text = ""
            for text_item in streamer:
                if text_item.endswith(PRXC.VALUE_END_OF_TEXT):
                    text_item = text_item[: -len(PRXC.VALUE_END_OF_TEXT)]
                    response_text += text_item
                    print(f"{text_item}", end="")
                    break
                response_text += text_item
                print(f"{text_item}", end="")

            print()
```


Below is an example how the unwanted output is being appended to my print statements:
```
Server heard: Loop 0 secs: 0.13939118385314941 device=cuda:7 input_ids device=cuda:7
please Loop 1 secs: 0.05998086929321289 device=cuda:7 input_ids device=cuda:7
end Loop 2 secs: 0.058597564697265625 device=cuda:7 input_ids device=cuda:7
this Loop 3 secs: 0.05765938758850098 device=cuda:7 input_ids device=cuda:7
conversation Loop 4 secs: 0.05775117874145508 device=cuda:7 input_ids device=cuda:7
Loop 5 secs: 0.058953285217285156 device=cuda:7 input_ids device=cuda:7
Loop total secs: 0.43303608894348145
```
It should have read:
Server heard: please end this conversation
e.g., each of the Loop... were from the underlying library print statements


Perhaps this doesn't happen in AsyncTextIteratorStreamer but only in the TextIteratorStreamer


@gante do you agree the print statements should be conditional to not interrupt the applications stdout output?

@wnm3 in theory yes. In practice, `TextStreamer` classes are a band-aid for streaming. Our code needs a refactor to allow the streaming of tokens (as opposed to printing text), so I'll not actively work to fix the issue. 

Happy to merge a PR with the fix if your are willing to work on it, though :)

I can not create a PR nor branch so I'm including proposed changes here. I have v 4.48.3 on a linux server where I observe the problem, and 4.49.0 on a macos MacBook. For the linux 4.48.3 version, the transformers/generation/utils.py also needed to be updated with logic like what follows though the print statements appear to have been removed from 4.49.0:

utils.py: _sample method lines 3318-3322:

```
            if streamer is not None and hasattr(streamer,"should_print"):
                if streamer.should_print():
                    print(f"Loop {cur_len} secs: {time.time()-then} device={self.device} input_ids device={input_ids.device}")
            else:
                print(f"Loop {cur_len} secs: {time.time()-then} device={self.device} input_ids device={input_ids.device}")
```

utils.py: _sample method lines 3329-3333:

```
        if streamers is not None and hasattr(streamer,"should_print"):
            if streamer.should_print():
                print(f"Loop total secs: {time.time()-then_1}")
        else:
            print(f"Loop total secs: {time.time()-then_1}")
```

For streamers.py I added print_text as an optional bool parameter defaulting to True to preserve current function, and I added a should_print method that is called in the on_finalized_text method. This allows me to instantiate as follows:

```
            streamer = AsyncTextIteratorStreamer(
                stt_tokenizer, skip_prompt=True, print_text=False
            )
```

Below is the diff output for 

```
diff --git a/src/transformers/generation/streamers.py b/src/transformers/generation/streamers.py
index c0fc26442b..2c7cc16113 100644
--- a/src/transformers/generation/streamers.py
+++ b/src/transformers/generation/streamers.py
@@ -53,6 +53,8 @@ class TextStreamer(BaseStreamer):
             The tokenized used to decode the tokens.
         skip_prompt (`bool`, *optional*, defaults to `False`):
             Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots.
+        print_text (`bool`, *optional*, defaults to `True`):
+            Whether text is printed to stdout during streaming
         decode_kwargs (`dict`, *optional*):
             Additional keyword arguments to pass to the tokenizer's `decode` method.
 
@@ -72,10 +74,11 @@ class TextStreamer(BaseStreamer):
         ```
     """
 
-    def __init__(self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, **decode_kwargs):
+    def __init__(self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, print_text: bool = True, **decode_kwargs):
         self.tokenizer = tokenizer
         self.skip_prompt = skip_prompt
         self.decode_kwargs = decode_kwargs
+        self.print_text = print_text
 
         # variables used in the streaming process
         self.token_cache = []
@@ -132,7 +135,8 @@ class TextStreamer(BaseStreamer):
 
     def on_finalized_text(self, text: str, stream_end: bool = False):
         """Prints the new text to stdout. If the stream is ending, also prints a newline."""
-        print(text, flush=True, end="" if not stream_end else None)
+        if self.should_print():
+            print(text, flush=True, end="" if not stream_end else None)
 
     def _is_chinese_char(self, cp):
         """Checks whether CP is the codepoint of a CJK character."""
@@ -179,6 +183,8 @@ class TextIteratorStreamer(TextStreamer):
         timeout (`float`, *optional*):
             The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions
             in `.generate()`, when it is called in a separate thread.
+        print_text (`bool`, *optional*, defaults to `True`):
+            Whether text is printed to stdout during streaming
         decode_kwargs (`dict`, *optional*):
             Additional keyword arguments to pass to the tokenizer's `decode` method.
 
@@ -206,9 +212,9 @@ class TextIteratorStreamer(TextStreamer):
     """
 
     def __init__(
-        self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, timeout: Optional[float] = None, **decode_kwargs
+        self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, timeout: Optional[float] = None, print_text: Optional[bool] = True, **decode_kwargs
     ):
-        super().__init__(tokenizer, skip_prompt, **decode_kwargs)
+        super().__init__(tokenizer, skip_prompt, print_text, **decode_kwargs)
         self.text_queue = Queue()
         self.stop_signal = None
         self.timeout = timeout
@@ -284,14 +290,15 @@ class AsyncTextIteratorStreamer(TextStreamer):
     """
 
     def __init__(
-        self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, timeout: Optional[float] = None, **decode_kwargs
+        self, tokenizer: "AutoTokenizer", skip_prompt: bool = False, timeout: Optional[float] = None, print_text: Optional[bool] = True, **decode_kwargs
     ):
-        super().__init__(tokenizer, skip_prompt, **decode_kwargs)
+        super().__init__(tokenizer, skip_prompt, print_text, **decode_kwargs)
         self.text_queue = asyncio.Queue()
         self.stop_signal = None
         self.timeout = timeout
         self.loop = asyncio.get_running_loop()
         self.has_asyncio_timeout = hasattr(asyncio, "timeout")
+        self.print_text = print_text
 
     def on_finalized_text(self, text: str, stream_end: bool = False):
         """Put the new text in the queue. If the stream is ending, also put a stop signal in the queue."""
@@ -299,6 +306,9 @@ class AsyncTextIteratorStreamer(TextStreamer):
         if stream_end:
             self.loop.call_soon_threadsafe(self.text_queue.put_nowait, self.stop_signal)
 
+    def should_print(self):
+        return self.print_text
+    
     def __aiter__(self):
         return self
 
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.