Repository: keras-team/keras

Issue:
UnicodeEncodeError when running keras.model.predict
The following problem occurs in keras version 3.1.1 (tensorflow version 2.16.1)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 19-38: character maps to <undefined>

prediction = trained_nn_model.predict(last_sequence)
A bit more details:
  File "...\lib\site-packages\keras\src\utils\traceback_utils.py", line 122, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "...\.pyenv\pyenv-win\versions\3.10.5\lib\encodings\cp1252.py", line 20, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 19-38: character maps to <undefined>

(The positions varies when tried with different test data).
The code works ok with keras version 2.15.0 (tensorflow 2.15.0)


Discussion:
Hi @geniusonanotherlevel ,

The issue seems to be with preprocessing steps. With the given info I am not sure to say whether this is Keras issue or not. 
It seems the error is related to encoding and may be adding `encoding="utf-8" ` might helps. If it not resolved with this 
 please provide more details along with a repro code snippet.

Thanks!
I’m seeing this in Windows 11, but not in Windows 10, or Mac.
It was working with Keras <3.0, but after updating to 3.1.1 it breaks for Windows 11 runs executed by Snakemake in a git bash conda environment.

If I copy/paste the same command Snakemake ran into the git bash conda environment and run it: it succeeds.

I have set the `KERAS_BACKEND` environment variable to `tensorflow`, but I suspect there are other environment shenanigans at play I am unaware of.

I get the same `19-38: character maps to <undefined>` error when running `model.fit` as the OP (exact position match).
Hi @SuryanarayanaY , thanks for the response. It can be noted that I did try to add encoding of utf-8 with the numpy array, but that wasn't an acceptable option as it only accepted ascii, latin & something else which I can't remember. But the fact that it works with an earlier version of tensorflow/keras while all the pre-processing steps and versions of other packages are exactly the same, is why I suspected keras. I've also noted your request for repro code, I will try and get that next week.

@ryand626 , I'm running on Windows 10.
Hi @geniusonanotherlevel ,

I will be waiting for your repro code. Thanks!
Hi @geniusonanotherlevel ,

Could you please provide repro for same. Thanks!
> Hi @geniusonanotherlevel ,
> 
> Could you please provide repro for same. Thanks!

Hi, thanks for the reminder. I don't have the code with me atm, so I will do when I go back to the office (I'm just not sure which day I'll be there yet... but soon) Thanks for your patience.
This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.
Same issue here on FastAPI. However, my `model.fit` will work as expected when running locally, just throw the `character maps to <undefined>` when on the FastAPI function called by client.
Got the same Problem with version keras 3.3.1 and tf  2.16.1 on windows 10. Unfortunately the stack trace of this error does not show the origin but with keras.config.disable_traceback_filtering() you can print out the whole stacktrace:

```python
 File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\backend\tensorflow\trainer.py", line 316, in fit
    callbacks.on_train_batch_end(step, logs)
  File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\callbacks\callback_list.py", line 106, in on_train_batch_end
    callback.on_train_batch_end(batch, logs=logs)
  File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\callbacks\progbar_logger.py", line 58, in on_train_batch_end
    self._update_progbar(batch, logs)
  File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\callbacks\progbar_logger.py", line 95, in _update_progbar
    self.progbar.update(self.seen, list(logs.items()), finalize=False)
  File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\utils\progbar.py", line 182, in update
    io_utils.print_msg(message, line_break=False)
  File "C:\Users\heisterkamp\Desktop\Git\tfx-pipeline\.venv\Lib\site-packages\keras\src\utils\io_utils.py", line 98, in print_msg
    sys.stdout.write(message)
  File "C:\Users\heisterkamp\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
```

The progress bar seems to be the problem here, maybe using the wrong character? but the problem itself seems to be on Keras side. I hope this will be fixed in the future, since the status printouts are helpful for quick testing. 

But for now verbose=0 has to do.
I had/have the same issue when using "shiny for Python".
Adding `verbose=0` to `model.predict` fixes the error.
> I had/have the same issue when using "shiny for Python". Adding `verbose=0` to `model.predict` fixes the error.

Thank you very much. My problem was fixed by adding `verbose=2` to both `model.fit` and `model.predict` functions.
Thanks. I had the same problem and it was solved in a similar way, after several hours of searching for the cause of the error



Patch:
diff --git a/keras/src/utils/io_utils.py b/keras/src/utils/io_utils.py
index 32322f405c33..f087ab6dd21a 100644
--- a/keras/src/utils/io_utils.py
+++ b/keras/src/utils/io_utils.py
@@ -91,10 +91,18 @@ def set_logging_verbosity(level):
 
 def print_msg(message, line_break=True):
     """Print the message to absl logging or stdout."""
+    message = str(message)
     if is_interactive_logging_enabled():
-        if line_break:
-            sys.stdout.write(message + "\n")
-        else:
+        message = message + "\n" if line_break else message
+        try:
+            sys.stdout.write(message)
+        except UnicodeEncodeError:
+            # If the encoding differs from UTF-8, `sys.stdout.write` may fail.
+            # To address this, replace special unicode characters in the
+            # message, and then encode and decode using the target encoding.
+            message = _replace_special_unicode_character(message)
+            message_bytes = message.encode(sys.stdout.encoding, errors="ignore")
+            message = message_bytes.decode(sys.stdout.encoding)
             sys.stdout.write(message)
         sys.stdout.flush()
     else:
@@ -123,3 +131,8 @@ def ask_to_proceed_with_overwrite(filepath):
         return False
     print_msg("[TIP] Next time specify overwrite=True!")
     return True
+
+
+def _replace_special_unicode_character(message):
+    message = str(message).replace("━", "=")  # Fall back to Keras2 behavior.
+    return message


Test patch:
diff --git a/keras/src/utils/io_utils_test.py b/keras/src/utils/io_utils_test.py
index 235314de3016..2fe1fbbea219 100644
--- a/keras/src/utils/io_utils_test.py
+++ b/keras/src/utils/io_utils_test.py
@@ -1,3 +1,5 @@
+import sys
+import tempfile
 from unittest.mock import patch
 
 from keras.src.testing import test_case
@@ -55,3 +57,13 @@ def test_ask_to_proceed_with_overwrite_invalid_then_yes(self, _):
     @patch("builtins.input", side_effect=["invalid", "n"])
     def test_ask_to_proceed_with_overwrite_invalid_then_no(self, _):
         self.assertFalse(io_utils.ask_to_proceed_with_overwrite("test_path"))
+
+    def test_print_msg_with_different_encoding(self):
+        # https://github.com/keras-team/keras/issues/19386
+        io_utils.enable_interactive_logging()
+        self.assertTrue(io_utils.is_interactive_logging_enabled())
+        ori_stdout = sys.stdout
+        with tempfile.TemporaryFile(mode="w", encoding="cp1251") as tmp:
+            sys.stdout = tmp
+            io_utils.print_msg("━")
+        sys.stdout = ori_stdout