Repository: gabrieldemarmiesse/python-on-whales (language: python)

Issue:
Request for improved handling of decode errors during builds.
Hi,

Recently, my team encountered an error during a `docker build` where we hit the docker build log limit, and the output was clipped; this resulted in a random non-utf-8 character being produced. The log limit non-deterministically clipped off content in the middle of a UTF-8 Character. In our logs, there was a UTF-8 Character ➤ present; this is made up of 3 bytes (see [Find all Unicode Characters from Hieroglyphs to Dingbats – Unicode Compart](https://www.compart.com/en/unicode/U+27A4) ) 0xE2 0x9E 0xA4. the 1MiB log clipping terminates the output at 0xE2 0x9E. It took us some time and troubleshooting to come to this finding. 

We are asking if there could be more graceful handling of these decoding errors implemented as we believe that the [current handling](https://github.com/gabrieldemarmiesse/python-on-whales/blob/58b52b62949bdbb3a583dbeeac045b34ca0234b5/python_on_whales/components/buildx/cli_wrapper.py#L561-L563) covered up the log limit error and produced a stack trace with the not-so-helpful `UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 10-11: invalid continuation byte`

Thank you!


Discussion / hints:
Thank you for the issue! Can you tell me how you called buildx exactly? which arguments in the function were used I mean. I'm trying to understand better the situation. Also since it happened to you, what do you think would be a good error message to display to the user in this situation? Thanks!
It might be better to use one of the other decode error handling options, e.g. ignore errors instead of raising https://docs.python.org/3/library/codecs.html#error-handlers. Or could explicitly handle cases where decode errors are 'expected' by checking it occurs at the end of the string and just trimming the end off.
We were calling `python_on_whales.docker.build` as such
```python
python_on_whales.docker.build(
                context_path=Path.cwd(),
                secrets=secrets,
                stream_logs=True,
                **docker_build_params.get_denormalized_params(...)
               )
```
**Note** get_denormalized_params() is a function we have that maps common build params for `python_on_whales.docker.build` or `docker-py.build`, which ever is being used; this is in a common use internal library we have.

We are currently working on adding `builder="larger_log"` to the function call to help with the build log limit.

But yes, something along @LewisGaul comment would do. Mainly a way that allows for the docker build error `[output clipped, log limit 1MiB reached] to be raised to the user.
Thank you for the feedback! It's much clearer. A pull request is welcomed if someone is motivated, otherwise I'll implement it when I have the time

Fix patch:
diff --git a/python_on_whales/components/buildx/cli_wrapper.py b/python_on_whales/components/buildx/cli_wrapper.py
index ea61a922..45c67381 100644
--- a/python_on_whales/components/buildx/cli_wrapper.py
+++ b/python_on_whales/components/buildx/cli_wrapper.py
@@ -560,4 +560,4 @@ def format_dict_for_buildx(options: Dict[str, str]) -> str:
 
 def stream_buildx_logs(full_cmd: list, env: Dict[str, str] = None) -> Iterator[str]:
     for origin, value in stream_stdout_and_stderr(full_cmd, env=env):
-        yield value.decode()
+        yield value.decode(errors="replace")


Test patch:
diff --git a/tests/python_on_whales/components/buildx/test_buildx_cli_wrapper.py b/tests/python_on_whales/components/buildx/test_buildx_cli_wrapper.py
index 1ee07aa5..a44d089a 100644
--- a/tests/python_on_whales/components/buildx/test_buildx_cli_wrapper.py
+++ b/tests/python_on_whales/components/buildx/test_buildx_cli_wrapper.py
@@ -45,6 +45,22 @@ def test_buildx_build_streaming_logs(tmp_path):
     assert "#6 DONE" in output[-1]
 
 
+@pytest.mark.usefixtures("with_docker_driver")
+def test_buildx_build_streaming_logs_with_decode_error_handling(tmp_path):
+    # This will simulate buildx clipping log output in the middle of a UTF-8 character
+    bad_encoding_dockerfile = (
+        dockerfile_content1
+        + """
+    RUN printf '\\xE2\\x9E'
+    """
+    )
+    (tmp_path / "Dockerfile").write_text(bad_encoding_dockerfile)
+    output = list(docker.buildx.build(tmp_path, cache=False, stream_logs=True))
+    assert output[0] == "#1 [internal] load build definition from Dockerfile\n"
+    assert len([x for x in output if "�" in x]) == 1
+    assert "#7 DONE" in output[-1]
+
+
 @pytest.mark.usefixtures("with_docker_driver")
 def test_buildx_build_load_docker_driver(tmp_path):
     (tmp_path / "Dockerfile").write_text(dockerfile_content1)