Repository: jcrist/msgspec (language: python)

Issue:
Encode_into might not write data at the intended offset location if buffer is too small.
### Description
When encoding into a buffer at an offset that is larger than the buffer size, the the msgpack `Encoder.encode_into()` function will start writing at `offset = -1 `instead.

Example:
```python
from msgspec.msgpack import Encoder

enc = Encoder()
buffer = bytearray(2) # bytearray of size 2

enc.encode_into("some serializable object", buffer, 4) # offset of size 4

print(buffer) # |> bytearray(b'\x00\x00\xb8some serializable object')
```

Notice it started at offset 2 instead of offset 4.

### Expected behaviour:
`encode_into` starts writing at the requested offset, no matter how large the buffer currently is.

### Note
Just mentioning this in the docs as intended behaviour is fine for me as longs as it's documented somewhere.



Discussion / hints:
This is the currently intended (or at least not unintended) behavior. In my mind there are 3 main use cases for `encode_into`, none of which would run into this issue:

1. Encoding into a pre-allocated buffer. In this case `offset` is never needed.

```python
buffer = bytearray(1024)
for msg in msgs:
    enc.encode_into(msg, buffer)
    socket.sendall(msg)
```

2. Encoding into a pre-allocated buffer, leaving space for a prefix. This is the [length-prefix framing example](https://jcristharif.com/msgspec/perf-tips.html#length-prefix-framing) in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.

```python
buffer = bytearray(64)
encoder.encode_into(msg, buffer, 4)  # offset for prefix, but 4 is less than 64 and both are hardcoded sizes (not data dependent)
n = len(buffer) - 4
buffer[:4] = n.to_bytes(4, "big")
socket.sendall(buffer)

3. Accumulating several writes into the same buffer, perhaps with a delimiter. In this case you're always writing to the end of the buffer, so offset is -1

```python
buffer = bytearray()
for msg in msgs:
    encoder.encode_into(msg, buffer, -1)
    buffer.append("\n")  # perhaps we're writing line-delimited json
```

---

That all said, I can see how the current behavior would be surprising, and would be open to changing it to automatically grow the buffer to support the specified `offset`.

Out of curiousity, what was your use case that led to finding this issue?
The use case was actually case 2.

> Encoding into a pre-allocated buffer, leaving space for a prefix. This is the [length-prefix framing example](https://jcristharif.com/msgspec/perf-tips.html#length-prefix-framing) in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.

I just forgot to add the size on one of my buffers. Python has no problem doing `x = bytearray()` without size, and `decode_into` has no issue writing to this buffer. Just the right amount of coincidences happend and my encoded message could still be decoded (at first).  It's only when I started to expand the tests that i suddenly got weird results, and messages that couldn't be decoded.

Since both just work, it was quite hard to pinpoint where my message got "corrupted". A mention about this in the docs would've saved me some time here.

For context i'm writing an ASGI RPC app (and client) using msgspec for serialisation.


Hmmm, in that case maybe we should error instead if `offset > len(buffer)`? That would have caught your issue and pointed you to a better usage pattern. On the other hand, making it work by expanding the buffer is also easy, and would have negligible performance impact (at least for reasonable offsets) since we'd have to grow the buffer anyway to write the message. Waffling a bit here on what the best UX is.
I can see benefits for both options. Expanding the buffer will "Just work" and I can't really see a use case where you might not want that to happen, especially since `encode_into` expands anyway when necessary. 

On the other hand going by the zen of python:

> Explicit is better than implicit.

and

> Special cases aren't special enough to break the rules

If anyone ever has a use case where they don't want this expanding to happen, it might be hard to figure out what is going on.

Fix patch:
diff --git a/msgspec/_core.c b/msgspec/_core.c
index 55f79aba..d2087b85 100644
--- a/msgspec/_core.c
+++ b/msgspec/_core.c
@@ -9401,8 +9401,10 @@ encoder_encode_into_common(
             PyErr_SetString(PyExc_ValueError, "offset must be >= -1");
             return NULL;
         }
-        if (offset > buf_size) {
-            offset = buf_size;
+
+        if (offset < buf_size) {
+            buf_size = Py_MAX(8, 1.5 * offset);
+            if (PyByteArray_Resize(buf, buf_size) < 0) return NULL;
         }
     }
 
@@ -9419,9 +9421,11 @@ encoder_encode_into_common(
         .max_output_len = buf_size,
         .resize_buffer = ms_resize_bytearray
     };
+
     if (encode(&state, obj) < 0) {
         return NULL;
     }
+
     FAST_BYTEARRAY_SHRINK(buf, state.output_len);
     Py_RETURN_NONE;
 }


Test patch:
diff --git a/tests/test_json.py b/tests/test_json.py
index d3a88f24..7c0af6fc 100644
--- a/tests/test_json.py
+++ b/tests/test_json.py
@@ -298,10 +298,11 @@ def test_encode_into_offset(self):
         enc.encode_into(msg, buf, 2)
         assert buf == b"01" + encoded
 
-        # Offset out of bounds appends to end
+        # Offset out of bounds extends
         buf = bytearray(b"01234")
-        enc.encode_into(msg, buf, 1000)
-        assert buf == b"01234" + encoded
+        enc.encode_into(msg, buf, 10)
+        assert buf[:5] == b"01234"
+        assert buf[10:] == encoded
 
         # Offset -1 means append at end
         buf = bytearray(b"01234")
diff --git a/tests/test_msgpack.py b/tests/test_msgpack.py
index f4467cb5..55db40f5 100644
--- a/tests/test_msgpack.py
+++ b/tests/test_msgpack.py
@@ -432,10 +432,11 @@ def test_encode_into_offset(self):
         enc.encode_into(msg, buf, 2)
         assert buf == b"01" + encoded
 
-        # Offset out of bounds appends to end
+        # Offset out of bounds extends
         buf = bytearray(b"01234")
-        enc.encode_into(msg, buf, 1000)
-        assert buf == b"01234" + encoded
+        enc.encode_into(msg, buf, 10)
+        assert buf[:5] == b"01234"
+        assert buf[10:] == encoded
 
         # Offset -1 means append at end
         buf = bytearray(b"01234")