Repository: ggml-org/llama.cpp
Pull request: ggml-rpc: chunk send()/recv() to avoid EINVAL for very large tensors over RPC (macOS & others)

Fixes #15055

This PR prevents send()/recv() from being called with extremely large buffers during RPC tensor transfer by chunking I/O into 1 GiB pieces. On macOS this avoids intermittent EINVAL errors that previously caused the client to abort when offloading very large models via RPC.

What’s the symptom?
Loading very large GGUFs via RPC would fail with:

client: send: Invalid argument

server: recv: Invalid argument

Repro seen with DeepSeek-R1-0528-* and Qwen3-480B-* at large quants.

Single-node load (no RPC) worked fine.

Root cause (observed)
OS-level limits on single send()/recv() buffer sizes; very large tensors were transmitted in one shot. Splitting into smaller chunks resolves the issue.

Changes
Add RPC_IO_CHUNK = 1 GiB.

Update send_data() and recv_data() to loop with chunked I/O.

Keep existing error logging; behavior is otherwise unchanged.

Why 1 GiB?
Empirically under the limits that triggered EINVAL on macOS.

Large enough to keep throughput good; easy to tune later if needed.

Testing
macOS (Metal): Previously failing large-model RPC offload now completes. Inference runs.

macOS (non-Metal): Build + basic RPC transfer OK.

Linux/Ubuntu: Not tested yet. Relying on CI and maintainer validation. (Happy to test on request; I can also try Docker later.)

Known quirk (non-blocking)
I still see an occasional non-fatal recv: Invalid argument before the big tensor transfer starts, but the run proceeds and finishes. I suspect a minor size-field mismatch during early handshake. If useful, I can follow up with a tiny patch that always serializes message lengths as uint64_t on the wire.

Performance / compatibility
No API changes.

Chunking is per-call looped send/recv; negligible overhead in my tests vs. “one big send”.

Should be safe across platforms.

Thanks!

---

Update: Verified cross-OS direction as well.

**Additional testing**
- Client: Ubuntu 22.04 (glibc), clang/gcc build, commit 0e7aa4e
- RPC server: macOS (Apple M3 Ultra, 512 GB RAM, Metal enabled)
- llama.cpp build: Release
- Model: DeepSeek-R1-0528-Q4_K_M (GGUF format, very large tensor size)
- Command (client):
  ./build/bin/llama-server \
    -m /path/to/DeepSeek-R1-0528-Q4_K_M.gguf \
    --rpc <mac-ip>:50052 -c 3000
- Command (server):
  ./build/bin/rpc-server -p 50052 --host <mac-ip>

**Result**
- Large tensor offload succeeds end-to-end. Inference runs normally.
- No client/server aborts observed.
- Occasionally still see a *non-fatal* `recv: Invalid argument` before the first large transfer; run proceeds normally.

**Notes**
- The chunked I/O change fixes the main crash.  
- If you prefer, I can follow up with a tiny patch to always serialize message length fields as `uint64_t` to silence the early handshake warning.


Changed files:
filename: ggml/src/ggml-rpc/ggml-rpc.cpp
patch: @@ -29,9 +29,12 @@
 #include <cstring>
 #include <fstream>
 #include <filesystem>
+#include <algorithm>
 
 namespace fs = std::filesystem;
 
+static constexpr size_t MAX_CHUNK_SIZE = 1024ull * 1024ull * 1024ull; // 1 GiB
+
 #ifdef _WIN32
 typedef SOCKET sockfd_t;
 using ssize_t = __int64;
@@ -323,23 +326,33 @@ static std::shared_ptr<socket_t> create_server_socket(const char * host, int por
 static bool send_data(sockfd_t sockfd, const void * data, size_t size) {
     size_t bytes_sent = 0;
     while (bytes_sent < size) {
-        ssize_t n = send(sockfd, (const char *)data + bytes_sent, size - bytes_sent, 0);
+        size_t size_to_send = std::min(size - bytes_sent, MAX_CHUNK_SIZE);
+        ssize_t n = send(sockfd, (const char *)data + bytes_sent, size_to_send, 0);
         if (n < 0) {
+            GGML_LOG_ERROR("send failed (bytes_sent=%zu, size_to_send=%zu)\n",
+                           bytes_sent, size_to_send);
             return false;
         }
-        bytes_sent += n;
+        bytes_sent += (size_t)n;
     }
     return true;
 }
 
 static bool recv_data(sockfd_t sockfd, void * data, size_t size) {
     size_t bytes_recv = 0;
     while (bytes_recv < size) {
-        ssize_t n = recv(sockfd, (char *)data + bytes_recv, size - bytes_recv, 0);
-        if (n <= 0) {
+        size_t size_to_recv = std::min(size - bytes_recv, MAX_CHUNK_SIZE);
+        ssize_t n = recv(sockfd, (char *)data + bytes_recv, size_to_recv, 0);
+        if (n < 0) {
+            GGML_LOG_ERROR("recv failed (bytes_recv=%zu, size_to_recv=%zu)\n",
+                           bytes_recv, size_to_recv);
+            return false;
+        }
+        if (n == 0) {
+            GGML_LOG_ERROR("recv returned 0 (peer closed?)\n");
             return false;
         }
-        bytes_recv += n;
+        bytes_recv += (size_t)n;
     }
     return true;
 }