Repository: pymodbus-dev/pymodbus

Issue:
ModbusSocketFramer bug when multiple modbus messages are received in 1 tcp payload
<!--
Before opening a new issue, make sure you do the following:

- Check that your issue isn't already filed: https://github.com/pymodbus-dev/pymodbus/issues
- Check the discussions forum https://github.com/pymodbus-dev/pymodbus/discussions
- Prepare a short, runnable example that reproduce the issue with the latest development version of Pymodbus
-->

### Versions

- Python: 3.8.x
- OS: Linux
- Pymodbus: 3.6.3
- Modbus Hardware (if used): modbus TCP

### Pymodbus Specific

- Server: tcp - async

### Description

When receiving multiple modbus frames in a single tcp frame, pymodbus throws an error. This happens cause of a misscalculation.
I fixed the issue by substracting 1 in the length calcuation of the frame in "pymodbus/framer/socket_framer.py" line 99:

```
    def getFrame(self):
        """Return the next frame from the buffered data.

        :returns: The next full frame buffer
        """
        length = self._hsize + self._header["len"] - 1
        print("getFrame", length, self._hsize, self._header['len'])
        return self._buffer[self._hsize : length]
```

This happens cause `self._hsize` and `self._header["len"]` have the `uid` byte.
This doesn't happen when there is only 1 modbus packet inside a TCP frame cause in python `my_list[:length] == my_list[:length+1]` is True.

### Code and Logs

Added some prints inside the framer funtions to debug

```python
datagram_received

2024-01-22 22:54:03,852 DEBUG logging:103 recv: 0x4 0x83 0x0 0x0 0x0 0x6 0x1 0x3 0x11 0x4d 0x0 0x32 0x4 0x84 0x0 0x0 0x0 0x6 0x1 0x3 0x30 0x1 0x0 0x4 0x4 0x85 0x0 0x0 0x0 0x6 0x1 0x3 0x31 0x5 0x0 0x11 old_data:  addr=None
2024-01-22 22:54:03,852 DEBUG logging:103 Handling data: 0x4 0x83 0x0 0x0 0x0 0x6 0x1 0x3 0x11 0x4d 0x0 0x32 0x4 0x84 0x0 0x0 0x0 0x6 0x1 0x3 0x30 0x1 0x0 0x4 0x4 0x85 0x0 0x0 0x0 0x6 0x1 0x3 0x31 0x5 0x0 0x11
2024-01-22 22:54:03,852 DEBUG logging:103 Processing: 0x4 0x83 0x0 0x0 0x0 0x6 0x1 0x3 0x11 0x4d 0x0 0x32 0x4 0x84 0x0 0x0 0x0 0x6 0x1 0x3 0x30 0x1 0x0 0x4 0x4 0x85 0x0 0x0 0x0 0x6 0x1 0x3 0x31 0x5 0x0 0x11
getFrame 13 7 6
_process b'\x03\x11M\x002\x04'
2024-01-22 22:54:03,853 DEBUG logging:103 Factory Request[ReadHoldingRegistersRequest': 3]
5 b'\x11M\x002\x04'
Traceback (most recent call last):
  File ".../MyModbusServer.py", line 111, in handle
    await self.inner_handle()
  File ".../MyModbusServer.py", line 101, in inner_handle
    self._framer.processIncomingPacket(
  File ".../pymodbus/framer/base.py", line 139, in processIncomingPacket
    self.frameProcessIncomingPacket(single, callback, slave, **kwargs)
  File ".../pymodbus/framer/socket_framer.py", line 141, in frameProcessIncomingPacket
    self._process(callback, tid)
  File ".../pymodbus/framer/socket_framer.py", line 147, in _process
    if (result := self.decoder.decode(data)) is None:
  File ".../pymodbus/factory.py", line 183, in decode
    return self._helper(message)
  File ".../pymodbus/factory.py", line 216, in _helper
    request.decode(data[1:])
  File ".../pymodbus/register_read_message.py", line 49, in decode
    self.address, self.count = struct.unpack(">HH", data)
struct.error: unpack requires a buffer of 4 bytes
2024-01-22 22:54:03,855 DEBUG logging:103 Client Disconnection server_listener due to unpack requires a buffer of 4 bytes
```



Discussion:
We have corrected a similar bug, and it is available on dev, however I think I will make a test case with multiple messages as well.

Can you please add a debug log, so I can see what is actually received. 

Most devices do not accept multiple frames on the same socket (one client connection), so what is the practical use case ?
I have a modbus slave which connects to a modbus master (instead of listening for clients to connect to me, I connect to the TCP server of the master). I changed the hard-coded `is_server` param to `False` in `ModbusBaseServer` `__init__` and extended the `ModbusTcpServer` to have same functionality as `ModbusServerRequestHandler` when handling messages. 

[pymodbus.log](https://github.com/pymodbus-dev/pymodbus/files/14019706/pymodbus.log)


```
class MyModbusServer(ModbusTcpServer):
    def __init__(self, *args, **kwargs):
        self.tls_setup = CommParams(
            comm_type=CommType.TCP,
            comm_name="server_listener",
            reconnect_delay=1,
            reconnect_delay_max=10,
            timeout_connect=1,
            host="192.168.11.1",
            port=502
        )
        super().__init__(*args, **kwargs)
        self.receive_queue = asyncio.Queue()

    def _log_exception(self):
        """Show log exception."""
        Log.debug(
            "Handler for stream [{}] has been canceled", self.comm_params.comm_name
        )

    def callback_connected(self) -> None:
        """Call when connection is succcesfull."""
        try:
            self.running = True
            self._framer = self.framer(
                self.decoder,
                client=None,
            )

            # schedule the connection handler on the event loop
            self.handler_task = asyncio.create_task(self.handle())
            self.handler_task.set_name("server connection handler")
        except Exception as exc:  # pragma: no cover pylint: disable=broad-except
            Log.error(
                "Server callback_connected exception: {}; {}",
                exc,
                traceback.format_exc(),
            )

    def callback_disconnected(self, call_exc: Exception | None) -> None:
        """Call when connection is lost."""
        try:
            if self.handler_task:
                self.handler_task.cancel()
            if hasattr(self, "on_connection_lost"):
                self.on_connection_lost()
            if call_exc is None:
                self._log_exception()
            else:
                Log.debug(
                    "Client Disconnection {} due to {}",
                    self.comm_params.comm_name,
                    call_exc,
                )
            self.running = False
        except Exception as exc:  # pylint: disable=broad-except
            Log.error(
                "Datastore unable to fulfill request: {}; {}",
                exc,
                traceback.format_exc(),
            )

    async def inner_handle(self):
        """Handle handler."""
        slaves = self.context.slaves()
        # this is an asyncio.Queue await, it will never fail
        data = await self._recv_()
        if isinstance(data, tuple):
            # addr is populated when talking over UDP
            data, *addr = data
        else:
            addr = (None,)  # empty tuple

        # if broadcast is enabled make sure to
        # process requests to address 0
        if self.broadcast_enable:  # pragma: no cover
            if 0 not in slaves:
                slaves.append(0)

        Log.debug("Handling data: {}", data, ":hex")

        single = self.context.single
        self._framer.processIncomingPacket(
            data=data,
            callback=lambda x: self.execute(x, *addr),
            slave=slaves,
            single=single,
        )

    async def handle(self):
        while self.running:
            try:
                await self.inner_handle()
            except asyncio.CancelledError:
                # catch and ignore cancellation errors
                if self.running:
                    # self._log_exception()
                    self.running = False
            except Exception as exc:  # pylint: disable=broad-except
                # force TCP socket termination as processIncomingPacket
                # should handle application layer errors
                import traceback
                traceback.print_exc()
                self.transport_close()
                self.callback_disconnected(exc)

    def execute(self, request, *addr):
        """Call with the resulting message.

        :param request: The decoded request message
        :param addr: the address
        """
        if self.request_tracer:
            self.request_tracer(request, *addr)

        broadcast = False
        try:
            if self.broadcast_enable and not request.slave_id:
                broadcast = True
                # if broadcasting then execute on all slave contexts,
                # note response will be ignored
                for slave_id in self.context.slaves():
                    response = request.execute(self.context[slave_id])
            else:
                context = self.context[request.slave_id]
                response = request.execute(context)
        except NoSuchSlaveException:
            Log.error("requested slave does not exist: {}", request.slave_id)
            if self.ignore_missing_slaves:
                return  # the client will simply timeout waiting for a response
            response = request.doException(merror.GatewayNoResponse)
        except Exception as exc:  # pylint: disable=broad-except
            Log.error(
                "Datastore unable to fulfill request: {}; {}",
                exc,
                traceback.format_exc(),
            )
            response = request.doException(merror.SlaveFailure)
        # no response when broadcasting
        if not broadcast:
            response.transaction_id = request.transaction_id
            response.slave_id = request.slave_id
            skip_encoding = False
            if self.response_manipulator:
                response, skip_encoding = self.response_manipulator(response)
            self.send(response, *addr, skip_encoding=skip_encoding)

    def send(self, message, addr, **kwargs):
        """Send message."""
        if kwargs.get("skip_encoding", False):
            self.transport_send(message, addr=addr)
        elif message.should_respond:
            pdu = self._framer.buildPacket(message)
            self.transport_send(pdu, addr=addr)
        else:
            Log.debug("Skipping sending response!!")

    async def _recv_(self):  # pragma: no cover
        """Receive data from the network."""
        try:
            result = await self.receive_queue.get()
        except RuntimeError:
            Log.error("Event loop is closed")
            result = None
        return result

    def callback_data(self, data: bytes, addr: tuple | None = ()) -> int:
        """Handle received data."""
        if addr != ():
            self.receive_queue.put_nowait((data, addr))
        else:
            self.receive_queue.put_nowait(data)
        return len(data)
```
Something is off here....a slave is pr modbus definition a server that listens and wait for clients to connect to it. It seems you have turned the modbus protocol upside down, that will have a lot of other implications like transaction handling, server/client decoder.
But anyhow your original statement about a bug in socket_framer, did you try dev as I suggested.
Yeah, I thought the same at first but the commercial device im trying to communicate with wants the slave to connect to itself so I had to implement a custom method as mentioned above.

Haven't tried the dev branch yet, will send an update when available
Please read the modbus protocol, a slave do not connect....but your device is at the rs485 level below modbus, and in rs485 you have one master (client in modbus) and slaves (server ind modbus).

I believe incorrect behavior can occur also for "standard" communication - communicating with slave, I send 10 (async) requests at the same time, Wireshark shows standard 1 Modbus response per TCP packet:
![Untitled](https://github.com/pymodbus-dev/pymodbus/assets/89193681/ca9cffc9-08da-4680-ae25-c2c5c82f7760)
However, in the log (please find it attached), seems 2 packets are aggregated, PyModbus ignores second part and retries request after timeout (3s).
[pymodbus.log](https://github.com/pymodbus-dev/pymodbus/files/14083922/pymodbus.log)

We do not support sending multiple requests in our clients, actually it should not be possible, if you use the standard calls.

Each client knows it has exactly 0 or 1 request outstanding, and depends on that information, so getting multiple responses is not supported.
Anyhow the software should not break down, but log an error.
Please send the pcap file, then it is easier for me to make a test case.
Just to be sure we are on the same page when it comes to multiple requests - what I do is something like (artificial example, but I used this line for gathering pcap data):
```python 
await asyncio.gather(*[client.read_holding_registers(address=x, count=2) for x in range(0, 1000, 100)])
```
I don't think I am doing any shenanigans with PyModbus API, the aggregation of requests in frame # 23 is done by TCP stack (Win10 x64+ CPython 3.11.6).

Because the slave is 3rd party device I can comment only on what I see in Wireshark and seems it sends one response at the time ("1" in the picture):
![image](https://github.com/pymodbus-dev/pymodbus/assets/89193681/dfbad89b-4034-4d82-9dd9-1ab0c2067117)
But the issue I see - PyModbus sometimes seems to lose a response and retransmission is required ("2" in the picture). Wireshark shows 10 separate responses, but in the log file from my previous comment we can see that PyModbus sometimes sees multiple responses "aggregated" and does not handle it correctly. The aggregation seems to be caused by OS/network stack, definitely not my application.

I belive the use case itself (send multiple async requests at once, wait for all responses) is valid - "Several MODBUS transactions can be activated simultaneously on the same TCP Connection." (after "4.2 TCP CONNECTION MANAGEMENT" from https://www.modbus.org/docs/Modbus_Messaging_Implementation_Guide_V1_0b.pdf ).

Attached please find [the anonymized pcap](https://github.com/pymodbus-dev/pymodbus/files/14089241/pymodbus.final.zip).

It might be you find it is a valid use case, but pymodbus still do not support it....just as many devices have problems when receiving a second request before responding to the first. This is because most devices actually do not care about the transaction id, but simply maps it into the response.

If you want pymodbus to support multiple parallel request, then pull requests are welcome. Please remember if you send 10 requests there are no guarantee that the responses are in the same order.

I do agree that pymodbus should not breakdown, and that is a case I will look at.

Resolving patch:
diff --git a/pymodbus/client/base.py b/pymodbus/client/base.py
--- a/pymodbus/client/base.py
+++ b/pymodbus/client/base.py
@@ -157,13 +157,13 @@ async def async_execute(self, request=None):
 
         count = 0
         while count <= self.retries:
+            req = self.build_response(request.transaction_id)
             if not count or not self.no_resend_on_retry:
                 self.transport_send(packet)
             if self.broadcast_enable and not request.slave_id:
                 resp = b"Broadcast write sent - no response expected"
                 break
             try:
-                req = self.build_response(request.transaction_id)
                 resp = await asyncio.wait_for(
                     req, timeout=self.comm_params.timeout_connect
                 )
diff --git a/pymodbus/framer/socket_framer.py b/pymodbus/framer/socket_framer.py
--- a/pymodbus/framer/socket_framer.py
+++ b/pymodbus/framer/socket_framer.py
@@ -77,7 +77,7 @@ def advanceFrame(self):
         it or determined that it contains an error. It also has to reset the
         current frame header handle
         """
-        length = self._hsize + self._header["len"]
+        length = self._hsize + self._header["len"] -1
         self._buffer = self._buffer[length:]
         self._header = {"tid": 0, "pid": 0, "len": 0, "uid": 0}
 
@@ -129,15 +129,16 @@ def frameProcessIncomingPacket(self, single, callback, slave, tid=None, **kwargs
         The processed and decoded messages are pushed to the callback
         function to process and send.
         """
-        if not self.checkFrame():
-            Log.debug("Frame check failed, ignoring!!")
-            return
-        if not self._validate_slave_id(slave, single):
-            header_txt = self._header["uid"]
-            Log.debug("Not a valid slave id - {}, ignoring!!", header_txt)
-            self.resetFrame()
-            return
-        self._process(callback, tid)
+        while True:
+            if not self.checkFrame():
+                Log.debug("Frame check failed, ignoring!!")
+                return
+            if not self._validate_slave_id(slave, single):
+                header_txt = self._header["uid"]
+                Log.debug("Not a valid slave id - {}, ignoring!!", header_txt)
+                self.resetFrame()
+                return
+            self._process(callback, tid)
 
     def _process(self, callback, tid, error=False):
         """Process incoming packets irrespective error condition."""
diff --git a/pymodbus/transport/stub.py b/pymodbus/transport/stub.py
--- a/pymodbus/transport/stub.py
+++ b/pymodbus/transport/stub.py
@@ -1,29 +1,47 @@
 """ModbusProtocol network stub."""
 from __future__ import annotations
 
-from pymodbus.transport.transport import ModbusProtocol
+from typing import Callable
+
+from pymodbus.transport.transport import CommParams, ModbusProtocol
 
 
 class ModbusProtocolStub(ModbusProtocol):
     """Protocol layer including transport."""
 
+    def __init__(
+        self,
+        params: CommParams,
+        is_server: bool,
+        handler: Callable[[bytes], bytes] | None = None,
+    ) -> None:
+        """Initialize a stub instance."""
+        self.stub_handle_data = handler if handler else self.dummy_handler
+        super().__init__(params, is_server)
+
+
     async def start_run(self):
         """Call need functions to start server/client."""
         if  self.is_server:
             return await self.transport_listen()
         return await self.transport_connect()
 
+
     def callback_data(self, data: bytes, addr: tuple | None = None) -> int:
         """Handle received data."""
         if (response := self.stub_handle_data(data)):
             self.transport_send(response)
         return len(data)
 
+    def callback_new_connection(self) -> ModbusProtocol:
+        """Call when listener receive new connection request."""
+        new_stub = ModbusProtocolStub(self.comm_params, False)
+        new_stub.stub_handle_data = self.stub_handle_data
+        return new_stub
+
     # ---------------- #
     # external methods #
     # ---------------- #
-    def stub_handle_data(self, data: bytes) -> bytes | None:
+    def dummy_handler(self, data: bytes) -> bytes | None:
         """Handle received data."""
-        if len(data) > 5:
-            return data
-        return None
+        return data


Test patch:
diff --git a/test/test_network.py b/test/test_network.py
--- a/test/test_network.py
+++ b/test/test_network.py
@@ -1,4 +1,7 @@
 """Test transport."""
+from __future__ import annotations
+
+import asyncio
 
 import pytest
 
@@ -24,5 +27,59 @@ async def test_stub(self, use_port, use_cls):
         stub = ModbusProtocolStub(use_cls, True)
         assert await stub.start_run()
         assert await client.connect()
+        test_data = b"Data got echoed."
+        client.transport.write(test_data)
+        client.transport_close()
+        stub.transport_close()
+
+    async def test_double_packet(self, use_port, use_cls):
+        """Test double packet on network."""
+        old_data = b''
+        client = AsyncModbusTcpClient(NULLMODEM_HOST, port=use_port, retries=0)
+
+        def local_handle_data(data: bytes) -> bytes | None:
+            """Handle server side for this test case."""
+            nonlocal old_data
+
+            addr = int(data[9])
+            response = data[0:5] + b'\x05\x00\x03\x02\x00' + (addr*10).to_bytes(1, 'big')
+
+            # 1, 4, 7 return correct data
+            # 2, 5 return NO data
+            # 3 return 2 + 3
+            # 6 return 6 + 6 (wrong order
+            # 8 return 7 + half 8
+            # 9 return second half 8 + 9
+            if addr in {2, 5}:
+                old_data = response
+                response = None
+            elif addr == 3:
+                response = old_data + response
+                old_data = b''
+            elif addr == 6:
+                response = response + old_data
+                old_data = b''
+            elif addr == 8:
+                x =  response
+                response = response[:7]
+                old_data = x[7:]
+            elif addr == 9:
+                response = old_data + response
+                old_data = b''
+            return response
+
+        async def local_call(addr: int) -> bool:
+            """Call read_holding_register and control."""
+            nonlocal client
+            reply = await client.read_holding_registers(address=addr, count=1)
+            assert not reply.isError(), f"addr {addr} isError"
+            assert reply.registers[0] == addr * 10, f"addr {addr} value"
+
+        stub = ModbusProtocolStub(use_cls, True, handler=local_handle_data)
+        stub.stub_handle_data = local_handle_data
+        await stub.start_run()
+
+        assert await client.connect()
+        await asyncio.gather(*[local_call(x) for x in range(1, 10)])
         client.transport_close()
         stub.transport_close()