Repository: pymodbus-dev/pymodbus

Issue:
Modbus RTU (serial) connection fails after first command in 2.5.1, (works in 2.3.0)
<!--
Please use the Pymodbus gitter channel at https://gitter.im/pymodbus_dev/Lobby or Stack Overflow(tag [pymodbus](https://stackoverflow.com/questions/tagged/pymodbus) for
support questions.

Before opening a new issue, make sure you do the following:
    * check that your issue isn't already filed: https://github.com/riptideio/pymodbus/issues
    * prepare a short, runnable example that reproduce the issue with the latest development version of Pymodbus
-->

### Versions

* Python: 3.8.1
* OS: Linux
* Pymodbus: 2.5.1
* Modbus Hardware (if used): 
_MCU via CP2102 (USB)
Rockwell, Siemens PLCs  (FTDI Adapter)
ESP32 over UART (FTDI Adapter)_

### Pymodbus Specific
* Client: rtu - sync

### Description

This issue was identified as a upstream issue with pymodbus from Home Assistant's Modbus Component. 
This issue emerged in Home Assistant 2021.5, when the pymodbus library was updated from 2.3.0 to 2.5.1.

In pymodbus 2.3.0 a Modbus device connected over serial worked but often produced an error on startup in Home Assistant:
`Cleanup recv buffer before send: 0x4d 0xfb` .

After this error occurred the Modbus client would continue to function correctly and the error would not occur again. Communication between the modbus device functioned as expected. 

In pymodbus 2.5.1, serial devices do not work after after the first modbus command. The first command always works but this error is produced and all future modbus commands fail:

```
2021-05-11 02:10:25 DEBUG (SyncWorker_1) [pymodbus.client.sync] New Transaction state 'SENDING'
2021-05-11 02:10:25 WARNING (SyncWorker_1) [pymodbus.client.sync] Cleanup recv buffer before send: 0x4d 0xfb
2021-05-11 02:10:25 ERROR (SyncWorker_1) [homeassistant.components.modbus.modbus] Pymodbus: Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response
```
Reverting back to pymodbus 2.30 fixes the issue. 

It is possible this issue is related: https://github.com/riptideio/pymodbus/issues/510

This is a significant issue, as Modbus RTU does not work after the initial command.
Testing was conducted with USB UART connections and RS485 serial connections to several microcontrollers and PLCs.


Speaking with @janiversen, (_maintainer of Home Assistant Modbus Component_) we identified the issue to be caused by updating to pymodbus 2.5.1.

### Code and Logs

Although configuration applies to Home Assistant, I have included it for reference. NOTE: I am not using pymodbus directly, when testing. 

```yaml
  modbus:
  - name: Curtain Controller
    type: serial
    method: rtu
    port: /dev/serial/by-id/usb-www.freetronics.com_0003_95437313934351505191-if00
    baudrate: 9600
    stopbits: 1
    bytesize: 8
    parity: N
    timeout: 10
    switches:
      #Curtain Controller
      - name: "Bedroom Curtain - UP"
        slave: 1
        address: 0
        write_type: coil
      - name: "Bedroom Curtain - Down"
        slave: 1
        address: 1
        write_type: coil
      - name: "Bedroom Curtain - Stop"
        slave: 1
        address: 2
        write_type: coil

```



Discussion:
👍 Lets hope this get solved fast and 2.5.2 are release so we can upgrade.

We want to use pymodbus async version, but once the 3.x is released.
@thehaxxa would you mind giving the complete logs with 2.5.1 and 2.3.0. Pymodbus 2.5.1 closes connection on encountering error. We have seen in some cases adding a small delay after an error is encountered fixes these sort of issue. There should also be kwarg which should disable disconnects after an erroneous transaction. Will try to find it.
We have the possibility to add a delay between the connect() and the first request, that might help as well.

@dhoomakethu you can see how we use pymodbus here:

[modbus.py.gz](https://github.com/riptideio/pymodbus/files/6459095/modbus.py.gz)

If you find the kwarg, it can easily be added.





> We have the possibility to add a delay between the connect() and the first request, that might help as well.

I can confirm the addition of `delay: 2` and `delay: 5` had no effect in this context. 
@dhoomakethu can we configure (with kwargs or similar) a delay after the error ?

@thehaxxa  No the delay needs to be after an error is received. I will try to see if I can reproduce this issue with the limited hardware I have at my  disposal. 
@janiversen we  can add additional configuration but it would be helpful if I can get some debug logs to figure out what could be going wrong. I will also take a look in to home assist modbus stack.
@dhoomakethu I sure can follow you on getting debug logs, that is a supporters nightmare 😁

I am sure @thehaxxa will supply you with those, I am "just" the maintainer of code and use Modbus-tcp.

> would you mind giving the complete logs with 2.5.1 and 2.3.0.

Unfortunately you have them, there is no logging about the Connection or RECEIVE feedback. @janiversen suggested this might be insightful in discussions with him.

I posted all the logging information I have.
I do believe it's a startup issue that may be mitigated by a delay. It would be great if this was configurable in either context. 

This is the only logging I have on 2.5.1:
```
2021-05-11 23:51:58 DEBUG (SyncWorker_1) [pymodbus.client.sync] New Transaction state 'SENDING'
2021-05-11 23:51:58 WARNING (SyncWorker_1) [pymodbus.client.sync] Cleanup recv buffer before send: 0x4d 0xfb
2021-05-11 23:51:58 ERROR (SyncWorker_1) [homeassistant.components.modbus.modbus] Pymodbus: Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response
```
and 2.3.0 has this:

```
2021-05-11 23:51:58 DEBUG (SyncWorker_2) [pymodbus.client.sync] New Transaction state 'SENDING'
2021-05-11 23:51:58 WARNING (SyncWorker_2) [pymodbus.client.sync] Cleanup recv buffer before send: 0x4d 0xfb
2021-05-11 23:51:58 ERROR (SyncWorker_2) [homeassistant.components.modbus.modbus] Pymodbus: Modbus Error: [Input/Output] No Response received from the remote unit/Unable to decode response

2021-05-11 23:52:00 DEBUG (SyncWorker_14) [pymodbus.client.sync] New Transaction state 'SENDING'
2021-05-11 23:52:00 WARNING (SyncWorker_14) [pymodbus.client.sync] Cleanup recv buffer before send: 0x20 0x43 0x75 0x72 0x74 0x61 0x69 0x6e 0x20 0x53 0x65 0x72 0x76 0x65 0x72 0xd 0xa 0x1 0x5 0x0 0x1 0xff 0x0 0xdd 0xfa
```
@dhoomakethu try to simulate your rs-485 delivers garble when opened (my rs485 - rs232 used to do that), as I read the pymodbus code that would lead to a disconnect in 2.5.1 whereas 2.30 would continue
The connection close was introduced to mitigate some nasty bugs around retries. I thought we finally hit the nail as there was not issue's reported on this topic with 2.5.1 until today 😁 . I will try to reproduce this and see if there is anything else we can do here.
@dhoomakethu without being deep into your code, allow me a suggestion:
after connect() in client mode, skip anything received until first request is sent. For server mode, you need to add "If not a valid modbus message".

After that I think disconnect() on bad messages is an excellent idea. 
If you want that tested I am sure @thehaxxa can help, if you send a patched python file.
> @dhoomakethu without being deep into your code, allow me a suggestion:
> after connect() in client mode, skip anything received until first request is sent. For server mode, you need to add "If not a valid modbus message".
> 
> After that I think disconnect() on bad messages is an excellent idea.

We are kind of doing it before every request [here](https://github.com/riptideio/pymodbus/blob/master/pymodbus/client/sync.py#L699)

Resolving patch:
diff --git a/pymodbus/repl/client/main.py b/pymodbus/repl/client/main.py
--- a/pymodbus/repl/client/main.py
+++ b/pymodbus/repl/client/main.py
@@ -233,9 +233,10 @@ def _process_args(args, string=True):
 @click.option("--retry-on-error", is_flag=True, default=False,
               help="Retry on error response")
 @click.option("--retries", default=3, help="Retry count")
+@click.option("--reset-socket/--no-reset-socket", default=True, help="Reset client socket on error")
 @click.pass_context
 def main(ctx, verbose, broadcast_support, retry_on_empty,
-         retry_on_error, retries):
+         retry_on_error, retries, reset_socket):
     if verbose:
         global log
         import logging
@@ -248,7 +249,8 @@ def main(ctx, verbose, broadcast_support, retry_on_empty,
         "broadcast": broadcast_support,
         "retry_on_empty": retry_on_empty,
         "retry_on_invalid": retry_on_error,
-        "retries": retries
+        "retries": retries,
+        "reset_socket": reset_socket
     }
 
 
diff --git a/pymodbus/repl/server/cli.py b/pymodbus/repl/server/cli.py
--- a/pymodbus/repl/server/cli.py
+++ b/pymodbus/repl/server/cli.py
@@ -25,7 +25,8 @@
  |     ___<   |  |/     \ /  _ \ / __ |  | __ \|  |  \/  ___/  \_____  \_/ __ \_  __ \  \/ // __ \_  __ \\
  |    |    \___  |  Y Y  (  <_> ) /_/ |  | \_\ \  |  /\___ \   /        \  ___/|  | \/\   /\  ___/|  | \/
  |____|    / ____|__|_|  /\____/\____ |  |___  /____//____  > /_______  /\___  >__|    \_/  \___  >__|
-           \/          \/            \/      \/           \/          \/     \/                 \/"""
+           \/          \/            \/      \/           \/          \/     \/                 \/
+"""
 
 SMALL_TITLE = "Pymodbus server..."
 BOTTOM_TOOLBAR = HTML('(MODBUS SERVER) <b><style bg="ansired">Press Ctrl+C or '
@@ -80,7 +81,9 @@
 
 
 def info(message):
-    click.secho(str(message), fg="green")
+    if not isinstance(message, str):
+        message = str(message)
+    click.secho(message, fg="green")
 
 
 def warning(message):
@@ -191,12 +194,12 @@ async def interactive_shell(server):
 
 
 async def main(server):
-    with patch_stdout():
-        try:
-            await interactive_shell(server)
-        finally:
-            pass
-        warning("Bye Bye!!!")
+    # with patch_stdout():
+    try:
+        await interactive_shell(server)
+    finally:
+        pass
+    warning("Bye Bye!!!")
 
 
 async def run_repl(server):
diff --git a/pymodbus/transaction.py b/pymodbus/transaction.py
--- a/pymodbus/transaction.py
+++ b/pymodbus/transaction.py
@@ -69,6 +69,7 @@ def __init__(self, client, **kwargs):
         self.retry_on_invalid = kwargs.get('retry_on_invalid',
                                            Defaults.RetryOnInvalid)
         self.retries = kwargs.get('retries', Defaults.Retries) or 1
+        self.reset_socket = kwargs.get('reset_socket', True)
         self._transaction_lock = RLock()
         self._no_response_devices = []
         if client:
@@ -217,7 +218,8 @@ def execute(self, request):
                                 "/Unable to decode response")
                             response = ModbusIOException(last_exception,
                                                          request.function_code)
-                        self.client.close()
+                        if self.reset_socket:
+                            self.client.close()
                     if hasattr(self.client, "state"):
                         _logger.debug("Changing transaction state from "
                                       "'PROCESSING REPLY' to "
@@ -230,7 +232,8 @@ def execute(self, request):
                 # Handle decode errors in processIncomingPacket method
                 _logger.exception(ex)
                 self.client.state = ModbusTransactionState.TRANSACTION_COMPLETE
-                self.client.close()
+                if self.reset_socket:
+                    self.client.close()
                 return ex
 
     def _retry_transaction(self, retries, reason,
@@ -294,7 +297,8 @@ def _transact(self, packet, response_length,
 
         except (socket.error, ModbusIOException,
                 InvalidMessageReceivedException) as msg:
-            self.client.close()
+            if self.reset_socket:
+                self.client.close()
             _logger.debug("Transaction failed. (%s) " % msg)
             last_exception = msg
             result = b''


Test patch: