Repository: pymeasure/pymeasure

Issue:
`VISAAdapter` still terminating on default term character in `read_bytes(-1)`
Pretty odd and specific issue, not sure if this belong here or on PyVisa.

When I try to read the complete buffer in a serial connection using the `VISAAdapter`, it still breaks on the byte corresponding to `\n`:

```
    def __init__(self, adapter, name="Velleman K8090", timeout=1000, **kwargs):
        super().__init__(
            adapter,
            name=name,
            asrl={"baud_rate": 19200},
            write_termination="",
            read_termination=chr(0x0F),
            timeout=timeout,
            **kwargs,
        )
      
    # ...

    def read(self):
        response = self.read_bytes(-1)

        # `response` will end with "\n", even though there are more bytes in the buffer!
```

Encountered in #859 .

It seems the issue is two fold: in this code any termchar should be ignored and it's even responding to the wrong termchar.

This is with the `pyvisa-py` backend.


Discussion:
`read_bytes()` currently has a bit of an odd structure:

```
    def _read_bytes(self, count, break_on_termchar=False, **kwargs):
        if count >= 0:
            return self.connection.read_bytes(count, break_on_termchar=break_on_termchar, **kwargs)
        elif break_on_termchar:
            return self.connection.read_raw(None, **kwargs)
        else:
            read_termination = self.connection.read_termination
            self.connection.read_termination = None
            # Try except allows to set the read_termination even after an error.
            try:
                return self.connection.read_raw(**kwargs)
            finally:
                self.connection.read_termination = read_termination
```

It looks like that when `self.connection.read_termination` is equal to `None` (in the last clause of the if-statement), a default temchar is used instead of none at all.
> read_bytes() currently has a bit of an odd structure:

What is odd with that structure?

> It looks like that when self.connection.read_termination is equal to None (in the last clause of the if-statement), a default temchar is used instead of none at all.

Oh, that should not be, at least if I understand VISA correctly (in `read_termination` property setter):
~~~python
self.set_visa_attribute(constants.ResourceAttribute.termchar, ord(self.LF))
self.set_visa_attribute(constants.ResourceAttribute.termchar_enabled, constants.VI_FALSE)
~~~
Using the application com0com I have this minimal example:

<details>
<summary>Code</summary>

```
#
# Use a loopback to create two linked serial devices: COM3 and COM4
#

import pytest

from serial import Serial
from pymeasure.adapters import VISAAdapter


@pytest.fixture()
def adapter():
    return VISAAdapter(
        "ASRL3::INSTR",
        visa_library="@py",
        asrl={"baud_rate": 19200},
        timeout=3000,
        read_termination=chr(0x0F),
        write_termination="",
    )


@pytest.fixture()
def loopback():
    return Serial("COM4", baudrate=19200)


def test_regular(adapter, loopback):
    """Regular read with fixed number of bytes."""
    loopback.write(
        bytearray.fromhex("0471FF")
    )
    result = adapter.read_bytes(3)

    assert len(result) == 3


def test_all(adapter, loopback):
    """Regular read with -1 number of bytes."""
    loopback.write(
        bytearray.fromhex("0471FF")
    )
    result = adapter.read_bytes(-1)

    assert len(result) == 3


def test_newline(adapter, loopback):
    """Regular read with -1 number of bytes."""
    loopback.write(
        bytearray.fromhex("04710AFFFFFFFFFF")
    )
    result = adapter.read_bytes(-1)

    assert len(result) == 8
```

</details>

There are two issues: in `test_all` there is a timeout instead of a full read and in `test_newline` the newline is used as terminating character.
> Oh, that should not be, at least if I understand VISA correctly:

Yeah, I was stepping through the code and `termchar_enabled` really does get disabled. I can't see where this goes wrong.
I think I have the culprit:
https://github.com/pyvisa/pyvisa-py/blob/b410a6e20a8593ceb515a880a22b2e0aa3c8a4e1/pyvisa_py/serial.py#L122-L168

It reads the asrl_end (which seems to be set with the term_char in the `read_termination` I quoted above) and therefore it stops reading, as it is the "end_checker" method.
Right now, `suppress_end_check` is always "False (second to last parameter in the `_read` call)".

So, either I misunderstand pyvisa/VISA, or pyvisa-py needs an update.

> Using the application com0com I have this minimal example:

Thanks for the code.
It is easier to understand the "string formatting" for bytes. For example `b"8123vbla\n"` is a bytes object with the appropriate ASCII chars (last one is a newline). If you have to define a bytes value (e.g. byte 0), you can use b"\x00".
Could you check the behaviour with another backend? (NI for example?)
- I opened a request for desired behaviour: https://github.com/pyvisa/pyvisa/issues/728
I was just about the post the same, with this link: https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/sessions.py#L798

Only the `checker` argument seems to be used, which does not return `False` when it should.

We can also work around this in `pymeasure` by disabling an extra attribute in the Visa resources, `asrl_end_in`:

```
    def _read_bytes(self, count, break_on_termchar=False, **kwargs):
        # ....
        if count >= 0:
            return self.connection.read_bytes(count, break_on_termchar=break_on_termchar, **kwargs)
        elif break_on_termchar:
            return self.connection.read_raw(None, **kwargs)
        else:
            read_termination = self.connection.read_termination
            self.connection.read_termination = None
            termination_type = self.connection.get_visa_attribute(constants.ResourceAttribute.asrl_end_in)
            self.connection.set_visa_attribute(constants.ResourceAttribute.asrl_end_in, constants.SerialTermination.none)
            # Try except allows to set the read_termination even after an error.
            try:
                return self.connection.read_raw(**kwargs)
            finally:
                self.connection.read_termination = read_termination
                self.connection.set_visa_attribute(constants.ResourceAttribute.asrl_end_in, termination_type)
```

I'll check other backends now, will have to install some NI stuff first.

Thank you for bearing with me in all this @bmoneke !
pyvisa's maintainer responded with a suggestion to look at the VISA attributes in pyvisa-py and NI and see if there is a difference.

Probably pyvisa needs a fix.

Resolving patch:
diff --git a/pymeasure/adapters/visa.py b/pymeasure/adapters/visa.py
--- a/pymeasure/adapters/visa.py
+++ b/pymeasure/adapters/visa.py
@@ -171,7 +171,7 @@ def _read_bytes(self, count, break_on_termchar=False, **kwargs):
         """Read a certain number of bytes from the instrument.
 
         :param int count: Number of bytes to read. A value of -1 indicates to
-            read from the whole read buffer.
+            read from the whole read buffer until timeout.
         :param bool break_on_termchar: Stop reading at a termination character.
         :param \\**kwargs: Keyword arguments for the connection itself.
         :returns bytes: Bytes response of the instrument (including termination).
@@ -181,13 +181,17 @@ def _read_bytes(self, count, break_on_termchar=False, **kwargs):
         elif break_on_termchar:
             return self.connection.read_raw(None, **kwargs)
         else:
-            read_termination = self.connection.read_termination
-            self.connection.read_termination = None
-            # Try except allows to set the read_termination even after an error.
-            try:
-                return self.connection.read_raw(**kwargs)
-            finally:
-                self.connection.read_termination = read_termination
+            # pyvisa's `read_raw` reads until newline, if no termination_character defined
+            # and if not configured to stop at a termination lane etc.
+            # see https://github.com/pyvisa/pyvisa/issues/728
+            result = bytearray()
+            while True:
+                try:
+                    result.extend(self.connection.read_bytes(1))
+                except pyvisa.errors.VisaIOError as exc:
+                    if exc.error_code == pyvisa.constants.StatusCode.error_timeout:
+                        return bytes(result)
+                    raise
 
     def ask(self, command):
         """ Writes the command to the instrument and returns the resulting


Test patch:
diff --git a/tests/adapters/test_visa.py b/tests/adapters/test_visa.py
--- a/tests/adapters/test_visa.py
+++ b/tests/adapters/test_visa.py
@@ -154,6 +154,13 @@ def test_read_no_break_on_termchar(self, adapterR):
         # `break_on_termchar=False` is default value
         assert adapterR.read_bytes(-1) == b"SCPI,MOCK,VERSION_1.0\n"
 
+    def test_read_no_break_on_newline(self, adapter):
+        # write twice to have two newline characters in the read buffer
+        adapter.write("*IDN?")
+        adapter.write("*IDN?")
+        # `break_on_termchar=False` is default value
+        assert adapter.read_bytes(-1) == b"SCPI,MOCK,VERSION_1.0\nSCPI,MOCK,VERSION_1.0\n"
+
 
 def test_visa_adapter(adapter):
     assert repr(adapter) == f"<VISAAdapter(resource='{SIM_RESOURCE}')>"