Repository: tomerfiliba-org/rpyc

Issue #558: Issues with get_id_pack, get_methods, a race and stdin, stdout issues
Created: 2024-06-15T18:04:55Z
Describe the issue briefly here, including:

expected result versus actual result
When working on a new Remote Library for Robot Framework (https://github.com/rlehfeld/robotframework-rpycremote) which takes the RPyC functionality into use, I run into several issues and also fixed them on my fork of RPyC. These changes I would like to describe but and if I remember right, I found 4 issues and unfortunately only one has a simple example which I will describe below.


for objects which do not  provide a  class property, RPyC is not usable. E.g. this is the case for some c-extensions or when it is hidden. (for this problem I will provide a simple example)
Here the problem is within  rpyc.lib.get_id_pack
When using Pydantic BaseModel objects, under some not completely understood cicumstances, the rpyc.lib.get_methods fails
There is a race in rpyc.core.protocol.Connection._unbox between the lines:

            if id_pack in self._proxy_cache:
                proxy = self._proxy_cache[id_pack]

as the garbage collector can and will kick in in between these two lines.
4. While fixing 3. and executing the unit tests I run into the following unrelated issue.
PipeStream and Win32PipeStream are taking ownership of stdin and stdout if from_std is used. When e.g. peer is closing the connection, the stdin and stdout of the python process is closed which can lead to exit code 1.
Further, I would like to say thank you for this super solution. Together with my Robot Framework Remote Library, this is a game changer.
Many thanks
René
Environment

rpyc version
pypy3.10-7.3.15-2.3.10.fc40.x86_64
python version
pypy3.10-7.3.15-2.3.10.fc40.x86_64
python3-3.12.2-2.fc40.x86_64
operating system
Fedora 40

Minimal example for problem 1
Server:
Objects from Dummy class below, cannot be used together with RPyC.
    class Dummy:
        """dummy class"""
        def __init__(self):
            self._value = 1
            self.value2 = 5

        def __getattribute__(self, attr):
            if attr in ('__class__'):
                raise AttributeError(f'catched {attr}')
            return super().__getattribute__(attr)

        @property
        def value(self):
            """value getter"""
            print('called value getter')
            return self._value

        @value.setter
        def value(self, v):
            print('called value setter')
            self._value = v

        def __call__(self, *args, **kwargs):
            """callable object"""
            print(f'called __call__({args}, {kwargs})')
            return '__call__'

        def method(self, *args, **kwargs):
            """and some callable method"""
            print(f'called method({args}, {kwargs})')
            return 'method'

Duplicate issue #526: Help with debugging a RPyC-related issue: KeyError in colls.py - how to debug this further?
Created: 2023-02-07T03:05:44Z
I'm looking for some help debugging this, please.
I'm using RPyC 5.3.0 in a Ubuntu 22.04 Docker container, with Python 3.9, and rarely (but too frequently) I see the following occur, when trying to read a value from a remote memoryview (the variable mv here, of type <netref class 'rpyc.core.netref.type'>). This is running under pytest:
memory_access.py:59: in read
    return mv[offset]
/home/docker/.local/lib/python3.9/site-packages/rpyc/core/netref.py:261: in method
    return syncreq(_self, consts.HANDLE_CALLATTR, name, args, kwargs)
/home/docker/.local/lib/python3.9/site-packages/rpyc/core/netref.py:63: in syncreq
    return conn.sync_request(handler, proxy, *args)
/home/docker/.local/lib/python3.9/site-packages/rpyc/core/protocol.py:718: in sync_request
    return _async_res.value
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <AsyncResult object (ready) at 0x7f08413a79f0>
    @property
    def value(self):
        """Returns the result of the operation. If the result has not yet
        arrived, accessing this property will wait for it. If the result does
        not arrive before the expiry time elapses, :class:`AsyncResultTimeout`
        is raised. If the returned result is an exception, it will be raised
        here. Otherwise, the result is returned directly.
        """
        self.wait()
        if self._is_exc:
>           raise self._obj
E           _get_exception_class.<locals>.Derived: ('subprocess.CompletedProcess', 187650931238928, 281473569049664)
E           
E           ========= Remote Traceback (1) =========
E           Traceback (most recent call last):
E             File "/usr/lib/python3.8/site-packages/rpyc/core/protocol.py", line 355, in _dispatch_request
E               args = self._unbox(args)
E             File "/usr/lib/python3.8/site-packages/rpyc/core/protocol.py", line 321, in _unbox
E               return tuple(self._unbox(item) for item in value)
E             File "/usr/lib/python3.8/site-packages/rpyc/core/protocol.py", line 321, in <genexpr>
E               return tuple(self._unbox(item) for item in value)
E             File "/usr/lib/python3.8/site-packages/rpyc/core/protocol.py", line 323, in _unbox
E               return self._local_objects[value]
E             File "/usr/lib/python3.8/site-packages/rpyc/lib/colls.py", line 110, in __getitem__
E               return self._dict[key][0]
E           KeyError: ('subprocess.CompletedProcess', 187650931238928, 281473569049664)
/home/docker/.local/lib/python3.9/site-packages/rpyc/core/async_.py:108: KeyError

My code performs many, many such read/write calls over a netref'd memoryview, and it's always one particular read that fails, if it fails at all. The access offset is correct.
I'm trying to gather clues so I have a few questions, please:

How should I interpret this trace? The client is expecting to read a value of type int from the memoryview (return mv[offset]), but the server throws this exception instead. What does that indicate went wrong? What could be happening to cause this?
is the key subprocess.CompletedProcess  coming from an internal, RPyC mechanism, or is that from my own code (or perhaps third-party code I call)? There is nothing to do with subprocess in any of my code to do with the memoryview, but there are calls to subprocess elsewhere in my program (on the server side).

Recently, I was using RPyC 5.1.0 instead, and I was seeing something similar - the same server exception was being raised (as a debug-level log entry), but the client was oblivious to this and did not see an exception. The correct value would be returned to the client. But occasionally when this happens, the client would get a None value back from return mv[offset], which is, of course, incorrect, and caused downstream code to fail. Updating to 5.3.0 seems to result in the exception propagating to the client, rather than returning None.

EDIT: I've put a try/except/breakpoint around line 322 in _unbox, to catch the error when label == consts.LABEL_LOCAL_REF - and  inspected self._local_objects. It's very large (589 entries) and has six that have subprocess.CompletedProcess in the key. All six of these have a match with the first number in the key's tuple, but none of them match with the second number (and the number in the key is significantly larger than any of the numbers in the _local_objects collection).
So it looks like this subprocess.CompletedProcess value is from my own code, but there seems to be a missing entry when RPyC unboxes it. What could lead to this happening? It seems to occur slightly more often if I load the server's CPU up to 100% (to slow it down a bit).

Closing comment (2024-06-16T03:23:54Z): I looks very much like Problem Number 3 I reported in #558 and in #559 I provide a fix. As this is a races with the garbage collector, it is more or less impossible to provide test code for this but looking at the code reveals that this is a race.
Maybe you can try my fork that I try to upstream with #559, if I am correct with my assumption.