Repository: tomerfiliba-org/rpyc

Issue #424: rpyc execution failed with EOFError: stream has been closed
Created: 2020-12-22T07:01:53Z
Steps

Started communicating to two ubuntu 16 machine parallelly (Executing some set of functions in thread)

Observation
Execution is failing with  EOFError: stream has been closed. Detailed trace is given below:
c:\python37\lib\site-packages\rpyc\core\stream.py:247: in read
    buf = self.sock.recv(min(self.MAX_IO_CHUNK, count))
E   ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:
lib\utils.py:105: in execute_remote_command
    output = eval("launch_command_remote({})".format(kwargs))
<string>:1: in <module>
    ???
c:\python37\lib\site-packages\rpyc\core\netref.py:247: in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
c:\python37\lib\site-packages\rpyc\core\netref.py:76: in syncreq
    return conn.sync_request(handler, proxy, *args)
c:\python37\lib\site-packages\rpyc\core\protocol.py:458: in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
c:\python37\lib\site-packages\rpyc\core\async_.py:100: in value
    self.wait()
c:\python37\lib\site-packages\rpyc\core\async_.py:47: in wait
    self._conn.serve(self._ttl)
c:\python37\lib\site-packages\rpyc\core\protocol.py:377: in serve
    data = self._channel.poll(timeout) and self._channel.recv()
c:\python37\lib\site-packages\rpyc\core\channel.py:55: in recv
    header = self.stream.read(self.FRAME_HEADER.size)
c:\python37\lib\site-packages\rpyc\core\stream.py:257: in read
    raise EOFError(ex)
E   EOFError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:
testcases\conftest.py:150: in InfraSetup
    location_type
steps\nas_steps.py:783: in activate_nas_proxies
    activation_token, build_path, location_type)
steps\nas_steps.py:721: in _check_connection_and_activate_nas_proxy
    nas_device_ip, share_location, agent_name, activation_token)
steps\nas_steps.py:750: in _activate_nas_proxy
    activation_token=activation_token)
lib\utils.py:116: in execute_remote_command
    output = eval("launch_command_remote({})".format(kwargs))
c:\python37\lib\site-packages\rpyc\core\netref.py:247: in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
c:\python37\lib\site-packages\rpyc\core\netref.py:76: in syncreq
    return conn.sync_request(handler, proxy, *args)
c:\python37\lib\site-packages\rpyc\core\protocol.py:458: in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
c:\python37\lib\site-packages\rpyc\core\protocol.py:479: in async_request
    self._async_request(handler, args, res)
c:\python37\lib\site-packages\rpyc\core\protocol.py:464: in _async_request
    self._send(consts.MSG_REQUEST, seq, (handler, self._box(args)))
c:\python37\lib\site-packages\rpyc\core\protocol.py:259: in _send
    self._channel.send(data)
c:\python37\lib\site-packages\rpyc\core\channel.py:75: in send
    self.stream.write(buf)
c:\python37\lib\site-packages\rpyc\core\stream.py:268: in write
    count = self.sock.send(data[:self.MAX_IO_CHUNK])
c:\python37\lib\site-packages\rpyc\core\stream.py:94: in __getattr__
    raise EOFError("stream has been closed")
E   EOFError: stream has been closed

Environment

rpyc version:  rpyc==4.1.1
python version: Python 3.7.4
operating system (Client): Windows Server 2012
operating system (server): Ubuntu16

Example
Client Connection:
def execute_remote_command(**kwargs):
    """Function to execute methods on remote machine or in local
    Usage:
    utils.execute_remote_command(
        host_ip=<Remote machine ip>, # Do not provide if want to run in local
        f_name=client_utils.create_new_file_with_size, #Function name
        path=file_name_create_path #Arguments
    )
    """
    host_ip = kwargs.get("host_ip", "127.0.0.1")
    port = kwargs.get("port", 18812)
    conn = rpyc.classic.connect(host_ip, port)
    conn._config['sync_request_timeout'] = None
    rpyc.core.stream.SocketStream.MAX_IO_CHUNK = 2000000    
    logger.info("Connection established with host {} at port {}".format(host_ip, port))
    conn.execute("from rpyc_classic import *")
    f_name = kwargs["f_name"]
    kwargs.pop("f_name")
    if "host_ip" in list(kwargs.keys()):
        kwargs.pop("host_ip")
   #To teleport function at server 
    teleport_supported_functions(conn)
    launch_command_remote = teleport_function(conn, f_name)
    logger.info("Executing remote method {}".format(f_name))
    if not kwargs:
        output = eval("launch_command_remote()")
    else:
        output = eval("launch_command_remote({})".format(kwargs))
    conn.close()
    return output

Duplicate issue #387: Attempt to execute a teleported function on remote host with different Python version results in SystemError: Unknown opcode
Created: 2020-05-23T11:13:09Z
Hello! I'm just getting started with RPyC library, using classic mode and I've attempted to teleport a simple function to execute on remote host. My server has a different Python version than client. Documentation isn't clear, whether there's any limitations on Python version regarding function teleportation, and looking at code in teleportation.py, I can assume this scenario is supported and it should even work on different OSes, so this looks like a bug.
---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-19-1f2268e10ba7> in <module>
----> 1 r('/download.py', '')

/usr/lib/python3.7/site-packages/rpyc/core/netref.py in __call__(_self, *args, **kwargs)
    251         def __call__(_self, *args, **kwargs):
    252             kwargs = tuple(kwargs.items())
--> 253             return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
    254         __call__.__doc__ = doc
    255         return __call__

/usr/lib/python3.7/site-packages/rpyc/core/netref.py in syncreq(proxy, handler, *args)
     74     """
     75     conn = object.__getattribute__(proxy, "____conn__")
---> 76     return conn.sync_request(handler, proxy, *args)
     77 
     78 

/usr/lib/python3.7/site-packages/rpyc/core/protocol.py in sync_request(self, handler, *args)
    467         """
    468         timeout = self._config["sync_request_timeout"]
--> 469         return self.async_request(handler, *args, timeout=timeout).value
    470 
    471     def _async_request(self, handler, args=(), callback=(lambda a, b: None)):  # serving

/usr/lib/python3.7/site-packages/rpyc/core/async_.py in value(self)
    100         self.wait()
    101         if self._is_exc:
--> 102             raise self._obj
    103         else:
    104             return self._obj

SystemError: unknown opcode

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/root/tmp.Q3FMlTXe62/rpyc/core/protocol.py", line 320, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/root/tmp.Q3FMlTXe62/rpyc/core/protocol.py", line 593, in _handle_call
    return obj(*args, **dict(kwargs))
  File "<ipython-input-17-df17b43e7680>", line 3, in calcMd5
SystemError: unknown opcode

Environment
RPyC 4.1.4 (Using zero-deploy, so both sides have same version)
Server Python: 3.6.6 (default, Mar 10 2019, 13:45:47) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)
Client Python: 3.7.7 (default, May 17 2020, 22:21:12) [GCC 9.3.0] + ipython3
Server OS: CentOS 7
Client OS: Gentoo Linux
Minimal example
Server:
Connection is established via Paramiko.
from plumbum.machines.paramiko_machine import ParamikoMachine                                                   

from rpyc.utils.zerodeploy import DeployedServer 

from paramiko.client import AutoAddPolicy 

mach = ParamikoMachine("myserver", user="user", password="password", load_system_host_keys=False, missing_host_policy=AutoAddPolicy)                                                                                       

server = DeployedServer(mach)                                                                                  

conn = server.classic_connect()

Then I define a simple file MD5 calculation function that I want to run on remote host.
def calcMd5(file_name, prefix): 
    import hashlib 
    md5obj = hashlib.md5(prefix.encode('utf-8')) 
    with open(file_name, 'rb') as f: 
        chunk_size = 1 << 20 
        while 1: 
            s = f.read(chunk_size) 
            if not s: 
                break 
            md5obj.update(s) 
    return md5obj.hexdigest()                                                                                                                

r = conn.teleport(calcMd5)                          

Then, attempting to invoke teleported function results in SystemError: Unknown opcode on remote:
In [19]: r('/download.py', '')
This yields an above stack track about unknown opcode. Still, simpler functions work perfectly.
In [8]: r = conn.teleport(lambda x: x**2)                                                                               

In [9]: r(2)                                                                                                            
Out[9]: 4

Closing comment (2020-12-26T06:34:50Z): When attempting to teleport to a function for a Windows client to a Linux server, a segmentation fault server side and a EOFError thrown on the client side. The issue reported in #424 is a duplicate of this issue.