Repository: secdev/scapy

Issue:
TestSockets are not closed in many tests
I've been investigating issues failures on PyPy and it appears it fails because we open too many pipes during the tests:

![image](https://user-images.githubusercontent.com/10530980/177212820-4fc0eb78-0dc0-42af-bf58-20106c315ff7.png)

Why PyPy and not CPython you may ask? Because PyPy has a different `gc` (see [this](https://doc.pypy.org/en/latest/cpython_differences.html)) that doesn't call `__del__` as early as CPython (as soon as the socket is unreferenced), so sockets pile up.

It's easy to find tons of unclosed sockets in the tests (`grep -R TestSocket` then a little bit of looking around)
https://github.com/secdev/scapy/blob/9420c2229bf5330c2cc580f114f63f920a68db10/test/contrib/isotpscan.uts#L72
https://github.com/secdev/scapy/blob/9420c2229bf5330c2cc580f114f63f920a68db10/test/contrib/isotpscan.uts#L163-L164

So we should probably perform some cleanups on those. Feel free to contribute !


Discussion:
Thanks a lot for this investigation.. I'll take care of it, since I caused these issues. 

What do you recommend? Using `with Testsocket() as sock:` all over the place?

You could do that or just call `.close()`, ideally in a `try` `finally` but as those are tests it's fine if you don't. (we would have a failure anyways)

It's alright those happen a lot. At some point automatons broke down entirely because of a similar issue.. they're pretty hard to troubleshoot
found a easy solution.. hope it holds. 
Okay that was clearly not enough. Tests on PyPy2 are still crashing after thousands of TestSocket are hold in memory.
There is a bug somewhere in an automotive test that keeps creating some and doesn't close them

For instance: https://github.com/secdev/scapy/runs/7296916202?check_suite_focus=true
Thanks.. I'll have a look!


This log looked like a race-condition was triggered... I've added some debug code to fail fast, once this error happens again. 
I do not believe that the issue here is that it fails to close the sockets, the real issue is that there are thousand and thousands of sockets to close in one go in the first place. You can't rely on `__del__` on PyPy to clear the sockets for you, because of limitations, so you're supposed to have closed them manually beforehand.
In the current version on master, I'm already calling `close` manually on all opened sockets after each testcase:
https://github.com/secdev/scapy/blob/2c92b0350ab5df8ea0adc164fb4441c979bec568/test/contrib/automotive/scanner/uds_scanner.uts#L82

This should ensure, that all sockets get closed, but maybe I'm missing something. 
The current CI results, clearly show, that OSX and Windows CI works as expected. Only on Linux based systems, somehow the objects doesn't get destroyed. 

Resolving patch:
diff --git a/scapy/contrib/automotive/scanner/executor.py b/scapy/contrib/automotive/scanner/executor.py
--- a/scapy/contrib/automotive/scanner/executor.py
+++ b/scapy/contrib/automotive/scanner/executor.py
@@ -160,16 +160,20 @@ def reconnect(self):
             raise Scapy_Exception(
                 "Socket closed even after reconnect. Stop scan!")
 
-    def execute_test_case(self, test_case):
-        # type: (AutomotiveTestCaseABC) -> None
+    def execute_test_case(self, test_case, kill_time=None):
+        # type: (AutomotiveTestCaseABC, Optional[float]) -> None
         """
         This function ensures the correct execution of a testcase, including
         the pre_execute, execute and post_execute.
         Finally the testcase is asked if a new edge or a new testcase was
         generated.
+
         :param test_case: A test case to be executed
+        :param kill_time: If set, this defines the maximum execution time for
+                          the current test_case
         :return: None
         """
+
         test_case.pre_execute(
             self.socket, self.target_state, self.configuration)
 
@@ -178,6 +182,12 @@ def execute_test_case(self, test_case):
         except KeyError:
             test_case_kwargs = dict()
 
+        if kill_time:
+            max_execution_time = max(int(kill_time - time.time()), 5)
+            cur_execution_time = test_case_kwargs.get("execution_time", 1200)
+            test_case_kwargs["execution_time"] = min(max_execution_time,
+                                                     cur_execution_time)
+
         log_interactive.debug("[i] Execute test_case %s with args %s",
                               test_case.__class__.__name__, test_case_kwargs)
 
@@ -220,13 +230,14 @@ def scan(self, timeout=None):
         :return: None
         """
         kill_time = time.time() + (timeout or 0xffffffff)
+        log_interactive.debug("[i] Set kill_time to %s" % time.ctime(kill_time))
         while kill_time > time.time():
             test_case_executed = False
             log_interactive.debug("[i] Scan paths %s", self.state_paths)
             for p, test_case in product(
                     self.state_paths, self.configuration.test_cases):
                 log_interactive.info("[i] Scan path %s", p)
-                terminate = kill_time < time.time()
+                terminate = kill_time <= time.time()
                 if terminate:
                     log_interactive.debug(
                         "[-] Execution time exceeded. Terminating scan!")
@@ -245,7 +256,7 @@ def scan(self, timeout=None):
                         continue
                     log_interactive.info(
                         "[i] Execute %s for path %s", str(test_case), p)
-                    self.execute_test_case(test_case)
+                    self.execute_test_case(test_case, kill_time)
                     test_case_executed = True
                 except (OSError, ValueError, Scapy_Exception) as e:
                     log_interactive.critical("[-] Exception: %s", e)


Test patch:
diff --git a/test/contrib/automotive/gm/scanner.uts b/test/contrib/automotive/gm/scanner.uts
--- a/test/contrib/automotive/gm/scanner.uts
+++ b/test/contrib/automotive/gm/scanner.uts
@@ -8,7 +8,7 @@
 import itertools
 from scapy.contrib.isotp import ISOTPMessageBuilder
 
-from test.testsocket import TestSocket, cleanup_testsockets
+from test.testsocket import TestSocket, cleanup_testsockets, open_test_sockets
 
 
 ############
@@ -43,6 +43,7 @@ def executeScannerInVirtualEnvironment(supported_responses, enumerators, **kwarg
                     ecu.close()
                 except Exception:
                     pass
+                assert len(open_test_sockets) < 3
                 ecu = TestSocket(GMLAN)
                 tester.pair(ecu)
                 answering_machine = EcuAnsweringMachine(supported_responses=supported_responses, main_socket=ecu, basecls=GMLAN, verbose=False)
@@ -55,11 +56,13 @@ def executeScannerInVirtualEnvironment(supported_responses, enumerators, **kwarg
     sim.start()
     answering_machine_started.wait(timeout=10)
     try:
+        assert len(open_test_sockets) < 3
         scanner = GMLAN_Scanner(
             tester, reset_handler=reset,
             test_cases=enumerators, timeout=0.2, retry_if_none_received=True,
             unittest=True, **kwargs)
         for i in range(12):
+            print("Starting scan")
             scanner.scan(timeout=10)
             if scanner.scan_completed:
                 print("Scan completed after %d iterations" % i)
diff --git a/test/contrib/automotive/scanner/uds_scanner.uts b/test/contrib/automotive/scanner/uds_scanner.uts
--- a/test/contrib/automotive/scanner/uds_scanner.uts
+++ b/test/contrib/automotive/scanner/uds_scanner.uts
@@ -7,7 +7,8 @@
 import io
 import pickle
 from scapy.contrib.isotp import ISOTPMessageBuilder
-from test.testsocket import TestSocket, cleanup_testsockets, UnstableSocket, open_test_sockets
+from test.testsocket import TestSocket, cleanup_testsockets, UnstableSocket
+from scapy.automaton import ObjectPipe
 
 ############
 ############
@@ -27,61 +28,51 @@ load_layer("can")
 = Define Testfunction
 
 def executeScannerInVirtualEnvironment(supported_responses, enumerators, unstable_socket=True, **kwargs):
+    tester_obj_pipe = ObjectPipe(name="TesterPipe")
+    ecu_obj_pipe = ObjectPipe(name="ECUPipe")
     TesterSocket = UnstableSocket if unstable_socket else TestSocket
-    answering_machine_started = threading.Event()
-    answering_machine_running = threading.Event()
-    answering_machine_running.set()
+    tester = TesterSocket(UDS, tester_obj_pipe)
+    ecu = TestSocket(UDS, ecu_obj_pipe)
+    tester.pair(ecu)
+    answering_machine = EcuAnsweringMachine(
+        supported_responses=supported_responses, main_socket=ecu,
+        basecls=UDS, verbose=False)
     def reset():
         answering_machine.state.reset()
         answering_machine.state["session"] = 1
         sniff(timeout=0.001, opened_socket=[ecu, tester])
     def reconnect():
-        global tester
         try:
             tester.close()
         except Exception:
             pass
-        tester = TesterSocket(UDS)
+        tester = TesterSocket(UDS, tester_obj_pipe)
         ecu.pair(tester)
         return tester
     def answering_machine_thread():
-        global ecu
-        global answering_machine
-        while answering_machine_running.is_set():
-            try:
-                try:
-                    ecu.close()
-                except Exception:
-                    pass
-                assert len(open_test_sockets) < 3
-                ecu = TestSocket(UDS)
-                tester.pair(ecu)
-                answering_machine = EcuAnsweringMachine(supported_responses=supported_responses, main_socket=ecu, basecls=UDS, verbose=False)
-                answering_machine_started.set()
-                answering_machine(timeout=120, stop_filter=lambda x: bytes(x) == b"\xff\xff\xff")
-                return
-            except OSError:
-                continue
+        answering_machine(
+            timeout=120, stop_filter=lambda x: bytes(x) == b"\xff\xff\xff")
     sim = threading.Thread(target=answering_machine_thread)
     try:
-        assert len(open_test_sockets) < 3
         sim.start()
-        answering_machine_started.wait(timeout=10)
         scanner = UDS_Scanner(
-            reconnect(), reset_handler=reset, reconnect_handler=reconnect, test_cases=enumerators, timeout=0.1,
+            tester, reset_handler=reset, reconnect_handler=reconnect,
+            test_cases=enumerators, timeout=0.1,
             retry_if_none_received=True, unittest=True,
             **kwargs)
-        for i in range(100):
+        for i in range(12):
+            print("Starting scan")
             scanner.scan(timeout=10)
             if scanner.scan_completed:
                 print("Scan completed after %d iterations" % i)
                 break
     finally:
-        answering_machine_running.clear()
         ecu.ins.send(Raw(b"\xff\xff\xff"))
         sim.join(timeout=2)
         assert not sim.is_alive()
         cleanup_testsockets()
+        tester_obj_pipe.close()
+        ecu_obj_pipe.close()
     if six.PY3 and LINUX:
         pickle_test(scanner)
     return scanner
@@ -156,22 +147,22 @@ assert e._retry_pkt[s] == None
 = Test UDS_SA_XOR_Enumerator stand alone mode
 
 TesterSocket = TestSocket
-ecu = TestSocket(UDS)
-tester = TesterSocket(UDS)
-ecu.pair(tester)
-answering_machine = EcuAnsweringMachine(supported_responses=mEcu.supported_responses, main_socket=ecu, basecls=UDS, verbose=False)
+ecu_sock = TestSocket(UDS)
+mTester = TesterSocket(UDS)
+ecu_sock.pair(mTester)
+answering_machine = EcuAnsweringMachine(supported_responses=mEcu.supported_responses, main_socket=ecu_sock, basecls=UDS, verbose=False)
 sim = threading.Thread(target=answering_machine, kwargs={'timeout': 1000, "stop_filter": lambda x: bytes(x) == b"\xff\xff\xff"})
 sim.start()
 try:
-    resp = tester.sr1(UDS()/UDS_TP(b"\x00"), verbose=False, timeout=1)
+    resp = mTester.sr1(UDS()/UDS_TP(b"\x00"), verbose=False, timeout=1)
     print(repr(resp))
     assert resp and resp.service != 0x7f
-    resp = tester.sr1(UDS()/UDS_DSC(diagnosticSessionType=3), verbose=False, timeout=1)
+    resp = mTester.sr1(UDS()/UDS_DSC(diagnosticSessionType=3), verbose=False, timeout=1)
     print(repr(resp))
     assert resp and resp.service != 0x7f
-    assert UDS_SA_XOR_Enumerator().get_security_access(tester, 1)
+    assert UDS_SA_XOR_Enumerator().get_security_access(mTester, 1)
 finally:
-    tester.send(Raw(b"\xff\xff\xff"))
+    mTester.send(Raw(b"\xff\xff\xff"))
     sim.join(timeout=2)
     cleanup_testsockets()
 
diff --git a/test/testsocket.py b/test/testsocket.py
--- a/test/testsocket.py
+++ b/test/testsocket.py
@@ -28,14 +28,17 @@ class TestSocket(SuperSocket):
 
     test_socket_mutex = Lock()
 
-    def __init__(self, basecls=None):
-        # type: (Optional[Type[Packet]]) -> None
+    def __init__(self,
+                 basecls=None,  # type: Optional[Type[Packet]]
+                 external_obj_pipe=None  # type: Optional[ObjectPipe[bytes]]
+                 ):
+        # type: (...) -> None
         global open_test_sockets
-        super(TestSocket, self).__init__()
         self.basecls = basecls
         self.paired_sockets = list()  # type: List[TestSocket]
-        self.closed = False
-        self.ins = self.outs = cast(socket, ObjectPipe(name="TestSocket"))
+        self.ins = external_obj_pipe or ObjectPipe(name="TestSocket")  # type: ignore
+        self._has_external_obj_pip = external_obj_pipe is not None
+        self.outs = None
         open_test_sockets.append(self)
 
     def __enter__(self):
@@ -51,27 +54,30 @@ def close(self):
         # type: () -> None
         global open_test_sockets
 
-        with self.test_socket_mutex:
-            if self.closed:
-                return
+        if self.closed:
+            return
 
-            self.closed = True
-            for s in self.paired_sockets:
-                try:
-                    s.paired_sockets.remove(self)
-                except (ValueError, AttributeError, TypeError):
-                    pass
-            super(TestSocket, self).close()
+        for s in self.paired_sockets:
             try:
-                open_test_sockets.remove(self)
+                s.paired_sockets.remove(self)
             except (ValueError, AttributeError, TypeError):
                 pass
 
+        if not self._has_external_obj_pip:
+            super(TestSocket, self).close()
+        else:
+            # We don't close external object pipes
+            self.closed = True
+
+        try:
+            open_test_sockets.remove(self)
+        except (ValueError, AttributeError, TypeError):
+            pass
+
     def pair(self, sock):
         # type: (TestSocket) -> None
-        with self.test_socket_mutex:
-            self.paired_sockets += [sock]
-            sock.paired_sockets += [self]
+        self.paired_sockets += [sock]
+        sock.paired_sockets += [self]
 
     def send(self, x):
         # type: (Packet) -> int
@@ -89,10 +95,6 @@ def recv_raw(self, x=MTU):
         """Returns a tuple containing (cls, pkt_data, time)"""
         return self.basecls, self.ins.recv(0), time.time()
 
-    def __del__(self):
-        # type: () -> None
-        self.close()
-
     @staticmethod
     def select(sockets, remain=conf.recv_poll_rate):
         # type: (List[SuperSocket], Optional[float]) -> List[SuperSocket]
@@ -105,9 +107,12 @@ class UnstableSocket(TestSocket):
     packets on recv.
     """
 
-    def __init__(self, basecls=None):
-        # type: (Optional[Type[Packet]]) -> None
-        super(UnstableSocket, self).__init__(basecls)
+    def __init__(self,
+                 basecls=None,  # type: Optional[Type[Packet]]
+                 external_obj_pipe=None  # type: Optional[ObjectPipe[bytes]]
+                 ):
+        # type: (...) -> None
+        super(UnstableSocket, self).__init__(basecls, external_obj_pipe)
         self.no_error_for_x_rx_pkts = 10
         self.no_error_for_x_tx_pkts = 10
 
@@ -147,9 +152,8 @@ def cleanup_testsockets():
     """
     Helper function to remove TestSocket objects after a test
     """
-    count = 100
-    while len(open_test_sockets) and count > 0:
-        print(open_test_sockets)
-        count -= 1
+    count = max(len(open_test_sockets), 1)
+    while len(open_test_sockets) and count:
         sock = open_test_sockets[0]
         sock.close()
+        count -= 1