Repository: secdev/scapy

Issue:
Timeout waiting for a UDS response
### Brief description

I have noticed that sometimes when I send a UDS packet using DoIP socket+sr1, scapy fails to give back the response even though the wireshark running in parallel shows that the response was sent from the ECU and received by the client machine.

There is no other network traffic on the interface (the interface handles UDS stuff exclusively) so I dont think sr1 is getting bogged down facing too much data.



### Scapy version

2.5.0

### Python version

3.11.2

### Operating system

Linux  5.15.32-v8+

### Additional environment information

_No response_

### How to reproduce

I have this example uds-dsc-test.py script which I run consecutively with a shell script.

```
from scapy.contrib.automotive.uds import *
from scapy.contrib.automotive.doip import *

IP = "169.254.98.64"
target_add = 0x1010

socket = DoIPSocket(IP)
pkt = DoIP(payload_type=0x8001, source_address=0xe80, target_address=target_add)/UDS()/UDS_DSC(diagnosticSessionType="extendedDiagnosticSession")
response = socket.sr1(pkt, verbose=False, timeout=4)
if response:
    print(response.show(dump=True))
else:
    print("UDS Timeout")
```
```
#!/bin/bash

CMD="python3 uds-dsc-test.py"

for ((i=1; i<=100; i++))
do
        # Run your command here
        echo "Running command: $CMD- $i"
        output=$($CMD)
        if grep -q "UDS Timeout" <<< "$output"; then
                echo "UDS Timeout"
                echo "$output" >  /home/sa/"$i"
		exit
        fi
        sleep 0.2
done
```

### Actual result

stdout

```
./test.sh 
Running command: python3 uds-dsc-test.py- 1
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 2
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 3
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 4
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 5
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 6
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 7
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 8
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 9
INFO: Routing activation successful! Target address set to: 0x3107
UDS Timeout
```

This is just output of one sample run. In other cases it might take 50 runs to see a timeout. But in everycase wireshark confirms thats the response packet was received. 

Also I can mitigate this behavior by using ```retry``` in sr1. It brings down the frequency of timeout but it still happens (approx once every 50 times). 

### Expected result

_No response_

### Related resources

_No response_


Discussion:
Thanks for reaching out. Debugging this kind of bug seems kinda difficult by the given information. Can you share a wireshark pcap? Can you check, if Scapy sees the response packet and it might not get matched by sendrecv. You could do this with a raw socket on the same interface. This would allow to localise the issue. Thanks On 6. Jun 2023, at 19:19, sa-isd ***@***.***> wrote:﻿
Brief description
I have noticed that sometimes when I send a UDS packet using DoIP socket+sr1, scapy fails to give back the response even though the wireshark running in parallel shows that the response was sent from the ECU and received by the client machine.
There is no other network traffic on the interface (the interface handles UDS stuff exclusively) so I dont think sr1 is getting bogged down facing too much data.
Scapy version
2.5.0
Python version
3.11.2
Operating system
Linux  5.15.32-v8+
Additional environment information
No response
How to reproduce
I have this example uds-dsc-test.py script which I run consecutively with a shell script.
from scapy.contrib.automotive.uds import *
from scapy.contrib.automotive.doip import *

IP = "169.254.98.64"
target_add = 0x1010

socket = DoIPSocket(IP)
pkt = DoIP(payload_type=0x8001, source_address=0xe80, target_address=target_add)/UDS()/UDS_DSC(diagnosticSessionType="extendedDiagnosticSession")
response = socket.sr1(pkt, verbose=False, timeout=4)
if response:
    print(response.show(dump=True))
else:
    print("UDS Timeout")

#!/bin/bash

CMD="python3 uds-dsc-test.py"

for ((i=1; i<=100; i++))
do
        # Run your command here
        echo "Running command: $CMD- $i"
        output=$($CMD)
        if grep -q "UDS Timeout" <<< "$output"; then
                echo "UDS Timeout"
                echo "$output" >  /home/sa/"$i"
		exit
        fi
        sleep 0.2
done

Actual result
stdout
./test.sh 
Running command: python3 uds-dsc-test.py- 1
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 2
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 3
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 4
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 5
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 6
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 7
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 8
INFO: Routing activation successful! Target address set to: 0x3107
Running command: python3 uds-dsc-test.py- 9
INFO: Routing activation successful! Target address set to: 0x3107
UDS Timeout

This is just output of one sample run. In other cases it might take 50 runs to see a timeout. But in everycase wireshark confirms thats the response packet was received.
Also I can mitigate this behavior by using retry in sr1. It brings down the frequency of timeout but it still happens (approx once every 50 times).
Expected result
No response
Related resources
No response

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>
sure...thanks for the hint... give me some time and I'll try to report back with some more info.
I have investigated the issue a little more and have found the following.

1. I started scapy ``sniff`` alongwith my test program to confirm that scapy is able to capture all packets and it did.
2. I enabled the ```conf.debug_match``` config option and printed the resulting packet in ```debug.recv``` which showed me a strange problem.

Following is the stdout of a normal request/reply which didnt have anytimeout
```
INFO: Routing activation successful! Target address set to: 0x3107
recv
###[ DoIP ]### 
  protocol_version= 0x2
  inverse_version= 0xfd
  payload_type= Diagnostic message ACK
  payload_length= 5
  source_address= 0x1010
  target_address= 0xe80
  ack_code  = ACK
  previous_msg= 

sent
match
0000 DoIP / UDS / UDS_DSC ==> DoIP / UDS / UDS_DSCPR
``` 
and this is the output incase of a timeout
```
UDS Timeout
recv
###[ DoIP ]### 
  protocol_version= 0x2
  inverse_version= 0xfd
  payload_type= Diagnostic message ACK
  payload_length= 5
  source_address= 0x1010
  target_address= 0xe80
  ack_code  = ACK
  previous_msg= 02fd80010000000a10100e805003003201f4
```
In every timeoutcase, the ```debug.recv``` has the above packet. The value ```02fd80010000000a10100e805003003201f4```  actually equals DoIP+(the expected reply) UDS. It seems scapy is concatenating the UDS reply to the previous DoIP ACK msg. 

I don't believe this is an ECU/network problem because I have recorded pcaps with both scapy and tcpdump and they all capture the packets separately and no packet has a ```previous_msg``` field. 

this is the test program I used for debugging
```
from scapy.contrib.automotive.uds import *
from scapy.contrib.automotive.doip import *
from scapy.config import conf
from scapy.all import *

conf.debug_match = True

IP = "169.254.98.64"
target_add = 0x1010

socket = DoIPSocket(IP)
pkt = DoIP(payload_type=0x8001, source_address=0xe80, target_address=target_add)/UDS()/UDS_DSC(diagnosticSessionType="extendedDiagnosticSession")
response = socket.sr1(pkt, verbose=False, timeout=4)
if response:
    # print(response.show(dump=True))
    print("recv")
    debug.recv[0].show()

    print("sent")
    debug.sent.show()

    print("match")
    debug.match.show()
else:
    print("UDS Timeout")
    print("recv")
    debug.recv[0].show()

    print("sent")
    debug.sent.show()

    print("match")
    debug.match.show()
```
Ok,Can you please check, if your issue is solved if you extend this line with your payload type code:         if self.payload_type == 0x8001:scapy/doip.py at a45836f97ef61baa2f17590d19c710414af6b9ef · secdev/scapygithub.com--  Mit freundlichen Grüßen  Dr. Nils WeißOn 9. Jun 2023, at 11:48, sa-isd ***@***.***> wrote:﻿
I have investigated the issue a little more and have found the following.

I started scapy sniff alongwith my test program to confirm that scapy is able to capture all packets and it did.
I enabled the conf.debug_match config option and printed the resulting packet in debug.recv which showed me a strange problem.

Following is the stdout of a normal request/reply which didnt have anytimeout
INFO: Routing activation successful! Target address set to: 0x3107
recv
###[ DoIP ]### 
  protocol_version= 0x2
  inverse_version= 0xfd
  payload_type= Diagnostic message ACK
  payload_length= 5
  source_address= 0x1010
  target_address= 0xe80
  ack_code  = ACK
  previous_msg= 

sent
match
0000 DoIP / UDS / UDS_DSC ==> DoIP / UDS / UDS_DSCPR

and this is the output incase of a timeout
UDS Timeout
recv
###[ DoIP ]### 
  protocol_version= 0x2
  inverse_version= 0xfd
  payload_type= Diagnostic message ACK
  payload_length= 5
  source_address= 0x1010
  target_address= 0xe80
  ack_code  = ACK
  previous_msg= 02fd80010000000a10100e805003003201f4

In every timeoutcase, the debug.recv has the above packet. The value 02fd80010000000a10100e805003003201f4  actually equals DoIP+(the expected reply) UDS. It seems scapy is concatenating the UDS reply to the previous DoIP ACK msg.
I don't believe this is an ECU/network problem because I have recorded pcaps with both scapy and tcpdump and they all capture the packets separately and no packet has a previous_msg field.
this is the test program I used for debugging
from scapy.contrib.automotive.uds import *
from scapy.contrib.automotive.doip import *
from scapy.config import conf
from scapy.all import *

conf.debug_match = True

IP = "169.254.98.64"
target_add = 0x1010

socket = DoIPSocket(IP)
pkt = DoIP(payload_type=0x8001, source_address=0xe80, target_address=target_add)/UDS()/UDS_DSC(diagnosticSessionType="extendedDiagnosticSession")
response = socket.sr1(pkt, verbose=False, timeout=4)
if response:
    # print(response.show(dump=True))
    print("recv")
    debug.recv[0].show()

    print("sent")
    debug.sent.show()

    print("match")
    debug.match.show()
else:
    print("UDS Timeout")
    print("recv")
    debug.recv[0].show()

    print("sent")
    debug.sent.show()

    print("match")
    debug.match.show()


—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
i added 0x8002  to the  2 ```if self.payload_type == 0x8001:```  lines in  doip.py but unfortunately the issue still persists.
```
    def answers(self, other):
        # type: (Packet) -> int
        """DEV: true if self is an answer from other"""
        if isinstance(other, type(self)):
            if self.payload_type == 0:
                return 1

            matches = [(4, 1), (4, 2), (4, 3), (6, 5), (8, 7),
                       (0x4002, 0x4001), (0x4004, 0x4003),
                       (0x8001, 0x8001), (0x8003, 0x8001)]
            if (self.payload_type, other.payload_type) in matches:
                # if self.payload_type == 0x8001:
                if self.payload_type == 0x8001 or self.payload_type == 0x8002: 
                    return self.payload.answers(other.payload)
                return 1
        return 0
```
and
```
    def extract_padding(self, s):
        # type: (bytes) -> Tuple[bytes, Optional[bytes]]
        # if self.payload_type == 0x8001: 
        if self.payload_type == 0x8001 or self.payload_type == 0x8002:
            return s[:self.payload_length - 4], None
        else:
            return b"", None
```
Sorry, my message was a little bit misleading. I just wanted you to change only the one line I marked in the link. Your other change could lead to new issues. Can you please share a pcap file so that I can investigate the issue?--  Mit freundlichen Grüßen  Dr. Nils WeißOn 9. Jun 2023, at 21:20, sa-isd ***@***.***> wrote:﻿
i added 0x8002  to the  2 if self.payload_type == 0x8001:  lines in  doip.py but unfortunately the issue still persists.
    def answers(self, other):
        # type: (Packet) -> int
        """DEV: true if self is an answer from other"""
        if isinstance(other, type(self)):
            if self.payload_type == 0:
                return 1

            matches = [(4, 1), (4, 2), (4, 3), (6, 5), (8, 7),
                       (0x4002, 0x4001), (0x4004, 0x4003),
                       (0x8001, 0x8001), (0x8003, 0x8001)]
            if (self.payload_type, other.payload_type) in matches:
                # if self.payload_type == 0x8001:
                if self.payload_type == 0x8001 or self.payload_type == 0x8002: 
                    return self.payload.answers(other.payload)
                return 1
        return 0

and
    def extract_padding(self, s):
        # type: (bytes) -> Tuple[bytes, Optional[bytes]]
        # if self.payload_type == 0x8001: 
        if self.payload_type == 0x8001 or self.payload_type == 0x8002:
            return s[:self.payload_length - 4], None
        else:
            return b"", None


—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>
pcap is available @ https://file.io/xnp10VwttskG

it shows 13 UDS requests and 13 replies. The last UDS reply is the one missed/concatenated by sr1.
The file has been deleted @sa-isd. Also, unrelated but you should not use `IP` as a constant name, since in Scapy `IP()` is the IP(v4) layer. It is not what causes your problem but it might get you in trouble later. Maybe `ADDR` would be a better choice?
sorry about that... uploaded again @ https://drive.google.com/file/d/1WISPwvBOdE3XPEa5Jfp-ZPxGK1EOL6jc/view?usp=sharing

thanks for the tip... ill rework the test program... 
I have the exact same issue. 
But it doesn't occur every time. What I noticed that this behavior occurs when TCP ACK's are send after DoIP and UDS response. (Sorry, can not provide pcap files, only screenshots).

Here Scapy gets the UDS response correctly:
![wireshark_doip_ok](https://github.com/secdev/scapy/assets/20972825/abcd3aaf-ad4a-4f2c-bb5d-06465158dba1)
![scapy_ok](https://github.com/secdev/scapy/assets/20972825/316963bd-e912-4ea9-9547-5fc4fd4e829c)


Here sr1 does not get any answer but socket gets DoIP packet with UDS response in "previous_msg":
![wireshark_doip_nok](https://github.com/secdev/scapy/assets/20972825/66b4077b-9cfc-4e50-a2e9-3d334fb5110f)
![scapy_nok](https://github.com/secdev/scapy/assets/20972825/de001ce7-ba67-42b9-8963-c6ca788ad2d1)



Seems for me that for "quick fix" it's needed to limit "previous_msg" length to 3 bytes, but I need to confirm it with ISO 13400-2 standard first.
Sorry for the delay. The issues is due to the use of a StreamSocket in DoIPSockets. Scapy is "Packet" oriented and always tries to read as much data as possible from a socket, which is "OK" if we are below layer 4. Since DoIPSockets are designed to be able to flawlessly communicate with an ECU, we are using TCPSockets. If your ECU is responding "to fast" one `recv()` call will consume multiple DoIP packets in Scapy. I didn't encountered this issue in the past on my ECUs, which I used for testing the implementation. 

However, to fix this issue, the DoIPSocket recv() function needs to be redesigned. Feel free to propose a fix. 

As soon as I find the time, I will work on a fix, as well. 
@polybassa an option would be, if that's possible, to `bind_layer()` the last layers to `Padding`.
Per ISO 13400-2, length of previous message can be from 0 to d(ata??) - 5
![Screenshot_20230622-112627_MuPDF viewer](https://github.com/secdev/scapy/assets/12693890/3ae5325d-5cd0-4cb0-aea4-15be72ce7991)
So maybe cutting previous_msg might help here.

Resolving patch:
diff --git a/scapy/contrib/automotive/bmw/hsfz.py b/scapy/contrib/automotive/bmw/hsfz.py
--- a/scapy/contrib/automotive/bmw/hsfz.py
+++ b/scapy/contrib/automotive/bmw/hsfz.py
@@ -86,6 +86,27 @@ def __init__(self, ip='127.0.0.1', port=6801):
         s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
         s.connect((self.ip, self.port))
         StreamSocket.__init__(self, s, HSFZ)
+        self.buffer = b""
+
+    def recv(self, x=MTU):
+        # type: (int) -> Optional[Packet]
+        if self.buffer:
+            len_data = self.buffer[:4]
+        else:
+            len_data = self.ins.recv(4, socket.MSG_PEEK)
+            if len(len_data) != 4:
+                return None
+
+        len_int = struct.unpack(">I", len_data)[0]
+        len_int += 6
+        self.buffer += self.ins.recv(len_int - len(self.buffer))
+
+        if len(self.buffer) != len_int:
+            return None
+
+        pkt = self.basecls(self.buffer)  # type: Packet
+        self.buffer = b""
+        return pkt
 
 
 class UDS_HSFZSocket(HSFZSocket):
diff --git a/scapy/contrib/automotive/doip.py b/scapy/contrib/automotive/doip.py
--- a/scapy/contrib/automotive/doip.py
+++ b/scapy/contrib/automotive/doip.py
@@ -287,12 +287,33 @@ def __init__(self, ip='127.0.0.1', port=13400, activate_routing=True,
         self.ip = ip
         self.port = port
         self.source_address = source_address
+        self.buffer = b""
         self._init_socket()
 
         if activate_routing:
             self._activate_routing(
                 source_address, target_address, activation_type, reserved_oem)
 
+    def recv(self, x=MTU):
+        # type: (int) -> Optional[Packet]
+        if self.buffer:
+            len_data = self.buffer[:8]
+        else:
+            len_data = self.ins.recv(8, socket.MSG_PEEK)
+            if len(len_data) != 8:
+                return None
+
+        len_int = struct.unpack(">I", len_data[4:8])[0]
+        len_int += 8
+        self.buffer += self.ins.recv(len_int - len(self.buffer))
+
+        if len(self.buffer) != len_int:
+            return None
+
+        pkt = self.basecls(self.buffer)  # type: Packet
+        self.buffer = b""
+        return pkt
+
     def _init_socket(self, sock_family=socket.AF_INET):
         # type: (int) -> None
         s = socket.socket(sock_family, socket.SOCK_STREAM)


Test patch:
diff --git a/test/contrib/automotive/bmw/hsfz.uts b/test/contrib/automotive/bmw/hsfz.uts
--- a/test/contrib/automotive/bmw/hsfz.uts
+++ b/test/contrib/automotive/bmw/hsfz.uts
@@ -74,3 +74,32 @@ assert pkt.dst == 0x10
 assert pkt.type == 1
 assert pkt.securitySeed == b"0" * 0xfff00
 
+= Test HSFZSocket
+
+
+server_up = threading.Event()
+def server():
+    buffer = bytes(HSFZ(type=1, src=0xf4, dst=0x10) / Raw(b'\x11\x22\x33' * 1024))
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    try:
+        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        sock.bind(('127.0.0.1', 6801))
+        sock.listen(1)
+        server_up.set()
+        connection, address = sock.accept()
+        connection.send(buffer[:1024])
+        time.sleep(0.1)
+        connection.send(buffer[1024:])
+        connection.close()
+    finally:
+        sock.close()
+
+server_thread = threading.Thread(target=server)
+server_thread.start()
+server_up.wait(timeout=1)
+sock = HSFZSocket()
+
+pkts = sock.sniff(timeout=1, count=1)
+assert len(pkts) == 1
+assert len(pkts[0]) > 2048
diff --git a/test/contrib/automotive/doip.uts b/test/contrib/automotive/doip.uts
--- a/test/contrib/automotive/doip.uts
+++ b/test/contrib/automotive/doip.uts
@@ -380,3 +380,30 @@ assert req.hashret() == resp.hashret()
 # exclude TCP layer from answers check
 assert resp[3].answers(req[3])
 assert not req[3].answers(resp[3])
+
+= Test DoIPSocket
+
+server_up = threading.Event()
+def server():
+    buffer = b'\x02\xfd\x80\x02\x00\x00\x00\x05\x00\x00\x00\x00\x00\x02\xfd\x80\x01\x00\x00\x00\n\x10\x10\x0e\x80P\x03\x002\x01\xf4'
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    try:
+        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
+        sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        sock.bind(('127.0.0.1', 13400))
+        sock.listen(1)
+        server_up.set()
+        connection, address = sock.accept()
+        connection.send(buffer)
+        connection.close()
+    finally:
+        sock.close()
+
+
+server_thread = threading.Thread(target=server)
+server_thread.start()
+server_up.wait(timeout=1)
+sock = DoIPSocket(activate_routing=False)
+
+pkts = sock.sniff(timeout=1, count=2)
+assert len(pkts) == 2