Repository: secdev/scapy

Issue:
TCP packet defragmentation using TCPSession does not show up in Wireshark.
### Brief description

<!-- describe the main issue in one sentence -->
<!-- if possible, describe what components / protocols could be affected by the issue (e.g. wrpcap() + IPv6, it is likely this also affects XXX) -->
At first, I parsed the `pcap` file through `rdpcap()`, however it can`t handle TCP defragmentation.

Under the guidance of #2763  and [https://scapy.readthedocs.io/en/latest/usage.html#advanced-sniffing-sniffing-sessions](advanced-sniffing-sniffing-sessions),  I use `pkt = sniff(offline="xxx.pcap", session=TLS_over_TCP)` to read and the parse pcap file. From the result of `pkt.show()`, it solves TCP fragmentation well.

However, when I try `wrpcap("sniff_dealed_xxx.pcap", pkt)`, the defragmentation results don`t show up in Wireshark.

### Environment

-   Scapy version: `2.4.5rc1.dev55` [6eef12d](https://github.com/secdev/scapy/commit/6eef12d2de02ec4822c19621fd3847a498ca1387)
-   Python version: `3.7.4`
-   Operating System: `Windows 10`

### How to reproduce

<!-- step-by-step explanation or a short script, may reference section 'Related resources' -->

**To be specific,** I used [192.168.165.217_49183_156.38.206.18_443_1593515552.pcap](https://drive.google.com/file/d/1iyIHMY5tWIuNQjggQMawIliQndh9ojB8/view?usp=sharing) provided by @knwng in #2763

You can run the following python code to reproduce the bug:

```python
from scapy.all import *
from scapy.utils import *
from scapy.layers.http import *  # load_layer("http")
load_layer("tls")
conf.tls_session_enable = True
print(conf.version)  # 2.4.5rc1.dev55

class TLS_over_TCP(TLSSession, TCPSession):
    pass

pkts = sniff(offline=r"192.168.165.217_49183_156.38.206.18_443_1593515552.pcap",
             session=TLS_over_TCP)

wrpcap("sniff_dealed_xxx.pcap", pkts)

pkts[5].show()
```


### Actual result


No.6 ~ No.9 packets in the original pcap file are automatically defragmented in Wireshark , as shown in the figure below.

![image](https://user-images.githubusercontent.com/20126304/124115368-4f00b100-daa0-11eb-8cf2-cd94a974f5f2.png)

After `wrpcap("sniff_dealed_xxx.pcap", pkt)`, use Wireshark to view `sniff_dealed_xxx.pcap`

![image](https://user-images.githubusercontent.com/20126304/124116200-3fce3300-daa1-11eb-8d78-fd6035611a9a.png)

We focus on No.6 packet.

It is found that it does merge data packets from No.6 to No.9 into No.6, but there are two problems:
1. [Expert Info (Error/Protocol): IPv4 total length exceeds packet length (781 bytes)]

![image](https://user-images.githubusercontent.com/20126304/124117975-5bd2d400-daa3-11eb-8c39-dd846bd0484d.png)


2. [Expert Info (Warning/Sequence): Previous segment(s) not captured (common at capture start)]

![image](https://user-images.githubusercontent.com/20126304/124118111-8b81dc00-daa3-11eb-858e-8db16b7adb4f.png)

- From the IP layer, the `Total Length：3681` that the TCP segment data is indeed merged
- But at the TCP layer the `TCP payload (741 bytes)` length is still the length of the last TCP packet  before the merge (No.9 in original pcap).

The result of `pkts[5].show()` is

```
###[ Ethernet ]### 
  dst       = 52:55:10:00:02:02
  src       = 52:54:00:12:34:56
  type      = IPv4
###[ IP ]### 
     version   = 4
     ihl       = 5
     tos       = 0x0
     len       = None
     id        = 3549
     flags     = 
     frag      = 0
     ttl       = 64
     proto     = tcp
     chksum    = None
     src       = 156.38.206.18
     dst       = 192.168.165.217
     \options   \
###[ TCP ]### 
        sport     = https
        dport     = 49183
        seq       = 7810902
        ack       = 62833955
        dataofs   = 5
        reserved  = 0
        flags     = PA
        window    = 9000
        chksum    = 0xe1de
        urgptr    = 0
        options   = []
###[ TLS ]### 
           type      = handshake
           version   = TLS 1.2
           len       = 3636
           iv        = b''
           \msg       \
            |###[ TLS Handshake - Server Hello ]### 
            |  msgtype   = server_hello
            |  msglen    = 77
            |  version   = TLS 1.2
            |  gmt_unix_time= Tue, 30 Jun 2020 03:46:06 +0800 (1593488766)
            |  random_bytes= 88dfdc237d27a0ffa2e2b5ec0e93a8e0de015b135b4615312078c6cc
            |  sidlen    = 32
            |  sid       = '`)\x00\x00\\x8aZ\\x90l\\xda\x0b\\xe1\\xec[i\x13\\xa7\\x8e\\xb9a\\x98"\\x8a7L\\x9d\\x90\\xe0\x01\x06c$9'
            |  cipher    = TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
            |  comp      = null
            |  extlen    = 5
            |  \ext       \
            |   |###[ TLS Extension - Renegotiation Indication ]### 
            |   |  type      = renegotiation_info
            |   |  len       = 1
            |   |  reneg_conn_len= 0
            |   |  renegotiated_connection= ''
            |###[ TLS Handshake - Certificate ]### 
            |  msgtype   = certificate
            |  msglen    = 3214
            |  certslen  = 3211
            |  certs     = [(1646, [X.509 Cert. Subject:/OU=PositiveSSL Wildcard/CN=*.mql5.net, Issuer:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA]), (1559, [X.509 Cert. Subject:/C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA, Issuer:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority])]
            |###[ TLS Handshake - Server Key Exchange ]### 
            |  msgtype   = server_key_exchange
            |  msglen    = 329
            |  \params    \
            |   |###[ Server ECDH parameters - Named Curve ]### 
            |   |  curve_type= named_curve
            |   |  named_curve= secp256r1
            |   |  pointlen  = 65
            |   |  point     = '\x04\x13\x1c\x02q\\xd4m\\x97\x01\\x99\\xcf\\xf2\\x80G\\xa8\\xe1\\xdf\x1ak\\xbf\x1fJ\\xf9\\x9e\\xd0\x02\x01W\\x9d\\xb8\\xbc*\\xf9S\\xb6\\xbf\\xb8\\xf1\\xc1\\x89͖C(\\xa8|\x189\x13\\xcd\\xc5\\xf7Q\x1e\\xe17h~\\x8c`\x1f8\\x8e\\xacq'
            |  \sig       \
            |   |###[ TLS Digital Signature ]### 
            |   |  sig_alg   = sha256+rsa
            |   |  sig_len   = 256
            |   |  sig_val   = '\\xc1R`\\xb8\x14!\\xed\\xb9\\xbca\\x9d0{\\xb7\\x95\\x94\\x80\x06\t.Â\\x99\\x89N_\\xa1\x08M%#\x1fg\\xb6\\xa2\\xfe\x00֨\\xe9\\x9fd\\x91O\\xdbzw\\xbfS\\x88?\\xeb[2\x7f\\xa1\\xeb\\xd1vmi_\\x95\\xd0A\x04`\x01+\x02\\\\x99\\xa0\\xe9\n\\xb5\\xb5j\\x85\\x89J\\x82\\xf8\x00\\xbb\\xa3%\x14\x15D\\xbf9\x12{\\x9e\\xca\x0e\\x92\u07fb\\xfd\\xd3\\xc8\x0ez\x04n \x12\x01\\xd2|\\xc6t\\xc36\\xce>:JÁ+d\\xbc\\xb1\x1d\\x8d\x00o\x00\\xc9\\xd4%\\xb6\\x90\x1f\\xe1\\xc5\x14\\xb5Qk\x06\x1e\\xf6{\\xbdJ\\xb2H\\xcbf\\xe9_mQ(\\x9e4\x10U#\\xcd4\\x88\x1c\\xfb\x03\\x80(Q:\\x9c\x0f\x16\\xed\\xad\\xb4\x18k\t\\xc5$\\x97}~s\\xc1ʮ\\x9d\\xd1q\\x94\\x9fi+Pj\\x80:v\\xc1z#\\xf6\\xee]ou~\\xa3\\xd9IθZ|\x1b\\x8ep\\xc6\x19\\xb4A\x03\\x92\x1bp\x16\x10\x0f\\x84\\xa9\\x9f\\xb7\\xc9\x01\\xc8^\\x93\\xaat\r\\x87\\x96\\x86\\xf6\\xc5\\xfe\\x88\x13\\xc3N'
            |###[ TLS Handshake - Server Hello Done ]### 
            |  msgtype   = server_hello_done
            |  msglen    = 0
           mac       = b''
           pad       = b''
           padlen    = None
```
At least in the Scapy data structure, it does merge, but why the strange phenomenon in Wireshark ?

### Expected result

The more reasonable results are as follows:

![image](https://user-images.githubusercontent.com/20126304/124117475-c8010800-daa2-11eb-9371-86b0b9773987.png)

I'm willing to provide a PR but it may take some time. I'm debugging the code implemented internally, hoping to find out why.


Discussion:
I found that just two changes will get the desired result.

I've already seen what the `tcp_reassemble` function does here. The merged TCP packet is formed from the last segment and splits the defragmenting payload.  The relevant code is as follows:

https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/sessions.py#L324-L336


Change 1:

https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/utils.py#L1895-L1899

![image](https://user-images.githubusercontent.com/20126304/124128229-2633e800-daaf-11eb-9c01-625ed9247c9b.png)

The `wirelen` obtained here is still the `wirelen` of the original last segmented TCP packet (Frame 9),  which will cause the TCP payload to show **only the first** `741 bytes` of the reassembled payload in Wireshark, with an expected value of `3641 bytes`

I changed the value of `wirelen` to `caplen` during debugging, and Wireshark successfully identified the TLS Layer.

Change 2:
[Expert Info (Warning/Sequence): Previous segment(s) not captured (common at capture start)]

For a similar reason, simply change `seq` of the reassembled packet to the `seq` of the **first** packet in the fragments,  instead of the `seq` of the **last** packet.

To record the `seq` field of the first packet in TCP fragments, add this information to the `metadata`, such as here
https://github.com/secdev/scapy/blob/6eef12d2de02ec4822c19621fd3847a498ca1387/scapy/sessions.py#L291-L302

```python
metadata["seq"] = pkt[TCP].seq
```

The `seq` needs to be corrected when returning the defragmented packet
```python
if packet: 
   pkt["TCP"].seq = metadata["seq"]  # before metadata clears, correct it !
   data.clear() 
   metadata.clear() 
   del self.tcp_frags[ident] 
   pay.underlayer.remove_payload()
   if IP in pkt: 
       pkt[IP].len = None 
       pkt[IP].chksum = None 
```

Resolving patch:
diff --git a/scapy/sessions.py b/scapy/sessions.py
--- a/scapy/sessions.py
+++ b/scapy/sessions.py
@@ -300,6 +300,7 @@ def _process_packet(self, pkt):
                 return pkt
             metadata["pay_class"] = pay_class
             metadata["tcp_reassemble"] = tcp_reassemble
+            metadata["seq"] = seq
         else:
             tcp_reassemble = metadata["tcp_reassemble"]
         # Get a relative sequence number for a storage purpose
@@ -326,6 +327,8 @@ def _process_packet(self, pkt):
             packet = tcp_reassemble(bytes(data), metadata)
         # Stack the result on top of the previous frames
         if packet:
+            if "seq" in metadata:
+                pkt[TCP].seq = metadata["seq"]
             data.clear()
             metadata.clear()
             del self.tcp_frags[ident]
@@ -333,7 +336,9 @@ def _process_packet(self, pkt):
             if IP in pkt:
                 pkt[IP].len = None
                 pkt[IP].chksum = None
-            return pkt / packet
+            pkt = pkt / packet
+            pkt.wirelen = None
+            return pkt
         return None
 
     def on_packet_received(self, pkt):


Test patch:
diff --git a/test/tls.uts b/test/tls.uts
--- a/test/tls.uts
+++ b/test/tls.uts
@@ -1509,6 +1509,9 @@ assert isinstance(a.msg[1], TLSCertificate)
 assert isinstance(a.msg[2], TLSServerKeyExchange)
 assert isinstance(a.msg[3], TLSServerHelloDone)
 
+assert a.wirelen is None
+assert a[TCP].seq == 7808002
+
 = Issue 2527
 
 p = TLS(b'\x16\x03\x00\x05\'\x02\x00\x00&\x03\x00\x00\x00\x00\x00\x7fk77\n\xe2\x1d\xdf\x82e\x06p$\xbaV7_\xa9\xb1\x03\x01\x0c\x0c\x18\x90\x00H\x01\x00\x00\x03\x00\x0b\x00\x03\xa8\x00\x03\xa5\x00\x03\xa20\x82\x03\x9e0\x82\x02\x86\xa0\x03\x02\x01\x02\x02\t\x00\xfe\x04W\r\xc7\'\xe9\xf60\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000T1\x0b0\t\x06\x03U\x04\x06\x13\x02MN1\x140\x12\x06\x03U\x04\x07\x0c\x0bUlaanbaatar1\x170\x15\x06\x03U\x04\x0b\x0c\x0eScapy Test PKI1\x160\x14\x06\x03U\x04\x03\x0c\rScapy Test CA0\x1e\x17\r160916102811Z\x17\r260915102811Z0X1\x0b0\t\x06\x03U\x04\x06\x13\x02MN1\x140\x12\x06\x03U\x04\x07\x0c\x0bUlaanbaatar1\x170\x15\x06\x03U\x04\x0b\x0c\x0eScapy Test PKI1\x1a0\x18\x06\x03U\x04\x03\x0c\x11Scapy Test Server0\x82\x01"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x82\x01\x0f\x000\x82\x01\n\x02\x82\x01\x01\x00\xcc\xf1\xf1\x9b`-`\xae\xf2\x98\r\')\xd9\xc0\tYL\x0fJ0\xa8R\xdf\xe5\xb1!\x9fO\xc3=V\x93\xdd_\xc6\xf7\xb3\xf6U\x8b\xe7\x92\xe2\xde\xf2\x85I\xb4\xa1,\xf4\xfdv\xa8g\xca\x04 `\x11\x18\xa6\xf2\xa9\xb6\xa6\x1d\xd9\xaa\xe5\xd9\xdb\xaf\xe6\xafUW\x9f\xffR\x89e\xe6\x80b\x80!\x94\xbc\xcf\x81\x1b\xcbg\xc2\x9d\xb5\x05w\x04\xa6\xc7\x88\x18\x80xh\x956\xde\x97\x1b\xb6a\x87B\x1au\x98E\x82\xeb>2\x11\xc8\x9b\x86B9\x8dM\x12\xb7X\x1b\x19\xf3\x9d+\xa1\x98\x82\xca\xd7;$\xfb\t9\xb0\xbc\xc2\x95\xcf\x82)u\x16)?B \x17+M@\x8cVl\xad\xba\x0f4\x85\xb1\x7f@yqx\xb7\xa5\x04\xbb\x94\xf7\xb5A\x95\xee|\xeb\x8d\x0cyhY\xef\xcb\xb3\xfa>x\x1e\xeegLz\xdd\xe0\x99\xef\xda\xe7\xef\xb2\t]\xbe\x80 !\x05\x83,D\xdb]*v)\xa5\xb0#\x88t\x07T"\xd6)z\x92\xf5o-\x9e\xe7\xf8&+\x9cXe\x02\x03\x01\x00\x01\xa3o0m0\t\x06\x03U\x1d\x13\x04\x020\x000\x0b\x06\x03U\x1d\x0f\x04\x04\x03\x02\x05\xe00\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14\xa1+ p\xd2k\x80\xe5e\xbc\xeb\x03\x0f\x88\x9ft\xad\xdd\xf6\x130\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14fS\x94\xf4\x15\xd1\xbdgh\xb0Q725\xe1\xa4\xaa\xde\x07|0\x13\x06\x03U\x1d%\x04\x0c0\n\x06\x08+\x06\x01\x05\x05\x07\x03\x010\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x00\x03\x82\x01\x01\x00\x81\x88\x92sk\x93\xe7\x95\xd6\xddA\xee\x8e\x1e\xbd\xa3HX\xa7A5?{}\xd07\x98\x0e\xb8,\x94w\xc8Q6@\xadY\t(\xc8V\xd6\xea[\xac\xb4\xd8?h\xb7f\xca\xe1V7\xa9\x00e\xeaQ\xc9\xec\xb2iI]\xf9\xe3\xc0\xedaT\xc9\x12\x9f\xc6\xb0\nsU\xe8U5`\xef\x1c6\xf0\xda\xd1\x90wV\x04\xb8\xab8\xee\xf7\t\xc5\xa5\x98\x90#\xea\x1f\xdb\x15\x7f2(\x81\xab\x9b\x85\x02K\x95\xe77Q{\x1bH.\xfb>R\xa3\r\xb4F\xa9\x92:\x1c\x1f\xd7\n\x1eXJ\xfa.Q\x8f)\xc6\x1e\xb8\x0e1\x0es\xf1\'\x88\x17\xca\xc8i\x0c\xfa\x83\xcd\xb3y\x0e\x14\xb0\xb8\x9b/:-\t\xe3\xfc\x06\xf0:n\xfd6;+\x1a\t*\xe8\xab_\x8c@\xe4\x81\xb2\xbc\xf7\x83g\x11nN\x93\xea"\xaf\xff\xa3\x9awWv\xd0\x0b8\xac\xf8\x8a\x945\x8e\xd7\xd4a\xcc\x01\xff$\xb4\x8fa#\xba\x88\xd7Y\xe4\xe9\xba*N\xb5\x15\x0f\x9c\xd0\xea\x06\x91\xd9\xde\xab\x0c\x00\x01I\x00@\xd1L\xf3\xe7\x8b\xdd\x98\xff\xb2\xf5Rd\xd6\x85\x0f\r{\x9f\xc2\xc0\x8aY\xbf.\xfb\xf0o\x96\xa5\xba;\x877qet\xe8\xe4K\xd7\xcb\xb8\xecAk>S\xe0\xa5\xc3\xfc\xe8\xde\xf1\xb0\xe5\x15s|\xb7\xe6D\x15+\x00\x03\x01\x00\x01\x01\x00H\xf1\x08\x88\xe9\xf8\xe6\xb2y\\\xf9\xf64\x95r\xf9\x8c]\x0b\x88%s\xee{\xd4\xa3{|Jd>\xfb\x01\x0b\xfdAf\xea\x13%\x1f\xcc\xba\xf8H\xed\xeb?u\x00\xc46\xe4\x9f!r\x99\xec\'!\xa1+\xe9\xcd;\xfa\x00a\xd1ME7\x9a\xc3C\xb2\xb0>\xec\x07\xff>\xb3\xa3\xbd\x8db\xa2\x17\x0b\xce\xe1H\xaf\xba_\xdc\x18\x83Fr^\xf6\xfd\x8f\xbd\xc1\xdf\xc3\xf9T\xc2RC\xfa1\xe1\x16\x94RgZ\xb1\xe8rycp\xaeEa@\xe2\xb7T\xe4\xaa7\x02\x1e\xb3\x0c_P\x14\xd9\x023]\xc9)\x1b\xd7]\xba\x8aS\x18\xe5\x88\x1e08W\xc7\xd5\xc0\x7f\xf6n\n>\x83_\r\t\x1f\x01\x99\xda\x88(\xbc\xd9\xb8!=\xb6%\x15wh\xacl)\xde\xb3-\x81M\xc6(,\xceom\x15W7\xcc\xd3\xe3\xc2e\xb4\x96\xf1\xfc\x1e\xa5?\xe1B\xbd\x00\x89\xc1\xd0t\xd6\xaa\xf8\xa7\x1f\xa1z}\x91M\x8egg\xa1}\x93\xaal\xec\x16@\xf3\xd7\x0b\x91\n\xcc\x0e\x00\x00\x00')