Repository: secdev/scapy

Issue:
packet.time_sent can't be set because of object copy during list comprehension
https://github.com/secdev/scapy/blob/254ab6d5ae443bb2190e85e4898bfdd8bace9c54/scapy/sendrecv.py#L186

Is this list comprehension necessary at this position? Without a list comprehension, the time_sent values in the objects to be sent will be propagated to the original object. Because of the deep copy in the list comprehension, the time_sent is only set in a temporary object and not in the original object. This applies for all sr and sr1 function calls.
Fix for issue #1199



Discussion:
What issue are you facing that will require changing this line? Please provide an example.
Hi, here is an example:
```
>>> TIMEOUT = 2
...:conf.verb = 0
...:packet = IP(dst="192.168.178.1", ttl=20)/ICMP()
...:reply = sr1(packet, timeout=TIMEOUT)
...:packet.sent_time is None
...:
True
```
Here is another example. SuperSocket.send() sets the sent_time correctly
```
>>> TIMEOUT = 2
...:conf.verb = 0
...:packet = IP(dst="192.168.178.1", ttl=20)/ICMP()
...:reply = sr1(packet, timeout=TIMEOUT)
...:print(packet.sent_time)
...:s = L2Socket()
...:s.send(packet)
...:print(packet.sent_time)
...:
...:
None
1521442246.051092
```
Issue confirmed. I don't have time to investigate it.
The problem ist not the mentioned line. 
The issue is probably in ```packet.__iter__```
# [Codecov](https://codecov.io/gh/secdev/scapy/pull/1276?src=pr&el=h1) Report
> Merging [#1276](https://codecov.io/gh/secdev/scapy/pull/1276?src=pr&el=desc) into [master](https://codecov.io/gh/secdev/scapy/commit/a13a05c19fca53e2ec61594566d95c284f3328c0?src=pr&el=desc) will **decrease** coverage by `5.53%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master    #1276      +/-   ##
==========================================
- Coverage   84.79%   79.25%   -5.54%     
==========================================
  Files         160      159       -1     
  Lines       38322    37544     -778     
==========================================
- Hits        32495    29757    -2738     
- Misses       5827     7787    +1960
```

| [Impacted Files](https://codecov.io/gh/secdev/scapy/pull/1276?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scapy/sendrecv.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvc2VuZHJlY3YucHk=) | `40.26% <100%> (-38%)` | :arrow_down: |
| [scapy/layers/tls/automaton.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvbGF5ZXJzL3Rscy9hdXRvbWF0b24ucHk=) | `15.78% <0%> (-64.92%)` | :arrow_down: |
| [scapy/arch/pcapdnet.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvYXJjaC9wY2FwZG5ldC5weQ==) | `12.62% <0%> (-56.15%)` | :arrow_down: |
| [scapy/as\_resolvers.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvYXNfcmVzb2x2ZXJzLnB5) | `32.07% <0%> (-49.06%)` | :arrow_down: |
| [scapy/layers/tls/automaton\_srv.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvbGF5ZXJzL3Rscy9hdXRvbWF0b25fc3J2LnB5) | `26.2% <0%> (-45.67%)` | :arrow_down: |
| [scapy/layers/tls/automaton\_cli.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvbGF5ZXJzL3Rscy9hdXRvbWF0b25fY2xpLnB5) | `31.88% <0%> (-45.15%)` | :arrow_down: |
| [scapy/supersocket.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvc3VwZXJzb2NrZXQucHk=) | `38.96% <0%> (-40.26%)` | :arrow_down: |
| [scapy/modules/nmap.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvbW9kdWxlcy9ubWFwLnB5) | `57.46% <0%> (-39.56%)` | :arrow_down: |
| [scapy/arch/linux.py](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree#diff-c2NhcHkvYXJjaC9saW51eC5weQ==) | `36.5% <0%> (-37.37%)` | :arrow_down: |
| ... and [41 more](https://codecov.io/gh/secdev/scapy/pull/1276/diff?src=pr&el=tree-more) | |

Thanks for the PR. Could you provide unit tests?

For example:
```
>>> packet = IP(dst="8.8.8.8")/ICMP() ; r = sr(packet); packet.sent_time is not None
>>> packet = IP(dst="8.8.8.8")/ICMP() ; r = sr1(packet); packet.sent_time is not None
```
It was only examples. Please remove the ';' and add assert before the tests for None. Thanks.
Sorry
Why not?:
```tobesent = list(pkt) if hasattr(pkt, "__iter__") else pkt```

I think it's safe to call simply `list(pkt)`:
- if pkt is a list, list() returns the same object
- if pkt is a packet, or a generator, __iter__ is implicitly called
Now I got a problem. If I do the following:
``` tobesent = [pkt] ```
this issue is fixed, but some other unit test will fail.
if I do 
``` tobesent = list(pkt) ```
this issue is still present, but other unit test will not fail.
This means that the fix is incorrect. this line is more complex than it looks, knowing that a packet can generate others depending on its parameters
> I think it's safe to call simply list(pkt):
>  if pkt is a list, list() returns the same object

It is not quite true.
list(list_object) returns a new list object (copy of list_object). Try:
a = [1,2,3,4]
b = list(a)
a is b
ok, the problem is probably somewhere inside``` packet.__iter__```
unfortunately, I don't know enough internals of scapy, to fix this issue


Resolving patch:
diff --git a/scapy/sendrecv.py b/scapy/sendrecv.py
--- a/scapy/sendrecv.py
+++ b/scapy/sendrecv.py
@@ -173,8 +173,8 @@ def sndrcv(pks, pkt, timeout=None, inter=0, verbose=None, chainCC=False,
     timeout:  how much time to wait after the last packet has been sent
     verbose:  set verbosity level
     multi:    whether to accept multiple answers for the same stimulus"""
-    if not isinstance(pkt, Gen):
-        pkt = SetGen(pkt)
+    is_single = isinstance(pkt, Gen)
+    pkts = [pkt] if is_single else pkt
     if verbose is None:
         verbose = conf.verb
     debug.recv = plist.PacketList([],"Unanswered")
@@ -183,15 +183,16 @@ def sndrcv(pks, pkt, timeout=None, inter=0, verbose=None, chainCC=False,
     nbrecv = 0
     ans = []
     # do it here to fix random fields, so that parent and child have the same
-    tobesent = [p for p in pkt]
+    tobesent = [p for p in (pkt if is_single else SetGen(pkt))]
     notans = len(tobesent)
 
     if retry < 0:
-        retry = -retry
-        autostop = retry
+        autostop = retry = -retry
     else:
         autostop = 0
 
+    for pkt in pkts:
+        pkt.sent_time = None
     while retry >= 0:
         if timeout is not None and timeout < 0:
             timeout = None
@@ -207,18 +208,28 @@ def sndrcv(pks, pkt, timeout=None, inter=0, verbose=None, chainCC=False,
             (rcv_pks or pks), tobesent, stopevent, nbrecv, notans, verbose, chainCC, multi,
         )
         thread.join()
+
         ans.extend(newans)
+        to_set_time = [pkt for pkt in pkts if pkt.sent_time is None]
+        if to_set_time:
+            try:
+                sent_time = min(p.sent_time for p in tobesent if getattr(p, "sent_time", None))
+            except ValueError:
+                pass
+            else:
+                for pkt in to_set_time:
+                    pkt.sent_time = sent_time
 
-        remain = list(itertools.chain(*six.itervalues(hsent)))
-        if multi:
-            remain = [p for p in remain if not hasattr(p, '_answered')]
+        remain = itertools.chain(*six.itervalues(hsent))
+        remain = [p for p in remain if not hasattr(p, '_answered')] if multi else list(remain)
 
-        if autostop and len(remain) > 0 and len(remain) != len(tobesent):
+        if not remain:
+            break
+
+        if autostop and len(remain) != len(tobesent):
             retry = autostop
             
         tobesent = remain
-        if len(tobesent) == 0:
-            break
         retry -= 1
 
     if conf.debug_match:
@@ -497,9 +508,9 @@ def srploop(pkts, *args, **kargs):
 def sndrcvflood(pks, pkt, inter=0, verbose=None, chainCC=False, prn=lambda x: x):
     if not verbose:
         verbose = conf.verb
-    if not isinstance(pkt, Gen):
-        pkt = SetGen(pkt)
-    tobesent = [p for p in pkt]
+    is_single = isinstance(pkt, Gen)
+    pkts = [pkt] if is_single else pkt
+    tobesent = [p for p in (pkt if is_single else SetGen(pkt))]
 
     stopevent = threading.Event()
     count_packets = six.moves.queue.Queue()
@@ -515,6 +526,8 @@ def send_in_loop(tobesent, stopevent, count_packets=count_packets):
 
     infinite_gen = send_in_loop(tobesent, stopevent)
 
+    for pkt in pkts:
+        pkt.sent_time = None
     # We don't use _sndrcv_snd verbose (it messes the logs up as in a thread that ends after recieving)
     thread = threading.Thread(
         target=_sndrcv_snd,
@@ -524,9 +537,19 @@ def send_in_loop(tobesent, stopevent, count_packets=count_packets):
 
     hsent, ans, nbrecv, notans = _sndrcv_rcv(pks, tobesent, stopevent, 0, len(tobesent), verbose, chainCC, False)
     thread.join()
+
+    ans = [(x, prn(y)) for (x, y) in ans]  # Apply prn
+    to_set_time = [pkt for pkt in pkts if pkt.sent_time is None]
+    if to_set_time:
+        try:
+            sent_time = min(p.sent_time for p in tobesent if getattr(p, "sent_time", None))
+        except ValueError:
+            pass
+        else:
+            for pkt in to_set_time:
+                pkt.sent_time = sent_time
+
     remain = list(itertools.chain(*six.itervalues(hsent)))
-    # Apply prn
-    ans = [(x, prn(y)) for (x, y) in ans]
 
     if verbose:
         print("\nReceived %i packets, got %i answers, remaining %i packets. Sent a total of %i packets." % (nbrecv+len(ans), len(ans), notans, count_packets.qsize()))


Test patch:
diff --git a/test/regression.uts b/test/regression.uts
--- a/test/regression.uts
+++ b/test/regression.uts
@@ -1223,6 +1223,34 @@ for i in range(10):
 response = res[0][1]
 assert response[ICMP].type == 0
 
+= Test set of sent_time by sr
+~ netaccess IP ICMP
+packet = IP(dst="8.8.8.8")/ICMP()
+r = sr(packet)
+assert packet.sent_time is not None
+
+= Test set of sent_time by sr (multiple packets)
+~ netaccess IP ICMP
+packet1 = IP(dst="8.8.8.8")/ICMP()
+packet2 = IP(dst="8.8.4.4")/ICMP()
+r = sr([packet1, packet2])
+assert packet1.sent_time is not None
+assert packet2.sent_time is not None
+
+= Test set of sent_time by srflood
+~ netaccess IP ICMP
+packet = IP(dst="8.8.8.8")/ICMP()
+r = srflood(packet)
+assert packet.sent_time is not None
+
+= Test set of sent_time by srflood (multiple packets)
+~ netaccess IP ICMP
+packet1 = IP(dst="8.8.8.8")/ICMP()
+packet2 = IP(dst="8.8.4.4")/ICMP()
+r = srflood([packet1, packet2])
+assert packet1.sent_time is not None
+assert packet2.sent_time is not None
+
 ############
 ############
 + ManuFDB tests