Repository: secdev/scapy

Issue:
NetflowTemplateFieldV9 cannot decode IPFIX template contains enterprise-specific Information Element
### Brief description

IPFIX is now re-using NetflowTemplateFieldV9 for template records. But IPFIX extends the format of Netflow protocol. According to [IPFIX RFC](https://tools.ietf.org/html/rfc7011#page-22), the first bit of Information Element id indicates whether it is IANA-assigned or enterprise-specific Information Elements. 

```
     |0| Information Element id. 1.2 |        Field Length 1.2       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |             ...               |              ...              |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |1| Information Element id. 1.N |        Field Length 1.N       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                    Enterprise Number  1.N                     |
```

NetflowTemplateFieldV9 in scapy now can only decode IANA-assigned template.


#### Environment

-   Scapy version: 2.4.3rc4.dev6
-   Python version: `e.g. 2.7`
-   Operating System: `CentOS 7`

#### How to reproduce

will upload a pcap later.



Discussion:
Hi here is the minimal code to reproduce the problem.

```
bind_bottom_up(UDP, NetflowHeader, dport=4739)
s=b'\x00\x00\x03\x04\x00\x06\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00E\x00\x00\x8c\x196@\x00@\x11\xe0\xad\x01\x01\x01\x01\xac\x18\x92c\xc2\xb8\x12\x83\x00xA\x07\x00\n\x00p]r\xa7\r\x00\x00\t$\x00\x00\x00\x01\x00\x02\x00`\x01\x07\x00\x12\x01\n\x00\x04\x84\x0c\x00\x02\x00\x00\x00\t\x01\n\x00&\x00\x0b\x00\x02\x00\x07\x00\x02\x00\x04\x00\x01\x00\x0c\x00\x04\x00\x08\x00\x04\x00\xea\x00\x02\x01\n\x00\x01\x84\x10\x00\x06\x00\x00\x00\t\x84\x0e\x00\x06\x00\x00\x00\t\x84\x0f\x00\x06\x00\x00\x00\t\x00\x01\x00\x04\x00\x02\x00\x04\x00\xf3\x00\x02\x00\x06\x00\x01\x01\n\x00#'
p = CookedLinux(s)
```
The output when decode it using scapy is:

```
<CookedLinux  pkttype=unicast lladdrtype=0x304 lladdrlen=6 src='' proto=IPv4 |
<IP  version=4 ihl=5 tos=0x0 len=140 id=6454 flags=DF frag=0 ttl=64 proto=udp chksum=0xe0ad src=1.1.1.1 dst=172.24.146.99 |
<UDP  sport=49848 dport=ipfix len=120 chksum=0x4107 |<NetflowHeader  version=10 |
<NetflowHeaderV10  length=112 ExportTime=Fri, 06 Sep 2019 18:35:57 -0800 (1567794957) flowSequence=2340 ObservationDomainID=1 |
<NetflowFlowsetV9  flowSetID=2 length=96 templates=[
    <NetflowTemplateV9  templateID=263 fieldCount=18 template_fields=[
        <NetflowTemplateFieldV9  fieldType=opaqueOctets fieldLength=4 |>, 
        <NetflowTemplateFieldV9  fieldType=33804 fieldLength=2 |>, 
        <NetflowTemplateFieldV9  fieldType=0 fieldLength=9 |>, 
        <NetflowTemplateFieldV9  fieldType=opaqueOctets fieldLength=38 |>, 
        <NetflowTemplateFieldV9  fieldType=L4_DST_PORT fieldLength=2 |>, 
        <NetflowTemplateFieldV9  fieldType=L4_SRC_PORT fieldLength=2 |>, 
        <NetflowTemplateFieldV9  fieldType=PROTOCOL fieldLength=1 |>, 
        <NetflowTemplateFieldV9  fieldType=IPV4_DST_ADDR fieldLength=4 |>, 
        <NetflowTemplateFieldV9  fieldType=IPV4_SRC_ADDR fieldLength=4 |>, 
        <NetflowTemplateFieldV9  fieldType=ingressVRFID fieldLength=2 |>, 
        <NetflowTemplateFieldV9  fieldType=opaqueOctets fieldLength=1 |>, 
        <NetflowTemplateFieldV9  fieldType=33808 fieldLength=6 |>, 
        <NetflowTemplateFieldV9  fieldType=0 fieldLength=9 |>, 
        <NetflowTemplateFieldV9  fieldType=33806 fieldLength=6 |>, 
        <NetflowTemplateFieldV9  fieldType=0 fieldLength=9 |>, 
        <NetflowTemplateFieldV9  fieldType=33807 fieldLength=6 |>, 
        <NetflowTemplateFieldV9  fieldType=0 fieldLength=9 |>, 
        <NetflowTemplateFieldV9  fieldType=IN_BYTES fieldLength=4 |>] |>, 
    <NetflowTemplateV9  templateID=2 fieldCount=4 template_fields=[
        <NetflowTemplateFieldV9  fieldType=dot1qVlanId fieldLength=2 |>, 
        <NetflowTemplateFieldV9  fieldType=TCP_FLAGS fieldLength=1 |>, 
        <NetflowTemplateFieldV9  fieldType=opaqueOctets fieldLength=35 |>] |>] |>>>>>>
```

Decoding with Wireshark, the output is 
```
Cisco NetFlow/IPFIX
    Version: 10
    Length: 112
    Timestamp: Sep  6, 2019 11:35:57.000000000 PDT
    FlowSequence: 2340
    Observation Domain Id: 1
    Set 1 [id=2] (Data Template): 263
        FlowSet Id: Data Template (V10 [IPFIX]) (2)
        FlowSet Length: 96
        Template (Id = 263, Count = 18)
            Template Id: 263
            Field Count: 18
            Field (1/18): opaqueOctets
            Field (2/18): 1036 [pen: ciscoSystems]
                1... .... .... .... = Pen provided: Yes
                .000 0100 0000 1100 = Type: 1036 [pen: ciscoSystems]
                Length: 2
                PEN: ciscoSystems (9)
            Field (3/18): opaqueOctets
            Field (4/18): L4_DST_PORT
            Field (5/18): L4_SRC_PORT
            Field (6/18): PROTOCOL
            Field (7/18): IP_DST_ADDR
            Field (8/18): IP_SRC_ADDR
            Field (9/18): ingressVRFID
            Field (10/18): opaqueOctets
            Field (11/18): 1040 [pen: ciscoSystems]
            Field (12/18): 1038 [pen: ciscoSystems]
            Field (13/18): 1039 [pen: ciscoSystems]
            Field (14/18): BYTES
            Field (15/18): PKTS
            Field (16/18): dot1qVlanId
            Field (17/18): TCP_FLAGS
            Field (18/18): opaqueOctets
```
The problem happenes in the 2nd element, when first bit of field_type is set to 1, there is an extra field for enterprise number. 
For exmaple, the first element of this template in hex is `01 0a 00 04`, `01 0a` is opaqueOctets(266), 04 is lenght. The seconde element in hex is  `84 0c 00 02 00 00 00 09`, the first bit is set, length is 2, enterprise number is `00 00 00 09` meaning ciscoSystem. 
Scapy only decodes the first 4 bytes, and then trys to decode the remaining 4 bytes as another template field. 

I think it can be easily fixed here https://github.com/secdev/scapy/blob/master/scapy/layers/netflow.py#L1331 . I'm writing a patch but don't know how to define a optional field. Could someone provide me an exmaple I can follow?





Resolving patch:
diff --git a/scapy/layers/netflow.py b/scapy/layers/netflow.py
--- a/scapy/layers/netflow.py
+++ b/scapy/layers/netflow.py
@@ -34,7 +34,7 @@
     FlagsField, IPField, IntField, MACField, \
     PacketListField, PadField, SecondsIntField, ShortEnumField, ShortField, \
     StrField, StrFixedLenField, ThreeBytesField, UTCTimeField, XByteField, \
-    XShortField, LongField
+    XShortField, LongField, BitField, ConditionalField, BitEnumField
 from scapy.packet import Packet, bind_layers, bind_bottom_up
 from scapy.plist import PacketList
 from scapy.sessions import IPSession, DefaultSession
@@ -1243,9 +1243,12 @@ class NetflowHeaderV10(Packet):
 
 class NetflowTemplateFieldV9(Packet):
     name = "Netflow Flowset Template Field V9/10"
-    fields_desc = [ShortEnumField("fieldType", None,
-                                  NetflowV910TemplateFieldTypes),
-                   ShortField("fieldLength", 0)]
+    fields_desc = [BitField("enterpriseBit", 0, 1),
+                   BitEnumField("fieldType", None, 15,
+                                NetflowV910TemplateFieldTypes),
+                   ShortField("fieldLength", 0),
+                   ConditionalField(IntField("enterpriseNumber", 0),
+                                    lambda p: p.enterpriseBit)]
 
     def __init__(self, *args, **kwargs):
         Packet.__init__(self, *args, **kwargs)
@@ -1520,9 +1523,12 @@ class NetflowOptionsRecordOptionV9(NetflowRecordV9):
 # Aka Set
 class NetflowOptionsFlowsetOptionV9(Packet):
     name = "Netflow Options Template FlowSet V9/10 - Option"
-    fields_desc = [ShortEnumField("optionFieldType", None,
-                                  NetflowV910TemplateFieldTypes),
-                   ShortField("optionFieldlength", 0)]
+    fields_desc = [BitField("enterpriseBit", 0, 1),
+                   BitEnumField("optionFieldType", None, 15,
+                                NetflowV910TemplateFieldTypes),
+                   ShortField("optionFieldlength", 0),
+                   ConditionalField(ShortField("enterpriseNumber", 0),
+                                    lambda p: p.enterpriseBit)]
 
     def default_payload_class(self, p):
         return conf.padding_layer


Test patch:
diff --git a/test/regression.uts b/test/regression.uts
--- a/test/regression.uts
+++ b/test/regression.uts
@@ -6750,6 +6750,12 @@ assert pkt3.scopes[0].scopeFieldType == 5
 assert pkt3.scopes[0].scopeFieldlength == 2
 assert pkt3[NetflowOptionsFlowset10].options[0].optionFieldType == 36
 
+# Templates with enterprise-specific Information Elements.
+s=b'\x01\x07\x00\x12\x01\n\x00\x04\x84\x0c\x00\x02\x00\x00\x00\t\x01\n\x00&\x00\x0b\x00\x02\x00\x07\x00\x02\x00\x04\x00\x01\x00\x0c\x00\x04\x00\x08\x00\x04\x00\xea\x00\x02\x01\n\x00\x01\x84\x10\x00\x06\x00\x00\x00\t\x84\x0e\x00\x06\x00\x00\x00\t\x84\x0f\x00\x06\x00\x00\x00\t\x00\x01\x00\x04\x00\x02\x00\x04\x00\xf3\x00\x02\x00\x06\x00\x01\x01\n\x00#'
+pkt4 = NetflowTemplateV9(s)
+assert len(pkt4.template_fields) == pkt4.fieldCount
+assert sum([template.fieldLength for template in pkt4.template_fields]) == 124
+
 ############
 ############
 + pcap / pcapng format support