Repository: vertexproject/synapse

Issue:
Srv4Type _norm_str returns bad secondary props
its
```        
    return (addr << 16) | port, {}
```
instead of
```        
    return (addr << 16) | port, {'port': port, 'ipv4': addr}
```

This means nodes made with this type are missing prop values.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=h1) Report
> Merging [#355](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=desc) into [master](https://codecov.io/gh/vertexproject/synapse/commit/ad375f072ebbc1e8f5bb792a3070fa915cdd0642?src=pr&el=desc) will **decrease** coverage by `1.06%`.
> The diff coverage is `29.16%`.

[![Impacted file tree graph](https://codecov.io/gh/vertexproject/synapse/pull/355/graphs/tree.svg?height=150&src=pr&width=650&token=c1t4BsMAUL)](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #355      +/-   ##
==========================================
- Coverage   89.93%   88.86%   -1.07%     
==========================================
  Files         123      122       -1     
  Lines       13735    13710      -25     
==========================================
- Hits        12352    12183     -169     
- Misses       1383     1527     +144
```


| [Impacted Files](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [synapse/models/inet.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9tb2RlbHMvaW5ldC5weQ==) | `90.45% <29.16%> (-7.6%)` | :arrow_down: |
| [synapse/compat.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9jb21wYXQucHk=) | `49.12% <0%> (-47.37%)` | :arrow_down: |
| [synapse/axon.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9heG9uLnB5) | `83.35% <0%> (-10.21%)` | :arrow_down: |
| [synapse/lib/threads.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9saWIvdGhyZWFkcy5weQ==) | `66.17% <0%> (-2.46%)` | :arrow_down: |
| [synapse/lib/moddef.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9saWIvbW9kZGVmLnB5) | `94.21% <0%> (-1.74%)` | :arrow_down: |
| [synapse/lib/persist.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9saWIvcGVyc2lzdC5weQ==) | `90.42% <0%> (-1.07%)` | :arrow_down: |
| [synapse/links/ssl.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9saW5rcy9zc2wucHk=) | `86.71% <0%> (-0.79%)` | :arrow_down: |
| [synapse/mindmeld.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9taW5kbWVsZC5weQ==) | `88.09% <0%> (-0.6%)` | :arrow_down: |
| [synapse/lib/socket.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9saWIvc29ja2V0LnB5) | `83.56% <0%> (-0.57%)` | :arrow_down: |
| [synapse/cores/lmdb.py](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree#diff-c3luYXBzZS9jb3Jlcy9sbWRiLnB5) | `90.74% <0%> (-0.23%)` | :arrow_down: |
| ... and [6 more](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=footer). Last update [ad375f0...6ec4219](https://codecov.io/gh/vertexproject/synapse/pull/355?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/synapse/devopsguides/model_format.rst b/docs/synapse/devopsguides/model_format.rst
index b7d76f133..c0035a27f 100644
--- a/docs/synapse/devopsguides/model_format.rst
+++ b/docs/synapse/devopsguides/model_format.rst
@@ -35,6 +35,11 @@ An example of a simple model definition file is the following ::
                 'types': (
                     ('foo:bar', {'subof': 'str', 'doc': 'A foo bar!'}),
                 ),
+                'forms': (
+                    ('foo:bar',
+                     {'ptype': 'foo:bar'},
+                     [])
+                )
             }
             name = 'foobar'
             return ((name, modl), )
@@ -52,7 +57,7 @@ a iterable of name, model tuples. This serves two purposes:
      On existing Cortexes which do not have nodes storing the model revision, the revision 0 of the model will be
      created and the base model automatically loaded.
 
-When model updates need to be done, these updates are be delivered as functions in the CoreModule sublcass.  These
+When model updates need to be done, these updates are to be delivered as functions in the CoreModule sublcass. These
 functions must be decorated with the @s_module.modelrev function.  This decorator takes two arguments:
 
   #. The name of the model to be updated.
@@ -89,7 +94,7 @@ will be updated to 201707210101 in the Cortex.::
                 ),
                 'forms': (
                     ('foo:bar',
-                     {'ptype': 'str'},
+                     {'ptype': 'foo:bar'},
                      [('duck', {'defval': 'mallard', 'ptype': 'str', 'doc': 'Duck type.'})]
                      ),
                 ),
@@ -159,11 +164,11 @@ An example of extending the previous example is shown below (minus migration fun
                 ),
                 'forms': (
                     ('foo:bar',
-                     {'ptype': 'str'},
+                     {'ptype': 'foo:bar'},
                      [('duck', {'defval': 'mallard', 'ptype': 'str', 'doc': 'Duck type.'})]
                      ),
                     ('foo:knight',
-                     {'ptype': 'str'},
+                     {'ptype': 'foo:knight'},
                      [('court', {'ptype': 'str', 'doc': 'Knight court'})]
                      ),
                 ),
diff --git a/synapse/models/inet.py b/synapse/models/inet.py
index e79805d66..ff1ab2b53 100644
--- a/synapse/models/inet.py
+++ b/synapse/models/inet.py
@@ -3,6 +3,7 @@ import socket
 import struct
 import hashlib
 
+import synapse.common as s_common
 import synapse.compat as s_compat
 import synapse.datamodel as s_datamodel
 import synapse.lib.socket as s_socket
@@ -109,6 +110,9 @@ def ipv6norm(text):
     return s_socket.inet_ntop(socket.AF_INET6, s_socket.inet_pton(socket.AF_INET6, text))
 
 class IPv6Type(DataType):
+    def repr(self, valu):
+        return self.norm(valu)[0]
+
     def norm(self, valu, oldval=None):
         try:
             return ipv6norm(valu), {}
@@ -147,7 +151,7 @@ class Srv4Type(DataType):
 
         addr = ipv4int(astr)
         port = int(pstr, 0)
-        return (addr << 16) | port, {}
+        return (addr << 16) | port, {'port': port, 'ipv4': addr}
 
 srv6re = re.compile('^\[([a-f0-9:]+)\]:(\d+)$')
 
@@ -160,6 +164,9 @@ class Srv6Type(DataType):
         ('ipv6', {'ptype': 'inet:ipv6'}),
     )
 
+    def repr(self, valu):
+        return self.norm(valu)[0]
+
     def norm(self, valu, oldval=None):
 
         valu = valu.lower()
@@ -337,6 +344,55 @@ class InetMod(CoreModule):
         for tufo in self.core.getTufosByProp('inet:fqdn:domain', fqdn):
             self.core.setTufoProp(tufo, 'zone', sfx)
 
+    @modelrev('inet', 201706201837)
+    def _revModl201706201837(self):
+        '''
+        Add :port and :ipv4 to inet:tcp4 and inet:udp4 nodes.
+        '''
+        tick = s_common.now()
+
+        forms = ('inet:tcp4', 'inet:udp4')
+        for form in forms:
+            adds = []
+            portprop = '{}:port'.format(form)
+            ipv4prop = '{}:ipv4'.format(form)
+
+            rows = self.core.getRowsByProp(form)
+            for i, p, v, _ in rows:
+                norm, subs = s_datamodel.tlib.getTypeNorm(form, v)
+
+                port = subs.get('port')
+                if port:
+                    adds.append((i, portprop, port, tick))
+
+                ipv4 = subs.get('ipv4')
+                if ipv4:
+                    adds.append((i, ipv4prop, ipv4, tick))
+
+            if adds:
+                self.core.addRows(adds)
+
+    @modelrev('inet', 201706121318)
+    def _revModl201706121318(self):
+
+        # account for the updated sub-property extraction for inet:url nodes
+        adds = []
+        rows = self.core.getRowsByProp('inet:url')
+
+        for i, p, v, t in rows:
+            norm, subs = s_datamodel.tlib.getTypeNorm('inet:url', v)
+
+            fqdn = subs.get('fqdn')
+            if fqdn is not None:
+                adds.append((i, 'inet:url:fqdn', fqdn, t))
+
+            ipv4 = subs.get('ipv4')
+            if ipv4 is not None:
+                adds.append((i, 'inet:url:ipv4', ipv4, t))
+
+        if adds:
+            self.core.addRows(adds)
+
     @staticmethod
     def getBaseModels():
         modl = {
@@ -682,24 +738,3 @@ class InetMod(CoreModule):
         }
         name = 'inet'
         return ((name, modl), )
-
-    @modelrev('inet', 201706121318)
-    def _revModl201706121318(self):
-
-        # account for the updated sub-property extraction for inet:url nodes
-        adds = []
-        rows = self.core.getRowsByProp('inet:url')
-
-        for i, p, v, t in rows:
-            norm, subs = s_datamodel.tlib.getTypeNorm('inet:url', v)
-
-            fqdn = subs.get('fqdn')
-            if fqdn is not None:
-                adds.append((i, 'inet:url:fqdn', fqdn, t))
-
-            ipv4 = subs.get('ipv4')
-            if ipv4 is not None:
-                adds.append((i, 'inet:url:ipv4', ipv4, t))
-
-        if adds:
-            self.core.addRows(adds)


Test patch:
diff --git a/synapse/tests/test_cortex.py b/synapse/tests/test_cortex.py
index 3cf75f5e8..45e80bb0e 100644
--- a/synapse/tests/test_cortex.py
+++ b/synapse/tests/test_cortex.py
@@ -63,7 +63,12 @@ class CoreTestDataModelModuleV0(s_module.CoreModule):
             'types': (
                 ('foo:bar', {'subof': 'str', 'doc': 'A foo bar!'}),
             ),
-            'forms': (),
+            'forms': (
+                ('foo:bar',
+                 {'ptype': 'foo:bar'},
+                 []
+                 ),
+            ),
         }
         name = 'test'
         return ((name, modl),)
@@ -78,7 +83,7 @@ class CoreTestDataModelModuleV1(s_module.CoreModule):
             ),
             'forms': (
                 ('foo:bar',
-                 {'ptype': 'str'},
+                 {'ptype': 'foo:bar'},
                  [('duck', {'defval': 'mallard', 'ptype': 'str', 'doc': 'Duck value!'})]
                  ),
             ),
diff --git a/synapse/tests/test_model_inet.py b/synapse/tests/test_model_inet.py
index ea2509101..f8a214bb2 100644
--- a/synapse/tests/test_model_inet.py
+++ b/synapse/tests/test_model_inet.py
@@ -1,7 +1,7 @@
 # -*- coding: UTF-8 -*-
 from __future__ import absolute_import, unicode_literals
 
-import synapse.cortex as s_cortex
+import synapse.lib.tufo as s_tufo
 
 from synapse.tests.common import *
 
@@ -43,6 +43,9 @@ class InetModelTest(SynTest):
             self.eq(t0[1].get('inet:ipv6'), '::1')
             self.eq(t0[1].get('inet:ipv6:asn'), -1)
 
+            self.eq(core.getTypeRepr('inet:ipv6', '0:0:0:0:0:0:0:1'), '::1')
+            self.eq(core.getTypeRepr('inet:ipv6', '::1'), '::1')
+
     def test_model_inet_cidr4(self):
 
         with s_cortex.openurl('ram:///') as core:
@@ -82,6 +85,56 @@ class InetModelTest(SynTest):
             self.eq(t1[1].get('inet:fqdn:sfx'), 0)
             self.eq(t1[1].get('inet:fqdn:zone'), 1)
 
+    def test_model_inet_srv4_types(self):
+        with s_cortex.openurl('ram:///') as core:
+            core.setConfOpt('enforce', 1)
+            t0 = core.formTufoByProp('inet:tcp4', '8.8.8.8:80')
+            form, pprop = s_tufo.ndef(t0)
+            self.eq(pprop, 8830587502672)
+            self.eq(t0[1].get('inet:tcp4:port'), 80)
+            self.eq(t0[1].get('inet:tcp4:ipv4'), core.getTypeNorm('inet:ipv4', '8.8.8.8')[0])
+
+            # 1.2.3.4:8443
+            t1 = core.formTufoByProp('inet:tcp4', 1108152164603)
+            self.eq(t1[1].get('inet:tcp4:port'), 8443)
+            self.eq(t1[1].get('inet:tcp4:ipv4'), core.getTypeNorm('inet:ipv4', '1.2.3.4')[0])
+
+            t2 = core.formTufoByProp('inet:udp4', '8.8.8.8:80')
+            form, pprop = s_tufo.ndef(t2)
+            self.eq(pprop, 8830587502672)
+            self.eq(t2[1].get('inet:udp4:port'), 80)
+            self.eq(t2[1].get('inet:udp4:ipv4'), core.getTypeNorm('inet:ipv4', '8.8.8.8')[0])
+
+            # 1.2.3.4:8443
+            t3 = core.formTufoByProp('inet:udp4', 1108152164603)
+            self.eq(t3[1].get('inet:udp4:port'), 8443)
+            self.eq(t3[1].get('inet:udp4:ipv4'), core.getTypeNorm('inet:ipv4', '1.2.3.4')[0])
+
+    def test_model_inet_srv6_types(self):
+        with s_cortex.openurl('ram:///') as core:
+            core.setConfOpt('enforce', 1)
+            t0 = core.formTufoByProp('inet:tcp6', '[0:0:0:0:0:0:0:1]:80')
+            form, pprop = s_tufo.ndef(t0)
+            self.eq(pprop, '[::1]:80')
+            self.eq(t0[1].get('inet:tcp6:port'), 80)
+            self.eq(t0[1].get('inet:tcp6:ipv6'), '::1')
+
+            t1 = core.formTufoByProp('inet:tcp6', '[0:0:0:0:0:3:2:1]:443')
+            form, pprop = s_tufo.ndef(t1)
+            self.eq(pprop, '[::3:2:1]:443')
+            self.eq(t1[1].get('inet:tcp6:port'), 443)
+            self.eq(t1[1].get('inet:tcp6:ipv6'), '::3:2:1')
+
+            t2 = core.formTufoByProp('inet:udp6', '[0:0:0:0:0:3:2:1]:5000')
+            form, pprop = s_tufo.ndef(t2)
+            self.eq(pprop, '[::3:2:1]:5000')
+            self.eq(t2[1].get('inet:udp6:port'), 5000)
+            self.eq(t2[1].get('inet:udp6:ipv6'), '::3:2:1')
+
+            self.eq(core.getTypeRepr('inet:tcp6', '[0:0:0:0:0:0:0:1]:80'), '[::1]:80')
+            self.eq(core.getTypeRepr('inet:tcp6', '[::1]:80'), '[::1]:80')
+            self.eq(core.getTypeRepr('inet:tcp6', '[0:0:0:0:0:3:2:1]:5000'), '[::3:2:1]:5000')
+
     def test_model_inet_fqdn_unicode(self):
 
         with s_cortex.openurl('ram:///') as core:
@@ -356,7 +409,6 @@ class InetModelTest(SynTest):
 
         byts = self.getRev0DbByts()
 
-        # Fake some nodes
         iden0 = guid()
         iden1 = guid()
         tick = now()
@@ -387,3 +439,44 @@ class InetModelTest(SynTest):
 
                 t1 = core.getTufoByIden(iden1)
                 self.eq(t1[1].get('inet:url:ipv4'), 0x01020304)
+
+    def test_model_inet_201706201837(self):
+
+        byts = self.getRev0DbByts()
+
+        # Fake some nodes
+        iden1 = guid()
+        iden2 = guid()
+        tick = now()
+        rows = [
+            (iden1, 'tufo:form', 'inet:tcp4', tick),
+            (iden1, 'inet:tcp4', '1.2.3.4:80', tick),
+            (iden2, 'tufo:form', 'inet:udp4', tick),
+            (iden2, 'inet:udp4', '1.2.3.4:443', tick),
+        ]
+
+        with self.getTestDir() as temp:
+            finl = os.path.join(temp, 'test.db')
+
+            with open(finl, 'wb') as fd:
+                fd.write(byts)
+
+            url = 'sqlite:///%s' % finl
+
+            # Add the nodes into the storage object
+            with s_cortex.openstore(url) as store:
+                store.addRows(rows)
+
+            # Open the cortex, applying the data model updates
+            # Validate our nodes now have the correct data
+            with s_cortex.openurl(url) as core:
+                modlrev = core.getModlVers('inet')
+                self.ge(modlrev, 201706201837)
+
+                t1 = core.getTufoByIden(iden1)
+                self.eq(t1[1].get('inet:tcp4:port'), 80)
+                self.eq(t1[1].get('inet:tcp4:ipv4'), 0x01020304)
+
+                t2 = core.getTufoByIden(iden2)
+                self.eq(t2[1].get('inet:udp4:port'), 443)
+                self.eq(t2[1].get('inet:udp4:ipv4'), 0x01020304)