Repository: vertexproject/synapse

Issue:
need inet:netuser logon nodes to track sources and times

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=h1) Report
> Merging [#404](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=desc) into [master](https://codecov.io/gh/vertexproject/synapse/commit/6c0d7366920ea5b5bdc39172f81bc951d0ea976f?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/vertexproject/synapse/pull/404/graphs/tree.svg?token=c1t4BsMAUL&src=pr&width=650&height=150)](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #404      +/-   ##
==========================================
- Coverage   90.45%   90.41%   -0.04%     
==========================================
  Files         127      127              
  Lines       14369    14369              
==========================================
- Hits        12997    12992       -5     
- Misses       1372     1377       +5
```


| [Impacted Files](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [synapse/models/inet.py](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree#diff-c3luYXBzZS9tb2RlbHMvaW5ldC5weQ==) | `98.42% <100%> (ø)` | :arrow_up: |
| [synapse/lib/socket.py](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree#diff-c3luYXBzZS9saWIvc29ja2V0LnB5) | `83.56% <0%> (-1.14%)` | :arrow_down: |
| [synapse/links/ssl.py](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree#diff-c3luYXBzZS9saW5rcy9zc2wucHk=) | `86.71% <0%> (-0.79%)` | :arrow_down: |
| [synapse/daemon.py](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree#diff-c3luYXBzZS9kYWVtb24ucHk=) | `83.22% <0%> (-0.21%)` | :arrow_down: |
| [synapse/cores/common.py](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=tree#diff-c3luYXBzZS9jb3Jlcy9jb21tb24ucHk=) | `94.07% <0%> (+0.08%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=footer). Last update [6c0d736...8f4b785](https://codecov.io/gh/vertexproject/synapse/pull/404?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/synapse/lib/types.py b/synapse/lib/types.py
index d3691b459..26250328c 100644
--- a/synapse/lib/types.py
+++ b/synapse/lib/types.py
@@ -109,7 +109,8 @@ class GuidType(DataType):
         # ( sigh... eventually everything will be a cortex... )
         node = self._getTufoByProp(self._guid_alias, valu[1:])
         if node is None:
-            self._raiseBadValu(valu, mesg='no result for guid resolver')
+            self._raiseBadValu(valu, mesg='no result for guid resolver',
+                               alias=self._guid_alias)
 
         iden = node[1].get(node[1].get('tufo:form'))
         return iden, {}
diff --git a/synapse/models/inet.py b/synapse/models/inet.py
index 2d949fa32..97d9311f8 100644
--- a/synapse/models/inet.py
+++ b/synapse/models/inet.py
@@ -320,7 +320,6 @@ class CidrType(DataType):
         return valu
 
 class InetMod(CoreModule):
-
     def initCoreModule(self):
         # add an inet:defang cast to swap [.] to .
         self.core.addTypeCast('inet:defang', castInetDeFang)
@@ -401,6 +400,47 @@ class InetMod(CoreModule):
     def _revModl201708231646(self):
         pass # for legacy/backward compat
 
+    @modelrev('inet', 201709181501)
+    def _revModl201709181501(self):
+        '''
+        Replace inet:whois:rec:ns<int> rows with inet:whos:recns nodes.
+        '''
+        adds = []
+        srcprops = ('inet:whois:rec:ns1', 'inet:whois:rec:ns2', 'inet:whois:rec:ns3', 'inet:whois:rec:ns4')
+        delprops = set()
+
+        tick = s_common.now()
+
+        # We could use the joins API but we would have to still fold rows into tufos for the purpose of migration.
+        nodes = self.core.getTufosByProp('inet:whois:rec')
+
+        for node in nodes:
+            rec = node[1].get('inet:whois:rec')
+            for prop in srcprops:
+                ns = node[1].get(prop)
+                if not ns:
+                    continue
+                delprops.add(prop)
+                iden = s_common.guid()
+                pprop = s_common.guid([ns, rec])
+                fqdn = node[1].get('inet:whois:rec:fqdn')
+                asof = node[1].get('inet:whois:rec:asof')
+                rows = [
+                    (iden, 'tufo:form', 'inet:whois:recns', tick),
+                    (iden, 'inet:whois:recns', pprop, tick),
+                    (iden, 'inet:whois:recns:ns', ns, tick),
+                    (iden, 'inet:whois:recns:rec', rec, tick),
+                    (iden, 'inet:whois:recns:rec:fqdn', fqdn, tick),
+                    (iden, 'inet:whois:recns:rec:asof', asof, tick),
+                ]
+                adds.extend(rows)
+
+        if adds:
+            self.core.addRows(adds)
+
+        for prop in delprops:
+            self.core.delRowsByProp(prop)
+
     @staticmethod
     def getBaseModels():
         modl = {
@@ -454,6 +494,8 @@ class InetMod(CoreModule):
 
                 ('inet:netuser', {'subof': 'sepr', 'sep': '/', 'fields': 'site,inet:fqdn|user,inet:user',
                                   'doc': 'A user account at a given web address', 'ex': 'twitter.com/invisig0th'}),
+                ('inet:web:logon', {'subof': 'guid',
+                                   'doc': 'An instance of a user account authenticating to a service.', }),
 
                 ('inet:netgroup', {'subof': 'sepr', 'sep': '/', 'fields': 'site,inet:fqdn|name,ou:name',
                                    'doc': 'A group within an online community'}),
@@ -478,6 +520,8 @@ class InetMod(CoreModule):
                 ('inet:whois:rec',
                  {'subof': 'sepr', 'sep': '@', 'fields': 'fqdn,inet:fqdn|asof,time', 'doc': 'A whois record',
                   'ex': ''}),
+                ('inet:whois:recns', {'subof': 'comp', 'fields': 'ns,inet:fqdn|rec,inet:whois:rec',
+                                      'doc': 'A nameserver associated with a given WHOIS record.'}),
 
                 ('inet:whois:contact', {'subof': 'comp', 'fields': 'rec,inet:whois:rec|type,str:lwr',
                                         'doc': 'A whois contact for a specific record'}),
@@ -525,6 +569,7 @@ class InetMod(CoreModule):
 
                 ('inet:asn', {'ptype': 'inet:asn', 'doc': 'An Autonomous System'}, (
                     ('name', {'ptype': 'str:lwr', 'defval': '??'}),
+                    ('owner', {'ptype': 'ou:org', 'doc': 'Organization which controls an ASN'}),
                 )),
 
                 ('inet:asnet4',
@@ -663,6 +708,16 @@ class InetMod(CoreModule):
                     ('seen:max', {'ptype': 'time:max'}),
                 ]),
 
+                ('inet:web:logon', {'ptype': 'inet:web:logon'}, [
+                    ('netuser', {'ptype': 'inet:netuser', 'doc': 'The netuser associated with the logon event.', }),
+                    ('netuser:site', {'ptype': 'inet:fqdn', }),
+                    ('netuser:user', {'ptype': 'inet:user', }),
+                    ('time', {'ptype': 'time', 'doc': 'The time the netuser logged into the service', }),
+                    ('ipv4', {'ptype': 'inet:ipv4', 'doc': 'The source IPv4 address of the logon.'}),
+                    ('ipv6', {'ptype': 'inet:ipv6', 'doc': 'The source IPv6 address of the logon.'}),
+                    ('logout', {'ptype': 'time', 'doc': 'The time the netuser logged out of the service.'})
+                ]),
+
                 ('inet:netgroup', {}, [
                     ('site', {'ptype': 'inet:fqdn', 'ro': 1}),
                     ('name', {'ptype': 'ou:name', 'ro': 1}),
@@ -760,10 +815,13 @@ class InetMod(CoreModule):
                     ('expires', {'ptype': 'time', 'doc': 'The "expires" time from the whois record'}),
                     ('registrar', {'ptype': 'inet:whois:rar', 'defval': '??'}),
                     ('registrant', {'ptype': 'inet:whois:reg', 'defval': '??'}),
-                    ('ns1', {'ptype': 'inet:fqdn'}),
-                    ('ns2', {'ptype': 'inet:fqdn'}),
-                    ('ns3', {'ptype': 'inet:fqdn'}),
-                    ('ns4', {'ptype': 'inet:fqdn'}),
+                ]),
+
+                ('inet:whois:recns', {}, [
+                    ('ns', {'ptype': 'inet:fqdn', 'ro': 1, 'doct': 'Nameserver for a given FQDN'}),
+                    ('rec', {'ptype': 'inet:whois:rec', 'ro': 1}),
+                    ('rec:fqdn', {'ptype': 'inet:fqdn', 'ro': 1}),
+                    ('rec:asof', {'ptype': 'time', 'ro': 1}),
                 ]),
 
                 ('inet:whois:contact', {}, [
@@ -797,4 +855,4 @@ class InetMod(CoreModule):
             ),
         }
         name = 'inet'
-        return ((name, modl), )
+        return ((name, modl),)


Test patch:
diff --git a/synapse/tests/test_model_inet.py b/synapse/tests/test_model_inet.py
index 2c07381e3..c8dda056c 100644
--- a/synapse/tests/test_model_inet.py
+++ b/synapse/tests/test_model_inet.py
@@ -70,6 +70,20 @@ class InetModelTest(SynTest):
             self.nn(core.getTufoByProp('inet:ipv4', 0x01020304))
             self.nn(core.getTufoByProp('inet:ipv4', 0x05060708))
 
+            o1 = core.formTufoByProp('ou:org', '*', alias='vertex')
+            _, o1pprop = s_tufo.ndef(o1)
+            t1 = core.formTufoByProp('inet:asn', 12345)
+            self.none(t1[1].get('inet:asn:owner'))
+            t1 = core.setTufoProps(t1, owner='$vertex')
+            self.eq(t1[1].get('inet:asn:owner'), o1pprop)
+            # TODO: Uncomment when we have a global alias resolver in place.
+            # self.nn(core.getTufoByProp('ou:alias', 'vertex'))
+            t2 = core.formTufoByProp('inet:asn', 12346, owner='$vertex')
+            self.eq(t2[1].get('inet:asn:owner'), o1pprop)
+            # Lift asn's by owner with guid resolver syntax
+            nodes = core.eval('inet:asn:owner=$vertex')
+            self.eq(len(nodes), 2)
+
     def test_model_inet_fqdn(self):
         with s_cortex.openurl('ram:///') as core:
             t0 = core.formTufoByProp('inet:fqdn', 'com', sfx=1)
@@ -390,6 +404,23 @@ class InetModelTest(SynTest):
             self.eq(len(core.eval('inet:whois:rec="woot.com@20501217"')), 1)
             self.eq(len(core.eval('inet:whois:contact:rec="woot.com@20501217"')), 1)
 
+    def test_model_inet_whois_recns(self):
+        with s_cortex.openurl('ram:///') as core:
+            core.setConfOpt('enforce', 1)
+
+            node = core.formTufoByProp('inet:whois:rec', 'woot.com@20501217')
+            form, pprop = s_tufo.ndef(node)
+            node = core.formTufoByProp('inet:whois:recns', ['ns1.woot.com', pprop])
+            self.eq(node[1].get('inet:whois:recns:ns'), 'ns1.woot.com')
+            self.eq(node[1].get('inet:whois:recns:rec'), pprop)
+            self.eq(node[1].get('inet:whois:recns:rec:fqdn'), 'woot.com')
+            self.eq(node[1].get('inet:whois:recns:rec:asof'), 2554848000000)
+            nodes = core.eval('inet:whois:recns:rec:fqdn=woot.com')
+            self.eq(node[0], nodes[0][0])
+            nodes = core.eval('inet:whois:rec:fqdn=woot.com inet:whois:rec->inet:whois:recns:rec')
+            self.eq(len(nodes), 1)
+            self.eq(node[0], nodes[0][0])
+
     def test_model_fqdn_punycode(self):
 
         with s_cortex.openurl('ram:///') as core:
@@ -405,6 +436,34 @@ class InetModelTest(SynTest):
 
             self.raises(BadTypeValu, core.getTypeNorm, 'inet:fqdn', '!@#$%')
 
+    def test_model_inet_weblogon(self):
+
+        with s_cortex.openurl('ram:///') as core:
+            core.setConfOpt('enforce', 1)
+            tick = now()
+
+            t0 = core.formTufoByProp('inet:web:logon', '*',
+                                     netuser='vertex.link/pennywise',
+                                     time=tick)
+
+            self.nn(t0)
+
+            self.eq(t0[1].get('inet:web:logon:time'), tick)
+            self.eq(t0[1].get('inet:web:logon:netuser'), 'vertex.link/pennywise')
+            self.eq(t0[1].get('inet:web:logon:netuser:user'), 'pennywise')
+            self.eq(t0[1].get('inet:web:logon:netuser:site'), 'vertex.link')
+
+            # Pivot from a netuser to the netlogon forms via storm
+            self.nn(core.getTufoByProp('inet:netuser', 'vertex.link/pennywise'))
+            nodes = core.eval('inet:netuser=vertex.link/pennywise inet:netuser -> inet:web:logon:netuser')
+            self.eq(len(nodes), 1)
+
+            t0 = core.setTufoProps(t0, ipv4=0x01020304, logout=tick + 1, ipv6='0:0:0:0:0:0:0:1')
+            self.eq(t0[1].get('inet:web:logon:ipv4'), 0x01020304)
+            self.eq(t0[1].get('inet:web:logon:logout'), tick + 1)
+            self.eq(t0[1].get('inet:web:logon:logout') - t0[1].get('inet:web:logon:time'), 1)
+            self.eq(t0[1].get('inet:web:logon:ipv6'), '::1')
+
     def test_model_inet_201706121318(self):
 
         iden0 = guid()
@@ -469,3 +528,49 @@ class InetModelTest(SynTest):
                 t2 = core.getTufoByIden(iden1)
                 self.eq(t2[1].get('inet:udp4:port'), 443)
                 self.eq(t2[1].get('inet:udp4:ipv4'), 0x01020304)
+
+    def test_model_inet_201709181501(self):
+        data = {}
+        iden0 = guid()
+        tick = now()
+        rows = [
+            (iden0, 'tufo:form', 'inet:whois:rec', tick),
+            (iden0, 'inet:whois:rec', 'vertex.link@2017/09/18 15:01:00.000', tick),  # 1505746860000,
+            (iden0, 'inet:whois:rec:fqdn', 'vertex.link', tick),
+            (iden0, 'inet:whois:rec:asof', 1505746860000, tick),
+            (iden0, 'inet:whois:rec:ns1', 'ns1.vertex.link', tick),
+            (iden0, 'inet:whois:rec:ns2', 'ns2.vertex.link', tick),
+            (iden0, 'inet:whois:rec:ns3', 'ns3.vertex.link', tick),
+            (iden0, 'inet:whois:rec:ns4', 'ns4.vertex.link', tick),
+        ]
+
+        with s_cortex.openstore('ram:///') as stor:
+
+            # force model migration callbacks
+            stor.setModlVers('inet', 0)
+
+            def addrows(mesg):
+                stor.addRows(rows)
+                data['added'] = True
+            stor.on('modl:vers:rev', addrows, name='inet', vers=201709181501)
+
+            with s_cortex.fromstore(stor) as core:
+
+                t_guid, _ = core.getTypeNorm('inet:whois:recns', ['ns1.vertex.link',
+                                                                  'vertex.link@2017/09/18 15:01:00.000'])
+
+                node = core.eval('inet:whois:rec')[0]
+                self.notin('inet:whois:rec:ns1', node[1])
+                self.notin('inet:whois:rec:ns2', node[1])
+                self.notin('inet:whois:rec:ns3', node[1])
+                self.notin('inet:whois:rec:ns4', node[1])
+
+                nodes = core.eval('inet:whois:recns')
+                self.eq(len(nodes), 4)
+
+                nodes = core.eval('inet:whois:recns={}'.format(t_guid))
+                self.eq(len(nodes), 1)
+                node = nodes[0]
+                self.eq(node[1].get('inet:whois:recns:ns'), 'ns1.vertex.link')
+                self.eq(node[1].get('inet:whois:recns:rec:fqdn'), 'vertex.link')
+                self.eq(node[1].get('inet:whois:recns:rec:asof'), 1505746860000)