Repository: vertexproject/synapse

Issue:
Creating nodes from secondary properties using autoadds may result in missing properties
When nodes are made from autoadds, those nodes are made using the system normalized primary property for the node and are missing all subs which may have been applicable for that node.  This primarily applies to nodes which have a primary property as a guid (either a GuidType or something that is a computed guid).  This leads to a issue where its possible to create meaningful nodes from API or Storm queries which lack properties, and in some cases cannot have those properties set because of ro flags.

An example of this behavior is the following block - an XREF node is used to directly make a inet:web:post node, however the inet:web:post node only would contain the primary property and would be missing the ``:acct`` and ``:text`` properties.

```
    def test_model_inet_postref_postmissingprops(self):
        with self.getRamCore() as core:

            postref_tufo = core.formTufoByProp('inet:web:postref', (('vertex.link/user', 'mypost 0.0.0.0'), ('inet:ipv4', 0)))
            post_tufo = core.formTufoByProp('inet:web:post', ('vertex.link/user', 'mypost 0.0.0.0'))
            # This feels wrong...

            self.eq(post_tufo[1].get('inet:web:post:acct'), 'vertex.link/user')
            self.eq(post_tufo[1].get('inet:web:post:text'), 'mypost 0.0.0.0')
```

Credit goes to @vertexmc  for finding this issue.

Discussion:
vEpiphyte: https://upload.wikimedia.org/wikipedia/commons/4/47/River_terrapin.jpg
codecov[bot]: # [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=h1) Report
> Merging [#478](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=desc) into [master](https://codecov.io/gh/vertexproject/synapse/commit/75b638425eb31e5a7b448f225f5883599eed7fab?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/vertexproject/synapse/pull/478/graphs/tree.svg?width=650&src=pr&token=c1t4BsMAUL&height=150)](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #478      +/-   ##
==========================================
- Coverage   91.22%   91.16%   -0.07%     
==========================================
  Files         123      122       -1     
  Lines       14621    14460     -161     
==========================================
- Hits        13338    13182     -156     
+ Misses       1283     1278       -5
```


| [Impacted Files](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [synapse/exc.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9leGMucHk=) | `94.26% <ø> (-0.05%)` | :arrow_down: |
| [synapse/cores/common.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9jb3Jlcy9jb21tb24ucHk=) | `93.75% <100%> (+0.04%)` | :arrow_up: |
| [synapse/lib/types.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9saWIvdHlwZXMucHk=) | `90.33% <100%> (+0.04%)` | :arrow_up: |
| [synapse/lib/persist.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9saWIvcGVyc2lzdC5weQ==) | `90.42% <0%> (-1.07%)` | :arrow_down: |
| [synapse/axon.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9heG9uLnB5) | `96.87% <0%> (ø)` | :arrow_up: |
| [synapse/models/infotech.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9tb2RlbHMvaW5mb3RlY2gucHk=) | `100% <0%> (ø)` | :arrow_up: |
| [synapse/lib/iq.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9saWIvaXEucHk=) | | |
| [synapse/lib/socket.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9saWIvc29ja2V0LnB5) | `83.33% <0%> (+0.32%)` | :arrow_up: |
| [synapse/lib/threads.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9saWIvdGhyZWFkcy5weQ==) | `69.11% <0%> (+0.98%)` | :arrow_up: |
| [synapse/eventbus.py](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=tree#diff-c3luYXBzZS9ldmVudGJ1cy5weQ==) | `90.9% <0%> (+1.29%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=footer). Last update [75b6384...ca8bee2](https://codecov.io/gh/vertexproject/synapse/pull/478?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/synapse/cores/common.py b/synapse/cores/common.py
index 097756bef..c9dfca297 100644
--- a/synapse/cores/common.py
+++ b/synapse/cores/common.py
@@ -261,7 +261,7 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
         node[1][form] = norm
         node[1]['tufo:form'] = form
-        node[1]['tufo:formed'] = s_common.now()
+        node[1]['node:created'] = s_common.now()
 
         self.runt_props[(form, None)].append(node)
         self.runt_props[(form, norm)].append(node)
@@ -2118,22 +2118,6 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
         return tufo
 
-    def addTufoEvent(self, form, **props):
-        '''
-        Add a "non-deconflicted" tufo by generating a guid
-
-        Example:
-
-            tufo = core.addTufoEvent('foo',bar=baz)
-
-        Notes:
-
-            If props contains a key "time" it will be used for
-            the cortex timestamp column in the row storage.
-
-        '''
-        return self.addTufoEvents(form, (props,))[0]
-
     def addJsonText(self, form, text):
         '''
         Add and fully index a blob of JSON text.
@@ -2290,82 +2274,6 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
         return props
 
-    def addTufoEvents(self, form, propss):
-        '''
-        Add a list of tufo events in bulk.
-
-        Example:
-
-            propss = [
-                {'foo':10,'bar':30},
-                {'foo':11,'bar':99},
-            ]
-
-            core.addTufoEvents('woot',propss)
-
-        '''
-        tname = self.getPropTypeName(form)
-        if tname is None and self.enforce:
-            raise s_common.NoSuchForm(name=form)
-
-        if tname and not self.isSubType(tname, 'guid'):
-            raise s_common.NotGuidForm(name=form)
-
-        with self.getCoreXact() as xact:
-
-            ret = []
-
-            for chunk in chunked(1000, propss):
-
-                adds = []
-                rows = []
-                allfulls = []
-
-                for props in chunk:
-
-                    iden = s_common.guid()
-
-                    fulls = self._normTufoProps(form, props, isadd=True)
-
-                    self._addDefProps(form, fulls)
-
-                    fulls[form] = iden
-                    fulls['tufo:form'] = form
-
-                    if self.autoadd:
-                        allfulls.append(fulls)
-
-                    # fire these immediately since we need them to potentially fill
-                    # in values before we generate rows for the new tufo
-                    self.fire('node:form', form=form, valu=iden, props=fulls)
-
-                    # Ensure we have ALL the required props after node:form is fired.
-                    self._reqProps(form, fulls)
-
-                    rows.extend([(iden, p, v, xact.tick) for (p, v) in fulls.items()])
-
-                    # sneaky ephemeral/hidden prop to identify newly created tufos
-                    fulls['.new'] = 1
-
-                    node = (iden, fulls)
-
-                    ret.append(node)
-                    adds.append((form, iden, props, node))
-
-                self.addRows(rows)
-
-                # fire splice events
-                for form, valu, props, node in adds:
-                    xact.fire('node:add', form=form, valu=valu, node=node)
-                    xact.spliced('node:add', form=form, valu=valu, props=props)
-                    xact.trigger(node, 'node:add', form=form)
-
-                if self.autoadd:
-                    for fulls in allfulls:
-                        self._runToAdd(fulls)
-
-        return ret
-
     def _reqProps(self, form, fulls):
         if not self.enforce:
             return
@@ -2450,7 +2358,7 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
     def formTufoByProp(self, prop, valu, **props):
         '''
         Form an (iden,info) tuple by atomically deconflicting
-        the existance of prop=valu and creating it if not present.
+        the existence of prop=valu and creating it if not present.
 
         Args:
 
@@ -2459,9 +2367,25 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
             **props:    Additional secondary properties for the node
 
         Example:
+            Make a node for the FQDN woot.com::
+
+                tufo = core.formTufoByProp('inet:fqdn','woot.com')
 
-            tufo = core.formTufoByProp('inet:fqdn','woot.com')
+        Notes:
+            When forming nodes whose primary property is derived from the
+            GuidType, deconfliction can be skipped if the value is set to
+            None. This allows for high-speed ingest of event type data which
+            does note require deconfliction.
+
+            This API will fire a ``node:form`` event prior to creating rows,
+            allowing callbacks to populate any additional properties on the
+            node.  After node creation is finished, ``node:add`` events are
+            fired on for the Cortex event bus, splices and triggers.
 
+        Returns:
+            ((str, dict)): The newly formed tufo, or the existing tufo if
+            the node already exists.  The ephemeral property ".new" can be
+            checked to see if the node was newly created or not.
         '''
         ctor = self.seedctors.get(prop)
         if ctor is not None:
@@ -2499,20 +2423,31 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
             props.update(subs)
 
-            fulls = self._normTufoProps(prop, props, isadd=True)
-
             # create a "full" props dict which includes defaults
+            fulls = self._normTufoProps(prop, props)
             self._addDefProps(prop, fulls)
 
             fulls[prop] = valu
             fulls['tufo:form'] = prop
-            fulls['tufo:formed'] = s_common.now()
+            fulls['node:created'] = s_common.now()
+
+            # Examine the fulls dictionary and identify any props which are
+            # themselves forms, and extract the form/valu/subs from the fulls
+            # dictionary so we can later make those nodes
+            toadds = None
+            if self.autoadd:
+                toadds = self._formToAdd(prop, fulls)
+
+            # Remove any non-model props present in the props and fulls
+            # dictionary which may have been added during _normTufoProps
+            self._pruneFulls(prop, fulls, props, isadd=True)
 
             # update our runtime form counters
             self.formed[prop] += 1
 
-            # fire these immediately since we need them to potentially fill
-            # in values before we generate rows for the new tufo
+            # Fire these immediately since we need them to potentially fill
+            # in values before we generate rows for the new tufo. It is
+            # possible this callback may generate extra-model values.
             self.fire('node:form', form=prop, valu=valu, props=fulls)
 
             # Ensure we have ALL the required props after node:form is fired.
@@ -2524,23 +2459,84 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
             tufo = (iden, fulls)
 
+            # Cache the tufo data now so we can avoid having a .new ephemeral
+            # property in the cache
             if self.caching:
-                # avoid .new in cache
                 cachefo = (iden, dict(fulls))
                 for p, v in fulls.items():
                     self._bumpTufoCache(cachefo, p, None, v)
 
-            # fire notification events
+            # Run any autoadd nodes we may have. In the event of autoadds being
+            # present, our subsequent node:add events are fired depth-first
+            if self.autoadd and toadds is not None:
+                self._runToAdd(toadds)
+
+            # fire the node:add events from the xact
             xact.fire('node:add', form=prop, valu=valu, node=tufo)
             xact.spliced('node:add', form=prop, valu=valu, props=props)
             xact.trigger(tufo, 'node:add', form=prop)
 
-            if self.autoadd:
-                self._runToAdd(fulls)
-
         tufo[1]['.new'] = True
         return tufo
 
+    def _pruneFulls(self, form, fulls, props, isadd=False):
+        '''
+        Modify fulls and props dicts in place to remove non-model props.
+
+        Args:
+            form (str): Form of the dictionary being operated on.
+            fulls (dict): Dictionary of full property name & valu pairs.
+            props (dict): Dictionary of property name & value pairs.
+            isadd (bool): Bool indicating if the data is newly being added or not.
+
+        Returns:
+            None
+        '''
+        splitp = form + ':'
+        for name in list(fulls.keys()):
+            if name in ('tufo:form', 'node:created', 'node:loc'):
+                continue
+            if not self.isSetPropOk(name, isadd):
+                prop = name.split(splitp)[1]
+                props.pop(prop, None)
+                fulls.pop(name, None)
+
+    def _formToAdd(self, form, fulls):
+        '''
+        Build a list of property, valu, **props from a dictionary of fulls.
+
+        Args:
+            form (str): Form of fulls
+            fulls (dict):
+
+        Returns:
+            list: List of tuples (prop,valu,**props) for consumption by formTufoByProp.
+        '''
+        ret = []
+        skips = ('tufo:form', 'node:created', 'node:loc')
+        valu = fulls.get(form)
+        for fprop, fvalu in fulls.items():
+            if fprop in skips:
+                continue
+            ptype = self.getPropTypeName(fprop)
+            prefix = fprop + ':'
+            plen = len(prefix)
+            for stype in self.getTypeOfs(ptype):
+                if self.isRuntForm(stype):
+                    continue
+                if self.isTufoForm(stype):
+                    # We don't want to recurse on forming ourself with all our same properties
+                    if stype == form and valu == fvalu:
+                        continue
+                    subs = {}
+
+                    for _fprop, _fvalu in fulls.items():
+                        if _fprop.startswith(prefix):
+                            k = _fprop[plen:]
+                            subs[k] = _fvalu
+                    ret.append((stype, fvalu, subs))
+        return ret
+
     def delTufo(self, tufo):
         '''
         Delete a tufo and it's associated props/lists/etc.
@@ -2720,47 +2716,36 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
         for k, v in self.getFormDefs(form):
             fulls.setdefault(k, v)
 
-    def _runToAdd(self, fulls):
-        toadd = set()
+    def _runToAdd(self, toadds):
+        for form, valu, props in toadds:
+            self.formTufoByProp(form, valu, **props)
 
-        for prop, valu in fulls.items():
-            ptype = self.getPropTypeName(prop)
-            for stype in self.getTypeOfs(ptype):
-
-                if self.isRuntForm(stype):
-                    continue
-
-                if self.isTufoForm(stype):
-                    toadd.add((stype, valu))
-
-        for form, valu in toadd:
-            self.formTufoByProp(form, valu)
-
-    def _normTufoProps(self, form, props, tufo=None, isadd=False):
+    def _normTufoProps(self, form, props, tufo=None):
         '''
         This will both return a dict of fully qualified props as
         well as modify the given props dict inband to normalize
         the values.
         '''
-
         fulls = {}
         for name in list(props.keys()):
 
             valu = props.get(name)
 
             prop = form + ':' + name
-            if not self.isSetPropOk(prop, isadd=isadd):
-                props.pop(name, None)
-                continue
 
             oldv = None
             if tufo is not None:
                 oldv = tufo[1].get(prop)
 
             valu, subs = self.getPropNorm(prop, valu, oldval=oldv)
-            if tufo is not None and tufo[1].get(prop) == valu:
-                props.pop(name, None)
-                continue
+            if tufo is not None:
+                if tufo[1].get(prop) == valu:
+                    props.pop(name, None)
+                    continue
+                _isadd = not bool(oldv)
+                if not self.isSetPropOk(prop, isadd=_isadd):
+                    props.pop(name, None)
+                    continue
 
             # any sub-properties to populate?
             for sname, svalu in subs.items():
@@ -2870,6 +2855,8 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
         Args:
             prop (str): Full property name to check.
+            isadd (bool): Bool indicating that the property check is being
+            done on a property which has not yet been set on a node.
 
         Examples:
             Check if a value is valid before calling a function.::
@@ -2970,10 +2957,14 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
         iden = tufo[0]
         valu = tufo[1].get(form)
 
+        toadds = None
+        if self.autoadd:
+            toadds = self._formToAdd(form, fulls)
+        self._pruneFulls(form, fulls, props, isadd=True)
+
         with self.getCoreXact() as xact:
 
             for p, v in fulls.items():
-
                 oldv = tufo[1].get(p)
                 self.setRowsByIdProp(iden, p, v)
 
@@ -2992,8 +2983,8 @@ class Cortex(EventBus, DataModel, Runtime, s_ingest.IngestApi):
 
                 xact.trigger(tufo, 'node:prop:set', form=form, prop=p)
 
-            if self.autoadd:
-                self._runToAdd(fulls)
+            if self.autoadd and toadds is not None:
+                self._runToAdd(toadds)
 
         return tufo
 
diff --git a/synapse/exc.py b/synapse/exc.py
index bf310f973..f7edb42a9 100644
--- a/synapse/exc.py
+++ b/synapse/exc.py
@@ -82,8 +82,6 @@ class BadCoreStore(SynErr):
 class CantDelProp(SynErr): pass
 class CantSetProp(SynErr): pass
 
-class NotGuidForm(SynErr): pass
-
 class MustBeLocal(SynErr): pass
 class MustBeProxy(SynErr): pass
 
diff --git a/synapse/lib/types.py b/synapse/lib/types.py
index 47bab12be..ae83de7e1 100644
--- a/synapse/lib/types.py
+++ b/synapse/lib/types.py
@@ -553,6 +553,10 @@ class XrefType(DataType):
             'xref': pvval,
         }
 
+        for k, v in vsub.items():
+            k = self._sorc_name + ':' + k
+            subs[k] = v
+
         for k, v in pvsub.items():
             k = 'xref:' + k
             subs[k] = v
diff --git a/synapse/models/inet.py b/synapse/models/inet.py
index 6f072b249..bde40731a 100644
--- a/synapse/models/inet.py
+++ b/synapse/models/inet.py
@@ -634,7 +634,7 @@ class InetMod(CoreModule):
                         break
 
                     for tufo in tufos:
-                        i, p, v, t = self.core.getRowsByIdProp(tufo[0], xref_prop_prop)[0] # unavoidable until we have `tufo:formed` prop
+                        i, p, v, t = self.core.getRowsByIdProp(tufo[0], xref_prop_prop)[0] # unavoidable until we have `node:created` prop
                         adds, dels = [], []
 
                         # modify :xref:prop
diff --git a/synapse/models/syn.py b/synapse/models/syn.py
index 2260e0168..d05a4847e 100644
--- a/synapse/models/syn.py
+++ b/synapse/models/syn.py
@@ -184,16 +184,16 @@ class SynMod(CoreModule):
             adds = []
             logger.debug('Lifting tufo:form rows')
             for i, _, v, t in self.core.store.getRowsByProp('tufo:form'):
-                adds.append((i, 'tufo:formed', t, now),)
-            logger.debug('Deleting existing tufo:formed rows')
-            self.core.store.delRowsByProp('tufo:formed')
+                adds.append((i, 'node:created', t, now),)
+            logger.debug('Deleting existing node:created rows')
+            self.core.store.delRowsByProp('node:created')
             if adds:
                 tot = len(adds)
-                logger.debug('Adding {:,d} tufo:formed rows'.format(tot))
+                logger.debug('Adding {:,d} node:created rows'.format(tot))
                 i = 0
                 n = 100000
                 for chunk in s_common.chunks(adds, n):
                     self.core.store.addRows(chunk)
                     i = i + len(chunk)
                     logger.debug('Loading {:,d} [{}%] rows into transaction'.format(i, int((i / tot) * 100)))
-        logger.debug('Finished adding tufo:formed rows to the Cortex')
+        logger.debug('Finished adding node:created rows to the Cortex')


Test patch:
diff --git a/synapse/tests/test_axon.py b/synapse/tests/test_axon.py
index f49dff25c..8f567436a 100644
--- a/synapse/tests/test_axon.py
+++ b/synapse/tests/test_axon.py
@@ -595,6 +595,7 @@ class AxonClusterTest(SynTest):
             blobs = axcluster.find('md5', craphash)
             self.eq(len(blobs), 0)
 
+            time.sleep(0.2)  # Yield to axon threads
             blobs = axcluster.find('md5', asdfhash)
             # We have two blobs for the same hash since the clone of axfo0 is up on host1/host2
             self.eq(len(blobs), 2)
diff --git a/synapse/tests/test_cortex.py b/synapse/tests/test_cortex.py
index c36c7be0a..565e4e211 100644
--- a/synapse/tests/test_cortex.py
+++ b/synapse/tests/test_cortex.py
@@ -695,7 +695,7 @@ class CortexBaseTest(SynTest):
         unodes = core.getTufosByProp('inet:tcp4:gatenumber')
         self.len(10, unodes)
         for node in unodes:
-            self.isin('tufo:formed', node[1])
+            self.isin('node:created', node[1])
             self.isin('inet:tcp4', node[1])
             self.isin('inet:tcp4:ipv4', node[1])
             self.isin('inet:tcp4:gatenumber', node[1])
@@ -1043,14 +1043,34 @@ class CortexTest(SynTest):
             self.eq(len(core.getTufosByProp('strform:foo', valu='zap')), 1)
             self.eq(len(core.getTufosByProp('strform:bar', valu='zap')), 1)
 
-        # Try using setprops with an built-in model which type subprops
-        t0 = core.formTufoByProp('inet:web:acct', 'vertex.link/pennywise')
-        self.notin('inet:web:acct:email', t0[1])
-        props = {'email': 'pennywise@vertex.link'}
-        core.setTufoProps(t0, **props)
-        self.isin('inet:web:acct:email', t0[1])
-        t1 = core.getTufoByProp('inet:email', 'pennywise@vertex.link')
-        self.nn(t1)
+            # Try using setprops with an built-in model which type subprops
+            t0 = core.formTufoByProp('inet:web:acct', 'vertex.link/pennywise')
+            self.notin('inet:web:acct:email', t0[1])
+            props = {'email': 'pennywise@vertex.link'}
+            core.setTufoProps(t0, **props)
+            self.isin('inet:web:acct:email', t0[1])
+            t1 = core.getTufoByProp('inet:email', 'pennywise@vertex.link')
+            self.nn(t1)
+
+            # Trying settufoprops on a ro prop doens't change anything
+            self.eq(t0[1].get('inet:web:acct:user'), 'pennywise')
+            t0 = core.setTufoProps(t0, user='ninja')
+            self.eq(t0[1].get('inet:web:acct:user'), 'pennywise')
+
+            # Try forming a node from its normalize value and then setting
+            # ro props after the fact. Also ensure those secondary props which
+            # may trigger autoadds are generating the autoadds and do not retain
+            # those non-model seconadry props.
+            valu, subs = core.getTypeNorm('inet:web:post', '(vertex.ninja/ninja,"Just ninja things.")')
+            t0 = core.formTufoByProp('inet:web:post', valu)
+            self.eq(t0[1].get('inet:web:post'), valu)
+            self.none(t0[1].get('inet:web:post:acct'))
+            self.none(t0[1].get('inet:web:post:text'))
+            t0 = core.setTufoProps(t0, **subs)
+            self.eq(t0[1].get('inet:web:post:acct'), 'vertex.ninja/ninja')
+            self.eq(t0[1].get('inet:web:post:text'), 'Just ninja things.')
+            t0 = core.setTufoProps(t0, text='Throwing stars are cool!')
+            self.eq(t0[1].get('inet:web:post:text'), 'Just ninja things.')
 
     def test_cortex_tufo_pop(self):
         with self.getRamCore() as core:
@@ -1339,8 +1359,8 @@ class CortexTest(SynTest):
         # we're also doing the same via storm
         self.eq(len(core.eval('[inet:fqdn=w00t.com +#some.tag]')), 1)
         self.eq(len(core.eval('inet:fqdn=w00t.com')), 1)
-        self.eq(len(core.eval('inet:fqdn=w00t.com +tufo:formed<1')), 0)
-        self.eq(len(core.eval('inet:fqdn=w00t.com +tufo:formed>1')), 1)
+        self.eq(len(core.eval('inet:fqdn=w00t.com +node:created<1')), 0)
+        self.eq(len(core.eval('inet:fqdn=w00t.com +node:created>1')), 1)
         self.eq(len(core.eval('inet:fqdn=w00t.com totags(leaf=0)')), 2)
         self.eq(len(core.eval('syn:tag=some')), 1)
         self.eq(len(core.eval('syn:tag=some.tag')), 1)
@@ -1403,6 +1423,50 @@ class CortexTest(SynTest):
             self.false(s_tags.tufoHasTag(tufo1, 'hehe'))
             self.false(s_tags.tufoHasTag(tufo1, 'hoho'))
 
+            # Add a complicated node which fires a bunch of autoadd nodes and
+            # ensure they are populated in the second core
+            postref_tufo = core0.formTufoByProp('inet:web:postref',
+                                                (('vertex.link/user', 'mypost 0.0.0.0'),
+                                                 ('inet:ipv4', 0)))
+            self.nn(core1.getTufoByProp('inet:web:post',
+                                        ('vertex.link/user', 'mypost 0.0.0.0')))
+            self.eq(postref_tufo[1]['tufo:form'], 'inet:web:postref')
+            self.eq(postref_tufo[1]['inet:web:postref'], '804ec63392f4ea031bb3fd004dee209d')
+            self.eq(postref_tufo[1]['inet:web:postref:post'], '68bc4607f0518963165536921d6e86fa')
+            self.eq(postref_tufo[1]['inet:web:postref:xref'], 'inet:ipv4=0.0.0.0')
+            self.eq(postref_tufo[1]['inet:web:postref:xref:prop'], 'inet:ipv4')
+            self.eq(postref_tufo[1]['inet:web:postref:xref:intval'], 0)
+
+            # Ensure we got the deconflicted node that was already made, not a new node
+            post_tufo = core1.formTufoByProp('inet:web:post',
+                                             ('vertex.link/user', 'mypost 0.0.0.0'))
+            self.notin('.new', post_tufo[1])
+            self.eq(post_tufo[1]['inet:web:post'], postref_tufo[1]['inet:web:postref:post'])
+            # Ensure that subs on the autoadd node are formed properly
+            self.eq(post_tufo[1].get('inet:web:post:acct'), 'vertex.link/user')
+            self.eq(post_tufo[1].get('inet:web:post:text'), 'mypost 0.0.0.0')
+            # Ensure multiple subs were made into nodes
+            self.nn(core1.getTufoByProp('inet:web:acct', 'vertex.link/user'))
+            self.nn(core1.getTufoByProp('inet:user', 'user'))
+            self.nn(core1.getTufoByProp('inet:fqdn', 'vertex.link'))
+            self.nn(core1.getTufoByProp('inet:fqdn', 'link'))
+
+            # Ensure that splices for changes on ro properties on a node are reflected
+            valu, subs = core0.getTypeNorm('inet:web:post',
+                                           '(vertex.ninja/ninja,"Just ninja things.")')
+            t0 = core0.formTufoByProp('inet:web:post', valu)
+            self.eq(t0[1].get('inet:web:post'), valu)
+            self.none(t0[1].get('inet:web:post:acct'))
+            self.none(t0[1].get('inet:web:post:text'))
+            core0.setTufoProps(t0, **subs)
+            t0 = core1.getTufoByProp('inet:web:post', valu)
+            self.eq(t0[1].get('inet:web:post:acct'), 'vertex.ninja/ninja')
+            self.eq(t0[1].get('inet:web:post:text'), 'Just ninja things.')
+            self.nn(core1.getTufoByProp('inet:web:acct', 'vertex.ninja/ninja'))
+            self.nn(core1.getTufoByProp('inet:user', 'ninja'))
+            self.nn(core1.getTufoByProp('inet:fqdn', 'vertex.ninja'))
+            self.nn(core1.getTufoByProp('inet:fqdn', 'ninja'))
+
     def test_cortex_dict(self):
         core = s_cortex.openurl('ram://')
         core.addTufoForm('foo:bar', ptype='int')
@@ -1884,22 +1948,6 @@ class CortexTest(SynTest):
             core.setConfOpt('enforce', 0)
             self.raises(BadPropValu, core.formTufoByProp, 'foo:bar', True)
 
-    def test_cortex_events(self):
-        with self.getRamCore() as core:
-
-            tick = now()
-
-            tufo0 = core.addTufoEvent('guidform', baz=10, foo='thing')
-
-            tock = now()
-
-            id0 = tufo0[0]
-            rows = core.getRowsById(id0)
-
-            self.eq(len(rows), 4)
-            self.true(rows[0][-1] >= tick)
-            self.true(rows[0][-1] <= tock)
-
     def test_cortex_splicefd(self):
         with self.getTestDir() as path:
             with genfile(path, 'savefile.mpk') as fd:
@@ -2000,8 +2048,8 @@ class CortexTest(SynTest):
                 tufo1 = core1.getTufoByProp('inet:fqdn', 'woot.com')
                 self.nn(tufo1)
 
-                # tufo:formed rows are not sent with the splice and will be created by the target core
-                self.gt(tufo1[1]['tufo:formed'], tufo0[1]['tufo:formed'])
+                # node:created rows are not sent with the splice and will be created by the target core
+                self.gt(tufo1[1]['node:created'], tufo0[1]['node:created'])
 
     def test_cortex_xact_deadlock(self):
         N = 100
@@ -2232,12 +2280,6 @@ class CortexTest(SynTest):
             with s_cortex.openurl('sqlite:///%s' % (path,)) as core:
                 self.false(core.isnew)
 
-    def test_cortex_notguidform(self):
-
-        with self.getRamCore() as core:
-
-            self.raises(NotGuidForm, core.addTufoEvents, 'inet:fqdn', [{}])
-
     def test_cortex_getbytag(self):
 
         with self.getRamCore() as core:
@@ -2301,7 +2343,7 @@ class CortexTest(SynTest):
 
             node = core.formTufoByProp('foo:bar', 'I am a bar foo.')
             self.eq(node[1].get('tufo:form'), 'foo:bar')
-            self.gt(node[1].get('tufo:formed'), 1483228800000)
+            self.gt(node[1].get('node:created'), 1483228800000)
             self.eq(node[1].get('foo:bar'), 'I am a bar foo.')
             self.none(node[1].get('foo:bar:duck'))
 
@@ -2351,7 +2393,7 @@ class CortexTest(SynTest):
 
                 node = core.formTufoByProp('foo:bar', 'I am a bar foo.')
                 self.eq(node[1].get('tufo:form'), 'foo:bar')
-                self.gt(node[1].get('tufo:formed'), 1483228800000)
+                self.gt(node[1].get('node:created'), 1483228800000)
                 self.eq(node[1].get('foo:bar'), 'I am a bar foo.')
                 self.none(node[1].get('foo:bar:duck'))
 
@@ -2396,7 +2438,7 @@ class CortexTest(SynTest):
 
                 node = core.formTufoByProp('foo:bar', 'I am a bar foo.')
                 self.eq(node[1].get('tufo:form'), 'foo:bar')
-                self.gt(node[1].get('tufo:formed'), 1483228800000)
+                self.gt(node[1].get('node:created'), 1483228800000)
                 self.eq(node[1].get('foo:bar'), 'I am a bar foo.')
                 self.none(node[1].get('foo:bar:duck'))
 
@@ -2518,14 +2560,14 @@ class CortexTest(SynTest):
                     self.isinstance(actual[0], tuple)
                     self.eq(len(actual[0]), 2)
                     self.eq(actual[0][1]['tufo:form'], 'inet:fqdn')
-                    self.gt(actual[0][1]['tufo:formed'], 1483228800000)
+                    self.gt(actual[0][1]['node:created'], 1483228800000)
                     self.eq(actual[0][1]['inet:fqdn'], 'vertex.link')
                     self.eq(actual[0][1]['inet:fqdn:zone'], 1)
 
                     self.isinstance(actual[1], tuple)
                     self.eq(actual[1][0], None)
                     self.eq(actual[1][1]['tufo:form'], 'syn:err')
-                    # NOTE: ephemeral data does not get tufo:formed
+                    # NOTE: ephemeral data does not get node:created
                     self.eq(actual[1][1]['syn:err'], 'BadTypeValu')
                     for s in ['BadTypeValu', 'name=', 'inet:url', 'valu=', 'bad']:
                         self.isin(s, actual[1][1]['syn:err:errmsg'])
@@ -2533,7 +2575,7 @@ class CortexTest(SynTest):
                     self.isinstance(actual[2], tuple)
                     self.eq(actual[2][0], None)
                     self.eq(actual[2][1]['tufo:form'], 'syn:err')
-                    # NOTE: ephemeral data does not get tufo:formed
+                    # NOTE: ephemeral data does not get node:created
                     self.eq(actual[2][1]['syn:err'], 'NoSuchForm')
                     for s in ['NoSuchForm', 'name=', 'bad']:
                         self.isin(s, actual[2][1]['syn:err:errmsg'])
@@ -2707,7 +2749,7 @@ class CortexTest(SynTest):
                                              'strform:haha': 1234, 'strform:foo': 'sup'}))
 
             form, valu = s_tufo.ndef(t1)
-            self.gt(t1[1]['tufo:formed'], 1483228800000)
+            self.gt(t1[1]['node:created'], 1483228800000)
             self.eq(form, 'strform')
             self.eq(valu, 'oh hai')
             self.eq(t1[1].get('strform:foo'), 'sup')
@@ -2756,7 +2798,7 @@ class StorageTest(SynTest):
                 tick = s_common.now()
                 rows.append(('1234', 'foo:bar:baz', 'yes', tick))
                 rows.append(('1234', 'tufo:form', 'foo:bar', tick))
-                rows.append(('1234', 'tufo:formed', 1483228800000, tick))
+                rows.append(('1234', 'node:created', 1483228800000, tick))
                 store.addRows(rows)
 
             # Retrieve the node via the Cortex interface
@@ -2764,7 +2806,7 @@ class StorageTest(SynTest):
                 node = core.getTufoByIden('1234')
                 self.nn(node)
                 self.eq(node[1].get('tufo:form'), 'foo:bar')
-                self.eq(node[1].get('tufo:formed'), 1483228800000)
+                self.eq(node[1].get('node:created'), 1483228800000)
                 self.eq(node[1].get('foo:bar:baz'), 'yes')
 
     def test_storage_row_manipulation(self):
@@ -2780,7 +2822,7 @@ class StorageTest(SynTest):
                 tick = s_common.now()
                 rows.append(('1234', 'foo:bar:baz', 'yes', tick))
                 rows.append(('1234', 'tufo:form', 'foo:bar', tick))
-                rows.append(('1234', 'tufo:formed', 1483228800000, tick))
+                rows.append(('1234', 'node:created', 1483228800000, tick))
                 store.addRows(rows)
 
             # Retrieve the node via the Cortex interface
@@ -2788,7 +2830,7 @@ class StorageTest(SynTest):
                 node = core.getTufoByIden('1234')
                 self.nn(node)
                 self.eq(node[1].get('tufo:form'), 'foo:bar')
-                self.eq(node[1].get('tufo:formed'), 1483228800000)
+                self.eq(node[1].get('node:created'), 1483228800000)
                 self.eq(node[1].get('foo:bar:baz'), 'yes')
 
     def test_storage_handler_misses(self):
diff --git a/synapse/tests/test_model_dns.py b/synapse/tests/test_model_dns.py
index 6317f3f0c..7dfdc1bb2 100644
--- a/synapse/tests/test_model_dns.py
+++ b/synapse/tests/test_model_dns.py
@@ -43,25 +43,25 @@ class DnsModelTest(SynTest):
 
             tick = now()
 
-            t0 = core.addTufoEvent('inet:dns:look', a='WOOT.com/1.002.3.4', time=tick)
+            t0 = core.formTufoByProp('inet:dns:look', '*', a='WOOT.com/1.002.3.4', time=tick)
             self.eq(t0[1].get('inet:dns:look:time'), tick)
             self.eq(t0[1].get('inet:dns:look:a'), 'woot.com/1.2.3.4')
             self.eq(t0[1].get('inet:dns:look:a:fqdn'), 'woot.com')
             self.eq(t0[1].get('inet:dns:look:a:ipv4'), 0x01020304)
 
-            t0 = core.addTufoEvent('inet:dns:look', ns='WOOT.com/ns.yermom.com', time=tick)
+            t0 = core.formTufoByProp('inet:dns:look', '*', ns='WOOT.com/ns.yermom.com', time=tick)
             self.eq(t0[1].get('inet:dns:look:time'), tick)
             self.eq(t0[1].get('inet:dns:look:ns'), 'woot.com/ns.yermom.com')
             self.eq(t0[1].get('inet:dns:look:ns:ns'), 'ns.yermom.com')
             self.eq(t0[1].get('inet:dns:look:ns:zone'), 'woot.com')
 
-            t0 = core.addTufoEvent('inet:dns:look', rev='1.2.3.4/WOOT.com', time=tick)
+            t0 = core.formTufoByProp('inet:dns:look', '*', rev='1.2.3.4/WOOT.com', time=tick)
             self.eq(t0[1].get('inet:dns:look:time'), tick)
             self.eq(t0[1].get('inet:dns:look:rev'), '1.2.3.4/woot.com')
             self.eq(t0[1].get('inet:dns:look:rev:fqdn'), 'woot.com')
             self.eq(t0[1].get('inet:dns:look:rev:ipv4'), 0x01020304)
 
-            t0 = core.addTufoEvent('inet:dns:look', aaaa='WOOT.com/FF::56', time=tick)
+            t0 = core.formTufoByProp('inet:dns:look', '*', aaaa='WOOT.com/FF::56', time=tick)
             self.eq(t0[1].get('inet:dns:look:time'), tick)
             self.eq(t0[1].get('inet:dns:look:aaaa'), 'woot.com/ff::56')
             self.eq(t0[1].get('inet:dns:look:aaaa:fqdn'), 'woot.com')
diff --git a/synapse/tests/test_model_files.py b/synapse/tests/test_model_files.py
index 534b8ba7f..aa5b025ca 100644
--- a/synapse/tests/test_model_files.py
+++ b/synapse/tests/test_model_files.py
@@ -58,6 +58,45 @@ class FileModelTest(SynTest):
             self.eq(n2def[1], stable_guid)
             self.eq(n1[0], n2[0])
 
+    def test_model_filepath_complex(self):
+        with self.getRamCore() as core:
+
+            node = core.formTufoByProp('file:path', '/Foo/Bar/Baz.exe')
+
+            self.nn(node)
+            self.eq(node[1].get('file:path:dir'), '/foo/bar')
+            self.eq(node[1].get('file:path:ext'), 'exe')
+            self.eq(node[1].get('file:path:base'), 'baz.exe')
+
+            node = core.getTufoByProp('file:path', '/foo')
+
+            self.nn(node)
+            self.none(node[1].get('file:path:ext'))
+
+            self.eq(node[1].get('file:path:dir'), '')
+            self.eq(node[1].get('file:path:base'), 'foo')
+
+            node = core.formTufoByProp('file:path', r'c:\Windows\system32\Kernel32.dll')
+
+            self.nn(node)
+            self.eq(node[1].get('file:path:dir'), 'c:/windows/system32')
+            self.eq(node[1].get('file:path:ext'), 'dll')
+            self.eq(node[1].get('file:path:base'), 'kernel32.dll')
+
+            self.nn(core.getTufoByProp('file:base', 'kernel32.dll'))
+
+            node = core.getTufoByProp('file:path', 'c:')
+
+            self.nn(node)
+            self.none(node[1].get('file:path:ext'))
+            self.eq(node[1].get('file:path:dir'), '')
+            self.eq(node[1].get('file:path:base'), 'c:')
+
+            node = core.formTufoByProp('file:path', r'/foo////bar/.././baz.json')
+
+            self.nn(node)
+            self.eq(node[1].get('file:path'), '/foo/baz.json')
+
     def test_filepath(self):
         with self.getRamCore() as core:
 
diff --git a/synapse/tests/test_model_inet.py b/synapse/tests/test_model_inet.py
index a00a97985..f40ca950f 100644
--- a/synapse/tests/test_model_inet.py
+++ b/synapse/tests/test_model_inet.py
@@ -594,6 +594,32 @@ class InetModelTest(SynTest):
             # We require the account to be present
             self.raises(PropNotFound, core.formTufoByProp, 'inet:web:chprofile', '*')
 
+    def test_model_inet_postref_postmissingprops(self):
+        with self.getRamCore() as core:
+
+            postref_tufo = core.formTufoByProp('inet:web:postref', (('vertex.link/user', 'mypost 0.0.0.0'), ('inet:ipv4', 0)))
+            self.nn(core.getTufoByProp('inet:web:post', ('vertex.link/user', 'mypost 0.0.0.0')))
+
+            self.eq(postref_tufo[1]['tufo:form'], 'inet:web:postref')
+            self.eq(postref_tufo[1]['inet:web:postref'], '804ec63392f4ea031bb3fd004dee209d')
+            self.eq(postref_tufo[1]['inet:web:postref:post'], '68bc4607f0518963165536921d6e86fa')
+            self.eq(postref_tufo[1]['inet:web:postref:xref'], 'inet:ipv4=0.0.0.0')
+            self.eq(postref_tufo[1]['inet:web:postref:xref:prop'], 'inet:ipv4')
+            self.eq(postref_tufo[1]['inet:web:postref:xref:intval'], 0)
+
+            post_tufo = core.formTufoByProp('inet:web:post', ('vertex.link/user', 'mypost 0.0.0.0'))
+            # Ensure we got the deconflicted node that was already made, not a new node
+            self.notin('.new', post_tufo[1])
+            self.eq(post_tufo[1]['inet:web:post'], postref_tufo[1]['inet:web:postref:post'])
+            # Ensure that subs on the autoadd node are formed properly
+            self.eq(post_tufo[1].get('inet:web:post:acct'), 'vertex.link/user')
+            self.eq(post_tufo[1].get('inet:web:post:text'), 'mypost 0.0.0.0')
+            # Ensure multiple subs were made into nodes
+            self.nn(core.getTufoByProp('inet:web:acct', 'vertex.link/user'))
+            self.nn(core.getTufoByProp('inet:user', 'user'))
+            self.nn(core.getTufoByProp('inet:fqdn', 'vertex.link'))
+            self.nn(core.getTufoByProp('inet:fqdn', 'link'))
+
     def test_model_inet_201706121318(self):
 
         iden0 = guid()
diff --git a/synapse/tests/test_model_infotech.py b/synapse/tests/test_model_infotech.py
index 9175efb92..2786adbe1 100644
--- a/synapse/tests/test_model_infotech.py
+++ b/synapse/tests/test_model_infotech.py
@@ -59,45 +59,6 @@ class InfoTechTest(SynTest):
             node = core.getTufoByProp('it:hostname', 'hehehaha')
             self.nn(node)
 
-    def test_model_infotech_filepath(self):
-        with self.getRamCore() as core:
-
-            node = core.formTufoByProp('file:path', '/Foo/Bar/Baz.exe')
-
-            self.nn(node)
-            self.eq(node[1].get('file:path:dir'), '/foo/bar')
-            self.eq(node[1].get('file:path:ext'), 'exe')
-            self.eq(node[1].get('file:path:base'), 'baz.exe')
-
-            node = core.getTufoByProp('file:path', '/foo')
-
-            self.nn(node)
-            self.none(node[1].get('file:path:ext'))
-
-            self.eq(node[1].get('file:path:dir'), '')
-            self.eq(node[1].get('file:path:base'), 'foo')
-
-            node = core.formTufoByProp('file:path', r'c:\Windows\system32\Kernel32.dll')
-
-            self.nn(node)
-            self.eq(node[1].get('file:path:dir'), 'c:/windows/system32')
-            self.eq(node[1].get('file:path:ext'), 'dll')
-            self.eq(node[1].get('file:path:base'), 'kernel32.dll')
-
-            self.nn(core.getTufoByProp('file:base', 'kernel32.dll'))
-
-            node = core.getTufoByProp('file:path', 'c:')
-
-            self.nn(node)
-            self.none(node[1].get('file:path:ext'))
-            self.eq(node[1].get('file:path:dir'), '')
-            self.eq(node[1].get('file:path:base'), 'c:')
-
-            node = core.formTufoByProp('file:path', r'/foo////bar/.././baz.json')
-
-            self.nn(node)
-            self.eq(node[1].get('file:path'), '/foo/baz.json')
-
     def test_model_infotech_itdev(self):
         with self.getRamCore() as core:
 
diff --git a/synapse/tests/test_model_syn.py b/synapse/tests/test_model_syn.py
index ae697374c..af8307a84 100644
--- a/synapse/tests/test_model_syn.py
+++ b/synapse/tests/test_model_syn.py
@@ -119,7 +119,7 @@ class SynModelTest(SynTest):
         data = {}
         old_formed = 12345
         iden0, iden1 = guid(), guid()
-        tick = now() - 10000  # putting tick in the past to show that preexisting tufo:formed rows will have their stamps removed
+        tick = now() - 10000  # putting tick in the past to show that preexisting node:created rows will have their stamps removed
         rows = [
             (iden0, 'inet:ipv4:type', '??', tick),
             (iden0, 'inet:ipv4', 16909060, tick),
@@ -131,7 +131,7 @@ class SynModelTest(SynTest):
             (iden1, 'file:bytes:mime', '??', tick),
             (iden1, 'file:bytes:md5', 'd41d8cd98f00b204e9800998ecf8427e', tick),
             (iden1, 'tufo:form', 'file:bytes', tick),
-            (iden1, 'tufo:formed', tick, tick),  # NOTE: this row should not exist pre-migration
+            (iden1, 'node:created', tick, tick),  # NOTE: this row should not exist pre-migration
         ]
 
         with s_cortex.openstore('ram:///') as stor:
@@ -147,28 +147,28 @@ class SynModelTest(SynTest):
             with s_cortex.fromstore(stor) as core:
 
                 # 1 file:bytes, 1 inet:ipv4
-                self.ge(len(core.eval('tufo:formed')), 3)
+                self.ge(len(core.eval('node:created')), 3)
 
-                tufos = core.eval('tufo:formed +tufo:form=inet:ipv4')
+                tufos = core.eval('node:created +tufo:form=inet:ipv4')
                 self.eq(len(tufos), 1)
                 iden, props = tufos[0]
                 self.eq(props['tufo:form'], 'inet:ipv4')
-                self.eq(props['tufo:formed'], tick)
+                self.eq(props['node:created'], tick)
                 self.eq(props['inet:ipv4'], 16909060)
                 self.eq(props['inet:ipv4:asn'], -1)
                 self.eq(props['inet:ipv4:cc'], '??')
-                rows = core.getRowsByIdProp(iden, 'tufo:formed')
+                rows = core.getRowsByIdProp(iden, 'node:created')
                 _, _, valu, stamp = rows[0]
-                self.gt(stamp, valu)  # tufo:formed row's stamp will be higher than its valu
+                self.gt(stamp, valu)  # node:created row's stamp will be higher than its valu
 
-                tufos = core.eval('tufo:formed +tufo:form=file:bytes')
+                tufos = core.eval('node:created +tufo:form=file:bytes')
                 self.eq(len(tufos), 1)
                 props = tufos[0][1]
                 self.eq(props['tufo:form'], 'file:bytes')
-                self.eq(props['tufo:formed'], tick)
+                self.eq(props['node:created'], tick)
                 self.eq(props['file:bytes'], 'd41d8cd98f00b204e9800998ecf8427e')
                 self.eq(props['file:bytes:mime'], '??')
                 self.eq(props['file:bytes:md5'], 'd41d8cd98f00b204e9800998ecf8427e')
-                rows = core.getRowsByIdProp(iden, 'tufo:formed')
+                rows = core.getRowsByIdProp(iden, 'node:created')
                 _, _, valu, stamp = rows[0]
-                self.gt(stamp, valu)  # tufo:formed row's stamp will be higher than its valu
+                self.gt(stamp, valu)  # node:created row's stamp will be higher than its valu