Repository: vertexproject/synapse

Issue:
Subsystem that parses ingest.json files calculates different file:bytes GUID (superhash) based on capitalization of hash value
Identified during testing for #334 .

Synapse calculates a GUID as a primary property for a file:bytes node based on the available hash value(s) for the bytes. If the bytes are present, all hashes are used; if we only have a hash and no bytes, the available hash is used.

The GUID should be calculated the same way regardless of whether the hash contains upper case or lower case alpha characters.  This is true when adding nodes via Storm:
```
cli> ask [file:bytes:sha256=6ACC29BFC5F8F772FA7AAF4A705F91CB68DC88CB22F4EF5101281DC42109A104]
file:bytes = 8a249054e1c0e455657867a817aa9fcc
(1 results)

cli> ask [file:bytes:sha256=6acc29bfc5f8f772fa7aaf4a705f91cb68dc88cb22f4ef5101281dc42109a104]
file:bytes = 8a249054e1c0e455657867a817aa9fcc
(1 results)
```

However, this is *not* the case when adding data via an ingest.json file.
1. Create two test ingest.json files, each adds only a single node based on file:bytes:sha256.
2. In one file (test1.json), create the SHA256 with upper case alpha characters.
3. In the other file (test2.json) create the SHA256 with lower case alpha characters.
Result:

A. Upper case hash:
```
py3 -m synapse.tools.ingest --verbose ~/research/test1.json --sync <cortex>
add: file:bytes=8a249054e1c0e455657867a817aa9fcc
       :mime = ??
       :sha256 = 6acc29bfc5f8f772fa7aaf4a705f91cb68dc88cb22f4ef5101281dc42109a104
add: hash:sha256=6acc29bfc5f8f772fa7aaf4a705f91cb68dc88cb22f4ef5101281dc42109a104
ingest took: 0.0028884410858154297 sec
```

B. Lower case hash:
```
py3 -m synapse.tools.ingest --verbose ~/research/test2.json --sync  <cortex>
add: file:bytes=ed73917b1dc4011627f7a101ace491c8
       :mime = ??
       :sha256 = 6acc29bfc5f8f772fa7aaf4a705f91cb68dc88cb22f4ef5101281dc42109a104
add: hash:sha256=6acc29bfc5f8f772fa7aaf4a705f91cb68dc88cb22f4ef5101281dc42109a104
ingest took: 0.0036017894744873047 sec
```

Note that it appears that the upper-case hash calculates the "correct" GUID (GUID value matches the one generated via Storm), but the lower-case hash does not.


Discussion:

Patch:
diff --git a/synapse/models/files.py b/synapse/models/files.py
index 63916defd..1dcf82c57 100644
--- a/synapse/models/files.py
+++ b/synapse/models/files.py
@@ -87,6 +87,8 @@ class FileMod(CoreModule):
         Hashes that we consider "cardinal enough" to pivot.
         '''
         name = prop.rsplit(':', 1)[-1]
+        # Normalize the valu before we go any further
+        valu, _ = self.core.getPropNorm(prop, valu)
         props[name] = valu
 
         # FIXME could we update additional hashes here and
@@ -107,10 +109,12 @@ class FileMod(CoreModule):
         return tufo
 
     def seedFileMd5(self, prop, valu, **props):
+        valu, _ = self.core.getPropNorm('file:bytes:md5', valu)
         props['md5'] = valu
         return self.core.formTufoByProp('file:bytes', valu, **props)
 
     def seedFileSha1(self, prop, valu, **props):
+        valu, _ = self.core.getPropNorm('file:bytes:sha1', valu)
         props['sha1'] = valu
         valu = guid(valu)
         return self.core.formTufoByProp('file:bytes', valu, **props)


Test patch:
diff --git a/synapse/tests/test_model_files.py b/synapse/tests/test_model_files.py
index 0b917710e..f42e5edb1 100644
--- a/synapse/tests/test_model_files.py
+++ b/synapse/tests/test_model_files.py
@@ -1,11 +1,11 @@
 from __future__ import absolute_import, unicode_literals
 
 import synapse.axon as s_axon
-import synapse.compat as s_compat
-import synapse.cortex as s_cortex
 import synapse.daemon as s_daemon
 import synapse.telepath as s_telepath
 
+import synapse.lib.tufo as s_tufo
+
 from synapse.tests.common import *
 
 class FileModelTest(SynTest):
@@ -44,6 +44,22 @@ class FileModelTest(SynTest):
             self.ne(t0[0], core.formTufoByProp('file:bytes:sha1', props.get('sha1'))[0])
             self.ne(t0[0], core.formTufoByProp('file:bytes:md5', props.get('md5'))[0])
 
+    def test_model_file_seeds_capitalization(self):
+        fhash = '6ACC29BFC5F8F772FA7AAF4A705F91CB68DC88CB22F4EF5101281DC42109A104'
+        fhash_lower = fhash.lower()
+        stable_guid = 'ed73917b1dc4011627f7a101ace491c8'
+
+        with s_cortex.openurl('ram:///') as core:
+
+            n1 = core.formTufoByProp('file:bytes:sha256', fhash)
+            n2 = core.formTufoByProp('file:bytes:sha256', fhash_lower)
+            # Sha256 should be lowercase since the prop type is lowercased
+            n1def = s_tufo.ndef(n1)
+            n2def = s_tufo.ndef(n2)
+            self.eq(n1def[1], stable_guid)
+            self.eq(n2def[1], stable_guid)
+            self.eq(n1[0], n2[0])
+
     def test_filepath(self):
 
         with s_cortex.openurl('ram:///') as core: