Repository: CERT-Polska/mwdb-core (language: python)

Issue:
Support search of multiple hashes
**Feature Category**

- [ ] Correctness
- [x] User Interface / User Experience
- [ ] Performance
- [ ] Other (please explain)

**Describe the problem**

Currently, there is no easy way to paste multiple hashes in the interface and get results back

**Describe the solution you'd like**

Similar to VT, I'd love to be able to paste multiple hashes (of different kinds) and search them at once.

The query can support these formats:

Space separation:
```
2a575f70de971c825a46c2f2f931a7228cedf117 e8b63a1b885fa91940bb5d53f1b6e67da2e11712e96166634d0ca71dcf948249 c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18 
```

New line separation:
```
2a575f70de971c825a46c2f2f931a7228cedf117
e8b63a1b885fa91940bb5d53f1b6e67da2e11712e96166634d0ca71dcf948249
c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18 
```

Random Bytes separation:

```
"2a575f70de971c825a46c2f2f931a7228cedf117", 13:00:1,; "e8b63a1b885fa91940bb5d53f1b6e67da2e11712e96166634d0ca71dcf948249", 17:09:00T01 "c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18"
```



![Peek 2021-01-05 12-03](https://user-images.githubusercontent.com/20182642/103633068-e223b880-4f4d-11eb-951d-3a4b374c988a.gif)

**Describe alternatives you've considered**
Stick to the less ux-friednly approach:
`file.sha256:51f14ae7f8e54c7fe9572f8c9d28a8a6b793d85fdbd72b56d4233db93efacae9 OR file.sha256:c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18`




Discussion / hints:
We should check which option is better:
- Include regex-based smart converter to the Lucene syntax form
- Implement field guessing as a part of search engine. AFAIK that space-separated form is native for Lucene syntax.

On the other hand, we can also use field grouping for that and I think it will be much easier to implement. The disadvantage is that MD5 and SHA256 must be grouped separately. What do you think about that alternative approach?
```
file.sha256:(51f14ae7f8e54c7fe9572f8c9d28a8a6b793d85fdbd72b56d4233db93efacae9 c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18) OR file:md5:(2a575f70de971c825a46c2f2f931a7228cedf117)
```
In general, I am against the options in which users should type anything other than <kbd>Ctrl</kbd>+<kbd>V</kbd>. 

That is what I really like about VT here, that I can take whatever source of text I get (slack message, shell output (sha256sum, ...), file names (.bin, .vir), snippets from tweets, etc) and throw it at the search box. 

Then, probably based on regex, the hashes are grabbed and queried
@psrok1 don't think it has to be a valid lucene syntax (`file.sha256:(xxxxxx yyyyy)` is super not what Itay had in mind). The idea is that you just copy-paste stuff and it just works. Imagine if you had to type `md5:cb411f088361c59d64044df6d60b02c2` instead of `cb411f088361c59d64044df6d60b02c2` every time

And that's a good lead. We already transform hash-like stuff (like `cb411f088361c59d64044df6d60b02c2`) into a valid internal query (like `md5:cb411f088361c59d64044df6d60b02c2`). So this feature request is just a natural extension of that.

WDYT?
Ok, I got the idea 😄 Still we can implement that in Lucene-compatible way. Let's try with the `Random bytes separation` proposed by Itay.

```
In [1]: from luqum.parser import parser

In [2]: parser.parse('"2a575f70de971c825a46c2f2f931a7228cedf117", 13:00:1,; "e8b
   ...: 63a1b885fa91940bb5d53f1b6e67da2e11712e96166634d0ca71dcf948249", 17:09:00
   ...: T01 "c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18"')
   ...: 
Illegal character ','
Illegal character ','
Out[2]: UnknownOperation(Phrase('"2a575f70de971c825a46c2f2f931a7228cedf117"'), SearchField('13', SearchField('00', Word('1,;'))), Phrase('"e8b63a1b885fa91940bb5d53f1b6e67da2e11712e96166634d0ca71dcf948249"'), SearchField('17', SearchField('09', Word('00T01'))), Phrase('"c3ad1e51953a3cc468f4b0bc17c2c918fff1c62ffa9c39d1db075d92c2d03f18"'))
```

As you can see, not all things can be parsed this way (e.g. datetime with ':' needs additional wrapping with double-quotes as they are parsed to `SearchField('13', SearchField('00'...`) but there's no problem with hashes and other word-like stuff.

We should just define UnknownOperation as OrOperation and introduce "default fields" based on guessed type (hashes for files and full-text search for configs/blobs).

So I think it's just a matter of guessing the field type and proper sanitization. It actually looks to be pretty easy :smile:
This sounds good to me 👍 
See also: https://github.com/CERT-Polska/mwdb-core/pull/297/files

Fix patch:
diff --git a/docs/user-guide/7-Lucene-search.rst b/docs/user-guide/7-Lucene-search.rst
index eff0771d6..bc7e078a5 100644
--- a/docs/user-guide/7-Lucene-search.rst
+++ b/docs/user-guide/7-Lucene-search.rst
@@ -385,6 +385,58 @@ Nested searches can be performed as well. If you want to find object which is pa
 
    child:(tag:emotet AND child:(config.family:emotet))
 
+Multi field (\ ``multi:``\ )
+------------------------------------------------------------
+
+Search engine supports ``multi:`` special field that is useful for filtering out objects using multiple
+type of object attributes.
+
+Depending on type of object, we can use field ``multi:`` applying various object attributes separated by spaces.
+Types of attributes are automatically recognised.
+Below allowable attributes were listed for different type of objects, which can be used in query building.
+
+* File (sample)
+
+  * all hashes values
+
+* Config
+
+  * dhash values
+  * extract of configuration content
+
+* Blob
+
+  * dhash values
+  * extract of Blob content
+
+
+If you want to search for samples that have 0cb988d042a7f28dd5fe2b55b3f5ac7a ``md5`` value or
+3b0ee981 ``crc32`` value use below query.
+
+.. code-block::
+
+   multi:"0cb988d042a7f28dd5fe2b55b3f5ac7a 3b0ee981"
+
+If query contains only hashes, field ``multi:`` can be omitted.
+
+.. code-block::
+
+   0cb988d042a7f28dd5fe2b55b3f5ac7a eb1c78d4994f7a107f78dec529988900e3601852ae0bfdefb3e15967c6d8f127
+
+If you want to search for Configs that have configuration content which contain strings "abcd" or "xyz" or
+
+``dhash`` value eb1c78d4994f7a107f78dec529988900e3601852ae0bfdefb3e15967c6d8f127 - use below query
+
+.. code-block::
+
+   multi:"abcd xyz eb1c78d4994f7a107f78dec529988900e3601852ae0bfdefb3e15967c6d8f127"
+
+.. warning::
+
+    Multi-query terms containing extract of Config configuration or Blob content must be separated by spaces end surrounded by double quotes. Field ``multi:`` also has to be explicitly used.
+
+    Wildcards are not allowed for field ``multi:``. They are automatically used for Config configuration and Blob content.
+
 Quick queries
 -------------
 
diff --git a/mwdb/core/search/fields.py b/mwdb/core/search/fields.py
index 861a863b7..94203b65c 100644
--- a/mwdb/core/search/fields.py
+++ b/mwdb/core/search/fields.py
@@ -13,11 +13,13 @@
 from mwdb.model import (
     Attribute,
     AttributeDefinition,
+    Config,
     File,
     Group,
     Member,
     Object,
     ObjectPermission,
+    TextBlob,
     User,
     db,
 )
@@ -545,6 +547,71 @@ def parse_upload_value(value):
             return self.column == upload_value
 
 
+class MultiField(BaseField):
+    @staticmethod
+    def get_column(queried_type: Type[Object], value: str):
+        if queried_type is File:
+            if re.match(r"^[0-9a-fA-F]{8}$", value):
+                return File.crc32
+            elif re.match(r"^[0-9a-fA-F]{32}$", value):
+                return File.md5
+            elif re.match(r"^[0-9a-fA-F]{40}$", value):
+                return File.sha1
+            elif re.match(r"^[0-9a-fA-F]{64}$", value):
+                return File.sha256
+            elif re.match(r"^[0-9a-fA-F]{128}$", value):
+                return File.sha512
+            else:
+                raise ObjectNotFoundException(f"{value} is not valid hash value")
+        elif queried_type is TextBlob:
+            if re.match(r"^[0-9a-fA-F]{64}$", value):
+                return TextBlob.dhash
+            else:
+                return TextBlob._content
+        elif queried_type is Config:
+            if re.match(r"^[0-9a-fA-F]{64}$", value):
+                return Config.dhash
+            else:
+                return Config._cfg
+        else:
+            raise ObjectNotFoundException(
+                f"{queried_type.__name__} is not valid data type"
+            )
+
+    def get_condition(self, expression: Expression, remainder: List[str]) -> Any:
+        string_column = ["TextBlob._content"]
+        json_column = ["Config._cfg"]
+
+        if remainder:
+            raise FieldNotQueryableException(
+                f"Field doesn't have subfields: {'.'.join(remainder)}"
+            )
+
+        value = get_term_value(expression).strip()
+        values_list = re.split("\\s+", value)
+
+        condition = None
+        for value in values_list:
+            column = MultiField.get_column(self.field_type, value)
+
+            if str(column) in string_column:
+                value = f"%{value}%"
+                condition = or_(condition, (column.like(value)))
+            elif str(column) in json_column:
+                value = f"%{value}%"
+                condition = or_(condition, (cast(column, String).like(value)))
+            else:
+                # hashes values
+                condition = or_(condition, (column == value))
+
+        if expression.has_wildcard():
+            raise UnsupportedGrammarException(
+                "Wildcards are not allowed for multi field"
+            )
+        else:
+            return condition
+
+
 class FileNameField(BaseField):
     def get_condition(self, expression: Expression, remainder: List[str]) -> Any:
         if remainder:
@@ -565,7 +632,6 @@ def get_condition(self, expression: Expression, remainder: List[str]) -> Any:
             )
 
             condition = or_(self.column.like(value), File.id.in_(file_id_matching))
-            return condition
         else:
             condition = or_((self.column == value), File.alt_names.any(value))
-            return condition
+        return condition
diff --git a/mwdb/core/search/mappings.py b/mwdb/core/search/mappings.py
index 79456b4c2..80dca277b 100644
--- a/mwdb/core/search/mappings.py
+++ b/mwdb/core/search/mappings.py
@@ -23,6 +23,7 @@
     FileNameField,
     JSONField,
     ListField,
+    MultiField,
     RelationField,
     ShareField,
     SizeField,
@@ -67,11 +68,13 @@
         "sha512": StringField(File.sha512),
         "ssdeep": StringField(File.ssdeep),
         "crc32": StringField(File.crc32),
+        "multi": MultiField(File.id),
     },
     Config.__name__: {
         "type": StringField(Config.config_type),
         "family": StringField(Config.family),
         "cfg": JSONField(Config.cfg),
+        "multi": MultiField(Config.id),
     },
     TextBlob.__name__: {
         "name": StringField(TextBlob.blob_name),
@@ -80,6 +83,7 @@
         "content": StringField(TextBlob._content),
         "first_seen": DatetimeField(TextBlob.upload_time),
         "last_seen": DatetimeField(TextBlob.last_seen),
+        "multi": MultiField(TextBlob.id),
     },
 }
 
diff --git a/mwdb/web/src/commons/helpers/search.js b/mwdb/web/src/commons/helpers/search.js
index a16f1e3bc..8002c6e9d 100644
--- a/mwdb/web/src/commons/helpers/search.js
+++ b/mwdb/web/src/commons/helpers/search.js
@@ -59,20 +59,20 @@ export function makeSearchDateLink(field, date, endpoint) {
     );
 }
 
-export function queryFromHash(query, dhashOnly) {
-    // SHA256
-    if (/^[0-9a-fA-F]{64}$/g.test(query))
-        if (dhashOnly) return `dhash:${query.toLowerCase()}`;
-        else return `sha256:${query.toLowerCase()}`;
-    // No SHA256 and dhashOnly? Time to stop
-    if (dhashOnly) return query;
-    // SHA1
-    if (/^[0-9a-fA-F]{40}$/g.test(query)) return `sha1:${query.toLowerCase()}`;
-    // MD5
-    if (/^[0-9a-fA-F]{32}$/g.test(query)) return `md5:${query.toLowerCase()}`;
-    // SHA512
-    if (/^[0-9a-fA-F]{128}$/g.test(query))
-        return `sha512:${query.toLowerCase()}`;
+export function isHash(element) {
+    return (
+        /^[0-9a-fA-F]{8}$/g.test(element) ||
+        /^[0-9a-fA-F]{32}$/g.test(element) ||
+        /^[0-9a-fA-F]{40}$/g.test(element) ||
+        /^[0-9a-fA-F]{64}$/g.test(element) ||
+        /^[0-9a-fA-F]{128}$/g.test(element)
+    );
+}
+
+export function multiFromHashes(query) {
+    const elements = query.split(/\s+/);
+    if (elements.every(isHash))
+        return `multi:"${elements.join(" ").toLowerCase()}"`;
     return query;
 }
 
diff --git a/mwdb/web/src/components/RecentBlobs.js b/mwdb/web/src/components/RecentBlobs.js
index b5ed0b39c..3ed0bee3d 100644
--- a/mwdb/web/src/components/RecentBlobs.js
+++ b/mwdb/web/src/components/RecentBlobs.js
@@ -165,7 +165,6 @@ export default function RecentBlobs(props) {
                 type="blob"
                 rowComponent={RecentBlobRow}
                 headerComponent={RecentBlobHeader}
-                dhashOnly
                 {...props}
             />
         </React.Fragment>
diff --git a/mwdb/web/src/components/RecentConfigs.js b/mwdb/web/src/components/RecentConfigs.js
index 4d49318ed..99b527109 100644
--- a/mwdb/web/src/components/RecentConfigs.js
+++ b/mwdb/web/src/components/RecentConfigs.js
@@ -124,7 +124,6 @@ export default function RecentConfigs(props) {
             type="config"
             rowComponent={RecentConfigRow}
             headerComponent={RecentConfigHeader}
-            dhashOnly
             {...props}
         />
     );
diff --git a/mwdb/web/src/components/RecentObjects.js b/mwdb/web/src/components/RecentObjects.js
index 6e582743c..62d8fe965 100644
--- a/mwdb/web/src/components/RecentObjects.js
+++ b/mwdb/web/src/components/RecentObjects.js
@@ -84,7 +84,6 @@ export default function RecentObjects(props) {
             type="object"
             rowComponent={RecentObjectRow}
             headerComponent={RecentObjectHeader}
-            dhashOnly
             {...props}
         />
     );
diff --git a/mwdb/web/src/components/RecentView/RecentView.js b/mwdb/web/src/components/RecentView/RecentView.js
index 9dc253aef..7747d4739 100644
--- a/mwdb/web/src/components/RecentView/RecentView.js
+++ b/mwdb/web/src/components/RecentView/RecentView.js
@@ -7,7 +7,7 @@ import { APIContext } from "@mwdb-web/commons/api/context";
 import {
     encodeSearchQuery,
     decodeSearchQuery,
-    queryFromHash,
+    multiFromHashes,
     addFieldToQuery,
 } from "@mwdb-web/commons/helpers";
 import { View } from "@mwdb-web/commons/ui";
@@ -59,8 +59,8 @@ export default function RecentView(props) {
     function setCurrentQuery(query) {
         // If query is already submitted: do nothing
         if (query === submittedQuery) return;
-        // Optionally convert query if only hash was provided
-        query = queryFromHash(query, props.dhashOnly);
+        // Optionally convert query if only hash or hashes were provided
+        query = multiFromHashes(query);
         // Set query in URL (currentQuery)
         history.push(getQueryURL(query));
     }


Test patch:
diff --git a/tests/backend/test_blobs.py b/tests/backend/test_blobs.py
index 28585a1ac..2fa1c4f4e 100644
--- a/tests/backend/test_blobs.py
+++ b/tests/backend/test_blobs.py
@@ -74,6 +74,46 @@ def test_blog_search_by_tags_transactional_added(admin_session):
     assert blob_search_2["id"] == blob["id"]
 
 
+def test_blob_search_multi(admin_session):
+    test = admin_session
+
+    blob_name_1 = rand_string(15)
+    blob_content_insert_1 = rand_string(15)
+    blob_content_1 = """
+    Binary junk: \x00\x01\x02\x03\x04\x05\x07
+    HELLO WORLD!
+    ========""" + blob_content_insert_1 + """
+    HELLO WORLD
+    """
+
+    blob_name_2 = rand_string(15)
+    blob_content_insert_2 = rand_string(15)
+    blob_content_2 = """
+    Binary junk: \x00\x01\x02\x03\x04\x05\x07
+    HELLO WORLD!
+    ========""" + blob_content_insert_2 + """
+    HELLO WORLD
+    """
+
+    blob_1 = test.add_blob(None, blobname=blob_name_1, blobtype="inject", content=blob_content_1)
+    blob_2 = test.add_blob(None, blobname=blob_name_2, blobtype="inject", content=blob_content_2)
+
+    # blob content and dhash search
+    query = f'blob.multi:"{blob_content_insert_1} {blob_2["id"]}"'
+    found_objs = test.search(query)
+    assert len(found_objs) == 2
+
+    # only blog content search in multi field
+    query = f'blob.multi:"{blob_content_insert_1}"'
+    found_obj = test.search(query)[0]
+    assert found_obj["id"] == blob_1["id"]
+
+    # only blog dhash search in multi field
+    query = f'blob.multi:"{blob_2["id"]}"'
+    found_obj = test.search(query)[0]
+    assert blob_2["id"] == found_obj["id"]
+
+
 def test_blob_search_upload_count(admin_session):
     blob_name = rand_string(15)
     blob_content = """
@@ -113,3 +153,4 @@ def test_blob_search_upload_count(admin_session):
 
     found_configs = users_session.search(f'tag:{tag} AND blob.upload_count:[* TO {len(test_users)}}}')
     assert len(found_configs) == 0
+
diff --git a/tests/backend/test_config.py b/tests/backend/test_config.py
index 7240fae79..2dc7c2cea 100644
--- a/tests/backend/test_config.py
+++ b/tests/backend/test_config.py
@@ -40,7 +40,6 @@ def test_adding_config_with_inblobs(admin_session):
 def test_config_search_by_tags_transactional_added(admin_session):
     test = admin_session
 
-    # blob_name = rand_string(15)
     config_json = {
         "cnc": [1, 2, 3],
         "raw_cfg": {
@@ -80,6 +79,66 @@ def test_config_search_by_tags_transactional_added(admin_session):
     assert config_search_2["id"] == config["id"]
 
 
+def test_config_search_multi(admin_session):
+    test = admin_session
+
+    config_json_insert_1 = rand_string(15)
+    config_json_1 = {
+        "cnc": [1, 2, config_json_insert_1],
+        "raw_cfg": {
+            "in-blob": {
+                "blob_name": "In blob name",
+                "blob_type": "Blob type",
+                "content": "Blob content"
+            }
+        },
+        "peers": {
+            "in-blob": {
+                "blob_name": "Peers blob name",
+                "blob_type": "Peers blob type",
+                "content": "Hello"
+            }
+        }
+    }
+
+    config_json_insert_2 = rand_string(15)
+    config_json_2 = {
+        "cnc": [1, 3, config_json_insert_2],
+        "raw_cfg": {
+            "in-blob": {
+                "blob_name": "In blob name",
+                "blob_type": "Blob type",
+                "content": "Blob content"
+            }
+        },
+        "peers": {
+            "in-blob": {
+                "blob_name": "Peers blob name",
+                "blob_type": "Peers blob type",
+                "content": "Hello world"
+            }
+        }
+    }
+
+    config_1 = test.add_config(None, "malwarex", config_json_1)
+    config_2 = test.add_config(None, "malwarex", config_json_2)
+
+    # config cfg and dhash search
+    query = f'config.multi:"{config_json_insert_1} {config_2["id"]}"'
+    found_objs = test.search(query)
+    assert len(found_objs) == 2
+
+    # only cfg content search in multi field
+    query = f'config.multi:"{config_json_insert_1}"'
+    found_obj = test.search(query)[0]
+    assert found_obj["id"] == config_1["id"]
+
+    # only blog dhash search in multi field
+    query = f'config.multi:"{config_2["id"]}"'
+    found_obj = test.search(query)[0]
+    assert config_2["id"] == found_obj["id"]
+
+
 def test_config_search_upload_count(admin_session):
     value_1 = rand_string()
     value_2 = rand_string()
@@ -131,3 +190,4 @@ def test_config_search_upload_count(admin_session):
 
     found_configs = users_session.search(f'tag:{tag} AND config.upload_count:[* TO {len(test_users)}}}')
     assert len(found_configs) == 0
+
diff --git a/tests/backend/test_search.py b/tests/backend/test_search.py
index 02a3fbbb6..7ed88412e 100644
--- a/tests/backend/test_search.py
+++ b/tests/backend/test_search.py
@@ -499,3 +499,28 @@ def test_uploader_query(admin_session):
         Bob.session.search(f"uploader:{Alice.identity}")
     ]
     assert sorted(results) == sorted([FileC.dhash])
+
+def test_search_multi(admin_session):
+    test = admin_session
+
+    # multi hashes search
+    samples = []
+    for i in range(5):
+        filename = base62uuid()
+        file_content = base62uuid()
+        sample = test.add_sample(filename, file_content)
+        samples.append(sample)
+
+    query = f'file.multi:"{samples[0].get("sha512")} {samples[1].get("md5")} {samples[2].get("sha1")} {samples[3].get("sha256")} {samples[4].get("crc32")}"'
+    found_objs = test.search(query)
+    assert len(found_objs) == 5
+
+    # incorrect hash check
+    incorrect_hash = samples[0].get("sha512")[:-2]
+    with ShouldRaise(status_code=400):
+        found_objs = test.search(f'file.multi:"{incorrect_hash}"')
+
+    # wildcard in hash
+    wildcard_hash = samples[0].get("sha512")[:-100] + "*"
+    with ShouldRaise(status_code=400):
+        found_objs = test.search(f'file.multi:"{wildcard_hash}"')