Repository: OpenCTI-Platform/connectors

Issue:
restore-files.py crashes when missing find_element returns None
## Description

When using restore-files to incrementally update a second instance of OpenCTI the process crashes whenever a referenced element is not in the data being restored (normally because it's already in the system). The return from find_element in the resolve_missing function (line 79 currently) is not checked for "None" (as is done in the restore_files function at line 121).

## Environment

1. OS (where OpenCTI server runs): Ubuntu 20.4
2. OpenCTI version: 5.3.11
3. OpenCTI client: python
4. Other environment details:

## Reproducible Steps

Steps to create the smallest reproducible scenario:
1. Run backup-files on system 1
2. copy output to system 2
3. Run restore-files on system 2

## Expected Output

Ingest of all elements.

## Actual Output

INFO:root:Restore run directory @ 20220314T200100Z
Traceback (most recent call last):
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 188, in <module>
    RestoreFilesInstance.start()
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 181, in start
    self.restore_files()
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 133, in restore_files
    self.resolve_missing(dir_date, ids, missing_element, acc)
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 87, in resolve_missing
    self.resolve_missing(dir_date, element_ids, missing_element, acc)
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 80, in resolve_missing
    refs = ref_extractors([data])
  File "/opt/opencti-highside-sync/connectors-master/external-import/restore-files/src/restore-files.py", line 17, in ref_extractors
    for key in data.keys():
AttributeError: 'NoneType' object has no attribute 'keys'
Killed



Discussion:
Backup restore is not designed to do incremental synchronization.

To do that you can use the builtin synchronizer.

Ok, now I'm using restore-files.py for just restoring a full backup. And I got this same error again. Note: it was after a restart of restore-files, but all of the input data was in place. It just needs: "if missing_element is not None:" after line 79 in resolve_missing (just like in restore_files).
Any chance to send me your backup directory to reproduce locally?
(We can of course sign an NDA or anything required)

Resolving patch:
diff --git a/external-import/restore-files/src/restore-files.py b/external-import/restore-files/src/restore-files.py
--- a/external-import/restore-files/src/restore-files.py
+++ b/external-import/restore-files/src/restore-files.py
@@ -77,8 +77,9 @@ def resolve_missing(self, dir_date, element_ids, data, acc=[]):
                 not_in = next((x for x in acc if x["id"] == ref), None)
                 if not_in is None:
                     missing_element = self.find_element(dir_date, ref)
-                    acc.insert(0, missing_element)
-                    self.resolve_missing(dir_date, element_ids, missing_element, acc)
+                    if missing_element is not None:
+                        acc.insert(0, missing_element)
+                        self.resolve_missing(dir_date, element_ids, missing_element, acc)
 
     def restore_files(self):
         stix2_splitter = OpenCTIStix2Splitter()


Test patch: