Repository: OpenCTI-Platform/connectors

Issue:
[Import Document] Connector does not process MD files
## Description

The Import Document connector currently supports plain/text media type, however files with the `.md` file extension are not recognized as a valid document. 

## Environment

1. OS (where OpenCTI server runs): AWS ECS Fargate
2. OpenCTI version: 5.1.4
3. OpenCTI client: python
4. Other environment details:

## Reproducible Steps

Steps to create the smallest reproducible scenario:
1. Run the Import External Reference connector to get a .md file OR just upload a .md file to the platform
2. Try to run an enrichment on the .md file

## Expected Output

I would expect that the Import connector would or could import a file, regardless of the file name.  

## Actual Output

There is no Output as the connector/platform doesn't recognize the .md file.  Only work around is to download the file, rename to a .txt file extension, and upload to the platform.
 
## Screenshots (optional)
<img width="1483" alt="Screen Shot 2022-04-28 at 9 24 53 AM" src="https://user-images.githubusercontent.com/30411037/165775435-87f694cf-ada9-439f-9cf7-246228283d80.png">
<img width="753" alt="Screen Shot 2022-04-28 at 9 24 20 AM" src="https://user-images.githubusercontent.com/30411037/165775444-fa1ade88-51f8-45a1-9fd8-f1d14002d903.png">




Discussion:
Hey @TechBurn0ut 

Could you please run the connector in debug mode and post the logs? It is just a bit odd, since .md files are also of exiftype 'text/plain' and should be parsed like .txt files.

Regards
Hey @TechBurn0ut 

I had a look at it again, it is not a connector issue, rather than something platform related, since the import connector button is grayed out and it is not the connector which doesn't allow the file to parse.

I'll move this to the platform issues, but I don't know why this issue arises. Normally files are allowed to be parsed if a connector registers the file's exif type. The exif type should be extension independent, but this is not the case in the screenshots you showed.

I'll need a further look at this...

EDIT: Maybe first try adding `text/markdown` to the defined scopes?

Regards
@nor3th Yeah I wasn’t sure if it was on the connector side or platform. Thanks for looking into it!

Resolving patch:
diff --git a/internal-import-file/import-document/src/reportimporter/constants.py b/internal-import-file/import-document/src/reportimporter/constants.py
--- a/internal-import-file/import-document/src/reportimporter/constants.py
+++ b/internal-import-file/import-document/src/reportimporter/constants.py
@@ -2,6 +2,7 @@
 MIME_TXT = "text/plain"
 MIME_HTML = "text/html"
 MIME_CSV = "text/csv"
+MIME_MD = "text/markdown"
 
 RESULT_FORMAT_TYPE = "type"
 RESULT_FORMAT_CATEGORY = "category"
diff --git a/internal-import-file/import-document/src/reportimporter/report_parser.py b/internal-import-file/import-document/src/reportimporter/report_parser.py
--- a/internal-import-file/import-document/src/reportimporter/report_parser.py
+++ b/internal-import-file/import-document/src/reportimporter/report_parser.py
@@ -19,6 +19,7 @@
     MIME_TXT,
     MIME_HTML,
     MIME_CSV,
+    MIME_MD,
     OBSERVABLE_DETECTION_CUSTOM_REGEX,
     OBSERVABLE_DETECTION_LIBRARY,
 )
@@ -51,6 +52,7 @@ def __init__(
             MIME_TXT: self._parse_text,
             MIME_HTML: self._parse_html,
             MIME_CSV: self._parse_text,
+            MIME_MD: self._parse_text,
         }
 
         self.library_lookup = library_mapping()


Test patch: