Repository: google/timesketch

Issue:
Import CSV: Checkbox for multiple fields selection
**Problem to solve**
When a user uploads a CSV file, Timesketch checks if the mandatory CSV headers are present or not. With the PR #2261 we partially solved this problem, i.e., the user can now map the missing headers with some existing ones. However, descriptive fields, such as "message" header, should be mapped to multiple fields to give the analyst the possibility to grab more information directly from that field.

**Possible solution**
The workflow is the same as #2261. However, a missing field can be mapped in two different manners: (1) using a drop down menu, or (2) using a list of check boxes. The image below shows a possible interface;

![image](https://user-images.githubusercontent.com/108743205/183947922-52aed58f-bab1-4b3a-94b8-d3c972077012.png)

When the user clicks on the button 'submit', the server will receive the following parameters:
* File's metadata,
* Headers mapping, and
* Delimiter.
The headers mapping is a list of dictionaries with the following structure:
* target : the name of the header that we want to insert into the CSV,
* source : the list of headers that we want to rename or combine. It may be null if we want to insert a new column by renaming/combining any columns,
* default_value : in case the previous value is null, then this expresses the default value of the new column.

For example, let's analyze the headers_mapping like the one below:

```
headers_mapping = [
                   {"source" : "datetime", "target" : ["data_UtcTime"], "default_value" : None},
                   {"source" : "message", "message" : ["event_ID", "event_ID_desc"], "default_value" : None},
                   {"source" : "timestamp_desc", "target" : None, "default_value" : Creation_time},
                   ] 
```
It means:

1. Rename column `data_UtcTime` into `datetime`.
2. Create column `message` combining `["event_ID", "event_ID_desc"]` columns' values.
3. Create column `timestamp_desc`. Fill this column's values with a default value equal to `Creation_time`.

We decided to:
* Rename a column if the source value in the python dictionary has only one element (first dictionary of the list),
* Create a column if the source value of a dictionary consists of multiple headers or it is none.

**Additional context**
This Issue extends #2261.


Discussion:

Resolving patch:
diff --git a/timesketch/api/v1/resources/upload.py b/timesketch/api/v1/resources/upload.py
--- a/timesketch/api/v1/resources/upload.py
+++ b/timesketch/api/v1/resources/upload.py
@@ -46,8 +46,7 @@ class UploadFileResource(resources.ResourceMixin, Resource):
     """Resource that processes uploaded files."""
 
     def _get_index(
-        self, name, description, sketch, index_name="",
-        data_label="", extension=""
+        self, name, description, sketch, index_name="", data_label="", extension=""
     ):
         """Returns a SearchIndex object to be used for uploads.
 
@@ -99,10 +98,7 @@ def _get_index(
 
         index_name = index_name or uuid.uuid4().hex
         searchindex = SearchIndex.get_or_create(
-            name=name,
-            index_name=index_name,
-            description=description,
-            user=current_user
+            name=name, index_name=index_name, description=description, user=current_user
         )
 
         searchindex.grant_permission(permission="read", user=current_user)
@@ -132,7 +128,7 @@ def _upload_and_index(
         events="",
         meta=None,
         headers_mapping=None,
-        delimiter=","
+        delimiter=",",
     ):
         """Creates a full pipeline for an uploaded file and returns the results.
 
@@ -153,7 +149,7 @@ def _upload_and_index(
                   included in the return.
             headers_mapping: list of dicts containing:
                              (i) target header we want to insert [key=target],
-                             (ii) source header we want to replace [key=source], and
+                             (ii) sources header we want to rename/combine [key=source],
                              (iii) def. value if we add a new column [key=default_value]
 
             delimiter: delimiter to read the CSV file
@@ -180,8 +176,7 @@ def _upload_and_index(
                 "google/timesketch/issues/new/choose",
             )
 
-        timelines = Timeline.query.filter_by(
-            name=timeline_name, sketch=sketch).all()
+        timelines = Timeline.query.filter_by(name=timeline_name, sketch=sketch).all()
 
         timeline = None
         for timeline_ in timelines:
@@ -200,8 +195,7 @@ def _upload_and_index(
                 )
             )
 
-            timeline_name = "{0:s}_{1:s}".format(
-                timeline_name, uuid.uuid4().hex[-5:])
+            timeline_name = "{0:s}_{1:s}".format(timeline_name, uuid.uuid4().hex[-5:])
             return self._upload_and_index(
                 file_extension=file_extension,
                 timeline_name=timeline_name,
@@ -215,7 +209,7 @@ def _upload_and_index(
                 events=events,
                 meta=meta,
                 headers_mapping=headers_mapping,
-                delimiter=delimiter
+                delimiter=delimiter,
             )
 
         searchindex.set_status("processing")
@@ -284,7 +278,7 @@ def _upload_and_index(
             only_index=enable_stream,
             timeline_id=timeline.id,
             headers_mapping=headers_mapping,
-            delimiter=delimiter
+            delimiter=delimiter,
         )
         task_id = uuid.uuid4().hex
         pipeline.apply_async(task_id=task_id)
@@ -293,10 +287,7 @@ def _upload_and_index(
             meta = {}
 
         meta["task_id"] = task_id
-        return self.to_json(
-            timeline,
-            status_code=HTTP_STATUS_CODE_CREATED,
-            meta=meta)
+        return self.to_json(timeline, status_code=HTTP_STATUS_CODE_CREATED, meta=meta)
 
     def _upload_events(self, events, form, sketch, index_name):
         """Upload a file like object.
@@ -334,7 +325,7 @@ def _upload_file(
         index_name,
         chunk_index_name="",
         headers_mapping=None,
-        delimiter=","
+        delimiter=",",
     ):
         """Upload a file.
 
@@ -347,7 +338,7 @@ def _upload_file(
                 chunks are used.
             headers_mapping: list of dicts containing:
                              (i) target header we want to insert [key=target],
-                             (ii) source header we want to replace [key=source], and
+                             (ii) sources header we want to rename/combine [key=source],
                              (iii) def. value if we add a new column [key=default_value]
             delimiter: delimiter to read the CSV file
 
@@ -375,19 +366,13 @@ def _upload_file(
         if isinstance(chunk_byte_offset, str) and chunk_byte_offset.isdigit():
             chunk_byte_offset = int(chunk_byte_offset)
         chunk_total_chunks = form.get("chunk_total_chunks")
-        if (
-            isinstance(chunk_total_chunks, str) and
-            chunk_total_chunks.isdigit()
-        ):
+        if isinstance(chunk_total_chunks, str) and chunk_total_chunks.isdigit():
             chunk_total_chunks = int(chunk_total_chunks)
         file_size = form.get("total_file_size")
         if isinstance(file_size, str) and file_size.isdigit():
             file_size = int(file_size)
         if file_size <= 0:
-            abort(
-                HTTP_STATUS_CODE_BAD_REQUEST,
-                "Unable to upload file. File is empty"
-            )
+            abort(HTTP_STATUS_CODE_BAD_REQUEST, "Unable to upload file. File is empty")
         enable_stream = form.get("enable_stream", False)
 
         data_label = form.get("data_label", "")
@@ -405,7 +390,7 @@ def _upload_file(
                 data_label=data_label,
                 enable_stream=enable_stream,
                 headers_mapping=headers_mapping,
-                delimiter=delimiter
+                delimiter=delimiter,
             )
 
         # For file chunks we need the correct filepath, otherwise each chunk
@@ -470,7 +455,7 @@ def _upload_file(
             enable_stream=enable_stream,
             meta=meta,
             headers_mapping=headers_mapping,
-            delimiter=delimiter
+            delimiter=delimiter,
         )
 
     @login_required
@@ -489,7 +474,7 @@ def post(self):
             form = request.form
 
         # headers mapping: map between mandatory headers and new ones
-        headers_mapping = json.loads(form.get("headersMapping", '{}')) or None
+        headers_mapping = json.loads(form.get("headersMapping", "{}")) or None
         delimiter = form.get("delimiter", ",")
 
         sketch_id = form.get("sketch_id", None)
@@ -533,7 +518,7 @@ def post(self):
                 sketch=sketch,
                 index_name=index_name,
                 headers_mapping=headers_mapping,
-                delimiter=delimiter
+                delimiter=delimiter,
             )
 
         events = form.get("events")
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -160,8 +160,7 @@ def _close_index(index_name, data_store, timeline_id):
         data_store.client.indices.close(index=index_name)
     except NotFoundError:
         logger.error(
-            "Unable to close index: {0:s} - index not " "found".format(
-                index_name)
+            "Unable to close index: {0:s} - index not " "found".format(index_name)
         )
 
 
@@ -239,7 +238,7 @@ def build_index_pipeline(
     only_index=False,
     timeline_id=None,
     headers_mapping=None,
-    delimiter=","
+    delimiter=",",
 ):
     """Build a pipeline for index and analysis.
 
@@ -267,8 +266,7 @@ def build_index_pipeline(
         task group.
     """
     if not (file_path or events):
-        raise RuntimeError(
-            "Unable to upload data, missing either a file or events.")
+        raise RuntimeError("Unable to upload data, missing either a file or events.")
     index_task_class = _get_index_task_class(file_extension)
     sketch_analyzer_chain = None
     searchindex = SearchIndex.query.filter_by(index_name=index_name).first()
@@ -283,16 +281,11 @@ def build_index_pipeline(
             file_extension,
             timeline_id,
             headers_mapping,
-            delimiter
+            delimiter,
         )
     else:
         index_task = index_task_class.s(
-            file_path,
-            events,
-            timeline_name,
-            index_name,
-            file_extension,
-            timeline_id
+            file_path, events, timeline_name, index_name, file_extension, timeline_id
         )
 
     # TODO: Check if a scenario is set or an investigative question
@@ -352,16 +345,14 @@ def build_sketch_analysis_pipeline(
     if not analyzer_names:
         analyzer_names = current_app.config.get("AUTO_SKETCH_ANALYZERS", [])
         if not analyzer_kwargs:
-            analyzer_kwargs = current_app.config.get(
-                "AUTO_SKETCH_ANALYZERS_KWARGS", {})
+            analyzer_kwargs = current_app.config.get("AUTO_SKETCH_ANALYZERS_KWARGS", {})
 
     # Exit early if no sketch analyzers are configured to run.
     if not analyzer_names:
         return None, None
 
     if not analyzer_kwargs:
-        analyzer_kwargs = current_app.config.get(
-            "ANALYZERS_DEFAULT_KWARGS", {})
+        analyzer_kwargs = current_app.config.get("ANALYZERS_DEFAULT_KWARGS", {})
 
     if user_id:
         user = User.query.get(user_id)
@@ -470,8 +461,7 @@ def run_email_result_task(index_name, sketch_id=None):
     """
     # We need to get a fake request context so that url_for() will work.
     with current_app.test_request_context():
-        searchindex = SearchIndex.query.filter_by(
-            index_name=index_name).first()
+        searchindex = SearchIndex.query.filter_by(index_name=index_name).first()
         sketch = None
 
         try:
@@ -498,16 +488,13 @@ def run_email_result_task(index_name, sketch_id=None):
                     '<a href="{0:s}">{1:s}</a>'.format(view_url, view_name)
                 )
 
-            body = body + \
-                "<br><br><b>Sketch</b><br>{0:s}".format(sketch.external_url)
+            body = body + "<br><br><b>Sketch</b><br>{0:s}".format(sketch.external_url)
 
             analysis_results = searchindex.description.replace("\n", "<br>")
-            body = body + \
-                "<br><br><b>Analysis</b>{0:s}".format(analysis_results)
+            body = body + "<br><br><b>Analysis</b>{0:s}".format(analysis_results)
 
             if view_links:
-                body = body + "<br><br><b>Views</b><br>" + \
-                    "<br>".join(view_links)
+                body = body + "<br><br><b>Views</b><br>" + "<br>".join(view_links)
 
         try:
             send_email(subject, body, to_username, use_html=True)
@@ -519,12 +506,7 @@ def run_email_result_task(index_name, sketch_id=None):
 
 @celery.task(track_started=True)
 def run_sketch_analyzer(
-    index_name,
-    sketch_id,
-    analysis_id,
-    analyzer_name,
-    timeline_id=None,
-    **kwargs
+    index_name, sketch_id, analysis_id, analyzer_name, timeline_id=None, **kwargs
 ):
     """Create a Celery task for a sketch analyzer.
 
@@ -540,10 +522,7 @@ def run_sketch_analyzer(
     """
     analyzer_class = manager.AnalysisManager.get_analyzer(analyzer_name)
     analyzer = analyzer_class(
-        sketch_id=sketch_id,
-        index_name=index_name,
-        timeline_id=timeline_id,
-        **kwargs
+        sketch_id=sketch_id, index_name=index_name, timeline_id=timeline_id, **kwargs
     )
 
     result = analyzer.run_wrapper(analysis_id)
@@ -552,14 +531,7 @@ def run_sketch_analyzer(
 
 
 @celery.task(track_started=True, base=SqlAlchemyTask)
-def run_plaso(
-    file_path,
-    events,
-    timeline_name,
-    index_name,
-    source_type,
-    timeline_id
-):
+def run_plaso(file_path, events, timeline_name, index_name, source_type, timeline_id):
     """Create a Celery task for processing Plaso storage file.
 
     Args:
@@ -578,10 +550,7 @@ def run_plaso(
     """
     if not plaso:
         raise RuntimeError(
-            (
-                "Plaso isn't installed, "
-                "unable to continue processing plaso files."
-            )
+            ("Plaso isn't installed, " "unable to continue processing plaso files.")
         )
 
     plaso_version = int(plaso.__version__)
@@ -628,8 +597,7 @@ def run_plaso(
             "process plaso file."
         )
 
-    opensearch = OpenSearchDataStore(
-        host=opensearch_server, port=opensearch_port)
+    opensearch = OpenSearchDataStore(host=opensearch_server, port=opensearch_port)
 
     try:
         opensearch.create_index(
@@ -638,24 +606,14 @@ def run_plaso(
     except errors.DataIngestionError as e:
         _set_timeline_status(timeline_id, status="fail", error_msg=str(e))
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         raise
 
-    except (
-        RuntimeError,
-        ImportError,
-        NameError,
-        UnboundLocalError,
-        RequestError
-    ) as e:
+    except (RuntimeError, ImportError, NameError, UnboundLocalError, RequestError) as e:
         _set_timeline_status(timeline_id, status="fail", error_msg=str(e))
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         raise
 
@@ -665,9 +623,7 @@ def run_plaso(
         _set_timeline_status(timeline_id, status="fail", error_msg=error_msg)
         logger.error("Error: {0!s}\n{1:s}".format(e, error_msg))
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         return None
 
@@ -718,15 +674,12 @@ def run_plaso(
 
     # Run psort.py
     try:
-        subprocess.check_output(
-            cmd, stderr=subprocess.STDOUT, encoding="utf-8")
+        subprocess.check_output(cmd, stderr=subprocess.STDOUT, encoding="utf-8")
     except subprocess.CalledProcessError as e:
         # Mark the searchindex and timelines as failed and exit the task
         _set_timeline_status(timeline_id, status="fail", error_msg=e.output)
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         return e.output
 
@@ -745,7 +698,7 @@ def run_csv_jsonl(
     source_type,
     timeline_id,
     headers_mapping=None,
-    delimiter=','
+    delimiter=",",
 ):
     """Create a Celery task for processing a CSV or JSONL file.
 
@@ -758,7 +711,7 @@ def run_csv_jsonl(
         timeline_id: ID of the timeline object this data belongs to.
         headers_mapping: list of dicts containing:
                          (i) target header we want to insert [key=target],
-                         (ii) source header we want to replace [key=source], and
+                         (ii) sources header we want to rename/combine [key=source],
                          (iii) def. value if we add a new column [key=default_value]
 
     Returns:
@@ -768,8 +721,7 @@ def run_csv_jsonl(
         file_handle = io.StringIO(events)
         source_type = "jsonl"
     else:
-        file_handle = codecs.open(
-            file_path, "r", encoding="utf-8", errors="replace")
+        file_handle = codecs.open(file_path, "r", encoding="utf-8", errors="replace")
 
     event_type = "generic_event"  # Document type for OpenSearch
     validators = {
@@ -819,7 +771,7 @@ def run_csv_jsonl(
         for event in read_and_validate(
             file_handle=file_handle,
             headers_mapping=headers_mapping,
-            delimiter=delimiter
+            delimiter=delimiter,
         ):
             opensearch.import_event(
                 index_name, event_type, event, timeline_id=timeline_id
@@ -839,21 +791,14 @@ def run_csv_jsonl(
     except errors.DataIngestionError as e:
         _set_timeline_status(timeline_id, status="fail", error_msg=str(e))
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         raise
 
-    except (
-        RuntimeError, ImportError, NameError,
-        UnboundLocalError, RequestError
-    ) as e:
+    except (RuntimeError, ImportError, NameError, UnboundLocalError, RequestError) as e:
         _set_timeline_status(timeline_id, status="fail", error_msg=str(e))
         _close_index(
-            index_name=index_name,
-            data_store=opensearch,
-            timeline_id=timeline_id
+            index_name=index_name, data_store=opensearch, timeline_id=timeline_id
         )
         raise
 
@@ -964,8 +909,7 @@ def find_data_task(
 
 
 def run_data_finder(
-    rule_names, sketch_id, start_date,
-    end_date, timeline_ids=None, parameters=None
+    rule_names, sketch_id, start_date, end_date, timeline_ids=None, parameters=None
 ):
     """Runs a task to find out if data exists in a dataset.
 
diff --git a/timesketch/lib/utils.py b/timesketch/lib/utils.py
--- a/timesketch/lib/utils.py
+++ b/timesketch/lib/utils.py
@@ -99,7 +99,7 @@ def _validate_csv_fields(mandatory_fields, data, headers_mapping=None):
         data: a DataFrame built from the ingested file.
         headers_mapping: list of dicts containing:
                          (i) target header we want to insert [key=target],
-                         (ii) source header we want to replace [key=source], and
+                         (ii) sources header we want to rename/combine [key=source],
                          (iii) def. value if we add a new column [key=default_value]
     Raises:
         RuntimeError: if there are missing fields.
@@ -132,15 +132,11 @@ def _validate_csv_fields(mandatory_fields, data, headers_mapping=None):
         parset_set_string = "None"
 
     raise RuntimeError(
-        """Missing mandatory CSV headers.
-        Mandatory headers: {0:s}
-        Headers found in the file: {1:s}
-        Headers provided in the mapping: {2:s}
-        Headers missing: {3:s}""".format(
-            ", ".join(list(mandatory_set)),
-            parset_set_string,
-            headers_mapping_string,
-            headers_missing_string)
+        f"Missing mandatory CSV headers."
+        f"Mandatory headers: {', '.join(list(mandatory_set))}"
+        f"Headers found in the file: {parset_set_string}"
+        f"Headers provided in the mapping: {headers_mapping_string}"
+        f"Headers missing: {headers_missing_string}"
     )
 
 
@@ -167,7 +163,7 @@ def check_mapping_errors(headers, headers_mapping):
         csv_headers: list of headers found in the CSV file.
         headers_mapping: list of dicts containing:
                          (i) target header we want to insert [key=target],
-                         (ii) source header we want to replace [key=source], and
+                         (ii) sources header we want to rename/combine [key=source],
                          (iii) def. value if we add a new column [key=default_value]
 
     Raises:
@@ -181,26 +177,32 @@ def check_mapping_errors(headers, headers_mapping):
         if mapping["target"] in headers:
             raise RuntimeError(
                 "Headers mapping is wrong.\n"
-                "Mapping done only if the mandatory header is missing")
-        if mapping["source"]:
-            # 3. Check if the header specified in headers mapping is in the headers list
-            if mapping["source"] not in headers:
-                raise RuntimeError(
-                "Value specified in the headers mapping not found in the CSV\n"
-                "Value specified in headers mapping: {0:s}\n"
-                "CSV columns: {1:s}".format(
-                    mapping["source"], ", ".join(headers))
+                "Mapping done only if the mandatory header is missing"
             )
-            # update the headers list that we will substitute
-            candidate_headers.append(mapping["source"])
+        if mapping["source"]:
+            # 3. Check if any of the headers specified in headers mapping
+            # is in the headers list
+            for source in mapping["source"]:
+                if source not in headers:
+                    raise RuntimeError(
+                        f"Value specified in the headers mapping not found in the CSV\n"
+                        f"Headers mapping: {', '.join(mapping['source'])}\n"
+                        f"Sources column/s: {source}\n"
+                        f"CSV columns: {', '.join(headers)}"
+                    )
+
+            # Update the headers list that we will substitute/rename
+            # we do this check only over the header column that will be renamed,
+            # i.e., when mapping["source"] has only 1 value
+            if len(mapping["source"]) == 1:
+                candidate_headers.append(mapping["source"][0])
 
         else:
             if not mapping["default_value"]:
                 raise RuntimeError(
-                    "Headers mapping is wrong.\n"
-                    "Error to create new column {0:s}. "
-                    "When create a new column, a default value must be assigned"
-                    .format(mapping["target"])
+                    f"Headers mapping is wrong.\n"
+                    f"Error to create new column {mapping['target']}. "
+                    f"When create a new column, a default value must be assigned"
                 )
     # 4. check if two or more mandatory headers are mapped
     #    to the same exisiting header
@@ -213,33 +215,41 @@ def check_mapping_errors(headers, headers_mapping):
 
 
 def rename_headers(chunk, headers_mapping):
-    """"Rename the headers of the dataframe
+    """ "Rename the headers of the dataframe
 
     Args:
         chunk: dataframe to be modified
         headers_mapping: list of dicts containing:
                          (i) target header we want to insert [key=target],
-                         (ii) source header we want to replace [key=source], and
+                         (ii) sources header we want to rename/combine [key=source],
                          (iii) def. value if we add a new column [key=default_value]
 
     Returns: the dataframe with renamed headers
     """
+    headers_mapping.sort(
+        key=lambda x: len(x["source"]) if x["source"] else 0, reverse=True
+    )
     for mapping in headers_mapping:
         if not mapping["source"]:
-            # add header and def values
+            # create new column with a given default value
             chunk[mapping["target"]] = mapping["default_value"]
+        elif len(mapping["source"]) > 1:
+            # concatanete multiple source headers into a new one
+            chunk[mapping["target"]] = ""
+            for column in mapping["source"]:
+                chunk[mapping["target"]] += (
+                    column + ":" + chunk[column].map(str) + " | "
+                )
         else:
             # just rename the header
             chunk.rename(
-                columns={mapping["source"]: mapping["target"]}, inplace=True)
+                columns={mapping["source"][0]: mapping["target"]}, inplace=True
+            )
     return chunk
 
 
 def read_and_validate_csv(
-    file_handle,
-    delimiter=",",
-    mandatory_fields=None,
-    headers_mapping=None
+    file_handle, delimiter=",", mandatory_fields=None, headers_mapping=None
 ):
     """Generator for reading a CSV file.
 
@@ -249,7 +259,7 @@ def read_and_validate_csv(
         mandatory_fields: list of fields that must be present in the CSV header
         headers_mapping: list of dicts containing:
                          (i) target header we want to insert [key=target],
-                         (ii) source header we want to replace [key=source], and
+                         (ii) sources header we want to rename/combine [key=source],
                          (iii) def. value if we add a new column [key=default_value]
     Raises:
         RuntimeError: when there are missing fields.
@@ -303,8 +313,7 @@ def read_and_validate_csv(
                             idx * reader.chunksize + num_chunk_rows,
                         )
                     )
-                chunk["timestamp"] = chunk["datetime"].dt.strftime(
-                    "%s%f").astype(int)
+                chunk["timestamp"] = chunk["datetime"].dt.strftime("%s%f").astype(int)
                 chunk["datetime"] = (
                     chunk["datetime"].apply(Timestamp.isoformat).astype(str)
                 )
@@ -312,8 +321,7 @@ def read_and_validate_csv(
                 logger.warning(
                     "Rows {0} to {1} skipped due to malformed "
                     "datetime values ".format(
-                        idx * reader.chunksize, idx *
-                        reader.chunksize + chunk.shape[0]
+                        idx * reader.chunksize, idx * reader.chunksize + chunk.shape[0]
                     )
                 )
                 continue
@@ -342,13 +350,9 @@ def read_and_validate_redline(file_handle):
     """
 
     csv.register_dialect(
-        "redlineDialect",
-        delimiter=",",
-        quoting=csv.QUOTE_ALL,
-        skipinitialspace=True
+        "redlineDialect", delimiter=",", quoting=csv.QUOTE_ALL, skipinitialspace=True
     )
-    reader = pandas.read_csv(file_handle, delimiter=",",
-                             dialect="redlineDialect")
+    reader = pandas.read_csv(file_handle, delimiter=",", dialect="redlineDialect")
 
     _validate_csv_fields(REDLINE_FIELDS, reader)
     for row in reader:
@@ -403,8 +407,7 @@ def read_and_validate_jsonl(file_handle, **kwargs):  # pylint: disable=unused-ar
             if "timestamp" not in ld_keys and "datetime" in ld_keys:
                 try:
                     linedict["timestamp"] = int(
-                        parser.parse(linedict["datetime"]
-                                     ).timestamp() * 1000000
+                        parser.parse(linedict["datetime"]).timestamp() * 1000000
                     )
                 except parser.ParserError:
                     logger.error(
@@ -423,15 +426,13 @@ def read_and_validate_jsonl(file_handle, **kwargs):  # pylint: disable=unused-ar
                 )
 
             if "tag" in linedict:
-                linedict["tag"] = [
-                    x for x in _parse_tag_field(linedict["tag"]) if x]
+                linedict["tag"] = [x for x in _parse_tag_field(linedict["tag"]) if x]
             _scrub_special_tags(linedict)
             yield linedict
 
         except ValueError as e:
             raise errors.DataIngestionError(
-                "Error parsing JSON at line {0:n}: {1:s}".format(
-                    lineno, str(e))
+                "Error parsing JSON at line {0:n}: {1:s}".format(lineno, str(e))
             )
 
 
@@ -482,10 +483,7 @@ def get_validated_indices(indices, sketch):
                         timelines.add(timeline_id)
                         indices.append(index)
 
-                    if (
-                        isinstance(item, str) and
-                        item.lower() == timeline_name.lower()
-                    ):
+                    if isinstance(item, str) and item.lower() == timeline_name.lower():
                         timelines.add(timeline_id)
                         indices.append(index)
 
@@ -508,8 +506,7 @@ def send_email(subject, body, to_username, use_html=False):
     email_enabled = current_app.config.get("ENABLE_EMAIL_NOTIFICATIONS")
     email_domain = current_app.config.get("EMAIL_DOMAIN")
     email_smtp_server = current_app.config.get("EMAIL_SMTP_SERVER")
-    email_from_user = current_app.config.get(
-        "EMAIL_FROM_ADDRESS", "timesketch")
+    email_from_user = current_app.config.get("EMAIL_FROM_ADDRESS", "timesketch")
     email_user_whitelist = current_app.config.get("EMAIL_USER_WHITELIST", [])
 
     if not email_enabled:


Test patch:
diff --git a/timesketch/lib/utils_test.py b/timesketch/lib/utils_test.py
--- a/timesketch/lib/utils_test.py
+++ b/timesketch/lib/utils_test.py
@@ -72,9 +72,9 @@ def test_invalid_headers_mapping(self):
         current_headers = ["DT", "message", "TD"]
 
         invalid_mapping_1 = [
-            {"target": "datetime", "source": "DT", "default_value": None},
-            {"target": "timestamp_desc", "source": "No.", "default_value": None},
-            {"target": "message", "source": "Source", "default_value": None}
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": ["No."], "default_value": None},
+            {"target": "message", "source": ["Source"], "default_value": None},
         ]
         # column message already exists
         with self.assertRaises(RuntimeError):
@@ -82,8 +82,12 @@ def test_invalid_headers_mapping(self):
             next(check_mapping_errors(current_headers, invalid_mapping_1))
 
         invalid_mapping_2 = [
-            {"target": "datetime", "source": "DT", "default_value": None},
-            {"target": "timestamp_desc", "source": "nope", "default_value": None}
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {
+                "target": "timestamp_desc",
+                "source": ["TD", "nope"],
+                "default_value": None,
+            },
         ]
         # nope columns does not exists
         with self.assertRaises(RuntimeError):
@@ -91,53 +95,61 @@ def test_invalid_headers_mapping(self):
             next(check_mapping_errors(current_headers, invalid_mapping_2))
 
         invalid_mapping_3 = [
-            {"target": "datetime", "source": "DT", "default_value": None},
-            {"target": "timestamp_desc", "source": "DT", "default_value": None}
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": ["DT"], "default_value": None},
         ]
         # 2 mandatory headers point to the same existing one
         with self.assertRaises(RuntimeError):
             # Call next to work around lazy generators.
             next(check_mapping_errors(current_headers, invalid_mapping_3))
 
-    def test_right_headers_mapping(self):
+    def test_valid_headers_mapping(self):
         """Test for valid headers mapping"""
         current_headers = ["DT", "message", "TD"]
 
         valid_mapping_1 = [
-            {"target": "datetime", "source": "DT", "default_value": None},
-            {"target": "timestamp_desc", "source": "TD", "default_value": None}
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": ["TD"], "default_value": None},
         ]
-        self.assertIs(check_mapping_errors(
-            current_headers,
-            valid_mapping_1), None)
+        self.assertIs(check_mapping_errors(current_headers, valid_mapping_1), None)
 
         valid_mapping_2 = [
-            {"target": "datetime", "source": "DT", "default_value": None},
-            {"target": "timestamp_desc", "source": None, "default_value": "a"}
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": None, "default_value": "a"},
         ]
-        self.assertIs(check_mapping_errors(
-            current_headers,
-            valid_mapping_2), None)
+        self.assertIs(check_mapping_errors(current_headers, valid_mapping_2), None)
+
+        current_headers = ["DT", "last_access", "TD", "file_path"]
+        valid_mapping_3 = [
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": None, "default_value": "a"},
+            {"target": "message", "source": ["TD", "file_path"], "default_value": None},
+        ]
+        self.assertIs(check_mapping_errors(current_headers, valid_mapping_3), None)
+
+        current_headers = ["DT", "last_access", "TD", "file_path", "T_desc"]
+        valid_mapping_4 = [
+            {"target": "datetime", "source": ["DT"], "default_value": None},
+            {"target": "timestamp_desc", "source": ["T_desc"], "default_value": None},
+            {
+                "target": "message",
+                "source": ["T_desc", "file_path"],
+                "default_value": None,
+            },
+        ]
+        self.assertIs(check_mapping_errors(current_headers, valid_mapping_4), None)
 
     def test_invalid_CSV_file(self):
         """Test for CSV with missing mandatory headers without mapping"""
         mandatory_fields = ["message", "datetime", "timestamp_desc"]
 
-        df_01 = pd.DataFrame({
-            'DT': ['test'],
-            'MSG': ['test'],
-            'TD': ['test']
-        })
+        df_01 = pd.DataFrame({"DT": ["test"], "MSG": ["test"], "TD": ["test"]})
 
         with self.assertRaises(RuntimeError):
             # Call next to work around lazy generators.
             next(_validate_csv_fields(mandatory_fields, df_01))
 
-        df_02 = pd.DataFrame({
-            'datetime': ['test'],
-            'MSG': ['test'],
-            'TD': ['test']
-        })
+        df_02 = pd.DataFrame({"datetime": ["test"], "MSG": ["test"], "TD": ["test"]})
         with self.assertRaises(RuntimeError):
             # Call next to work around lazy generators.
             next(_validate_csv_fields(mandatory_fields, df_02))