Repository: matchms/matchms
Pull request #480 (2023-08-14 10:11:31)

Add individual filters in yaml
Solves #478 

Instead of storing "default" in the pipeline we will store the actual filters called in the order they are run in a yaml file. 

This makes it possible to run a yaml file of an older version of matchms (with different default filters) and still getting the same results. 
It also gives me a quick check that the default filters actually do what I expect them to do as an user, without having to dive into the matchms code.


Patch:
diff --git a/.github/workflows/CI_build.yml b/.github/workflows/CI_build.yml
index f8e3c7cd..09a25c89 100644
--- a/.github/workflows/CI_build.yml
+++ b/.github/workflows/CI_build.yml
@@ -1,8 +1,9 @@
 name: CI Build
 
 on:
+  push:
   pull_request:
-    types: [opened, ready_for_review, closed]
+    branches: [ master ]
   workflow_dispatch:
     inputs:
       tag:
@@ -11,6 +12,7 @@ on:
         default: ''
 
 jobs:
+
   first_check:
     name: first code check / python-3.9 / ubuntu-latest
     runs-on: ubuntu-latest
diff --git a/.github/workflows/CI_first_code_check.yml b/.github/workflows/CI_first_code_check.yml
deleted file mode 100644
index d667c4bb..00000000
--- a/.github/workflows/CI_first_code_check.yml
+++ /dev/null
@@ -1,60 +0,0 @@
-name: CI first code check
-
-on:
-  push:
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: Tag for manually running CI first code check workflow
-        required: False
-        default: ''
-
-jobs:
-
-  first_check:
-    name: first code check / python-3.9 / ubuntu-latest
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v3
-      - name: Set up Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: 3.9
-      - name: Python info
-        run: |
-          which python
-          python --version
-      - name: Build package and create dev environment
-        run: |
-          python -m pip install --upgrade pip poetry
-          poetry install
-      - name: Show pip list
-        run: |
-          pip list
-      - name: Test with coverage
-        run: |
-          poetry run pytest --cov --cov-report term --cov-report xml --junitxml=xunit-result.xml
-      - name: Correct coverage paths
-        run: sed -i "s+$PWD/++g" coverage.xml
-      - name: Check style against standards using prospector
-        shell: bash -l {0}
-        run: prospector -o grouped -o pylint:pylint-report.txt
-      - name: Store PR metadata in json file
-        env:
-          PR_NUMBER: ${{ github.event.pull_request.number }}
-          HEAD_REF: ${{ github.event.pull_request.head.ref }}
-          BASE_REF: ${{ github.event.pull_request.base.ref }}
-        run: |
-          jq -n --arg pr_number "$PR_NUMBER"\
-                --arg head_ref "$HEAD_REF"\
-                --arg base_ref "$BASE_REF"\
-            '{"pr_number": $pr_number, "head_ref": $head_ref, "base_ref": $base_ref}' > pr_metadata.json
-      - uses: actions/upload-artifact@v3
-        with:
-          name: sonarcloud-data-pip
-          path: |
-            coverage.xml
-            xunit-result.xml
-            pr_metadata.json
-      - name: Check whether import statements are used consistently
-        run: isort . --check-only --diff
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 76da7629..32b7b238 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,16 +5,6 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [unreleased]
-
-### Added
-
-- New `SpectrumProcessing` class to be the central hub for all filter functions [#455](https://github.com/matchms/matchms/pull/455). Also takes care that filters are executed in a useful order. This is also integrated into the `Pipeline` class.
-
-### Changed
-
-- Extensive code refactoring and cleaning.
-
 ## [0.21.2] - 2023-08-01
 
 ### Added
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 00000000..e69de29b
diff --git a/README.rst b/README.rst
index 3ebdd02e..0250c3bb 100644
--- a/README.rst
+++ b/README.rst
@@ -137,8 +137,6 @@ To date we are aware of:
 
 + `matchmsextras <https://github.com/matchms/matchmsextras>`_ which contains additional functions to create networks based on spectral similarities, to run spectrum searchers against `PubChem`, or additional plotting methods.
 
-+ `MS2Query <https://github.com/iomega/ms2query>`_ Reliable and fast MS/MS spectral-based analogue search, running on top op matchms.
-
 + `memo <https://github.com/mandelbrot-project/memo>`_ a method allowing a Retention Time (RT) agnostic alignment of metabolomics samples using the fragmentation spectra (MS2) of their consituents.
 
 *(if you know of any other packages that are fully compatible with matchms, let us know!)*
diff --git a/matchms/Pipeline.py b/matchms/Pipeline.py
index 01cd774a..17f04d0c 100644
--- a/matchms/Pipeline.py
+++ b/matchms/Pipeline.py
@@ -30,10 +30,10 @@ def create_workflow(yaml_file_name=None,
     workflow = OrderedDict()
     workflow["importing"] = {"queries": query_file_name,
                              "references": reference_file_name}
-    workflow["predefined_processing_queries"] = predefined_processing_queries
-    workflow["predefined_processing_references"] = predefined_processing_reference
-    workflow["additional_processing_queries"] = additional_filters_queries
-    workflow["additional_processing_references"] = additional_filters_references
+    queries_processor = initialize_spectrum_processor(predefined_processing_queries, additional_filters_queries)
+    workflow["query_filters"] = queries_processor.processing_steps
+    queries_processor = initialize_spectrum_processor(predefined_processing_reference, additional_filters_references)
+    workflow["reference_filters"] = queries_processor.processing_steps
     workflow["score_computations"] = score_computations
     if yaml_file_name is not None:
         with open(yaml_file_name, 'w', encoding="utf-8") as file:
@@ -44,6 +44,16 @@ def create_workflow(yaml_file_name=None,
     return workflow
 
 
+def initialize_spectrum_processor(predefined_workflow, additional_filters):
+    """Initialize spectrum processing workflow."""
+    processor = SpectrumProcessor(predefined_workflow)
+    for step in additional_filters:
+        if isinstance(step, (tuple, list)) and len(step) == 1:
+            step = step[0]
+        processor.add_filter(step)
+    return processor
+
+
 def load_in_workflow_from_yaml_file(yaml_file):
     with open(yaml_file, 'r', encoding="utf-8") as file:
         workflow = ordered_load(file, yaml.SafeLoader)
@@ -131,13 +141,14 @@ class Pipeline:
                                        ["filter_by_range", {"name": "Spec2Vec", "low": 0.3}]]
 
     """
-    def __init__(self, workflow: OrderedDict, progress_bar=True):
+    def __init__(self, workflow, progress_bar=True):
         """
         Parameters
         ----------
-        workflow:
-            Contains an orderedDict containing the workflow settings. Can be created using create_workflow.
-        progress_bar:
+        config_file
+            Filename of config file (yaml file) to define pipeline. Default is None
+            in which case the pipeline should be defined via a Python script.
+        progress_bar
             Default is True. Set to False if no progress bar should be displayed.
         """
         self.spectrums_queries = []
@@ -155,25 +166,33 @@ class Pipeline:
     def _initialize_spectrum_processor_queries(self):
         """Initialize spectrum processing workflow for the query spectra."""
         self.processing_queries = initialize_spectrum_processor(
-            self.workflow["predefined_processing_queries"],
-            self.workflow["additional_processing_queries"]
+            None,
+            self.workflow["query_filters"]
             )
+
         self.write_to_logfile(str(self.processing_queries))
+        if self.processing_queries.processing_steps != self.workflow["query_filters"]:
+            #todo save the yaml file in a new file
+            logger.warning("The order of the filters has been changed compared to the Yaml file.")
 
     def _initialize_spectrum_processor_references(self):
         """Initialize spectrum processing workflow for the reference spectra."""
         self.processing_references = initialize_spectrum_processor(
-            self.workflow["predefined_processing_references"],
-            self.workflow["additional_processing_references"]
+            None,
+            self.workflow["reference_filters"]
             )
         self.write_to_logfile(str(self.processing_references))
+        self.write_to_logfile(str(self.processing_queries))
+        if self.processing_queries.processing_steps != self.workflow["query_filters"]:
+            #todo save the yaml file in a new file
+            logger.warning("The order of the filters has been changed compared to the Yaml file.")
 
     def complete_workflow(self):
         """Define Pipeline workflow based on a yaml file (config_file).
         """
         #todo add a check if all the fields are given that are needed.
-        if self.workflow["additional_processing_references"] == "processing_queries":
-            self.workflow["additional_processing_references"] = self.workflow["additional_processing_queries"]
+        if self.workflow["reference_filters"] == "processing_queries":
+            self.workflow["reference_filters"] = self.workflow["query_filters"]
         if "logging" not in self.workflow:
             self.workflow["logging"] = {}
         if self.logging_level is None:
@@ -390,13 +409,3 @@ class Pipeline:
     @score_computations.setter
     def score_computations(self, computations_list):
         self.workflow["score_computations"] = computations_list
-
-
-def initialize_spectrum_processor(predefined_workflow, additional_filters):
-    """Initialize spectrum processing workflow."""
-    processor = SpectrumProcessor(predefined_workflow)
-    for step in additional_filters:
-        if isinstance(step, (tuple, list)) and len(step) == 1:
-            step = step[0]
-        processor.add_filter(step)
-    return processor


Test patch:
diff --git a/tests/test_pipeline.yaml b/tests/test_pipeline.yaml
index d0a2ef58..1a3ca9b9 100644
--- a/tests/test_pipeline.yaml
+++ b/tests/test_pipeline.yaml
@@ -4,13 +4,27 @@
 importing:
   queries: ./tests/testdata/massbank_five_spectra.msp
   references:
-predefined_processing_queries: default
-predefined_processing_references: default
-additional_processing_queries:
+query_filters:
+- - make_charge_int
+- - interpret_pepmass
+- - derive_ionmode
+- - correct_charge
+- - add_compound_name
+- - derive_adduct_from_name
+- - derive_formula_from_name
+- - clean_compound_name
+- - add_precursor_mz
+- - require_precursor_mz
+- - add_parent_mass
+- - harmonize_undefined_inchikey
+- - harmonize_undefined_inchi
+- - harmonize_undefined_smiles
+- - repair_inchi_inchikey_smiles
+- - repair_parent_mass_match_smiles_wrapper
 - - normalize_intensities
 - - select_by_intensity
   - intensity_from: 0.001
-additional_processing_references: processing_queries
+reference_filters: processing_queries
 score_computations:
 - - precursormzmatch
   - tolerance: 120.0