Repository: SymbiFlow/FPGA-Tool-Performance-Visualization-Library

Issue:
Compare a given list of evaluations
Given a list of evaluations, produce series of tables (or rows) showing metrics with respect to the first eval e.g.


| Eval | Placement Time | Routing Time | CPD |
|-|-|-|-|
| 1 | 100s | 200s | 18ns |
| 2 | 50s (2x) | 150 (1.3x) | 18ns (1x) |
| 3 | 25s (4x) | 100 (2x) | 15ns (1.2x) |



Discussion:

Patch:
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
new file mode 100644
index 0000000..877eaf1
--- /dev/null
+++ b/.github/workflows/python-package.yml
@@ -0,0 +1,37 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Python package
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pylint pytest
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        pylint -E ftpvl/
+    - name: Test with pytest
+      run: |
+        pytest
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
new file mode 100644
index 0000000..f9fedba
--- /dev/null
+++ b/.github/workflows/python-publish.yml
@@ -0,0 +1,30 @@
+# This workflows will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish
+      env:
+        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+      run: |
+        python setup.py upload
diff --git a/README.md b/README.md
index 78773c8..7d84811 100644
--- a/README.md
+++ b/README.md
@@ -12,8 +12,9 @@ Take a look at the notebooks below to demonstrate the functionality of FTPVL.
 
 1. [Using `HydraFetcher` and Processors](https://colab.research.google.com/drive/1BIQ-iulDFpzcve7lGJPwLePJ5ETBJ6Ut?usp=sharing)
 2. [Styling tables with `SingleTableVisualizer`](https://colab.research.google.com/drive/1u3EnmIYnTBk-LXZhqNHt_h4aMuq-_cWq?usp=sharing)
-3. [Comparing two different Evaluations](https://colab.research.google.com/drive/1I7InmA6210vIIwdQ7TGHE6aF_WwIm1dM?usp=sharing)
+3. [Comparing two Evaluations using the internal dataframe](https://colab.research.google.com/drive/1I7InmA6210vIIwdQ7TGHE6aF_WwIm1dM?usp=sharing)
 4. [Filtering and Aggregating an Evaluation](https://colab.research.google.com/drive/1DDwlQFS81RGLL-q8DsgICF-HOC5ir6oS?usp=sharing)
+5. [Comparing multiple Evaluations](https://colab.research.google.com/drive/1kSF3bEjG_c6bLh9PLus9GPk5aLrtYhly?usp=sharing)
 
 ## Documentation
 Extensive documentation, including a *Getting Started* guide, is available on
@@ -41,11 +42,14 @@ make html
 * `pylint`: for linting ([website](https://www.pylint.org/))
 * `pytest`: testrunner ([website](https://docs.pytest.org/en/stable/))
 * `coverage`: for coverage testing ([website](https://coverage.readthedocs.io/))
-* `black`: for auto-formatting ([website](https://github.com/psf/black))
 * `sphinx`: for documentation generation ([website](https://www.sphinx-doc.org/en/master/))
 * `sphinx-rtd-theme`: for documentation generation (theme) ([website](https://github.com/readthedocs/sphinx_rtd_theme))
 
 ## Changes
+### 0.2.0
+* Added evaluation concatenation and `CompareToFirst` processors for relative comparisons between evaluations.
+* Fixed `HydraFetcher` issues when fetching an older evaluation that contains more than one build artifact.
+
 ### 0.1.6
 * Added support for filter and aggregator processors, fixes [#9](https://github.com/SymbiFlow/FPGA-Tool-Performance-Visualization-Library/issues/9)
 
diff --git a/docs/intro/firststeps.rst b/docs/intro/firststeps.rst
index 2af8cfb..5557161 100644
--- a/docs/intro/firststeps.rst
+++ b/docs/intro/firststeps.rst
@@ -230,7 +230,7 @@ Styling
 =======
 
 Now that the Evaluation has been processed, we can add styling so that
-important information stands out in the final visualization. This is achived
+important information stands out in the final visualization. This is achieved
 through a special type of Processor called :ref:`topics-styles`. 
 
 Styles are also run in a processing pipeline, but they always output CSS strings.
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 63231e0..2de1096 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -76,6 +76,8 @@ Processors API
 .. autoclass:: ftpvl.processors.GeomeanAggregate
     :members:
 
+.. autoclass:: ftpvl.processors.CompareToFirst
+
 .. _topics-api-styles:
 
 Styles API
@@ -99,4 +101,21 @@ Visualizers API
     :members:
 
 .. autoclass:: ftpvl.visualizers.SingleTableVisualizer
-    :members:
\ No newline at end of file
+    :members:
+
+Enums
+=====
+Direction
+*********
+.. autoclass:: ftpvl.processors.Direction
+    :members:
+
+    .. autoattribute:: MAXIMIZE
+    
+        Indicates that the corresponding metric is optimized by 
+        *maximizing* the value
+    
+    .. autoattribute:: MINIMIZE
+    
+        Indicates that the corresponding metric is optimized by
+        *minimizing* the value
diff --git a/docs/topics/evaluation.rst b/docs/topics/evaluation.rst
index 0a0c88d..2687637 100644
--- a/docs/topics/evaluation.rst
+++ b/docs/topics/evaluation.rst
@@ -21,6 +21,30 @@ transform the evaluation one-at-a-time.
 Extracting the internal dataframe
 =================================
 Evaluations store the test results internally using a Pandas dataframe. You can
-retrive this dataframe by using the ``get_df()`` method.
+retrieve this dataframe by using the ``get_df()`` method. This can be used to
+manipulate the underlying data without needing to use the built-in FTPVL processors.
 
-.. automethod:: ftpvl.evaluation.Evaluation.get_df
\ No newline at end of file
+.. note:: ``get_df()`` returns a defensive copy of the dataframe, so any 
+    mutations to the returned dataframe will not be reflected in the original
+    Evaluation. Instead, you must instantiate a new ``Evaluation`` by passing
+    in the dataframe and evaluation ID.
+
+.. automethod:: ftpvl.evaluation.Evaluation.get_df
+
+Example
+*******
+
+.. code-block:: python
+
+    >>> eval1 = Evaluation(pd.DataFrame([{"a": 1, "b": 2}, {"a": 4, "b": 5}]), eval_id=1)
+    >>> df = eval1.get_df() # extract Pandas dataframe
+    >>> df
+        a   b
+    0   1   2
+    1   4   5
+    >>> df["c"] = [3, 6] # add a new column
+    >>> eval2 = Evaluation(df, eval_id=eval1.get_eval_id()) # create new evaluation
+    >>> eval2.get_df()
+        a   b   c
+    0   1   2   3
+    1   4   5   6
\ No newline at end of file
diff --git a/docs/topics/visualizers.rst b/docs/topics/visualizers.rst
index c4006ef..7c65188 100644
--- a/docs/topics/visualizers.rst
+++ b/docs/topics/visualizers.rst
@@ -16,7 +16,7 @@ You can choose to show the version info of each test result by setting the
 the display of the version info that is provided by Hydra.
 
 
-Displaying the Visualizaton
+Displaying the Visualization
 ============================
 After instantiating the desired visualizer, you can call the 
 ``get_visualization()`` method to return an object that can be displayed in an 
diff --git a/ftpvl/evaluation.py b/ftpvl/evaluation.py
index 7ff0a61..9611cb8 100644
--- a/ftpvl/evaluation.py
+++ b/ftpvl/evaluation.py
@@ -34,6 +34,12 @@ class Evaluation():
         Returns the ID number of the evaluation if specified, otherwise None
         """
         return self._eval_id
+    
+    def get_copy(self) -> 'Evaluation':
+        """
+        Returns a deep copy of the Evaluation instance
+        """
+        return Evaluation(self.get_df(), self.get_eval_id())
 
     def process(self, pipeline: List['Processor']) -> 'Evaluation':
         """
@@ -47,3 +53,37 @@ class Evaluation():
             an Evaluation instance that was processed by the pipeline
         """
         return reduce(lambda r, p: p.process(r), pipeline, self)
+
+    def __add__(self, other: 'Evaluation') -> 'Evaluation':
+        """
+        Magic method for concatenating two Evaluations, returning an Evaluation
+        with dataframe (self + other).
+
+        Args
+        ------
+            other: the other Evaluation to concatenate
+        
+        Returns:
+            a new Evaluation that consists of the two Evaluations concatenated
+        """
+        if not isinstance(other, Evaluation):
+            raise TypeError(f"can only concatenate Evaluation (not {type(other).__name__}) to Evaluation")
+        new_df = pd.concat([self.get_df(), other.get_df()], ignore_index=True)
+        return Evaluation(new_df)
+
+    def __radd__(self, other: 'Evaluation') -> 'Evaluation':
+        """
+        Magic method for reverse concatenating two Evaluations, returning an
+        Evaluation with dataframe (other + self).
+
+        Args
+        ------
+            other: the other Evaluation to concatenate
+        
+        Returns:
+            a new Evaluation that consists of the two Evaluations concatenated
+        """
+        # handle default start value of sum() is `0`
+        if other == 0:
+            return self.get_copy()
+        return self.__add__(other)
\ No newline at end of file
diff --git a/ftpvl/processors.py b/ftpvl/processors.py
index eb96e3e..c1c5b97 100644
--- a/ftpvl/processors.py
+++ b/ftpvl/processors.py
@@ -1,12 +1,21 @@
 """ Processors transform Evaluations to be more useful when visualized. """
 import math
 from typing import Any, Callable, Dict, List, Union
+from enum import Enum
 
 import numpy as np
 import pandas as pd
 from ftpvl.evaluation import Evaluation
 from scipy import stats
 
+class Direction(Enum):
+    """
+    Represents the optimization direction for certain test metrics. For example,
+    runtime is usually minimized, while frequency is maximized.
+    """
+    MINIMIZE = -1
+    MAXIMIZE = 1
+
 
 class Processor:
     """
@@ -141,19 +150,22 @@ class AddNormalizedColumn(Processor):
     output_col_name : str
         the column to write the normalized values to
 
-    direction : int (either 1 or -1)
-        the direction to find the best value to normalize against. If 1, then
-        the best value is the max and all other values are compared to it. If -1,
-        the best value is the min and all other values are compared to it.
+    direction : Direction
+        specifies how to find the 'best' value to normalize against. By default
+        MAXIMIZE, all values will be compared to the max value of the input
+        column.
     """
 
-    def __init__(self, groupby: str, input_col_name: str, output_col_name: str, direction: int = 1):
-
+    def __init__(
+        self,
+        groupby: str,
+        input_col_name: str,
+        output_col_name: str,
+        direction: Direction = Direction.MAXIMIZE
+    ):
         self._groupby = groupby
         self._input_col_name = input_col_name
         self._output_col_name = output_col_name
-        
-        assert direction in [1, -1], "direction must be either 1 or -1"
         self._direction = direction
 
     def _normalize(self, input_df: pd.DataFrame):
@@ -161,7 +173,7 @@ class AddNormalizedColumn(Processor):
         Given a dataframe, find the max value of the input col name and
         create a new column with the normalized value of each row
         """
-        if self._direction == 1:
+        if self._direction == Direction.MAXIMIZE:
             max_val = input_df[self._input_col_name].max()
         else:
             max_val = input_df[self._input_col_name].min()
@@ -284,11 +296,10 @@ class NormalizeAround(Processor):
 
     Parameters
     ----------
-    normalize_direction : dict
-        a dictionary mapping column names to 1 or -1. If a value is
-        optimized when smaller, set the negation to 1. If it is optimized
-        when larger, set the negation to -1. If there is no entry,
-        normalization is skipped.
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
 
     group_by : str
         the column name used to group results before finding the baseline
@@ -303,16 +314,25 @@ class NormalizeAround(Processor):
         the value of the baseline result at idx_name
     """
 
-    def __init__(self, normalize_direction: dict, group_by: str, idx_name: str, idx_value: str):
+    def __init__(
+        self,
+        normalize_direction: Dict[str, Direction],
+        group_by: str,
+        idx_name: str,
+        idx_value: str
+    ):
         self._groupby = group_by
         self._idx_name = idx_name
         self._idx_value = idx_value
 
         self._column_names = []
         self._column_negations = []
-        for name, negation in normalize_direction.items():
+        for name, direction in normalize_direction.items():
             self._column_names.append(name)
-            self._column_negations.append(negation)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(1)
+            else:
+                self._column_negations.append(-1)
 
     def _normalize_around(self, input_df):
         """
@@ -357,19 +377,22 @@ class Normalize(Processor):
 
     Parameters
     ----------
-    normalize_direction : dict
-        a dictionary mapping column names to 1 or -1. If a value is optimized
-        when smaller, set the negation to 1. If it is optimized when larger, set
-        the negation to -1. If there is no entry, normalization is skipped.
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
     """
 
-    def __init__(self, normalize_direction: dict):
+    def __init__(self, normalize_direction: Dict[str, Direction]):
 
         self._column_names = []
         self._column_negations = []
-        for name, negation in normalize_direction.items():
+        for name, direction in normalize_direction.items():
             self._column_names.append(name)
-            self._column_negations.append(negation)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(1)
+            else:
+                self._column_negations.append(-1)
 
     def _normalize(self, input_df):
         """
@@ -543,3 +566,82 @@ class GeomeanAggregate(Aggregate):
             x = x.dropna()
             return stats.gmean(x) if not x.empty else math.nan
         super().__init__(geomean)
+
+
+class CompareToFirst(Processor):
+    """
+    Processor that compares numeric rows in an evaluation to the first row by
+    adding columns that specify the relative difference between the first row
+    and all other rows.
+
+    You can specify the direction that improvements should be outputted. For
+    example, a change from `100` to `50` may be a 2x change if the objective is
+    minimization, while it may be a 0.5x change if the objective is
+    maximization.
+
+    Parameters
+    ----------
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
+    suffix : str
+        the suffix to use when creating new columns that contain the relative
+        comparison to the first row, by default ".relative"
+    
+    Examples
+    --------
+    >>> a = Evaluation(pd.DataFrame(
+    ... data=[
+    ...     {"x": 1, "y": 8},
+    ...     {"x": 4, "y": 8}
+    ... ]))
+    >>> direction = {"x": Direction.MAXIMIZE, "y": Direction.MAXIMIZE}
+    >>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
+        x   x.diff  y   y.diff
+    0   1   1.00    8   1.0
+    1   4   4.00    8   1.0
+
+    >>> a = Evaluation(pd.DataFrame(
+    ... data=[
+    ...     {"x": 1, "y": 8},
+    ...     {"x": 4, "y": 8}
+    ... ]))
+    >>> direction = {"x": Direction.MINIMIZE, "y": Direction.MINIMIZE}
+    >>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
+        x   x.diff  y   y.diff
+    0   1   1.00    8   1.0
+    1   4   0.25    8   1.0
+    """
+
+    def __init__(self, normalize_direction: Dict[str, Direction], suffix: str = ".relative"):
+        self._column_names = []
+        self._column_negations = []
+        for name, direction in normalize_direction.items():
+            self._column_names.append(name)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(-1)
+            else:
+                self._column_negations.append(1)
+
+        self._suffix = suffix
+
+    def _compare_to_first(self, input_df):
+        """
+        Given a dataframe, sets the first row as the baseline and compares all
+        rows to the first row, outputting the relative difference between each
+        value as a new column.
+        """
+        new_cols = []
+        for col in self._column_names:
+            new_cols.extend([col, col + self._suffix])
+
+        base = input_df.loc[0, self._column_names]
+        baseline_ratio = (input_df[self._column_names] / base) ** self._column_negations
+        renamed_ratio = baseline_ratio.rename(lambda col: col + self._suffix, axis=1)
+        return pd.concat([input_df, renamed_ratio], axis=1)[new_cols]
+
+    def process(self, input_eval: Evaluation) -> Evaluation:
+        input_df = input_eval.get_df()
+        new_df = self._compare_to_first(input_df)
+        return Evaluation(new_df, input_eval.get_eval_id())
diff --git a/requirements.txt b/requirements.txt
index 079aa6b..3bbd156 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -7,6 +7,5 @@ scipy
 pylint
 pytest
 coverage
-black
 sphinx
 sphinx-rtd-theme
\ No newline at end of file
diff --git a/setup.py b/setup.py
index a23faf3..7c657f5 100644
--- a/setup.py
+++ b/setup.py
@@ -18,7 +18,7 @@ URL = 'https://github.com/TypingKoala/FPGA-Tool-Performance-Visualization-Librar
 EMAIL = 'me@johnnybui.com'
 AUTHOR = 'Johnny Bui'
 REQUIRES_PYTHON = '>=3.6.0'
-VERSION = '0.1.6'
+VERSION = '0.2.0'
 
 # What packages are required for this module to be executed?
 REQUIRED = [


Test patch:
diff --git a/tests/test_evaluation.py b/tests/test_evaluation.py
index 60af16b..3bd41ec 100644
--- a/tests/test_evaluation.py
+++ b/tests/test_evaluation.py
@@ -6,6 +6,7 @@ import unittest
 import pandas as pd
 from ftpvl.evaluation import Evaluation
 from ftpvl.processors import MinusOne
+from pandas.testing import assert_frame_equal
 
 
 class TestEvaluation(unittest.TestCase):
@@ -15,8 +16,13 @@ class TestEvaluation(unittest.TestCase):
         get_df()
             defensive copying
         get_eval_num()
+        get_copy()
         process(List[Processor])
             0, 1, 1+ processors
+        __add__()
+            direct add
+            reverse add
+            sum
     """
 
     def test_evaluation_get_df_equality(self):
@@ -53,6 +59,23 @@ class TestEvaluation(unittest.TestCase):
         result = Evaluation(df, eval_id=0)
         assert result.get_eval_id() == 0
 
+    def test_evaluation_get_copy(self):
+        """
+        get_copy() should return a deep copy of Evaluation
+        """
+        df = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        eval1 = Evaluation(df, eval_id=1000)
+
+        result = eval1.get_copy()
+        assert result.get_eval_id() == 1000
+        assert_frame_equal(result.get_df(), df)
+
+        # change original
+        df.iloc[0]["a"] = 0
+        
+        # assert that copy has not changed
+        assert result.get_df().iloc[0]["a"] == 1
+
     def test_evaluation_process_empty(self):
         """
         Calling process() with an empty list of processors should return the
@@ -98,3 +121,68 @@ class TestEvaluation(unittest.TestCase):
 
         # check if output eval has correct values
         self.assertTrue(result.equals(expected))
+
+    def test_evaluation_add(self):
+        """
+        Using the + magic method should return a new Evaluation that consists
+        of the concatenated Evaluations.
+        """
+        df1 = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        result1 = Evaluation(df1)
+
+        df2 = pd.DataFrame([{"a": 5, "b": 6}, {"a": 7, "b": 8}])
+        result2 = Evaluation(df2)
+
+        sum_result = result1 + result2
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5, "b": 6},
+            {"a": 7, "b": 8}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
+
+    def test_evaluation_add_different_columns(self):
+        """
+        Using the + magic method should return a new Evaluation that consists
+        of the concatenated Evaluations, even if columns don't match
+        """
+        df1 = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        result1 = Evaluation(df1)
+
+        df2 = pd.DataFrame([{"a": 5}, {"a": 7}])
+        result2 = Evaluation(df2)
+
+        sum_result = result1 + result2
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5},
+            {"a": 7}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
+
+    def test_evaluation_add_multiple(self):
+        """
+        Using the sum() built-in function should return a new Evaluation
+        that consists of concatenated Evaluations
+        """
+        eval1 = Evaluation(pd.DataFrame([{"a": 1, "b": 2}]))
+        eval2 = Evaluation(pd.DataFrame([{"a": 3, "b": 4}]))
+        eval3 = Evaluation(pd.DataFrame([{"a": 5, "b": 6}]))
+        eval4 = Evaluation(pd.DataFrame([{"a": 7, "b": 8}]))
+
+        sum_result = sum([eval1, eval2, eval3, eval4])
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5, "b": 6},
+            {"a": 7, "b": 8}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
diff --git a/tests/test_processor.py b/tests/test_processor.py
index ca7250d..c1bf247 100644
--- a/tests/test_processor.py
+++ b/tests/test_processor.py
@@ -26,6 +26,7 @@ class TestProcessor:
     FilterByIndex()
     Aggregate()
     GeomeanAggregate()
+    CompareToFirst()
     """
 
     def test_minusone(self):
@@ -191,7 +192,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         # direction is -1 => normalize around minimum
-        pipeline = [AddNormalizedColumn("group", "value", "normalized", -1)]
+        pipeline = [AddNormalizedColumn("group", "value", "normalized", Direction.MINIMIZE)]
         result = eval1.process(pipeline)
         expected = pd.DataFrame(
             [
@@ -330,7 +331,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": 1
+            "value": Direction.MINIMIZE
         }
         pipeline = [NormalizeAround(
             normalize_direction,
@@ -376,7 +377,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": -1
+            "value": Direction.MAXIMIZE
         }
         pipeline = [NormalizeAround(
             normalize_direction,
@@ -415,7 +416,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": 1
+            "value": Direction.MINIMIZE
         }
         pipeline = [Normalize(normalize_direction)]
 
@@ -638,4 +639,102 @@ class TestProcessor:
             ]
         )
         assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst(self):
+        """ Test if CompareToFirst works with default params """
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MAXIMIZE,
+            "b": Direction.MAXIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction)]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.relative": 1.0/1, "b": 5, "b.relative": 5.0/5},
+                {"a": 2, "a.relative": 2.0/1, "b": 4, "b.relative": 4.0/5},
+                {"a": 3, "a.relative": 3.0/1, "b": 3, "b.relative": 3.0/5},
+                {"a": 4, "a.relative": 4.0/1, "b": 2, "b.relative": 2.0/5},
+                {"a": 5, "a.relative": 5.0/1, "b": 1, "b.relative": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst_dir_subset(self):
+        """ Test if CompareToFirst works with different direction and subset"""
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MINIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction)]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.relative": 1.0/1},
+                {"a": 2, "a.relative": 1.0/2},
+                {"a": 3, "a.relative": 1.0/3},
+                {"a": 4, "a.relative": 1.0/4},
+                {"a": 5, "a.relative": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst_suffix(self):
+        """ Test if CompareToFirst works with different suffix """
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MAXIMIZE,
+            "b": Direction.MAXIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction, suffix=".diff")]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.diff": 1.0/1, "b": 5, "b.diff": 5.0/5},
+                {"a": 2, "a.diff": 2.0/1, "b": 4, "b.diff": 4.0/5},
+                {"a": 3, "a.diff": 3.0/1, "b": 3, "b.diff": 3.0/5},
+                {"a": 4, "a.diff": 4.0/1, "b": 2, "b.diff": 2.0/5},
+                {"a": 5, "a.diff": 5.0/1, "b": 1, "b.diff": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
         assert eval1.get_eval_id() == 20
\ No newline at end of file