Repository: SymbiFlow/FPGA-Tool-Performance-Visualization-Library

Issue:
Add support for hydrafetcher getting by absolute eval id
Hydrafetcher currently supports getting evaluations based on how recently they occurred. To ensure that the future dashboard is repeatable, we must be able to specify exact evaluation IDs to use when fetching.

Functionality must be implemented to support this type of use case. An example:

```python
HydraFetcher(
    eval_id=973, # not the same as eval_num
    project="dusty",
    jobset="fpga-tool-perf",
    ...
)
```

Discussion:

Patch:
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
new file mode 100644
index 0000000..877eaf1
--- /dev/null
+++ b/.github/workflows/python-package.yml
@@ -0,0 +1,37 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: Python package
+
+on:
+  push:
+    branches: [ master ]
+  pull_request:
+    branches: [ master ]
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.6, 3.7, 3.8]
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pylint pytest
+        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+    - name: Lint with flake8
+      run: |
+        # stop the build if there are Python syntax errors or undefined names
+        pylint -E ftpvl/
+    - name: Test with pytest
+      run: |
+        pytest
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
new file mode 100644
index 0000000..f9fedba
--- /dev/null
+++ b/.github/workflows/python-publish.yml
@@ -0,0 +1,30 @@
+# This workflows will upload a Python Package using Twine when a release is created
+# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
+
+name: Upload Python Package
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  deploy:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.x'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install setuptools wheel twine
+    - name: Build and publish
+      env:
+        TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
+        TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+      run: |
+        python setup.py upload
diff --git a/README.md b/README.md
index 78773c8..7d84811 100644
--- a/README.md
+++ b/README.md
@@ -12,8 +12,9 @@ Take a look at the notebooks below to demonstrate the functionality of FTPVL.
 
 1. [Using `HydraFetcher` and Processors](https://colab.research.google.com/drive/1BIQ-iulDFpzcve7lGJPwLePJ5ETBJ6Ut?usp=sharing)
 2. [Styling tables with `SingleTableVisualizer`](https://colab.research.google.com/drive/1u3EnmIYnTBk-LXZhqNHt_h4aMuq-_cWq?usp=sharing)
-3. [Comparing two different Evaluations](https://colab.research.google.com/drive/1I7InmA6210vIIwdQ7TGHE6aF_WwIm1dM?usp=sharing)
+3. [Comparing two Evaluations using the internal dataframe](https://colab.research.google.com/drive/1I7InmA6210vIIwdQ7TGHE6aF_WwIm1dM?usp=sharing)
 4. [Filtering and Aggregating an Evaluation](https://colab.research.google.com/drive/1DDwlQFS81RGLL-q8DsgICF-HOC5ir6oS?usp=sharing)
+5. [Comparing multiple Evaluations](https://colab.research.google.com/drive/1kSF3bEjG_c6bLh9PLus9GPk5aLrtYhly?usp=sharing)
 
 ## Documentation
 Extensive documentation, including a *Getting Started* guide, is available on
@@ -41,11 +42,14 @@ make html
 * `pylint`: for linting ([website](https://www.pylint.org/))
 * `pytest`: testrunner ([website](https://docs.pytest.org/en/stable/))
 * `coverage`: for coverage testing ([website](https://coverage.readthedocs.io/))
-* `black`: for auto-formatting ([website](https://github.com/psf/black))
 * `sphinx`: for documentation generation ([website](https://www.sphinx-doc.org/en/master/))
 * `sphinx-rtd-theme`: for documentation generation (theme) ([website](https://github.com/readthedocs/sphinx_rtd_theme))
 
 ## Changes
+### 0.2.0
+* Added evaluation concatenation and `CompareToFirst` processors for relative comparisons between evaluations.
+* Fixed `HydraFetcher` issues when fetching an older evaluation that contains more than one build artifact.
+
 ### 0.1.6
 * Added support for filter and aggregator processors, fixes [#9](https://github.com/SymbiFlow/FPGA-Tool-Performance-Visualization-Library/issues/9)
 
diff --git a/docs/intro/firststeps.rst b/docs/intro/firststeps.rst
index 2af8cfb..5557161 100644
--- a/docs/intro/firststeps.rst
+++ b/docs/intro/firststeps.rst
@@ -230,7 +230,7 @@ Styling
 =======
 
 Now that the Evaluation has been processed, we can add styling so that
-important information stands out in the final visualization. This is achived
+important information stands out in the final visualization. This is achieved
 through a special type of Processor called :ref:`topics-styles`. 
 
 Styles are also run in a processing pipeline, but they always output CSS strings.
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 63231e0..2de1096 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -76,6 +76,8 @@ Processors API
 .. autoclass:: ftpvl.processors.GeomeanAggregate
     :members:
 
+.. autoclass:: ftpvl.processors.CompareToFirst
+
 .. _topics-api-styles:
 
 Styles API
@@ -99,4 +101,21 @@ Visualizers API
     :members:
 
 .. autoclass:: ftpvl.visualizers.SingleTableVisualizer
-    :members:
\ No newline at end of file
+    :members:
+
+Enums
+=====
+Direction
+*********
+.. autoclass:: ftpvl.processors.Direction
+    :members:
+
+    .. autoattribute:: MAXIMIZE
+    
+        Indicates that the corresponding metric is optimized by 
+        *maximizing* the value
+    
+    .. autoattribute:: MINIMIZE
+    
+        Indicates that the corresponding metric is optimized by
+        *minimizing* the value
diff --git a/docs/topics/evaluation.rst b/docs/topics/evaluation.rst
index 0a0c88d..2687637 100644
--- a/docs/topics/evaluation.rst
+++ b/docs/topics/evaluation.rst
@@ -21,6 +21,30 @@ transform the evaluation one-at-a-time.
 Extracting the internal dataframe
 =================================
 Evaluations store the test results internally using a Pandas dataframe. You can
-retrive this dataframe by using the ``get_df()`` method.
+retrieve this dataframe by using the ``get_df()`` method. This can be used to
+manipulate the underlying data without needing to use the built-in FTPVL processors.
 
-.. automethod:: ftpvl.evaluation.Evaluation.get_df
\ No newline at end of file
+.. note:: ``get_df()`` returns a defensive copy of the dataframe, so any 
+    mutations to the returned dataframe will not be reflected in the original
+    Evaluation. Instead, you must instantiate a new ``Evaluation`` by passing
+    in the dataframe and evaluation ID.
+
+.. automethod:: ftpvl.evaluation.Evaluation.get_df
+
+Example
+*******
+
+.. code-block:: python
+
+    >>> eval1 = Evaluation(pd.DataFrame([{"a": 1, "b": 2}, {"a": 4, "b": 5}]), eval_id=1)
+    >>> df = eval1.get_df() # extract Pandas dataframe
+    >>> df
+        a   b
+    0   1   2
+    1   4   5
+    >>> df["c"] = [3, 6] # add a new column
+    >>> eval2 = Evaluation(df, eval_id=eval1.get_eval_id()) # create new evaluation
+    >>> eval2.get_df()
+        a   b   c
+    0   1   2   3
+    1   4   5   6
\ No newline at end of file
diff --git a/docs/topics/visualizers.rst b/docs/topics/visualizers.rst
index c4006ef..7c65188 100644
--- a/docs/topics/visualizers.rst
+++ b/docs/topics/visualizers.rst
@@ -16,7 +16,7 @@ You can choose to show the version info of each test result by setting the
 the display of the version info that is provided by Hydra.
 
 
-Displaying the Visualizaton
+Displaying the Visualization
 ============================
 After instantiating the desired visualizer, you can call the 
 ``get_visualization()`` method to return an object that can be displayed in an 
diff --git a/ftpvl/evaluation.py b/ftpvl/evaluation.py
index 7ff0a61..9611cb8 100644
--- a/ftpvl/evaluation.py
+++ b/ftpvl/evaluation.py
@@ -34,6 +34,12 @@ class Evaluation():
         Returns the ID number of the evaluation if specified, otherwise None
         """
         return self._eval_id
+    
+    def get_copy(self) -> 'Evaluation':
+        """
+        Returns a deep copy of the Evaluation instance
+        """
+        return Evaluation(self.get_df(), self.get_eval_id())
 
     def process(self, pipeline: List['Processor']) -> 'Evaluation':
         """
@@ -47,3 +53,37 @@ class Evaluation():
             an Evaluation instance that was processed by the pipeline
         """
         return reduce(lambda r, p: p.process(r), pipeline, self)
+
+    def __add__(self, other: 'Evaluation') -> 'Evaluation':
+        """
+        Magic method for concatenating two Evaluations, returning an Evaluation
+        with dataframe (self + other).
+
+        Args
+        ------
+            other: the other Evaluation to concatenate
+        
+        Returns:
+            a new Evaluation that consists of the two Evaluations concatenated
+        """
+        if not isinstance(other, Evaluation):
+            raise TypeError(f"can only concatenate Evaluation (not {type(other).__name__}) to Evaluation")
+        new_df = pd.concat([self.get_df(), other.get_df()], ignore_index=True)
+        return Evaluation(new_df)
+
+    def __radd__(self, other: 'Evaluation') -> 'Evaluation':
+        """
+        Magic method for reverse concatenating two Evaluations, returning an
+        Evaluation with dataframe (other + self).
+
+        Args
+        ------
+            other: the other Evaluation to concatenate
+        
+        Returns:
+            a new Evaluation that consists of the two Evaluations concatenated
+        """
+        # handle default start value of sum() is `0`
+        if other == 0:
+            return self.get_copy()
+        return self.__add__(other)
\ No newline at end of file
diff --git a/ftpvl/fetchers.py b/ftpvl/fetchers.py
index 489026a..e70bc9d 100644
--- a/ftpvl/fetchers.py
+++ b/ftpvl/fetchers.py
@@ -18,7 +18,7 @@ class Fetcher:
     """
 
     def __init__(self):
-        self._eval_id = None
+        self._abs_eval_id = None
 
     def _download(self) -> Any:
         """
@@ -38,7 +38,7 @@ class Fetcher:
         """
         data = self._download()
         preprocessed_df = self._preprocess(data)
-        return Evaluation(preprocessed_df, eval_id=self._eval_id)
+        return Evaluation(preprocessed_df, eval_id=self._abs_eval_id)
 
 
 class HydraFetcher(Fetcher):
@@ -48,13 +48,19 @@ class HydraFetcher(Fetcher):
 
     Parameters
     ----------
-    project : str, optional
+    project : str
         The project name to use when fetching from Hydra
-    jobset : str, optional
+    jobset : str
         The jobset name to use when fetching from Hydra
     eval_num : int, optional
-        A non-negative integer for the evaluation number to download,
-        with `0` being the latest evaluation, by default 0
+        An integer that specifies the evaluation to download. Functionality
+        differs depending on whether `absolute_eval_num` is True, by default 0
+    absolute_eval_num : bool, optional
+        Flag that specifies if the eval_num is an absolute identifier instead of
+        a relative identifier. If True, the fetcher will find an evaluation
+        with the exact ID in `eval_num`. If False, `eval_num` should be a
+        non-negative integer with `0` being the latest evaluation and `1` being
+        the second latest evaluation, etc. By default False.
     mapping : dict, optional
         A dictionary mapping input column names to output
         column names, if needed for remapping, by default None
@@ -68,16 +74,80 @@ class HydraFetcher(Fetcher):
         project: str,
         jobset: str,
         eval_num: int = 0,
+        absolute_eval_num: bool = False,
         mapping: dict = None,
         hydra_clock_names: list = None
     ) -> None:
-        super().__init__()
+        super().__init__() # inits self._abs_eval_id
         self.project = project
         self.jobset = jobset
         self.eval_num = eval_num
+        self.absolute_eval_num = absolute_eval_num
         self.mapping = mapping
         self.hydra_clock_names = hydra_clock_names
 
+    def _get_builds(self, eval_num: int, params: str = "") -> List[int]:
+        """
+        Recursive function that returns a list of build numbers given an eval_num
+        and whether it is an absolute eval num.
+
+        Parameters
+        ----------
+        eval_num : int
+            An integer that specifies the evaluation to download. Functionality
+            differs depending on whether `absolute_eval_num` is True
+        params : str
+            A string of query parameters used when fetching the evaluations.
+            Most commonly used for pagination. By default, "".
+
+        Returns
+        -------
+        List[int]
+            A list of builds that correspond with the eval_num
+
+        Raises
+        ------
+        ConnectionError
+            Raised if the HTTP request to get the evaluations fails.
+        IndexError
+            Raised if the relative eval_num is not valid.
+        ValueError
+            Raised if the eval_num has no associated builds.
+        """
+        resp = requests.get(
+            f"https://hydra.vtr.tools/jobset/{self.project}/{self.jobset}/evals{params}",
+            headers={"Content-Type": "application/json"},
+        )
+        if resp.status_code != 200:
+            raise ConnectionError("Unable to get evals from server.")
+        evals_json = resp.json()
+
+        if self.absolute_eval_num:
+            self._abs_eval_id = eval_num # set absolute id for eval
+            for eval_data in evals_json["evals"]:
+                if eval_data["id"] == eval_num:
+                    return eval_data["builds"]
+            if "next" in evals_json:
+                return self._get_builds(
+                    eval_num,
+                    evals_json["next"] # query param for next page
+                )
+            else: # if couldn't find ID and there is no next page
+                raise ValueError(f"Unable to find absolute eval_num {eval_num}")
+
+        else: # if not absolute eval number
+            if eval_num >= len(evals_json["evals"]):
+                # check if there is a next page
+                if "next" in evals_json:
+                    return self._get_builds(
+                        eval_num - len(evals_json["evals"]),
+                        evals_json["next"] # query param for next page
+                    )
+                else:
+                    raise IndexError(f"Unable to find relative eval_num {self.eval_num}")
+            self._abs_eval_id = evals_json["evals"][eval_num]["id"] # set absolute id for eval
+            return evals_json["evals"][eval_num]["builds"]
+
     def _download(self) -> List[Dict]:
         """
         Fetches data from Hydra, returning a list of decoded meta.json dicts
@@ -103,16 +173,7 @@ class HydraFetcher(Fetcher):
             Raised if all builds in a given eval failed.
         """
         # get build numbers from eval_num
-        resp = requests.get(
-            f"https://hydra.vtr.tools/jobset/{self.project}/{self.jobset}/evals",
-            headers={"Content-Type": "application/json"},
-        )
-        if resp.status_code != 200:
-            raise ConnectionError("Unable to get evals from server.")
-        evals_json = resp.json()
-        if self.eval_num >= len(evals_json["evals"]):
-            raise IndexError(f"Invalid eval_num: {self.eval_num}")
-        build_nums = evals_json["evals"][self.eval_num]["builds"]
+        build_nums = self._get_builds(self.eval_num)
 
         # fetch build info and download 'meta.json'
         data = []
@@ -165,7 +226,6 @@ class HydraFetcher(Fetcher):
         if len(data) == 0:
             raise ValueError(f"Unable to get any successful builds from eval_num {self.eval_num}.")
 
-        self._eval_id = evals_json["evals"][self.eval_num]["id"]
         return data
 
     def _preprocess(self, data: List[Dict]) -> pd.DataFrame:
diff --git a/ftpvl/processors.py b/ftpvl/processors.py
index eb96e3e..c1c5b97 100644
--- a/ftpvl/processors.py
+++ b/ftpvl/processors.py
@@ -1,12 +1,21 @@
 """ Processors transform Evaluations to be more useful when visualized. """
 import math
 from typing import Any, Callable, Dict, List, Union
+from enum import Enum
 
 import numpy as np
 import pandas as pd
 from ftpvl.evaluation import Evaluation
 from scipy import stats
 
+class Direction(Enum):
+    """
+    Represents the optimization direction for certain test metrics. For example,
+    runtime is usually minimized, while frequency is maximized.
+    """
+    MINIMIZE = -1
+    MAXIMIZE = 1
+
 
 class Processor:
     """
@@ -141,19 +150,22 @@ class AddNormalizedColumn(Processor):
     output_col_name : str
         the column to write the normalized values to
 
-    direction : int (either 1 or -1)
-        the direction to find the best value to normalize against. If 1, then
-        the best value is the max and all other values are compared to it. If -1,
-        the best value is the min and all other values are compared to it.
+    direction : Direction
+        specifies how to find the 'best' value to normalize against. By default
+        MAXIMIZE, all values will be compared to the max value of the input
+        column.
     """
 
-    def __init__(self, groupby: str, input_col_name: str, output_col_name: str, direction: int = 1):
-
+    def __init__(
+        self,
+        groupby: str,
+        input_col_name: str,
+        output_col_name: str,
+        direction: Direction = Direction.MAXIMIZE
+    ):
         self._groupby = groupby
         self._input_col_name = input_col_name
         self._output_col_name = output_col_name
-        
-        assert direction in [1, -1], "direction must be either 1 or -1"
         self._direction = direction
 
     def _normalize(self, input_df: pd.DataFrame):
@@ -161,7 +173,7 @@ class AddNormalizedColumn(Processor):
         Given a dataframe, find the max value of the input col name and
         create a new column with the normalized value of each row
         """
-        if self._direction == 1:
+        if self._direction == Direction.MAXIMIZE:
             max_val = input_df[self._input_col_name].max()
         else:
             max_val = input_df[self._input_col_name].min()
@@ -284,11 +296,10 @@ class NormalizeAround(Processor):
 
     Parameters
     ----------
-    normalize_direction : dict
-        a dictionary mapping column names to 1 or -1. If a value is
-        optimized when smaller, set the negation to 1. If it is optimized
-        when larger, set the negation to -1. If there is no entry,
-        normalization is skipped.
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
 
     group_by : str
         the column name used to group results before finding the baseline
@@ -303,16 +314,25 @@ class NormalizeAround(Processor):
         the value of the baseline result at idx_name
     """
 
-    def __init__(self, normalize_direction: dict, group_by: str, idx_name: str, idx_value: str):
+    def __init__(
+        self,
+        normalize_direction: Dict[str, Direction],
+        group_by: str,
+        idx_name: str,
+        idx_value: str
+    ):
         self._groupby = group_by
         self._idx_name = idx_name
         self._idx_value = idx_value
 
         self._column_names = []
         self._column_negations = []
-        for name, negation in normalize_direction.items():
+        for name, direction in normalize_direction.items():
             self._column_names.append(name)
-            self._column_negations.append(negation)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(1)
+            else:
+                self._column_negations.append(-1)
 
     def _normalize_around(self, input_df):
         """
@@ -357,19 +377,22 @@ class Normalize(Processor):
 
     Parameters
     ----------
-    normalize_direction : dict
-        a dictionary mapping column names to 1 or -1. If a value is optimized
-        when smaller, set the negation to 1. If it is optimized when larger, set
-        the negation to -1. If there is no entry, normalization is skipped.
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
     """
 
-    def __init__(self, normalize_direction: dict):
+    def __init__(self, normalize_direction: Dict[str, Direction]):
 
         self._column_names = []
         self._column_negations = []
-        for name, negation in normalize_direction.items():
+        for name, direction in normalize_direction.items():
             self._column_names.append(name)
-            self._column_negations.append(negation)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(1)
+            else:
+                self._column_negations.append(-1)
 
     def _normalize(self, input_df):
         """
@@ -543,3 +566,82 @@ class GeomeanAggregate(Aggregate):
             x = x.dropna()
             return stats.gmean(x) if not x.empty else math.nan
         super().__init__(geomean)
+
+
+class CompareToFirst(Processor):
+    """
+    Processor that compares numeric rows in an evaluation to the first row by
+    adding columns that specify the relative difference between the first row
+    and all other rows.
+
+    You can specify the direction that improvements should be outputted. For
+    example, a change from `100` to `50` may be a 2x change if the objective is
+    minimization, while it may be a 0.5x change if the objective is
+    maximization.
+
+    Parameters
+    ----------
+    normalize_direction : Dict[str, Direction]
+        a dictionary mapping column names to the optimization direction of
+        the column. Used to determine if increases or decreases to baseline are
+        perceived to be 'better'.
+    suffix : str
+        the suffix to use when creating new columns that contain the relative
+        comparison to the first row, by default ".relative"
+    
+    Examples
+    --------
+    >>> a = Evaluation(pd.DataFrame(
+    ... data=[
+    ...     {"x": 1, "y": 8},
+    ...     {"x": 4, "y": 8}
+    ... ]))
+    >>> direction = {"x": Direction.MAXIMIZE, "y": Direction.MAXIMIZE}
+    >>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
+        x   x.diff  y   y.diff
+    0   1   1.00    8   1.0
+    1   4   4.00    8   1.0
+
+    >>> a = Evaluation(pd.DataFrame(
+    ... data=[
+    ...     {"x": 1, "y": 8},
+    ...     {"x": 4, "y": 8}
+    ... ]))
+    >>> direction = {"x": Direction.MINIMIZE, "y": Direction.MINIMIZE}
+    >>> a.process([CompareToFirst(direction, suffix=".diff")).get_df()
+        x   x.diff  y   y.diff
+    0   1   1.00    8   1.0
+    1   4   0.25    8   1.0
+    """
+
+    def __init__(self, normalize_direction: Dict[str, Direction], suffix: str = ".relative"):
+        self._column_names = []
+        self._column_negations = []
+        for name, direction in normalize_direction.items():
+            self._column_names.append(name)
+            if direction == Direction.MINIMIZE:
+                self._column_negations.append(-1)
+            else:
+                self._column_negations.append(1)
+
+        self._suffix = suffix
+
+    def _compare_to_first(self, input_df):
+        """
+        Given a dataframe, sets the first row as the baseline and compares all
+        rows to the first row, outputting the relative difference between each
+        value as a new column.
+        """
+        new_cols = []
+        for col in self._column_names:
+            new_cols.extend([col, col + self._suffix])
+
+        base = input_df.loc[0, self._column_names]
+        baseline_ratio = (input_df[self._column_names] / base) ** self._column_negations
+        renamed_ratio = baseline_ratio.rename(lambda col: col + self._suffix, axis=1)
+        return pd.concat([input_df, renamed_ratio], axis=1)[new_cols]
+
+    def process(self, input_eval: Evaluation) -> Evaluation:
+        input_df = input_eval.get_df()
+        new_df = self._compare_to_first(input_df)
+        return Evaluation(new_df, input_eval.get_eval_id())
diff --git a/requirements.txt b/requirements.txt
index 079aa6b..3bbd156 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -7,6 +7,5 @@ scipy
 pylint
 pytest
 coverage
-black
 sphinx
 sphinx-rtd-theme
\ No newline at end of file
diff --git a/setup.py b/setup.py
index a23faf3..7c657f5 100644
--- a/setup.py
+++ b/setup.py
@@ -18,7 +18,7 @@ URL = 'https://github.com/TypingKoala/FPGA-Tool-Performance-Visualization-Librar
 EMAIL = 'me@johnnybui.com'
 AUTHOR = 'Johnny Bui'
 REQUIRES_PYTHON = '>=3.6.0'
-VERSION = '0.1.6'
+VERSION = '0.2.0'
 
 # What packages are required for this module to be executed?
 REQUIRED = [


Test patch:
diff --git a/tests/sample_data/evals.small.2.json b/tests/sample_data/evals.small.2.json
new file mode 100644
index 0000000..31a9228
--- /dev/null
+++ b/tests/sample_data/evals.small.2.json
@@ -0,0 +1,34 @@
+{
+    "evals": [
+        {
+            "builds": [
+                12,
+                13,
+                14,
+                15
+            ],
+            "id": 4
+        },
+        {
+            "builds": [
+                16,
+                17,
+                18,
+                19
+            ],
+            "id": 5
+        },
+        {
+            "builds": [
+                20,
+                21,
+                22,
+                23
+            ],
+            "id": 6
+        }
+    ],
+    "previous": "?page=1",
+    "first": "?page=1",
+    "last": "?page=2"
+}
\ No newline at end of file
diff --git a/tests/sample_data/evals.small.json b/tests/sample_data/evals.small.json
index 24ffffd..d12cb8f 100644
--- a/tests/sample_data/evals.small.json
+++ b/tests/sample_data/evals.small.json
@@ -27,5 +27,8 @@
             ],
             "id": 3
         }
-    ]
+    ],
+    "next": "?page=2",
+    "last": "?page=2",
+    "first": "?page=1"
 }
\ No newline at end of file
diff --git a/tests/test_evaluation.py b/tests/test_evaluation.py
index 60af16b..3bd41ec 100644
--- a/tests/test_evaluation.py
+++ b/tests/test_evaluation.py
@@ -6,6 +6,7 @@ import unittest
 import pandas as pd
 from ftpvl.evaluation import Evaluation
 from ftpvl.processors import MinusOne
+from pandas.testing import assert_frame_equal
 
 
 class TestEvaluation(unittest.TestCase):
@@ -15,8 +16,13 @@ class TestEvaluation(unittest.TestCase):
         get_df()
             defensive copying
         get_eval_num()
+        get_copy()
         process(List[Processor])
             0, 1, 1+ processors
+        __add__()
+            direct add
+            reverse add
+            sum
     """
 
     def test_evaluation_get_df_equality(self):
@@ -53,6 +59,23 @@ class TestEvaluation(unittest.TestCase):
         result = Evaluation(df, eval_id=0)
         assert result.get_eval_id() == 0
 
+    def test_evaluation_get_copy(self):
+        """
+        get_copy() should return a deep copy of Evaluation
+        """
+        df = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        eval1 = Evaluation(df, eval_id=1000)
+
+        result = eval1.get_copy()
+        assert result.get_eval_id() == 1000
+        assert_frame_equal(result.get_df(), df)
+
+        # change original
+        df.iloc[0]["a"] = 0
+        
+        # assert that copy has not changed
+        assert result.get_df().iloc[0]["a"] == 1
+
     def test_evaluation_process_empty(self):
         """
         Calling process() with an empty list of processors should return the
@@ -98,3 +121,68 @@ class TestEvaluation(unittest.TestCase):
 
         # check if output eval has correct values
         self.assertTrue(result.equals(expected))
+
+    def test_evaluation_add(self):
+        """
+        Using the + magic method should return a new Evaluation that consists
+        of the concatenated Evaluations.
+        """
+        df1 = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        result1 = Evaluation(df1)
+
+        df2 = pd.DataFrame([{"a": 5, "b": 6}, {"a": 7, "b": 8}])
+        result2 = Evaluation(df2)
+
+        sum_result = result1 + result2
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5, "b": 6},
+            {"a": 7, "b": 8}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
+
+    def test_evaluation_add_different_columns(self):
+        """
+        Using the + magic method should return a new Evaluation that consists
+        of the concatenated Evaluations, even if columns don't match
+        """
+        df1 = pd.DataFrame([{"a": 1, "b": 2}, {"a": 3, "b": 4}])
+        result1 = Evaluation(df1)
+
+        df2 = pd.DataFrame([{"a": 5}, {"a": 7}])
+        result2 = Evaluation(df2)
+
+        sum_result = result1 + result2
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5},
+            {"a": 7}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
+
+    def test_evaluation_add_multiple(self):
+        """
+        Using the sum() built-in function should return a new Evaluation
+        that consists of concatenated Evaluations
+        """
+        eval1 = Evaluation(pd.DataFrame([{"a": 1, "b": 2}]))
+        eval2 = Evaluation(pd.DataFrame([{"a": 3, "b": 4}]))
+        eval3 = Evaluation(pd.DataFrame([{"a": 5, "b": 6}]))
+        eval4 = Evaluation(pd.DataFrame([{"a": 7, "b": 8}]))
+
+        sum_result = sum([eval1, eval2, eval3, eval4])
+        expected = pd.DataFrame([
+            {"a": 1, "b": 2},
+            {"a": 3, "b": 4},
+            {"a": 5, "b": 6},
+            {"a": 7, "b": 8}
+        ])
+
+        assert_frame_equal(sum_result.get_df(), expected)
+        assert sum_result.get_eval_id() is None
diff --git a/tests/test_fetcher_large.py b/tests/test_fetcher_large.py
index aa14b01..b116c75 100644
--- a/tests/test_fetcher_large.py
+++ b/tests/test_fetcher_large.py
@@ -2,6 +2,7 @@
 
 """ Tests for Evaluation class """
 import unittest
+import pytest
 import json
 
 import pandas as pd
@@ -60,6 +61,8 @@ class TestHydraFetcherLarge(unittest.TestCase):
         """
         get_evaluation() should return an Evaluation corresponding to the real
         dataset and mapping.
+
+        Tests the use of relative eval_id.
         """
         with requests_mock.Mocker() as m:
             evals_fn = 'tests/sample_data/evals.large.json'
@@ -107,6 +110,78 @@ class TestHydraFetcherLarge(unittest.TestCase):
                     expected_num_rows = len(evals_json_decoded['evals'][eval_num]['builds'])
                     self.assertEqual(len(result.index), expected_num_rows)
 
+    def test_hydrafetcher_get_evaluation_absolute(self):
+        """
+        get_evaluation() should return an Evaluation corresponding to the real
+        dataset and mapping.
+
+        Tests the use of absolute eval_id.
+        """
+        with requests_mock.Mocker() as m:
+            evals_fn = 'tests/sample_data/evals.large.json'
+            evals_url = 'https://hydra.vtr.tools/jobset/dusty/fpga-tool-perf/evals'
+
+            build_fn = 'tests/sample_data/build.large.json'
+
+            meta_fn = 'tests/sample_data/meta.large.json'
+
+            # setup /evals request mock
+            evals_json_decoded = None
+            with open(evals_fn, "r") as f:
+                evals_json_encoded = f.read()
+                evals_json_decoded = json.loads(evals_json_encoded)
+                m.get(evals_url, text=evals_json_encoded)
+            
+            # read sample build response
+            build_json_encoded = None
+            with open(build_fn, "r") as f:
+                build_json_encoded = f.read()
+
+            # setup /build and /meta.json request mock
+            with open(meta_fn, "r") as f:
+                meta_json_encoded = f.read()
+                for eval_obj in evals_json_decoded["evals"]:
+                    for build_num in eval_obj["builds"]:
+                        # setup /build
+                        m.get(f'https://hydra.vtr.tools/build/{build_num}', text=build_json_encoded)
+                        # setup meta.json
+                        url = f'https://hydra.vtr.tools/build/{build_num}/download/5/meta.json'
+                        m.get(url, text=meta_json_encoded)
+
+            # map absolute eval_num to eval_data
+            eval_num_to_data = {}
+            for eval_data in evals_json_decoded["evals"]:
+                eval_num_to_data[eval_data["id"]] = eval_data
+
+            # run tests on different eval_num
+            for eval_num in [eval_data["id"] for eval_data in evals_json_decoded["evals"]]:
+                with self.subTest(eval_num=eval_num):
+                    result = None
+
+                    if len(eval_num_to_data[eval_num]["builds"]) == 0:
+                        with pytest.raises(ValueError): # expected exception
+                            result = HydraFetcher(
+                                project="dusty",
+                                jobset="fpga-tool-perf",
+                                eval_num=eval_num,
+                                absolute_eval_num=True).get_evaluation()
+                        continue # skip the rest of the test
+                    else:
+                        result = HydraFetcher(
+                            project="dusty",
+                            jobset="fpga-tool-perf",
+                            eval_num=eval_num,
+                            absolute_eval_num=True).get_evaluation()
+
+                    df = result.get_df()
+                    
+                    # find expected eval
+                    expected_eval_data = eval_num_to_data[eval_num]
+                    expected_num_rows = len(expected_eval_data["builds"])
+                    assert len(df.index) == expected_num_rows
+
+                    assert result.get_eval_id() == eval_num
+
 
 class TestJSONFetcherLarge(unittest.TestCase):
     """
diff --git a/tests/test_fetcher_small.py b/tests/test_fetcher_small.py
index f97ba85..80d8fdc 100644
--- a/tests/test_fetcher_small.py
+++ b/tests/test_fetcher_small.py
@@ -18,8 +18,10 @@ class TestHydraFetcherSmall(unittest.TestCase):
         __init__(eval_num, mapping_dict, hydra_clock_names)
         get_evaluation()
             different eval_num
+                relative, absolute
             different mapping
                 exclusion, renaming
+            pagination
     """
 
     def test_hydrafetcher_init(self):
@@ -43,10 +45,11 @@ class TestHydraFetcherSmall(unittest.TestCase):
         get_evaluation() should return an Evaluation corresponding to the small
         dataset and the specified eval_num.
 
-        Tests different eval_num functionality.
+        Tests different eval_num functionality along with pagination
         """
         with requests_mock.Mocker() as m:
             evals_fn = 'tests/sample_data/evals.small.json'
+            evals_fn2 = 'tests/sample_data/evals.small.2.json'
             evals_url = 'https://hydra.vtr.tools/jobset/dusty/fpga-tool-perf/evals'
 
             build_fn = 'tests/sample_data/build.small.json'
@@ -54,13 +57,18 @@ class TestHydraFetcherSmall(unittest.TestCase):
             # setup /evals request mock
             with open(evals_fn, "r") as f:
                 json_data = f.read()
-                m.get(evals_url, text=json_data) 
+                m.get(evals_url, text=json_data)
+            
+            # setup /evals?page=2 request mock
+            with open(evals_fn2, "r") as f:
+                json_data = f.read()
+                m.get(evals_url + "?page=2", text=json_data)
 
             # setup /build and /meta.json request mock
             with open(build_fn, "r") as f:
                 json_data = f.read()
 
-                for build_num in range(12):
+                for build_num in range(24):
                     # /build/:buildid
                     build_url = f'https://hydra.vtr.tools/build/{build_num}'
                     m.get(build_url, text=json_data)
@@ -71,7 +79,7 @@ class TestHydraFetcherSmall(unittest.TestCase):
                     m.get(meta_url, json=payload) # setup /meta.json request mock
 
             # run tests on different eval_num
-            for eval_num in range(0, 3):
+            for eval_num in range(0, 6):
                 with self.subTest(eval_num=eval_num):
                     # if mapping is not defined, should not remap
                     hf = HydraFetcher(
@@ -87,6 +95,63 @@ class TestHydraFetcherSmall(unittest.TestCase):
                     eval_id = hf.get_evaluation().get_eval_id()
                     assert eval_id == eval_num + 1
 
+    def test_hydrafetcher_get_evaluation_eval_num_absolute(self):
+        """
+        get_evaluation() should return an Evaluation corresponding to the small
+        dataset and the specified *absolute* eval_num.
+
+        Tests whether the fetcher is able to get an exact evaluation by absolute
+        eval num and pagination.
+        """
+        with requests_mock.Mocker() as m:
+            evals_fn = 'tests/sample_data/evals.small.json'
+            evals_fn2 = 'tests/sample_data/evals.small.2.json'
+            evals_url = 'https://hydra.vtr.tools/jobset/dusty/fpga-tool-perf/evals'
+
+            build_fn = 'tests/sample_data/build.small.json'
+
+            # setup /evals request mock
+            with open(evals_fn, "r") as f:
+                json_data = f.read()
+                m.get(evals_url, text=json_data) 
+
+            # setup /evals?page=2 request mock
+            with open(evals_fn2, "r") as f:
+                json_data = f.read()
+                m.get(evals_url + "?page=2", text=json_data)
+
+            # setup /build and /meta.json request mock
+            with open(build_fn, "r") as f:
+                json_data = f.read()
+
+                for build_num in range(24):
+                    # /build/:buildid
+                    build_url = f'https://hydra.vtr.tools/build/{build_num}'
+                    m.get(build_url, text=json_data)
+                    
+                    # /meta.json
+                    meta_url = f'https://hydra.vtr.tools/build/{build_num}/download/5/meta.json'
+                    payload = {"build_num": build_num}
+                    m.get(meta_url, json=payload) # setup /meta.json request mock
+
+            # run tests on different eval_num
+            for eval_num in range(1, 7):
+                with self.subTest(eval_num=eval_num):
+                    # if mapping is not defined, should not remap
+                    hf = HydraFetcher(
+                        project="dusty",
+                        jobset="fpga-tool-perf",
+                        eval_num=eval_num,
+                        absolute_eval_num=True)
+                    result = hf.get_evaluation().get_df()
+
+                    col = [x for x in range(eval_num * 4 - 4, eval_num * 4)]
+                    expected = pd.DataFrame({"build_num": col})
+                    assert_frame_equal(result, expected)
+
+                    eval_id = hf.get_evaluation().get_eval_id()
+                    assert eval_id == eval_num
+
     def test_hydrafetcher_get_evaluation_mapping(self):
         """
         get_evaluation() should return an Evaluation corresponding to the small
diff --git a/tests/test_processor.py b/tests/test_processor.py
index ca7250d..c1bf247 100644
--- a/tests/test_processor.py
+++ b/tests/test_processor.py
@@ -26,6 +26,7 @@ class TestProcessor:
     FilterByIndex()
     Aggregate()
     GeomeanAggregate()
+    CompareToFirst()
     """
 
     def test_minusone(self):
@@ -191,7 +192,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         # direction is -1 => normalize around minimum
-        pipeline = [AddNormalizedColumn("group", "value", "normalized", -1)]
+        pipeline = [AddNormalizedColumn("group", "value", "normalized", Direction.MINIMIZE)]
         result = eval1.process(pipeline)
         expected = pd.DataFrame(
             [
@@ -330,7 +331,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": 1
+            "value": Direction.MINIMIZE
         }
         pipeline = [NormalizeAround(
             normalize_direction,
@@ -376,7 +377,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": -1
+            "value": Direction.MAXIMIZE
         }
         pipeline = [NormalizeAround(
             normalize_direction,
@@ -415,7 +416,7 @@ class TestProcessor:
         eval1 = Evaluation(df, eval_id=10)
 
         normalize_direction = {
-            "value": 1
+            "value": Direction.MINIMIZE
         }
         pipeline = [Normalize(normalize_direction)]
 
@@ -638,4 +639,102 @@ class TestProcessor:
             ]
         )
         assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst(self):
+        """ Test if CompareToFirst works with default params """
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MAXIMIZE,
+            "b": Direction.MAXIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction)]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.relative": 1.0/1, "b": 5, "b.relative": 5.0/5},
+                {"a": 2, "a.relative": 2.0/1, "b": 4, "b.relative": 4.0/5},
+                {"a": 3, "a.relative": 3.0/1, "b": 3, "b.relative": 3.0/5},
+                {"a": 4, "a.relative": 4.0/1, "b": 2, "b.relative": 2.0/5},
+                {"a": 5, "a.relative": 5.0/1, "b": 1, "b.relative": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst_dir_subset(self):
+        """ Test if CompareToFirst works with different direction and subset"""
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MINIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction)]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.relative": 1.0/1},
+                {"a": 2, "a.relative": 1.0/2},
+                {"a": 3, "a.relative": 1.0/3},
+                {"a": 4, "a.relative": 1.0/4},
+                {"a": 5, "a.relative": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
+        assert eval1.get_eval_id() == 20
+
+    def test_comparetofirst_suffix(self):
+        """ Test if CompareToFirst works with different suffix """
+        df = pd.DataFrame(
+            [
+                {"a": 1, "b": 5},
+                {"a": 2, "b": 4},
+                {"a": 3, "b": 3},
+                {"a": 4, "b": 2},
+                {"a": 5, "b": 1},
+            ]
+        )
+        eval1 = Evaluation(df, eval_id=20)
+
+        direction = {
+            "a": Direction.MAXIMIZE,
+            "b": Direction.MAXIMIZE
+        }
+
+        pipeline = [CompareToFirst(direction, suffix=".diff")]
+        eval1 = eval1.process(pipeline)
+
+        expected_df = pd.DataFrame(
+            [
+                {"a": 1, "a.diff": 1.0/1, "b": 5, "b.diff": 5.0/5},
+                {"a": 2, "a.diff": 2.0/1, "b": 4, "b.diff": 4.0/5},
+                {"a": 3, "a.diff": 3.0/1, "b": 3, "b.diff": 3.0/5},
+                {"a": 4, "a.diff": 4.0/1, "b": 2, "b.diff": 2.0/5},
+                {"a": 5, "a.diff": 5.0/1, "b": 1, "b.diff": 1.0/5},
+            ]
+        )
+        assert_frame_equal(eval1.get_df(), expected_df)
         assert eval1.get_eval_id() == 20
\ No newline at end of file