Repository: RecList/reclist (language: python)

Issue:
Dev session



Discussion / hints:

Fix patch:
diff --git a/README.rst b/README.rst
index 5fd2918..496cf0b 100644
--- a/README.rst
+++ b/README.rst
@@ -54,8 +54,7 @@ Quick Links
 Project updates
 ~~~~~~~~~~~~~~~
 
-*Nov. 2021*: the library is currently in alpha. We welcome contributions and feedback, but please be advised that the package
-may change substantially in the upcoming months.
+*Nov. 2021*: the library is currently in alpha (i.e. enough working code to finish the paper and tinker with it). We welcome early feedback, but please be advised that the package may change substantially in the upcoming months ("If you're not embarrassed by the first version, you've launched too late").
 
 As the project is in active development, come back often for updates.
 
diff --git a/docs/abstractions.rst b/docs/abstractions.rst
deleted file mode 100644
index 8e75836..0000000
--- a/docs/abstractions.rst
+++ /dev/null
@@ -1,37 +0,0 @@
-============
-Abstractions
-============
-
-
-RecTest
--------
-
-The RecTest is probably the most fundamental abstraction in RecList.
-
-
-.. code-block:: python
-
-    class MyRecListRecList(RecList):
-
-        @rec_test(test_type='stats')
-        def basic_stats(self):
-            """
-            Basic statistics on training, test and prediction data
-            """
-            from reclist.metrics.standard_metrics import statistics
-            return statistics(self._x_train,
-                              self._y_train,
-                              self._x_test,
-                              self._y_test,
-                              self._y_preds)
-
-
-
-.. automodule:: reclist.abstractions
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
-
-
diff --git a/docs/core.rst b/docs/core.rst
new file mode 100644
index 0000000..77e6346
--- /dev/null
+++ b/docs/core.rst
@@ -0,0 +1,107 @@
+Core Components
+===============
+
+To understand how to use RecList you need to know the core components of the package.
+After reading this page you should be comfortable enough to play and modify the tutorial and also create your
+reclists for evaluation.
+
+Two Paths
+~~~~~~~~~
+
+
+Say you have a new recommender and you want to validate on behavioural tests. We have different datasets you can play
+with. If you want to follow this path, you just need to get the training data from the datastets, train the model and then
+run the reclist to evaluate your model.
+
+Another possible use case is that you might want to create a RecList for a new dataset you have.
+
+RecTest
+~~~~~~~
+
+The RecTest is probably the most fundamental abstraction in RecList. The RecTest is essentially a decorator used to
+evaluate the various datasets.
+
+
+.. code-block:: python
+
+    class MyRecListRecList(RecList):
+
+        @rec_test(test_type='stats')
+        def basic_stats(self):
+            """
+            Basic statistics on training, test and prediction data
+            """
+            from reclist.metrics.standard_metrics import statistics
+            return statistics(self._x_train,
+                              self._y_train,
+                              self._x_test,
+                              self._y_test,
+                              self._y_preds)
+
+
+RecDataset
+~~~~~~~~~~
+
+The dataset is a simple abstraction you might want to implement if you are playing with other datasets.
+You will see that we just need to instantiate the main parameters.
+
+
+.. code-block:: python
+
+    class RecDataset(ABC):
+    """
+    Implements an abstract class for the dataset
+    """
+    def __init__(self, force_download=False):
+        """
+        :param force_download: allows to force the download of the dataset in case it is needed.
+        :type: force_download: bool, optional
+        """
+        self._x_train = None
+        self._y_train = None
+        self._x_test = None
+        self._y_test = None
+        self._catalog = None
+        self.force_download = force_download
+        self.load()
+
+    @abstractmethod
+    def load(self):
+        """
+        Abstract method that should implement dataset loading
+        @return:
+        """
+        return
+
+
+RecModel
+~~~~~~~~
+
+
+
+.. code-block:: python
+
+    class MyRecModel(RecModel):
+        """
+        My Recommender Model
+        """
+        def __init__(self, **kwargs):
+            super().__init__(**kwargs)
+
+        model_name = "mymodel"
+
+        def train(self, products):
+
+            self._model = my_training_function(products)
+
+        def predict(self, prediction_input: list, *args, **kwargs):
+
+            predictions = self.model.predict(prediction_input)
+
+            return predictions
+
+        def get_vector(self, product_sku):
+            try:
+                return list(self._model.get_vector(product_sku))
+            except Exception as e:
+                return []
diff --git a/docs/index.rst b/docs/index.rst
index e0b19b4..7f2fb75 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,7 +6,7 @@ Welcome to RecList's documentation!
    :caption: Contents:
 
    readme
-   abstractions
+   core
    modules
    contributing
    authors
diff --git a/docs/modules.rst b/docs/modules.rst
index 2a48eda..3b3ef68 100644
--- a/docs/modules.rst
+++ b/docs/modules.rst
@@ -1,7 +1,37 @@
-reclist
+Modules
 =======
 
 .. toctree::
    :maxdepth: 4
 
    reclist
+
+
+Abstractions
+------------
+
+
+.. automodule:: reclist.abstractions
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Datasets
+--------
+
+Data sets currently available
+
+.. automodule:: reclist.datasets
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+Metrics
+--------
+
+Metrics
+
+.. automodule:: reclist.metrics
+    :members:
+    :undoc-members:
+    :show-inheritance:
diff --git a/docs/reclist.metrics.rst b/docs/reclist.metrics.rst
deleted file mode 100644
index 7b29844..0000000
--- a/docs/reclist.metrics.rst
+++ /dev/null
@@ -1,78 +0,0 @@
-reclist.metrics package
-=======================
-
-Submodules
-----------
-
-reclist.metrics.cosine\_distance\_metrics module
-------------------------------------------------
-
-.. automodule:: reclist.metrics.cosine_distance_metrics
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.generic\_cosine\_distance module
-------------------------------------------------
-
-.. automodule:: reclist.metrics.generic_cosine_distance
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.graph\_distance\_test module
---------------------------------------------
-
-.. automodule:: reclist.metrics.graph_distance_test
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.hits\_distribution module
------------------------------------------
-
-.. automodule:: reclist.metrics.hits_distribution
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.hits\_slice module
-----------------------------------
-
-.. automodule:: reclist.metrics.hits_slice
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.perturbation module
------------------------------------
-
-.. automodule:: reclist.metrics.perturbation
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.price\_homogeneity module
------------------------------------------
-
-.. automodule:: reclist.metrics.price_homogeneity
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.metrics.standard\_metrics module
-----------------------------------------
-
-.. automodule:: reclist.metrics.standard_metrics
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
-Module contents
----------------
-
-.. automodule:: reclist.metrics
-    :members:
-    :undoc-members:
-    :show-inheritance:
diff --git a/docs/reclist.recommenders.rst b/docs/reclist.recommenders.rst
deleted file mode 100644
index 83b3249..0000000
--- a/docs/reclist.recommenders.rst
+++ /dev/null
@@ -1,22 +0,0 @@
-reclist.recommenders package
-============================
-
-Submodules
-----------
-
-reclist.recommenders.prod2vec module
-------------------------------------
-
-.. automodule:: reclist.recommenders.prod2vec
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
-Module contents
----------------
-
-.. automodule:: reclist.recommenders
-    :members:
-    :undoc-members:
-    :show-inheritance:
diff --git a/docs/reclist.rst b/docs/reclist.rst
deleted file mode 100644
index 95cfca0..0000000
--- a/docs/reclist.rst
+++ /dev/null
@@ -1,47 +0,0 @@
-reclist package
-===============
-
-Subpackages
------------
-
-.. toctree::
-
-    reclist.metrics
-    reclist.recommenders
-    reclist.utils
-
-Submodules
-----------
-
-reclist.abstractions module
----------------------------
-
-.. automodule:: reclist.abstractions
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.datasets module
------------------------
-
-.. automodule:: reclist.datasets
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.reclist module
-----------------------
-
-.. automodule:: reclist.reclist
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
-Module contents
----------------
-
-.. automodule:: reclist
-    :members:
-    :undoc-members:
-    :show-inheritance:
diff --git a/docs/reclist.utils.rst b/docs/reclist.utils.rst
deleted file mode 100644
index 67a646b..0000000
--- a/docs/reclist.utils.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-reclist.utils package
-=====================
-
-Submodules
-----------
-
-reclist.utils.config module
----------------------------
-
-.. automodule:: reclist.utils.config
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-reclist.utils.train\_w2v module
--------------------------------
-
-.. automodule:: reclist.utils.train_w2v
-    :members:
-    :undoc-members:
-    :show-inheritance:
-
-
-Module contents
----------------
-
-.. automodule:: reclist.utils
-    :members:
-    :undoc-members:
-    :show-inheritance:
diff --git a/examples/movielens_similar_rec.py b/examples/movielens_similar_rec.py
new file mode 100644
index 0000000..fc3c6ed
--- /dev/null
+++ b/examples/movielens_similar_rec.py
@@ -0,0 +1,38 @@
+"""
+    This is a sample script showing how to get started with reclist. In particular,
+    we are targeting a "similar items" use cases, such as for example a item recommender:
+    if a shopper is browsing a product listing and clicked on item X, what other item
+    might they be interested in that is similar in type to item X?
+
+    We train a simple, yet effective prod2vec baseline model (https://arxiv.org/abs/2007.14906),
+    re-using for convenience a "training embedding" function already implemented by recsys. The script
+    show how easy it is to run behavioral tests on a target dataset, in this case a wrapper around a large
+    dataset widely used for benchmarking recommendation systems (MovieLens 25M dataset: https://grouplens.org/datasets/movielens/)
+
+    If you want to run your own model on the same dataset and the same behavioral test suite, just implement
+    your model prediction using the RecModel, and pass it to the RecList.
+"""
+from reclist.datasets import MovieLensDataset
+from reclist.recommenders.prod2vec import MovieLensP2VRecModel
+from reclist.reclist import MovieLensSimilarItemRecList
+
+
+if __name__ == "__main__":
+
+    # get the MovieLens 25M dataset as a RecDataset object
+    movielens_dataset = MovieLensDataset()
+
+    # re-use a skip-gram model from reclist to train a latent product space, to be used
+    # (through knn) to build a recommender
+    model = MovieLensP2VRecModel()
+    model.train(movielens_dataset.x_train)
+
+    # instantiate rec_list object, prepared with standard quantitative tests
+    # and sensible behavioral tests (check the paper for details!)
+    rec_list = MovieLensSimilarItemRecList(
+        model=model,
+        dataset=movielens_dataset
+    )
+
+    # invoke rec_list to run tests
+    rec_list(verbose=True)
diff --git a/examples/spotify_session_rec.py b/examples/spotify_session_rec.py
new file mode 100644
index 0000000..b7617bd
--- /dev/null
+++ b/examples/spotify_session_rec.py
@@ -0,0 +1,35 @@
+"""
+    This is a sample script showing how to get started with reclist. In particular,
+    we are targeting the use case of session-based recommendations.
+
+    We train a simple, yet effective prod2vec baseline model (https://arxiv.org/abs/2007.14906),
+    re-using for convenience a "training embedding" function already implemented by recsys. The script
+    shows how easy it is to run behavioral tests on a target dataset, in this case a wrapper around a large
+    music dataset (the Spotify Million Playlist dataset:
+    https://www.aicrowd.com/challenges/spotify-million-playlist-dataset-challenge).
+
+    If you want to run your own model on the same dataset and the same behavioral test suite, just implement
+    your model prediction using the RecModel, and pass it to the RecList.
+"""
+from reclist.datasets import SpotifyDataset
+from reclist.recommenders.prod2vec import SpotifyP2VRecModel
+from reclist.reclist import SpotifySessionRecList
+
+if __name__ == "__main__":
+
+    # get the Spotify million playlist dataset as a RecDataset object
+    spotify_dataset = SpotifyDataset()
+
+    # re-use a skip-gram model from reclist to train a latent product space, to be used
+    # (through knn) to build a recommender
+    model = SpotifyP2VRecModel()
+    model.train(spotify_dataset.x_train)
+
+    # instantiate rec_list object, prepared with standard quantitative tests
+    # and sensible behavioral tests (check the paper for details!)
+    rec_list = SpotifySessionRecList(
+        model=model,
+        dataset=spotify_dataset
+    )
+    # invoke rec_list to run tests
+    rec_list(verbose=True)
diff --git a/reclist/abstractions.py b/reclist/abstractions.py
index b597b6d..4514f40 100644
--- a/reclist/abstractions.py
+++ b/reclist/abstractions.py
@@ -1,7 +1,5 @@
-from abc import ABC, abstractmethod
-import ast
+
 from datetime import datetime
-import inspect
 import os
 from abc import ABC, abstractmethod
 from functools import wraps
@@ -9,7 +7,7 @@
 import time
 import json
 from reclist.utils.train_w2v import train_embeddings
-
+from reclist.current import current
 
 class RecDataset(ABC):
     """
@@ -32,7 +30,7 @@ def __init__(self, force_download=False):
     def load(self):
         """
         Abstract method that should implement dataset loading
-        @return:
+        :return:
         """
         return
 
@@ -122,6 +120,7 @@ def __init__(self, model: RecModel, dataset: RecDataset, y_preds: list = None):
         :param dataset:
         :param y_preds:
         """
+
         self.name = self.__class__.__name__
         self._rec_tests = self.get_tests()
         self._x_train = dataset.x_train
@@ -164,6 +163,16 @@ def get_tests(self):
 
     def __call__(self, verbose=True, *args, **kwargs):
         run_epoch_time_ms = round(time.time() * 1000)
+        # create datastore
+        current._report_path = os.path.join(self.META_DATA_FOLDER,
+                                        self.name,
+                                        self.rec_model.__class__.__name__,
+                                        str(run_epoch_time_ms))
+
+        Path(os.path.join(current.report_path, 'artifacts')).mkdir(parents=True, exist_ok=True)
+        Path(os.path.join(current.report_path, 'results')).mkdir(parents=True, exist_ok=True)
+        Path(os.path.join(current.report_path, 'plots')).mkdir(parents=True, exist_ok=True)
+
         # iterate through tests
         for test_func_name, test in self._rec_tests.items():
             test_result = test(*args, **kwargs)
@@ -198,9 +207,7 @@ def generate_report(self, epoch_time_ms: int):
         self.store_artifacts(report_path)
 
     def store_artifacts(self, report_path: str):
-        target_path = os.path.join(report_path, 'artifacts')
-        # make sure the folder is there, with all intermediate parents
-        Path(target_path).mkdir(parents=True, exist_ok=True)
+        target_path = os.path.join(current.report_path, 'artifacts')
         # store predictions
         with open(os.path.join(target_path, 'model_predictions.json'), 'w') as f:
             json.dump({
diff --git a/reclist/current.py b/reclist/current.py
new file mode 100644
index 0000000..b3dbc3c
--- /dev/null
+++ b/reclist/current.py
@@ -0,0 +1,10 @@
+class Current():
+    def __init__(self):
+        self._report_path = None
+
+    @property
+    def report_path(self):
+        return self._report_path
+
+
+current = Current()
diff --git a/reclist/datasets.py b/reclist/datasets.py
index 2905008..ccb3e34 100644
--- a/reclist/datasets.py
+++ b/reclist/datasets.py
@@ -1,7 +1,7 @@
 import json
 import tempfile
 import zipfile
-import random
+import os
 from reclist.abstractions import RecDataset
 from reclist.utils.config import *
 
@@ -33,8 +33,19 @@ def load(self):
         self._y_train = None
         self._x_test = data["x_test"]
         self._y_test = data["y_test"]
-        self._catalog = data["catalog"]
-
+        self._catalog = self._convert_catalog_keys(data["catalog"])
+        
+    def _convert_catalog_keys(self, catalog):
+        """
+        Convert catalog keys from string to integer type
+
+        JSON encodes all keys to strings, so the catalog dictionary
+        will be loaded up string representation of movie IDs.
+        """
+        converted_catalog = {}
+        for k, v in catalog.items():
+            converted_catalog[int(k)] = v
+        return converted_catalog
 
 class CoveoDataset(RecDataset):
     """
@@ -64,23 +75,25 @@ def load(self):
 
 class SpotifyDataset(RecDataset):
 
-    def __init__(self, k: int = 5, **kwargs):
-        self.k = k
+    def __init__(self, **kwargs):
         super().__init__(**kwargs)
 
     def load(self):
         data = self.load_spotify_playlist_dataset()
-        x_test, y_test = self.preprocess_spotify_playlist_data()
-        self._x_train = data["x_train"]
+        self._x_train = data["train"]
         self._y_train = None
-        self._x_test = x_test
-        self._y_test = y_test
-        self._catalog = data["metadata"]
+        self._x_test = data['test']
+        self._y_test = None
+        self._catalog = data["catalog"]
+
+        # generate NEP dataset here for now
+        test_pairs = [(playlist[:-1], [playlist[-1]])  for playlist in self._x_test if len(playlist) > 1]
+        self._x_test, self._y_test = zip(*test_pairs)
 
     def load_spotify_playlist_dataset(self):
 
         cache_directory = get_cache_directory()
-        filename = os.path.join(cache_directory, "spotify_playlist.zip")   # TODO: make var somewhere
+        filename = os.path.join(cache_directory, "small_spotify_playlist.zip")   # TODO: make var somewhere
 
         if not os.path.exists(filename) or self.force_download:
             download_with_progress(SPOTIFY_PLAYLIST_DATASET_S3_URL, filename)
@@ -91,31 +104,3 @@ def load_spotify_playlist_dataset(self):
             with open(os.path.join(temp_dir, 'dataset.json')) as f:
                 data = json.load(f)
         return data
-
-    def preprocess_spotify_playlist_data(
-        self,
-        shuffle: bool = False,
-        seed: int = 0
-    ):
-        x_test = []
-        y_test = []
-        for playlist in self.data['test']:
-            if len(playlist['tracks']) <= self.k:
-                continue
-            if shuffle:
-                random.seed(seed)
-                all_idx = list(range(len(playlist['tracks'])))
-                seeded_idx = random.sample(all_idx, k=self.k)
-                held_out_idx = set(all_idx) - set(seeded_idx)
-                seeded_tracks = [playlist['tracks'][idx] for idx in seeded_idx]
-                held_out_tracks = [playlist['tracks'][idx]
-                                   for idx in held_out_idx]
-                assert len(seeded_tracks) + \
-                    len(held_out_tracks) == len(playlist['tracks'])
-            else:
-                seeded_tracks = playlist['tracks'][:self.k]
-                held_out_tracks = playlist['tracks'][self.k:]
-            x_test.append(seeded_tracks)
-            y_test.append(held_out_tracks)
-
-        return x_test, y_test
diff --git a/reclist/metrics/cosine_distance_metrics.py b/reclist/metrics/cosine_distance_metrics.py
deleted file mode 100644
index 966314e..0000000
--- a/reclist/metrics/cosine_distance_metrics.py
+++ /dev/null
@@ -1,80 +0,0 @@
-import numpy as np
-from reclist.metrics.standard_metrics import sample_misses_at_k
-from scipy.spatial.distance import cosine
-import matplotlib.pyplot as plt
-from statistics import mean
-
-def error_by_cosine_distance(model, y_test, y_preds, k=3, bins=25, debug=False):
-    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):
-        error_msg = "Error : Model {} does not support retrieval of vector embeddings".format(model.__class__)
-        print(error_msg)
-        return error_msg
-    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
-    cos_distances = []
-    for m in misses:
-        if m['Y_PRED']:
-            vector_test = model.get_vector(m['Y_TEST'][0])
-            vector_pred = model.get_vector(m['Y_PRED'][0])
-            if vector_pred and vector_test:
-                cos_dist = cosine(vector_pred, vector_test)
-                cos_distances.append(cos_dist)
-
-    histogram = np.histogram(cos_distances, bins=bins, density=False)
-    # cast to list
-    histogram = (histogram[0].tolist(), histogram[1].tolist())
-    # debug / viz
-    if debug:
-        plt.hist(cos_distances, bins=bins)
-        plt.title('dist over cosine distance prod space')
-        plt.show()
-
-    return {'mean': np.mean(cos_distances), 'histogram': histogram}
-
-def distance_to_query(model, x_test, y_test, y_preds, k=3, bins=25, debug=False):
-    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):
-        error_msg = "Error : Model {} does not support retrieval of vector embeddings".format(model.__class__)
-        print(error_msg)
-        return error_msg
-    misses = sample_misses_at_k(y_preds, y_test, x_test=x_test, k=k, size=-1)
-    x_to_y_cos = []
-    x_to_p_cos = []
-    for m in misses:
-        if m['Y_PRED']:
-            vector_x = model.get_vector(m['X_TEST'][0])
-            vector_y = model.get_vector(m['Y_TEST'][0])
-            vectors_p = [model.get_vector(_) for _ in m['Y_PRED']]
-            c_dists =[]
-            if not vector_x or not vector_y:
-                continue
-            x_to_y_cos.append(cosine(vector_x, vector_y))
-            for v_p in vectors_p:
-                if not v_p:
-                    continue
-                cos_dist = cosine(v_p, vector_x)
-                if cos_dist:
-                    c_dists.append(cos_dist)
-            if c_dists:
-                x_to_p_cos.append(mean(c_dists))
-
-    h_xy = np.histogram(x_to_y_cos, bins=bins, density=False)
-    h_xp = np.histogram(x_to_p_cos, bins=bins, density=False)
-
-    h_xy = (h_xy[0].tolist(), h_xy[1].tolist())
-    h_xp = (h_xp[0].tolist(), h_xp[1].tolist())
-
-    # debug / viz
-    if debug:
-        plt.hist(x_to_y_cos, bins=bins, alpha=0.5)
-        plt.hist(x_to_p_cos, bins=bins, alpha=0.5)
-        plt.title('distribution of distance to input')
-        plt.legend(['Distance from Input to Label',
-                    'Distance from Input to Label'],
-                   loc='upper right')
-        plt.show()
-
-    return {
-        'histogram_x_to_y': h_xy,
-        'histogram_x_to_p': h_xp,
-        'raw_distances_x_to_y': x_to_y_cos,
-        'raw_distances_x_to_p': x_to_p_cos,
-    }
diff --git a/reclist/metrics/distance_metrics.py b/reclist/metrics/distance_metrics.py
new file mode 100644
index 0000000..c89c1d4
--- /dev/null
+++ b/reclist/metrics/distance_metrics.py
@@ -0,0 +1,179 @@
+from scipy.spatial.distance import cosine
+import matplotlib.pyplot as plt
+from reclist.current import current
+import os
+import json
+import networkx as nx
+from networkx.algorithms.shortest_paths.generic import shortest_path
+from statistics import mean
+from reclist.metrics.standard_metrics import sample_misses_at_k, sample_hits_at_k
+import numpy as np
+
+def error_by_cosine_distance(model, y_test, y_preds, k=3, bins=25, debug=False):
+    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):
+        error_msg = "Error : Model {} does not support retrieval of vector embeddings".format(model.__class__)
+        print(error_msg)
+        return error_msg
+    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
+    cos_distances = []
+    for m in misses:
+        if m['Y_PRED']:
+            vector_test = model.get_vector(m['Y_TEST'][0])
+            vector_pred = model.get_vector(m['Y_PRED'][0])
+            if vector_pred and vector_test:
+                cos_dist = cosine(vector_pred, vector_test)
+                cos_distances.append(cos_dist)
+
+    histogram = np.histogram(cos_distances, bins=bins, density=False)
+    # cast to list
+    histogram = (histogram[0].tolist(), histogram[1].tolist())
+    # debug / viz
+    if debug:
+        plt.hist(cos_distances, bins=bins)
+        plt.title('dist over cosine distance prod space')
+        plt.savefig(os.path.join(current.report_path,
+                                 'plots',
+                                 'distance_to_predictions.png'))
+        plt.clf()
+        # plt.show()
+
+    return {'mean': np.mean(cos_distances), 'histogram': histogram}
+
+def distance_to_query(model, x_test, y_test, y_preds, k=3, bins=25, debug=False):
+    if not(hasattr(model.__class__, 'get_vector') and callable(getattr(model.__class__, 'get_vector'))):
+        error_msg = "Error : Model {} does not support retrieval of vector embeddings".format(model.__class__)
+        print(error_msg)
+        return error_msg
+    misses = sample_misses_at_k(y_preds, y_test, x_test=x_test, k=k, size=-1)
+    x_to_y_cos = []
+    x_to_p_cos = []
+    for m in misses:
+        if m['Y_PRED']:
+            vector_x = model.get_vector(m['X_TEST'][0])
+            vector_y = model.get_vector(m['Y_TEST'][0])
+            vectors_p = [model.get_vector(_) for _ in m['Y_PRED']]
+            c_dists =[]
+            if not vector_x or not vector_y:
+                continue
+            x_to_y_cos.append(cosine(vector_x, vector_y))
+            for v_p in vectors_p:
+                if not v_p:
+                    continue
+                cos_dist = cosine(v_p, vector_x)
+                if cos_dist:
+                    c_dists.append(cos_dist)
+            if c_dists:
+                x_to_p_cos.append(mean(c_dists))
+
+    h_xy = np.histogram(x_to_y_cos, bins=bins, density=False)
+    h_xp = np.histogram(x_to_p_cos, bins=bins, density=False)
+
+    h_xy = (h_xy[0].tolist(), h_xy[1].tolist())
+    h_xp = (h_xp[0].tolist(), h_xp[1].tolist())
+
+    # debug / viz
+    if debug:
+        plt.hist(x_to_y_cos, bins=bins, alpha=0.5)
+        plt.hist(x_to_p_cos, bins=bins, alpha=0.5)
+        plt.title('distribution of distance to input')
+        plt.legend(['Distance from Input to Label',
+                    'Distance from Input to Label'],
+                   loc='upper right')
+        # plt.show()
+        plt.savefig(os.path.join(current.report_path,
+                                 'plots',
+                                 'distance_to_query.png'))
+        plt.clf()
+
+    return {
+        'histogram_x_to_y': h_xy,
+        'histogram_x_to_p': h_xp,
+        'raw_distances_x_to_y': x_to_y_cos,
+        'raw_distances_x_to_p': x_to_p_cos,
+    }
+
+
+
+def shortest_path_length():
+    pass
+
+
+get_nodes = lambda nodes, ancestors="": [] if not nodes else ['_'.join([ancestors, nodes[0]])] + \
+                                                             get_nodes(nodes[1:], '_'.join([ancestors, nodes[0]]))
+
+
+def graph_distance_test(y_test, y_preds, product_data, k=3):
+    path_lengths = []
+    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
+    for _y, _y_p in zip([_['Y_TEST'] for _ in misses],
+                        [_['Y_PRED'] for _ in misses]):
+        if not _y_p:
+            continue
+        _y_sku = _y[0]
+        _y_p_sku = _y_p[0]
+
+        if _y_sku not in product_data or _y_p_sku not in product_data:
+            continue
+        if not product_data[_y_sku]['CATEGORIES'] or not product_data[_y_p_sku]['CATEGORIES']:
+            continue
+        # extract graph information
+        catA = json.loads(product_data[_y_sku]['CATEGORIES'])
+        catB = json.loads(product_data[_y_p_sku]['CATEGORIES'])
+        catA_nodes = [get_nodes(c) for c in catA]
+        catB_nodes = [get_nodes(c) for c in catB]
+        all_nodes = list(set([n for c in catA_nodes + catB_nodes for n in c]))
+        all_edges = [(n1, n2) for c in catA_nodes + catB_nodes for n1, n2 in zip(c[:-1], c[1:])]
+        all_edges = list(set(all_edges))
+
+        # build graph
+        G = nx.Graph()
+        G.add_nodes_from(all_nodes)
+        G.add_edges_from(all_edges)
+
+        # get leaves
+        cat1_leaves = [c[-1] for c in catA_nodes]
+        cat2_leaves = [c[-1] for c in catB_nodes]
+
+        all_paths = [shortest_path(G, c1_l, c2_l) for c1_l in cat1_leaves for c2_l in cat2_leaves]
+        s_path = min(all_paths, key=len)
+        s_path_len = len(s_path) - 1
+        path_lengths.append(s_path_len)
+
+    histogram = np.histogram(path_lengths, bins=np.arange(0, max(path_lengths)))
+    histogram = (histogram[0].tolist(), histogram[1].tolist())
+    return {'mean': mean(path_lengths), 'hist': histogram}
+
+
+
+def generic_cosine_distance(embeddings: dict,
+                            type_fn,
+                            y_test,
+                            y_preds,
+                            k=10,
+                            bins=25,
+                            debug=False):
+
+    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
+    cos_distances = []
+    for m in misses:
+        if m['Y_TEST'] and m['Y_PRED'] and type_fn(m['Y_TEST'][0]) and type_fn(m['Y_PRED'][0]):
+            vector_test = embeddings.get(type_fn(m['Y_TEST'][0]), None)
+            vector_pred = embeddings.get(type_fn(m['Y_PRED'][0]), None)
+            if vector_pred and vector_test:
+                cos_dist = cosine(vector_pred, vector_test)
+                cos_distances.append(cos_dist)
+
+    # TODO: Maybe sample some examples from the bins
+    histogram = np.histogram(cos_distances, bins=bins, density=False)
+    # cast to list
+    histogram = (histogram[0].tolist(), histogram[1].tolist())
+    # debug / viz
+    if debug:
+        plt.hist(cos_distances, bins=bins)
+        plt.title('cosine distance misses')
+        plt.savefig(os.path.join(current.report_path,
+                                 'plots',
+                                 'cosine_distance_over_type.png'))
+        plt.clf()
+
+    return {'mean': np.mean(cos_distances), 'histogram': histogram}
diff --git a/reclist/metrics/generic_cosine_distance.py b/reclist/metrics/generic_cosine_distance.py
deleted file mode 100644
index 1b352c4..0000000
--- a/reclist/metrics/generic_cosine_distance.py
+++ /dev/null
@@ -1,34 +0,0 @@
-import numpy as np
-from reclist.metrics.standard_metrics import sample_misses_at_k, sample_hits_at_k
-from scipy.spatial.distance import cosine
-import matplotlib.pyplot as plt
-
-def generic_cosine_distance(embeddings: dict,
-                            type_fn,
-                            y_test,
-                            y_preds,
-                            k=10,
-                            bins=25,
-                            debug=False):
-
-    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
-    cos_distances = []
-    for m in misses:
-        if m['Y_TEST'] and m['Y_PRED'] and type_fn(m['Y_TEST'][0]) and type_fn(m['Y_PRED'][0]):
-            vector_test = embeddings.get(type_fn(m['Y_TEST'][0]), None)
-            vector_pred = embeddings.get(type_fn(m['Y_PRED'][0]), None)
-            if vector_pred and vector_test:
-                cos_dist = cosine(vector_pred, vector_test)
-                cos_distances.append(cos_dist)
-
-    # TODO: Maybe sample some examples from the bins
-    histogram = np.histogram(cos_distances, bins=bins, density=False)
-    # cast to list
-    histogram = (histogram[0].tolist(), histogram[1].tolist())
-    # debug / viz
-    if debug:
-        plt.hist(cos_distances, bins=bins)
-        plt.title('cosine distance misses')
-        plt.show()
-    return {'mean': np.mean(cos_distances), 'histogram': histogram}
-
diff --git a/reclist/metrics/hits.py b/reclist/metrics/hits.py
new file mode 100644
index 0000000..8061be0
--- /dev/null
+++ b/reclist/metrics/hits.py
@@ -0,0 +1,84 @@
+from collections import Counter, defaultdict
+import numpy as np
+import math
+from collections import defaultdict
+from reclist.metrics.standard_metrics import hit_rate_at_k, sample_hits_at_k, sample_misses_at_k
+import matplotlib.pyplot as plt
+import os
+from reclist.current import current
+
+
+def roundup(x: int):
+    div = 10.0 ** (len(str(x)))
+    return int(math.ceil(x / div)) * div
+
+
+def hits_distribution(x_train, x_test, y_test, y_preds, k=3, debug=False):
+    # get product interaction frequency
+    prod_interaction_cnt = Counter([_ for x in x_train for _ in x])
+    hit_per_interaction_cnt = defaultdict(list)
+    for _x, _y_test, _y_pred in zip(x_test, y_test, y_preds):
+        _x_cnt = prod_interaction_cnt[_x[0]]
+        # TODO: allow for generic metric
+        hit_per_interaction_cnt[_x_cnt].append(hit_rate_at_k([_y_pred], [_y_test], k=k))
+    # get max product frequency
+    max_cnt = prod_interaction_cnt.most_common(1)[0][1]
+    # round up to nearest place
+    max_cnt = int(roundup(max_cnt))
+    # generate log-bins
+    indices = np.logspace(1, np.log10(max_cnt), num=int(np.log10(max_cnt))).astype(np.int64)
+    indices = np.concatenate(([0], indices))
+    counts_per_bin = [[_ for i in range(low, high) for _ in hit_per_interaction_cnt[i]]
+                      for low, high in zip(indices[:-1], indices[1:])]
+
+    histogram = [np.mean(counts) if counts else 0 for counts in counts_per_bin]
+    count = [len(counts) for counts in counts_per_bin]
+
+    if debug:
+        # debug / visualization
+        plt.bar(indices[1:], histogram, width=-np.diff(indices) / 1.05, align='edge')
+        plt.xscale('log', base=10)
+        plt.title('HIT Distribution Across Product Frequency')
+        # plt.show()
+        plt.savefig(os.path.join(current.report_path, 'plots', 'hit_distribution.png'))
+        plt.clf()
+
+    return {
+        'histogram': {int(k): v for k, v in zip(indices[1:], histogram)},
+        'counts': {int(k): v for k, v in zip(indices[1:], count)}
+    }
+
+
+def hits_distribution_by_slice(slices,
+                               y_test,
+                               y_preds,
+                               k=3,
+                               sample_size=3,
+                               debug=False):
+    hit_rate_per_slice = defaultdict(dict)
+    for slice_name, slice_idx in slices.items():
+        # get predictions for slice
+        slice_y_preds = [y_preds[_] for _ in slice_idx]
+        # get labels for slice
+        slice_y_test = [y_test[_] for _ in slice_idx]
+        # TODO: We may want to allow for generic metric to be used here
+        slice_hr = hit_rate_at_k(slice_y_preds, slice_y_test, k=k)
+        # store results
+        hit_rate_per_slice[slice_name]['hit_rate'] = slice_hr
+        # hit_rate_per_slice[slice_name]['hits'] = sample_hits_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)
+        # hit_rate_per_slice[slice_name]['misses'] = sample_misses_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)
+
+    # debug / visualization
+    if debug:
+        x_tick_names = list(hit_rate_per_slice.keys())
+        x_tick_idx = list(range(len(x_tick_names)))
+        plt.figure(dpi=150)
+        plt.bar(x_tick_idx, [_['hit_rate'] for _ in hit_rate_per_slice.values()], align='center')
+        plt.xticks(list(range(len(hit_rate_per_slice))), x_tick_names, rotation=45, fontsize=5)
+        plt.savefig(os.path.join(current.report_path,
+                                 'plots',
+                                 'hit_distribution_slice.png'))
+        plt.clf()
+
+    # cast to normal dict
+    return dict(hit_rate_per_slice)
diff --git a/reclist/metrics/hits_distribution.py b/reclist/metrics/hits_distribution.py
deleted file mode 100644
index e4a29ee..0000000
--- a/reclist/metrics/hits_distribution.py
+++ /dev/null
@@ -1,44 +0,0 @@
-from collections import Counter, defaultdict
-from reclist.metrics.standard_metrics import hit_rate_at_k
-import matplotlib.pyplot as plt
-import numpy as np
-import math
-
-
-def roundup(x: int):
-    div = 10.0 ** (len(str(x)))
-    return int(math.ceil(x / div)) * div
-
-
-def hits_distribution(x_train, x_test, y_test, y_preds, k=3, debug=False):
-    # get product interaction frequency
-    prod_interaction_cnt = Counter([_ for x in x_train for _ in x])
-    hit_per_interaction_cnt = defaultdict(list)
-    for _x, _y_test, _y_pred in zip(x_test, y_test, y_preds):
-        _x_cnt = prod_interaction_cnt[_x[0]]
-        # TODO: allow for generic metric
-        hit_per_interaction_cnt[_x_cnt].append(hit_rate_at_k([_y_pred], [_y_test], k=k))
-    # get max product frequency
-    max_cnt = prod_interaction_cnt.most_common(1)[0][1]
-    # round up to nearest place
-    max_cnt = int(roundup(max_cnt))
-    # generate log-bins
-    indices = np.logspace(1, np.log10(max_cnt), num=int(np.log10(max_cnt))).astype(np.int64)
-    indices = np.concatenate(([0], indices))
-    counts_per_bin = [[_ for i in range(low, high) for _ in hit_per_interaction_cnt[i]]
-                      for low, high in zip(indices[:-1], indices[1:])]
-
-    histogram = [np.mean(counts) if counts else 0 for counts in counts_per_bin]
-    count = [len(counts) for counts in counts_per_bin]
-
-    if debug:
-        # debug / visualization
-        plt.bar(indices[1:], histogram, width=-np.diff(indices)/1.05, align='edge')
-        plt.xscale('log', base=10)
-        plt.title('HIT Distribution Across Product Frequency')
-        plt.show()
-
-    return {
-             'histogram': {int(k): v for k, v in zip(indices[1:], histogram)},
-             'counts':  {int(k): v for k, v in zip(indices[1:], count)}
-           }
diff --git a/reclist/metrics/hits_rating.py b/reclist/metrics/hits_rating.py
new file mode 100644
index 0000000..2f36024
--- /dev/null
+++ b/reclist/metrics/hits_rating.py
@@ -0,0 +1,46 @@
+from collections import defaultdict
+import os
+
+import matplotlib.pyplot as plt
+
+from reclist.current import current
+
+
+def hits_distribution_by_rating(y_test, y_preds, debug=False):
+    """
+    Calculates the distribution of hit-rate across the movie ratings in testing data
+    """
+    hits = defaultdict(int)
+    total = defaultdict(int)
+
+    for target, pred in zip(y_test, y_preds):
+        target_movie_id, target_rating = target[0]["movieId"], target[0]["rating"]
+        for movie in pred:
+            pred_movie_id, _ = movie["movieId"], movie["rating"]
+            if target_movie_id == pred_movie_id:
+                hits[target_rating] += 1
+                break
+        total[target_rating] += 1
+    hits_distribution_by_rating = {}
+    for target_rating in sorted(hits.keys()):
+        hit_rate = hits[target_rating] / total[target_rating]
+        hits_distribution_by_rating[target_rating] = hit_rate
+
+    if debug:
+        x_tick_names = list(hits_distribution_by_rating.keys())
+        x_tick_idx = list(range(len(x_tick_names)))
+        plt.figure(dpi=150)
+        plt.bar(
+            x_tick_idx,
+            [hit_rate for hit_rate in hits_distribution_by_rating.values()],
+            align='center'
+        )
+        plt.xticks(
+            list(range(len(hits_distribution_by_rating))), x_tick_names, fontsize=10
+        )
+        plt.savefig(os.path.join(current.report_path,
+                                'plots',
+                                'hit_distribution_rating.png'))
+        plt.clf()
+
+    return hits_distribution_by_rating
\ No newline at end of file
diff --git a/reclist/metrics/hits_slice.py b/reclist/metrics/hits_slice.py
deleted file mode 100644
index c934232..0000000
--- a/reclist/metrics/hits_slice.py
+++ /dev/null
@@ -1,38 +0,0 @@
-from collections import Counter, defaultdict
-from reclist.metrics.standard_metrics import hit_rate_at_k, sample_hits_at_k, sample_misses_at_k
-import matplotlib.pyplot as plt
-
-
-def hits_distribution_by_slice(slice_fns: dict,
-                               y_test,
-                               y_preds,
-                               product_data,
-                               k=3,
-                               sample_size=3,
-                               debug=False):
-
-    hit_rate_per_slice = defaultdict(dict)
-    for slice_name, filter_fn in slice_fns.items():
-        # get indices for slice
-        slice_idx = [idx for idx,_y in enumerate(y_test) if _y[0] in product_data and filter_fn(product_data[_y[0]])]
-        # get predictions for slice
-        slice_y_preds = [y_preds[_] for _ in slice_idx]
-        # get labels for slice
-        slice_y_test = [y_test[_] for _ in slice_idx]
-        # TODO: We may want to allow for generic metric to be used here
-        slice_hr = hit_rate_at_k(slice_y_preds, slice_y_test,k=3)
-        # store results
-        hit_rate_per_slice[slice_name]['hit_rate'] = slice_hr
-        hit_rate_per_slice[slice_name]['hits'] = sample_hits_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)
-        hit_rate_per_slice[slice_name]['misses'] = sample_misses_at_k(slice_y_preds, slice_y_test, k=k, size=sample_size)
-
-    # debug / visualization
-    if debug:
-        x_tick_names = list(hit_rate_per_slice.keys())
-        x_tick_idx = list(range(len(x_tick_names)))
-        plt.bar(x_tick_idx, hit_rate_per_slice.values(), align='center')
-        plt.xticks(list(range(len(hit_rate_per_slice))), x_tick_names)
-        plt.show()
-
-    # cast to normal dict
-    return dict(hit_rate_per_slice)
diff --git a/reclist/metrics/perturbation.py b/reclist/metrics/perturbation.py
index 4b16940..2cd9b87 100644
--- a/reclist/metrics/perturbation.py
+++ b/reclist/metrics/perturbation.py
@@ -1,54 +1,28 @@
-import random
 import numpy as np
-from tqdm import tqdm
 
-# TODO: We might need to enforce some standardization of how CATEGORY is represented
-def get_item_with_category(product_data: dict, category: set, to_ignore=None):
-    to_ignore = [] if to_ignore is None else to_ignore
-    skus = [_ for _ in product_data if product_data[_]['category_hash'] == category and _ not in to_ignore]
-    if skus:
-        return random.choice(skus)
-    return []
 
-
-def perturb_session(session, product_data):
-    last_item = session[-1]
-    if last_item not in product_data:
-        return []
-    last_item_category = product_data[last_item]['category_hash']
-    similar_item = get_item_with_category(product_data, last_item_category, to_ignore=[last_item])
-    if similar_item:
-        new_session = session[:-1] + [similar_item]
-        return new_session
-    return []
-
-
-def session_perturbation_test(model, x_test, y_preds, product_data):
+def session_perturbation_test(model,
+                              x_test,
+                              y_preds,
+                              perturbation_fn,
+                              id_fn,
+                              k):
+    # generate perturbations
+    perturbed_pairs = [(perturbation_fn(_x), _y_p) for _x, _y_p in zip(x_test, y_preds) if perturbation_fn(_x)]
+    # extract perturbed x and original y
+    x_test_p, y_preds_o = zip(*perturbed_pairs)
+    # make new predictions over perturbed inputs
+    y_preds_n = model.predict(x_test_p)
+    # extract atomic unit for comparison
+    y_preds_o, y_preds_n = id_fn(y_preds_o), id_fn(y_preds_n)
+    # check for overlapping predictions
     overlap_ratios = []
-    # print(product_data)
-    y_p = []
-    s_perturbs = []
-
-    # generate a batch of perturbations
-    for idx, (s, _y_p) in enumerate(tqdm(zip(x_test,y_preds))):
-        # perturb last item in session
-        s = [ _.split('_')[0] for _ in s]
-        s_perturb = perturb_session(s, product_data)
-        if not s_perturb:
-            continue
-
-        s_perturb = ['_'.join([_,'add']) for _ in s_perturb]
-
-        s_perturbs.append(s_perturb)
-        y_p.append(_y_p)
-
-    y_perturbs = model.predict(s_perturbs)
-
-    for _y_p, _y_perturb in zip(y_p, y_perturbs):
-        if _y_p and _y_perturb:
+    for _y_p_n, _y_p_o in zip(y_preds_o, y_preds_n):
+        if _y_p_n and _y_p_o:
             # compute prediction intersection
-            intersection = set(_y_perturb).intersection(_y_p)
-            overlap_ratio = len(intersection)/len(_y_p)
+            intersection = set(_y_p_o[:k]).intersection(_y_p_n[:k])
+            overlap_ratio = len(intersection) / len(_y_p_n[:k])
             overlap_ratios.append(overlap_ratio)
-
+        elif _y_p_n or _y_p_o:
+            overlap_ratios.append(0)
     return np.mean(overlap_ratios)
diff --git a/reclist/metrics/price_homogeneity.py b/reclist/metrics/price_homogeneity.py
index 77493ff..d21659c 100644
--- a/reclist/metrics/price_homogeneity.py
+++ b/reclist/metrics/price_homogeneity.py
@@ -1,7 +1,19 @@
 import numpy as np
 import matplotlib.pyplot as plt
+import os
+from reclist.current import current
 
-def price_homogeneity_test(y_test, y_preds, product_data, bins=25, debug=True, key='PRICE'):
+def price_homogeneity_test(y_test, y_preds, product_data, price_sel_fn, bins=25, debug=True):
+    """
+
+    :param y_test: test data
+    :param y_preds: predicted data
+    :param product_data: catalog data
+    :param bins: bins on which we split the data for the product comparison
+    :param debug: shows plots
+    :param key: key from the product data to find the product price
+    @return:
+    """
     abs_log_price_diff = []
     for idx, (_y, _y_pred) in enumerate(zip(y_test, y_preds)):
         # need >=1 predictions
@@ -10,19 +22,24 @@ def price_homogeneity_test(y_test, y_preds, product_data, bins=25, debug=True, k
         # item must be in product data
         if _y[0] not in product_data or _y_pred[0] not in product_data:
             continue
-        if product_data[_y[0]][key] and product_data[_y_pred[0]][key]:
-            pred_item_price = float(product_data[_y_pred[0]][key])
-            y_item_price = float(product_data[_y[0]][key])
-            if pred_item_price and y_item_price:
-                abs_log_price_diff.append(np.abs(np.log10(pred_item_price)-(np.log10(y_item_price))))
+        _y_info = product_data[_y[0]]
+        _y_pred_info = product_data[_y_pred[0]]
+        if price_sel_fn(_y_info) and price_sel_fn(_y_pred_info):
+            y_item_price = price_sel_fn(_y_info)
+            pred_item_price = price_sel_fn(_y_pred_info)
+            price_diff = np.abs(np.log10(pred_item_price/y_item_price))
+            abs_log_price_diff.append(price_diff)
 
     histogram = np.histogram(abs_log_price_diff, bins=bins, density=False)
     histogram = (histogram[0].tolist(), histogram[1].tolist())
     if debug:
-        # debug / viz
         plt.hist(abs_log_price_diff, bins=25)
-        plt.show()
+        plt.savefig(os.path.join(current.report_path,
+                                 'plots',
+                                 'price_homogeneity.png'))
+        plt.clf()
+
     return {
         'mean': np.mean(abs_log_price_diff),
         'histogram': histogram
-    }
\ No newline at end of file
+    }
diff --git a/reclist/metrics/standard_metrics.py b/reclist/metrics/standard_metrics.py
index 42ec376..3f83fc3 100644
--- a/reclist/metrics/standard_metrics.py
+++ b/reclist/metrics/standard_metrics.py
@@ -50,14 +50,31 @@ def sample_misses_at_k(y_preds, y_test, x_test=None, k=3, size=3):
     return random.sample(misses, k=size)
 
 
+def hit_rate_at_k_nep(y_preds, y_test, k=3):
+    y_test = [[k] for k in y_test]
+    return hit_rate_at_k(y_preds, y_test, k=k)
+
+
 def hit_rate_at_k(y_preds, y_test, k=3):
     hits = 0
     for _p, _y in zip(y_preds, y_test):
-        if _y[0] in _p[:k]:
+        if len(set(_p[:k]).intersection(set(_y))) > 0:
             hits += 1
     return hits / len(y_test)
 
 
+def mrr_at_k_nep(y_preds, y_test, k=3):
+    """
+    Computes MRR
+
+    :param y_preds: predictions, as lists of lists
+    :param y_test: target data, as lists of lists (eventually [[sku1], [sku2],...]
+    :param k: top-k
+    """
+    y_test = [[k] for k in y_test]
+    return mrr_at_k(y_preds, y_test, k=k)
+
+
 def mrr_at_k(y_preds, y_test, k=3):
     """
     Computes MRR
@@ -67,13 +84,14 @@ def mrr_at_k(y_preds, y_test, k=3):
     :param k: top-k
     """
     rr = []
-    y_test = [k[0] for k in y_test]
     for _p, _y in zip(y_preds, y_test):
-        if _y in _p[:k+1]:
-            rank = _p[:k+1].index(_y) + 1
-            rr.append(1/rank)
+        for rank, p in enumerate(_p[:k], start=1):
+            if p in _y:
+                rr.append(1 / rank)
+                break
         else:
             rr.append(0)
+    assert len(rr) == len(y_preds)
     return np.mean(rr)
 
 
@@ -100,3 +118,13 @@ def popularity_bias_at_k(y_preds, x_train, k=3):
         average_pop = sum(pop_map.get(_, 0.0) for _ in p[:k]) / len(p) if len(p) > 0 else 0
         all_popularity.append(average_pop)
     return sum(all_popularity) / len(y_preds)
+
+
+def precision_at_k(y_preds, y_test, k=3):
+    precision_ls = [len(set(_y).intersection(set(_p[:k]))) / len(_p) if _p else 1 for _p, _y in zip(y_preds, y_test)]
+    return np.average(precision_ls)
+
+
+def recall_at_k(y_preds, y_test, k=3):
+    recall_ls = [len(set(_y).intersection(set(_p[:k]))) / len(_y) if _y else 1 for _p, _y in zip(y_preds, y_test)]
+    return np.average(recall_ls)
\ No newline at end of file
diff --git a/reclist/reclist.py b/reclist/reclist.py
index c01ab86..8cc9968 100644
--- a/reclist/reclist.py
+++ b/reclist/reclist.py
@@ -1,5 +1,8 @@
+import collections
 from reclist.abstractions import RecList, rec_test
 from typing import List
+import random
+
 
 class CoveoCartRecList(RecList):
 
@@ -15,6 +18,20 @@ def basic_stats(self):
                           self._y_test,
                           self._y_preds)
 
+    @rec_test(test_type='price_homogeneity')
+    def price_test(self):
+        """
+        Measures the absolute log ratio of ground truth and prediction price
+        """
+        from reclist.metrics.price_homogeneity import price_homogeneity_test
+        return price_homogeneity_test(y_test=self.sku_only(self._y_test),
+                                      y_preds=self.sku_only(self._y_preds),
+                                      product_data=self.product_data,
+                                      price_sel_fn=lambda x: float(x['price_bucket'])
+                                                             if x['price_bucket']
+                                                             else None
+                                      )
+
     @rec_test(test_type='Coverage@10')
     def coverage_at_k(self):
         """
@@ -22,7 +39,6 @@ def coverage_at_k(self):
         recommends based on a set of sessions
         """
         from reclist.metrics.standard_metrics import coverage_at_k
-
         return coverage_at_k(self.sku_only(self._y_preds),
                              self.product_data,
                              k=10)
@@ -42,7 +58,7 @@ def hits_distribution(self):
         """
         Compute the distribution of hit-rate across product frequency in training data
         """
-        from reclist.metrics.hits_distribution import hits_distribution
+        from reclist.metrics.hits import hits_distribution
         return hits_distribution(self.sku_only(self._x_train),
                                  self.sku_only(self._x_test),
                                  self.sku_only(self._y_test),
@@ -55,11 +71,214 @@ def dist_to_query(self):
         """
         Compute the distribution of distance from query to label and query to prediction
         """
-        from reclist.metrics.cosine_distance_metrics import distance_to_query
+        from reclist.metrics.distance_metrics import distance_to_query
         return distance_to_query(self.rec_model,
                                  self.sku_only(self._x_test),
                                  self.sku_only(self._y_test),
                                  self.sku_only(self._y_preds), k=10, bins=25, debug=True)
 
-    def sku_only(self, l:List[List]):
+    def sku_only(self, l: List[List]):
         return [[e['product_sku'] for e in s] for s in l]
+
+
+class SpotifySessionRecList(RecList):
+
+    @rec_test(test_type='basic_stats')
+    def basic_stats(self):
+        """
+        Basic statistics on training, test and prediction data for Next Event Prediction
+        """
+        from reclist.metrics.standard_metrics import statistics
+        return statistics(self._x_train,
+                          self._y_train,
+                          self._x_test,
+                          self._y_test,
+                          self._y_preds)
+
+    @rec_test(test_type='HR@10')
+    def hit_rate_at_k(self):
+        """
+        Compute the rate at which the top-k predictions contain the item to be predicted
+        """
+        from reclist.metrics.standard_metrics import hit_rate_at_k
+        return hit_rate_at_k(self.uri_only(self._y_preds),
+                             self.uri_only(self._y_test),
+                             k=10)
+
+    @rec_test(test_type='perturbation_test')
+    def perturbation_at_k(self):
+        """
+        Compute average consistency in model predictions when inputs are perturbed
+        """
+        from reclist.metrics.perturbation import session_perturbation_test
+        from collections import defaultdict
+        from functools import partial
+
+        # Step 1: Generate a map from artist uri to track uri
+        substitute_mapping = defaultdict(list)
+        for track_uri, row in self.product_data.items():
+            substitute_mapping[row['artist_uri']].append(track_uri)
+
+        # Step 2: define a custom perturbation function
+        def perturb(session, sub_map):
+            last_item = session[-1]
+            last_item_artist = self.product_data[last_item['track_uri']]['artist_uri']
+            substitutes = set(sub_map.get(last_item_artist,[])) - {last_item['track_uri']}
+            if substitutes:
+                similar_item = random.sample(substitutes, k=1)
+                new_session = session[:-1] + [{"track_uri": similar_item[0]}]
+                return new_session
+            return []
+
+        # Step 3: call test
+        return session_perturbation_test(self.rec_model,
+                                         self._x_test,
+                                         self._y_preds,
+                                         partial(perturb, sub_map=substitute_mapping),
+                                         self.uri_only,
+                                         k=10)
+
+    @rec_test(test_type='shuffle_session')
+    def perturbation_shuffle_at_k(self):
+        """
+        Compute average consistency in model predictions when inputs are re-ordered
+        """
+        from reclist.metrics.perturbation import session_perturbation_test
+
+        # Step 1: define a custom perturbation function
+        def perturb(session):
+            return random.sample(session, len(session))
+
+        # Step 2: call test
+        return session_perturbation_test(self.rec_model,
+                                         self._x_test,
+                                         self._y_preds,
+                                         perturb,
+                                         self.uri_only,
+                                         k=10)
+
+    @rec_test(test_type='hits_distribution_by_slice')
+    def hits_distribution_by_slice(self):
+        """
+        Compute the distribution of hit-rate across various slices of data
+        """
+        from reclist.metrics.hits import hits_distribution_by_slice
+
+        len_map = collections.defaultdict(list)
+        for idx, playlist in enumerate(self._x_test):
+            len_map[len(playlist)].append(idx)
+        slices = collections.defaultdict(list)
+        bins = [(x * 5, (x + 1) * 5) for x in range(max(len_map) // 5 + 1)]
+        for bin_min, bin_max in bins:
+            for i in range(bin_min + 1, bin_max + 1, 1):
+                slices[f'({bin_min}, {bin_max}]'].extend(len_map[i])
+                del len_map[i]
+        assert len(len_map) == 0
+
+        return hits_distribution_by_slice(slices,
+                                          self.uri_only(self._y_test),
+                                          self.uri_only(self._y_preds),
+                                          debug=True)
+
+    @rec_test(test_type='Coverage@10')
+    def coverage_at_k(self):
+        """
+        Coverage is the proportion of all possible products which the RS
+        recommends based on a set of sessions
+        """
+        from reclist.metrics.standard_metrics import coverage_at_k
+        return coverage_at_k(self.uri_only(self._y_preds),
+                             self.product_data,
+                             # this contains all the track URIs from train and test sets
+                             k=10)
+
+    @rec_test(test_type='Popularity@10')
+    def popularity_bias_at_k(self):
+        """
+        Compute average frequency of occurrence across recommended items in training data
+        """
+        from reclist.metrics.standard_metrics import popularity_bias_at_k
+        return popularity_bias_at_k(self.uri_only(self._y_preds),
+                                    self.uri_only(self._x_train),
+                                    k=10)
+
+    @rec_test(test_type='MRR@10')
+    def mrr_at_k(self):
+        """
+        MRR calculates the mean reciprocal of the rank at which the first
+        relevant item was retrieved
+        """
+        from reclist.metrics.standard_metrics import mrr_at_k
+        return mrr_at_k(self.uri_only(self._y_preds),
+                        self.uri_only(self._y_test))
+
+    def uri_only(self, playlists: List[dict]):
+        return [[track['track_uri'] for track in playlist] for playlist in playlists]
+
+
+class MovieLensSimilarItemRecList(RecList):
+    @rec_test(test_type="stats")
+    def basic_stats(self):
+        """
+        Basic statistics on training, test and prediction data
+        """
+        from reclist.metrics.standard_metrics import statistics
+        return statistics(
+            self._x_train,
+            self._y_train,
+            self._x_test,
+            self._y_test,
+            self._y_preds
+        )
+
+    @rec_test(test_type='HR@10')
+    def hit_rate_at_k(self):
+        """
+        Compute the rate at which the top-k predictions contain the item to be predicted
+        """
+        from reclist.metrics.standard_metrics import hit_rate_at_k
+        return hit_rate_at_k(
+            self.movie_only(self._y_preds),
+            self.movie_only(self._y_test),
+            k=10
+        )
+
+    @rec_test(test_type='Coverage@10')
+    def coverage_at_k(self):
+        """
+        Coverage is the proportion of all possible products which the RS
+        recommends based on a set of movies and their respective ratings
+        """
+        from reclist.metrics.standard_metrics import coverage_at_k
+        return coverage_at_k(
+            self.movie_only(self._y_preds),
+            self.product_data,
+            k=10
+        )
+
+    @rec_test(test_type='hits_distribution')
+    def hits_distribution(self):
+        """
+        Compute the distribution of hit-rate across product frequency in training data
+        """
+        from reclist.metrics.hits import hits_distribution
+        return hits_distribution(
+            self.movie_only(self._x_train),
+            self.movie_only(self._x_test),
+            self.movie_only(self._y_test),
+            self.movie_only(self._y_preds),
+            k=10,
+            debug=True
+        )
+
+    @rec_test(test_type="hits_distribution_by_rating")
+    def hits_distribution_by_rating(self):
+        from reclist.metrics.hits_rating import hits_distribution_by_rating
+        return hits_distribution_by_rating(
+            self._y_test,
+            self._y_preds,
+            debug=True
+        )
+
+    def movie_only(self, movies):
+        return [[x["movieId"] for x in y] for y in movies]
diff --git a/reclist/recommenders/prod2vec.py b/reclist/recommenders/prod2vec.py
index b1eaf95..0d5d20c 100644
--- a/reclist/recommenders/prod2vec.py
+++ b/reclist/recommenders/prod2vec.py
@@ -1,3 +1,4 @@
+import numpy as np
 from reclist.abstractions import RecModel
 from reclist.utils.train_w2v import train_embeddings
 
@@ -6,15 +7,13 @@ class CoveoP2VRecModel(RecModel):
     """
     Implement of the prod2vec model through the standard RecModel interface.
 
-    Since init is ok, we just need to overwrite the prediction methods to get predictions
-    out of it.
     """
     def __init__(self, **kwargs):
         super().__init__(**kwargs)
 
     model_name = "prod2vec"
 
-    def train(self, products):
+    def train(self, products, iterations=15):
         """
         Takes a list of products in the COVEO dataset (with coveo dataset format) and trains a model
 
@@ -22,7 +21,7 @@ def train(self, products):
         :return:
         """
         x_train_skus = [[e['product_sku'] for e in s] for s in products]
-        self._model = train_embeddings(x_train_skus)
+        self._model = train_embeddings(x_train_skus, iterations=iterations)
 
     def predict(self, prediction_input: list, *args, **kwargs):
         """
@@ -41,7 +40,7 @@ def predict(self, prediction_input: list, *args, **kwargs):
             key_item = _x[0]['product_sku']
             nn_products = self._model.most_similar(key_item, topn=10) if key_item in self._model else None
             if nn_products:
-                predictions.append([{'product_sku':_[0]} for _ in nn_products])
+                predictions.append([{'product_sku': _[0]} for _ in nn_products])
             else:
                 predictions.append([])
 
@@ -52,3 +51,100 @@ def get_vector(self, product_sku):
             return list(self._model.get_vector(product_sku))
         except Exception as e:
             return []
+
+
+class SpotifyP2VRecModel(RecModel):
+    """
+    Implement of the prod2vec model through the standard RecModel interface.
+
+    Since init is ok, we just need to overwrite the prediction methods to get predictions
+    out of it.
+    """
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+    model_name = "prod2vec"
+
+    def train(self, playlists, iterations=15):
+        x_train_uris = [[track['track_uri'] for track in playlist] for playlist in playlists]
+        self._model = train_embeddings(x_train_uris, iterations=iterations)
+
+    def predict(self, prediction_input: list, *args, **kwargs):
+        """
+        Implement the abstract method.
+        NEP following industry best practices mentioned in https://arxiv.org/abs/2007.14906:
+        given a trained prod2vec, take all the seeded (or before-last) tracks to construct a
+        playlist vector by average pooling, and use kNN to predict the next items (or the last
+        item).
+        """
+        predictions = []
+        for _x in prediction_input:
+            embeddings = [self._model[track['track_uri']] for track in _x if track['track_uri'] in self._model]
+            if embeddings:
+                avg_playlist_vector = np.average(embeddings, axis=0)
+                nn_tracks = self._model.similar_by_vector(avg_playlist_vector, topn=100)
+                predictions.append([{'track_uri':_[0]} for _ in nn_tracks])
+            else:
+                predictions.append([])
+
+        return predictions
+
+    def get_vector(self, track_uri):
+        try:
+            return list(self._model.get_vector(track_uri))
+        except Exception as e:
+            return []
+
+class MovieLensP2VRecModel(RecModel):
+    """
+    Prod2Vec implementation for MovieLens 25M dataset
+    """
+    model_name = "prod2vec"
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+    def train(self, movies, iterations=15):
+        # Get the movie ID and rating for each movie for each unique user
+        x_train = [[(x["movieId"], x["rating"]) for x in y] for y in movies]
+        self._model = train_embeddings(x_train, iterations=15)
+
+    def predict(self, prediction_input, *args, **kwargs):
+        """
+        Predicts the top 10 similar items recommended for each user according
+        to the movies that they've watched and the ratings that they've given
+
+        :param prediction_input: a list of lists containing a dictionary for
+                                 each movie watched by that user
+        :return:
+        """
+        all_predictions = []
+        for movies in prediction_input:
+            predictions = []
+            emb_vecs = []
+            for movie in movies:
+                emb_vec = self.get_vector(movie)
+                if emb_vec:
+                    emb_vecs.append(emb_vec)
+            if emb_vecs:
+                # Calculate the average of all the latent vectors representing
+                # the movies watched by the user as is done in https://arxiv.org/abs/2007.14906
+                avg_emb_vec = np.mean(emb_vecs, axis=0)
+                nn_products = self.model.similar_by_vector(avg_emb_vec, topn=10)
+                for elem in nn_products:
+                    predictions.append({"movieId": elem})
+            all_predictions.append(predictions)
+        return all_predictions
+
+    def get_vector(self, x):
+        """
+        Returns the latent vector that corresponds to the movie ID
+
+        :param x:
+        :return:
+        """
+        movie_id = x["movieId"]
+        try:
+            return list(self.model.get_vector(movie_id))
+        except Exception as e:
+            return []
diff --git a/reclist/utils/config.py b/reclist/utils/config.py
index 99099ea..d7d8fe6 100644
--- a/reclist/utils/config.py
+++ b/reclist/utils/config.py
@@ -5,7 +5,7 @@
 from enum import Enum
 
 COVEO_INTERACTION_DATASET_S3_URL = 'https://reclist-datasets-6d3c836d-6djh887d.s3.us-west-2.amazonaws.com/coveo_sigir.zip'
-SPOTIFY_PLAYLIST_DATASET_S3_URL = 'https://reclist-datasets-6d3c836d-6djh887d.s3.us-west-2.amazonaws.com/spotify_playlist.zip'
+SPOTIFY_PLAYLIST_DATASET_S3_URL = 'https://reclist-datasets-6d3c836d-6djh887d.s3.us-west-2.amazonaws.com/small_spotify_playlist.zip'
 MOVIELENS_DATASET_S3_URL = "https://reclist-datasets-6d3c836d-6djh887d.s3.us-west-2.amazonaws.com/movielens_25m.zip"
 
 
diff --git a/reclist/utils/train_w2v.py b/reclist/utils/train_w2v.py
index 4e1b94b..f59ab54 100644
--- a/reclist/utils/train_w2v.py
+++ b/reclist/utils/train_w2v.py
@@ -30,6 +30,6 @@ def train_embeddings(
                                    ns_exponent=ns_exponent)
 
     if is_debug:
-        print("# products in the space: {}".format(len(model.wv.index_to_key)))
+        print("# items in the space: {}".format(len(model.wv.index_to_key)))
 
     return model.wv


Test patch:
diff --git a/reclist/metrics/graph_distance_test.py b/reclist/metrics/graph_distance_test.py
deleted file mode 100644
index be2c43f..0000000
--- a/reclist/metrics/graph_distance_test.py
+++ /dev/null
@@ -1,56 +0,0 @@
-import json
-import networkx as nx
-from networkx.algorithms.shortest_paths.generic import shortest_path
-from statistics import mean
-from reclist.metrics.standard_metrics import sample_misses_at_k, sample_hits_at_k
-import numpy as np
-
-
-def shortest_path_length():
-    pass
-
-
-get_nodes = lambda nodes, ancestors="": [] if not nodes else ['_'.join([ancestors, nodes[0]])] + \
-                                                             get_nodes(nodes[1:], '_'.join([ancestors, nodes[0]]))
-
-
-def graph_distance_test(y_test, y_preds, product_data, k=3):
-    path_lengths = []
-    misses = sample_misses_at_k(y_preds, y_test, k=k, size=-1)
-    for _y, _y_p in zip([_['Y_TEST'] for _ in misses],
-                        [_['Y_PRED'] for _ in misses]):
-        if not _y_p:
-            continue
-        _y_sku = _y[0]
-        _y_p_sku = _y_p[0]
-
-        if _y_sku not in product_data or _y_p_sku not in product_data:
-            continue
-        if not product_data[_y_sku]['CATEGORIES'] or not product_data[_y_p_sku]['CATEGORIES']:
-            continue
-        # extract graph information
-        catA = json.loads(product_data[_y_sku]['CATEGORIES'])
-        catB = json.loads(product_data[_y_p_sku]['CATEGORIES'])
-        catA_nodes = [get_nodes(c) for c in catA]
-        catB_nodes = [get_nodes(c) for c in catB]
-        all_nodes = list(set([n for c in catA_nodes + catB_nodes for n in c]))
-        all_edges = [(n1, n2) for c in catA_nodes + catB_nodes for n1, n2 in zip(c[:-1], c[1:])]
-        all_edges = list(set(all_edges))
-
-        # build graph
-        G = nx.Graph()
-        G.add_nodes_from(all_nodes)
-        G.add_edges_from(all_edges)
-
-        # get leaves
-        cat1_leaves = [c[-1] for c in catA_nodes]
-        cat2_leaves = [c[-1] for c in catB_nodes]
-
-        all_paths = [shortest_path(G, c1_l, c2_l) for c1_l in cat1_leaves for c2_l in cat2_leaves]
-        s_path = min(all_paths, key=len)
-        s_path_len = len(s_path) - 1
-        path_lengths.append(s_path_len)
-
-    histogram = np.histogram(path_lengths, bins=np.arange(0, max(path_lengths)))
-    histogram = (histogram[0].tolist(), histogram[1].tolist())
-    return {'mean': mean(path_lengths), 'hist': histogram}
diff --git a/tests/test_reclist.py b/tests/test_reclist.py
index c4735e3..19d53bd 100644
--- a/tests/test_reclist.py
+++ b/tests/test_reclist.py
@@ -4,23 +4,26 @@
 
 from reclist.datasets import *
 from reclist.metrics.standard_metrics import mrr_at_k
-from reclist.datasets import CoveoDataset
-from reclist.recommenders.prod2vec import CoveoP2VRecModel
-from reclist.reclist import CoveoCartRecList
+from reclist.datasets import CoveoDataset, SpotifyDataset, MovieLensDataset
+from reclist.recommenders.prod2vec import CoveoP2VRecModel, SpotifyP2VRecModel, MovieLensP2VRecModel
+from reclist.reclist import CoveoCartRecList, SpotifySessionRecList, MovieLensSimilarItemRecList
+import random
 
 def test_basic_dataset_downloading():
     CoveoDataset()
     MovieLensDataset()
+    MovieLensDataset()
 
 
-def test_examples():
+def test_coveo_example():
     # get the coveo data challenge dataset as a RecDataset object
     coveo_dataset = CoveoDataset()
 
     # re-use a skip-gram model from reclist to train a latent product space, to be used
     # (through knn) to build a recommender
     model = CoveoP2VRecModel()
-    model.train(coveo_dataset.x_train)
+    x_train = random.sample(coveo_dataset.x_train, 2000)
+    model.train(x_train, iterations=1)
 
     # instantiate rec_list object, prepared with standard quantitative tests
     # and sensible behavioral tests (check the paper for details!)
@@ -28,6 +31,7 @@ def test_examples():
         model=model,
         dataset=coveo_dataset
     )
+
     # invoke rec_list to run tests
     rec_list(verbose=True)
 
@@ -40,6 +44,6 @@ def test_mrr():
     list_b = [[0, 1], [1, 0]]
     list_c = [[2, 3], [0, 1]]
 
-    assert mrr_at_k(list_b, list_a, 0) == 1
-    assert mrr_at_k(list_c, list_a, 20) == 0.25
-    assert mrr_at_k(list_c, list_a, 0) == 0
+    assert mrr_at_k(list_b, list_a, 1) == 1
+    assert mrr_at_k(list_c, list_a, 21) == 0.25
+    assert mrr_at_k(list_c, list_a, 1) == 0