Repository: pass-culture/data-gcp
Pull request #3981: (DS) feat(metrics): enhanced metrics plots readibility and added baseline metrics to plots
Author: login: cibrahim-pass
Created: 2025-03-28T15:23:18Z | Merged: 2025-03-31T07:46:34Z

…for baselines

# DS PR

## Describe your changes

Please include a summary of the changes:

* This PR removes novelty from previous metrics plots to a different plot 
* new plots for each basline
* a separate plot to compare recall between baseline models and two tower model

## Jira ticket number and/or notion link

JIRA-ticket_number

### Type of change

* [ ] hotfix (non-breaking change which fixes an issue)
* [ ] New model
* [ ] Training
* [x] New features
* [ ] Bug Fix
* [ ] Code Refacto
* [ ] Performance Improvements
* [ ] Test
* [ ] CI
* [ ] Config

### Checklist before requesting a review

* [x] I have performed a self-review of my code
* [x] My code passes CI/CD tests
* [ ] I updated README.md
* [ ] I have updated the dag
* [ ] If my changes concern incremental table, I have altered their schema to accomodate with field's creation/deletion
* [ ] I have documented the corresponding [notion page](https://www.notion.so/passcultureapp/Team-Data-engineering-Data-science-22ab0eb5ddf34dc2a854d9f0e596e91b)
* [ ] I will create a review on slack and ensure to specify the duration of the review task: short (<10min), medium (<30min), long (>30min)

### Added tests?

* [x] 👍 yes
* [ ] 🙅 no, because they aren't needed
* [ ] 🙋 no, because I need help
* [ ] ⏰ no, but I created a ticket

### PR title format (except for MEP)

There is a linter on the PR title format. Please respect the following format:

<details>
<summary>(ticket) type(topic): comment</summary>

* ticket surrounded by parenthesis, with optionnaly a hyphen followed by one or more digits (e.g., -1234). The first part must be one of the following strings:
  * DA
  * DE
  * AE
  * DS
  * HF
  * BSR
  * PC

* type :
The second part to specify the type of change one of the following :
  * build
  * lint
  * ci
  * docs
  * feat
  * fix
  * perf
  * refactor
  * test
  * chore
  * dbt

* topic within parenthesis: 1 word e.g., (dag)

* comment: tell us your life

examples:

* :white_check_mark: (DE-124) refactor(firebase): update source field
* :x: (DE-124) refactor (firebase): update source field **(space between type and topic)**
* :x: (DE-124) airflow(firebase): update source fiedd in DAG **(wrong type)**
* :x: (DE-124) (DE-124) refactor(firebase refacto): update source field **(topic in two words)**
* :white_check_mark: (BSR) docs(github): add PR title valid format in template

</details>



Diff:
diff --git a/jobs/ml_jobs/algo_training/two_towers_model/evaluate.py b/jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
index b58508d494..26d7a319c7 100644
--- a/jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
+++ b/jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
@@ -22,6 +22,7 @@
 from two_towers_model.utils.evaluate import (
     evaluate,
     plot_metrics_evolution,
+    plot_recall_comparison,
     save_pca_representation,
 )
 
@@ -96,10 +97,17 @@ def main(
         item_data=item_data,
         figures_folder=pca_plots_path,
     )
+
     # Create metrics evolution plots
     metrics_plots_path = f"{MODEL_DIR}/metrics_plots/"
     os.makedirs(metrics_plots_path, exist_ok=True)
-    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path)
+    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="")
+
+    # Create metrics evolution plots for popular and random baselines if dummy is True
+    if dummy:
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="popular_")
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="random_")
+        plot_recall_comparison(metrics, LIST_K, metrics_plots_path)
 
     connect_remote_mlflow()
     with mlflow.start_run(
diff --git a/jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py b/jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py
index 31c6ebfc5c..228a0dbca3 100644
--- a/jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py
+++ b/jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py
@@ -481,63 +481,135 @@ def save_pca_representation(
     fig.savefig(figures_folder + "ALL_CATEGORIES.pdf")
 
 
-def plot_metrics_evolution(metrics, list_k, figures_folder):
+def plot_metrics_evolution(
+    metrics: Dict[str, float], list_k: List[int], figures_folder: str, prefix: str
+):
     """
-    Plot the evolution of metrics with different k values
+    Plot the evolution of precision, recall, coverage and novelty with different k values in two separate plots
+    (one for precision, recall and coverage, one for novelty)
+    The prefix is used to identify the metrics to plot (popular_, random_, or empty string for the two towers model)
 
     Args:
         metrics: Dictionary containing metrics for different k values
+        list_k: List of k values
         figures_folder: Folder to save the plot
+        prefix: Prefix of the metrics (popular_, random_, or empty string)
 
     Returns:
         None
     """
-    logger.info("Creating metrics evolution plot")
+    logger.info("Creating metrics evolution plots")
 
     # Prepare data for plotting
     precision_values = [
         metrics[metric_name]
         for metric_name in metrics.keys()
-        if metric_name.startswith("precision")
+        if metric_name.startswith(f"{prefix}precision")
     ]
     recall_values = [
         metrics[metric_name]
         for metric_name in metrics.keys()
-        if metric_name.startswith("recall")
+        if metric_name.startswith(f"{prefix}recall")
     ]
     coverage_values = [
         metrics[metric_name]
         for metric_name in metrics.keys()
-        if metric_name.startswith("coverage")
+        if metric_name.startswith(f"{prefix}coverage")
     ]
     novelty_values = [
         metrics[metric_name]
         for metric_name in metrics.keys()
-        if metric_name.startswith("novelty")
+        if metric_name.startswith(f"{prefix}novelty")
     ]
 
-    # Create plot
+    # Create plot for precision, recall and coverage
     fig, ax = plt.subplots(figsize=(12, 8))
 
     # Plot each metric
     ax.plot(list_k, precision_values, marker="o", label="Precision")
     ax.plot(list_k, recall_values, marker="s", label="Recall")
     ax.plot(list_k, coverage_values, marker="^", label="Coverage")
-    ax.plot(list_k, novelty_values, marker="d", label="Novelty")
 
     ax.set_xlabel("k (Number of recommendations)")
     ax.set_ylabel("Score")
-    ax.set_title("Evolution of Metrics with k")
+    ax.set_title(f"Evolution of {prefix} Metrics with the value of k")
     ax.grid(True)
     ax.legend()
 
-    # Save the plot
-    plot_path = f"{figures_folder}/metrics_evolution.png"
+    # Save the plot for precision, recall and coverage
+    plot_path = f"{figures_folder}/{prefix}metrics_evolution.png"
+    fig.savefig(plot_path)
+
+    # Create plot for novelty
+    fig, ax = plt.subplots(figsize=(12, 8))
+    ax.plot(list_k, novelty_values, marker="d", label="Novelty")
+    ax.set_xlabel("k (Number of recommendations)")
+    ax.set_ylabel("Score")
+    ax.set_title(f"Evolution of {prefix} Novelty with the value of k")
+    ax.grid(True)
+    ax.legend()
+
+    # Save the plot for novelty
+    plot_path = f"{figures_folder}/{prefix}novelty_evolution.png"
     fig.savefig(plot_path)
-    logger.info(f"Metrics evolution plot saved to {plot_path}")
 
+    logger.info(f"Metrics evolution plots saved to {figures_folder}")
+
+
+def plot_recall_comparison(
+    metrics: Dict[str, float], list_k: List[int], figures_folder: str
+):
+    """
+    Plot the evolution of recall with different k values for the two towers model and the baselines (random and popular)
+    Only relevant if dummy is True
+
+    Args:
+        metrics: Dictionary containing metrics for different k values
+        list_k: List of k values
+        figures_folder: Folder to save the plot
+    """
+    logger.info("Creating recall comparison plots")
+
+    # Prepare data for plotting
+    two_towers_recall = [
+        metrics[metric_name]
+        for metric_name in metrics.keys()
+        if metric_name.startswith("recall")
+    ]
+    random_recall = [
+        metrics[metric_name]
+        for metric_name in metrics.keys()
+        if metric_name.startswith("random_recall")
+    ]
+    popular_recall = [
+        metrics[metric_name]
+        for metric_name in metrics.keys()
+        if metric_name.startswith("popular_recall")
+    ]
+
+    # Create plot for recall comparison
+    fig, ax = plt.subplots(figsize=(12, 8))
+
+    # Plot two towers recall
+    ax.plot(list_k, two_towers_recall, marker="o", label="Two Towers")
+
+    # Plot random recall
+    ax.plot(list_k, random_recall, marker="s", label="Random")
+
+    # Plot popular recall
+    ax.plot(list_k, popular_recall, marker="^", label="Popular")
+
+    ax.set_xlabel("k (Number of recommendations)")
+    ax.set_ylabel("Recall")
+    ax.set_title("Recall Comparison between Two Towers and Baselines")
+    ax.grid(True)
+    ax.legend()
+
+    # Save the plot
+    plot_path = f"{figures_folder}/recall_comparison.png"
+    fig.savefig(plot_path)
 
-## Baselines: random and most popular
+    logger.info(f"Recall comparison plot saved to {figures_folder}")
 
 
 def generate_random_baseline(


Reviews:
user: login: lmontier-pass
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-31T06:54:33Z
user: login: cibrahim-pass
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T07:21:58Z
user: login: cibrahim-pass
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T07:44:16Z
user: login: lmontier-pass
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-31T07:45:20Z
user: login: lmontier-pass
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-31T07:45:44Z

Review comments:
diff_hunk: @@ -96,10 +97,17 @@ def main(
         item_data=item_data,
         figures_folder=pca_plots_path,
     )
+
     # Create metrics evolution plots
     metrics_plots_path = f"{MODEL_DIR}/metrics_plots/"
     os.makedirs(metrics_plots_path, exist_ok=True)
-    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path)
+    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="")
+
+    # Create metrics evolution plots for popular and random baselines if dummy is True
+    if dummy:
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="popular_")
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="random_")
+        plot_recall_comparison(metrics, LIST_K, metrics_plots_path)
path: jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
user: login: lmontier-pass
body: y a pas un doublon ici ? l.104 vs l.110?
created_at: 2025-03-31T06:53:35Z
author_association: COLLABORATOR
diff_hunk: @@ -481,63 +481,131 @@ def save_pca_representation(
     fig.savefig(figures_folder + "ALL_CATEGORIES.pdf")
 
 
-def plot_metrics_evolution(metrics, list_k, figures_folder):
+def plot_metrics_evolution(metrics, list_k, figures_folder, prefix):
path: jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py
user: login: lmontier-pass
body: un petit typing de la fct?
created_at: 2025-03-31T06:53:55Z
author_association: COLLABORATOR
diff_hunk: @@ -96,10 +97,17 @@ def main(
         item_data=item_data,
         figures_folder=pca_plots_path,
     )
+
     # Create metrics evolution plots
     metrics_plots_path = f"{MODEL_DIR}/metrics_plots/"
     os.makedirs(metrics_plots_path, exist_ok=True)
-    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path)
+    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="")
+
+    # Create metrics evolution plots for popular and random baselines if dummy is True
+    if dummy:
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="popular_")
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="random_")
+        plot_recall_comparison(metrics, LIST_K, metrics_plots_path)
path: jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
user: login: cibrahim-pass
body: Non, le plot_metrics_evolution fait deux plots pour un même dataframe de prédictions (TT, random ou popular), un plot pour recall, precision et coverage et un autre pour novelty (novelty a des valeur >> 1 donc écrase les autres métriques sur un même plot). la fonction plot_recall_comparison ne plot que le recall entre les 3 différents datafarles de pred (tt, random et baseline) pour pouvoir comparer le recall des 3 "modèles".
created_at: 2025-03-31T07:21:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -481,63 +481,131 @@ def save_pca_representation(
     fig.savefig(figures_folder + "ALL_CATEGORIES.pdf")
 
 
-def plot_metrics_evolution(metrics, list_k, figures_folder):
+def plot_metrics_evolution(metrics, list_k, figures_folder, prefix):
path: jobs/ml_jobs/algo_training/two_towers_model/utils/evaluate.py
user: login: cibrahim-pass
body: done
created_at: 2025-03-31T07:44:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -96,10 +97,17 @@ def main(
         item_data=item_data,
         figures_folder=pca_plots_path,
     )
+
     # Create metrics evolution plots
     metrics_plots_path = f"{MODEL_DIR}/metrics_plots/"
     os.makedirs(metrics_plots_path, exist_ok=True)
-    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path)
+    plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="")
+
+    # Create metrics evolution plots for popular and random baselines if dummy is True
+    if dummy:
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="popular_")
+        plot_metrics_evolution(metrics, LIST_K, metrics_plots_path, prefix="random_")
+        plot_recall_comparison(metrics, LIST_K, metrics_plots_path)
path: jobs/ml_jobs/algo_training/two_towers_model/evaluate.py
user: login: lmontier-pass
body: ah pardon mal lu :) 
created_at: 2025-03-31T07:45:20Z
author_association: COLLABORATOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-03-31T07:45:05Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=pass-culture_data-gcp&pullRequest=3981) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=pass-culture_data-gcp&pullRequest=3981&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=pass-culture_data-gcp&pullRequest=3981&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=pass-culture_data-gcp&pullRequest=3981&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=pass-culture_data-gcp&pullRequest=3981&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=pass-culture_data-gcp&pullRequest=3981&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=pass-culture_data-gcp&pullRequest=3981)


author_association: NONE