Repository: ydataai/ydata-profiling

Issue:
High correlation warning printed multiple times
I get the same warning "High correlation" with the same other column four times in the report.

Looks like a bug where the warning is accidentally generated multiple times or not de-duplicated properly.

Is it easy to spot the issue or reproduce? Or should I try to extract a standalone test case?

This is with pandas 1.3.0 and pandas-profiling 3.0.0.

<img width="572" alt="Screenshot 2021-09-05 at 18 54 44" src="https://user-images.githubusercontent.com/852409/132135015-45c0a273-763a-430e-b12f-d340e79b3ea7.png">



Discussion:
The alert is probably generated for each active correlation. This in itself is not wrong, however it should be clear. This can be improved by either merging the alerts or clarifying with respect to which coefficient there is a high correlation.

Feel free to contribute a PR.
What do you mean by "active correlation"?

There are multiple alerts under "alite" saying multiple times `HIGH_CORRELATION` with the same column "belite".
That should be a bug, no?

I tried to have a quick look at the code to see if I can spot how this can happen:
https://github.com/pandas-profiling/pandas-profiling/blob/113277ede126e52717c1ee54014ed8f91fce5d02/src/pandas_profiling/model/alerts.py#L293-L309

Not clear to me what's going on. I can try to debug this and create a test case in the next days.

Resolving patch:
diff --git a/src/pandas_profiling/model/alerts.py b/src/pandas_profiling/model/alerts.py
--- a/src/pandas_profiling/model/alerts.py
+++ b/src/pandas_profiling/model/alerts.py
@@ -293,19 +293,24 @@ def check_variable_alerts(config: Settings, col: str, description: dict) -> List
 def check_correlation_alerts(config: Settings, correlations: dict) -> List[Alert]:
     alerts = []
 
+    correlations_consolidated = {}
     for corr, matrix in correlations.items():
         if config.correlations[corr].warn_high_correlations:
             threshold = config.correlations[corr].threshold
             correlated_mapping = perform_check_correlation(matrix, threshold)
-            if len(correlated_mapping) > 0:
-                for k, v in correlated_mapping.items():
-                    alerts.append(
-                        Alert(
-                            column_name=k,
-                            alert_type=AlertType.HIGH_CORRELATION,
-                            values={"corr": corr, "fields": v},
-                        )
-                    )
+            for col, fields in correlated_mapping.items():
+                set(fields).update(set(correlated_mapping.get(col, [])))
+                correlations_consolidated[col] = fields
+
+    if len(correlations_consolidated) > 0:
+        for col, fields in correlations_consolidated.items():
+            alerts.append(
+                Alert(
+                    column_name=col,
+                    alert_type=AlertType.HIGH_CORRELATION,
+                    values={"corr": 'Overall', "fields": fields},
+                )
+            )
     return alerts
 
 


Test patch:
diff --git a/tests/issues/test_issue824.py b/tests/issues/test_issue824.py
new file mode 100644
--- /dev/null
+++ b/tests/issues/test_issue824.py
@@ -0,0 +1,44 @@
+"""
+Test for issue 824:
+https://github.com/ydataai/pandas-profiling/issues/824
+
+High correlation warning printed multiple times
+
+Problem :
+The alert is probably generated for each active correlation calculation
+
+Solution :
+Field will be marked once is labelled as highly correlated for any calculation
+"""
+import pandas as pd
+import pytest
+
+import pandas_profiling
+from pandas_profiling import ProfileReport
+from pandas_profiling.model.alerts import AlertType
+
+
+@pytest.mark.skip()
+def test_issue824():
+
+    # Minimal reproducible code
+
+    df = pd.DataFrame.from_dict(
+        {
+
+            "integer1": pd.Series([3, 4, 5, 6], dtype="int"),
+            "integer2": pd.Series([3, 4, 5, 6], dtype="int"),
+            "integer3": pd.Series([3, 4, 5, 6], dtype="int"),
+            "integer4": pd.Series([3, 4, 5, 6], dtype="int"),
+            "integer5": pd.Series([3, 4, 5, 6], dtype="int"),
+
+
+        }
+    )
+
+
+    report = ProfileReport(df)
+    _ = report.description_set
+
+    assert len([a for a in report.description_set['alerts'] if a.alert_type == AlertType.HIGH_CORRELATION]) == 5
+