Repository: mlflow/mlflow

Issue:
[BUG] Can't install on python 3.11
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

- Client: 2.1.1
- Tracking server: n/a


### System information

- macOS 13.1:
- Python Version: 3.11.1:


### Describe the problem

Can't install mlflow because numba is required which depends on llvmlite which is not yet python 3.11 compatible (https://github.com/numba/numba/issues/8304) (Can this numba dependency be optional for the time being?)
The error is that llvmlite explicitly mentions that it needs python <3.11 when installing mlflow.

### What language(s) does this bug affect?

- [x] `language/python`: Python APIs and clients


Discussion:
+1 for moving shap to an extra in general if there's a non-intrusive way to do so. It brings in quite a heavy transitive deps chain that only appears needed in one spot. 

* Dropping [numba](https://pypi.org/project/numba/#files) and [llvmlite](https://pypi.org/project/llvmlite/#files) could shave about 25MB off the download size for those who don't need it. 
* Numba/llvmlite make handling numpy updates more difficult since they require pretty specific versions. An [example pr](https://github.com/numba/numba/pull/8620) adding support for numpy 1.24.

Example at the time of writing this:

```bash
python --version
Python 3.10.9

# Fails to compile from source on windows
python -m pip install mlflow==2.1.1 numpy==1.24.1

# Fails to compile numba and llvmlite from source 
python -m pip install numba llvmlite numpy==1.24.1

# works as expected with prior versions of numpy
python -m pip install numba llvmlite 'numpy>=1.23.0'
```

Not sure how one might pull this off without "breaking" changes though.
I have windows 10.  I can try and install it there if that would help.  I haven't been able to get numba to install, though.  I have python 3.11.1.
It's unlikely that it is different depending on the OS, no testing needed. Numba/llvmlite have not yet finished adding support for python 3.11. As that might take a while, making it an optional dependency is maybe the better way to fix this now.
Is there a pip command to install this now or can't it install because of Numba/llvmlite?
@kjyv Looks like shap depends on numba. Can you raise an issue in the shap repository?

cc @sunishsheth2009 can you file a ticket to track moving SHAP to an extra (like it was before) and gate recipes functionality based on whether or not shap is installed (if not installed, MLflow Recipes won't log model explanations).
<!-- assign-maintainer -->
@BenWilson2 @dbczumar @harupy @WeichenXu123 Please assign a maintainer and start triaging this issue.
llvmlite error is also occurs on python3.10
+1 for moving shap to extras
+1

I'd like to do a simple `pip install mlflow` without dealing with complex C/C++ dependencies.
+1
> cc @sunishsheth2009 can you file a ticket to track moving SHAP to an extra (like it was before) and gate recipes functionality based on whether or not shap is installed (if not installed, MLflow Recipes won't log model explanations).

@dbczumar is there work internally for this? I am happy to start a PR but don't want to duplicate anything.

Resolving patch:
diff --git a/mlflow/recipes/steps/evaluate.py b/mlflow/recipes/steps/evaluate.py
--- a/mlflow/recipes/steps/evaluate.py
+++ b/mlflow/recipes/steps/evaluate.py
@@ -391,25 +391,37 @@ def get_icon(validated):
                 classifiers_plot_tab.add_image("ROC_CURVE_PLOT", roc_curve_path, width=400)
 
         # Tab 3: SHAP plots.
-        shap_plot_tab = card.add_tab(
-            "Feature Importance",
-            '<h3 class="section-title">Feature Importance on Validation Dataset</h3>'
-            '<h3 class="section-title">SHAP Bar Plot</h3>{{SHAP_BAR_PLOT}}'
-            '<h3 class="section-title">SHAP Beeswarm Plot</h3>{{SHAP_BEESWARM_PLOT}}',
-        )
+        def _add_shap_plots(card):
+            """Contingent on shap being installed."""
+            shap_plot_tab = card.add_tab(
+                "Feature Importance",
+                '<h3 class="section-title">Feature Importance on Validation Dataset</h3>'
+                '<h3 class="section-title">SHAP Bar Plot</h3>{{SHAP_BAR_PLOT}}'
+                '<h3 class="section-title">SHAP Beeswarm Plot</h3>{{SHAP_BEESWARM_PLOT}}',
+            )
 
-        shap_bar_plot_path = os.path.join(
-            output_directory,
-            "eval_validation/artifacts",
-            f"{_VALIDATION_METRIC_PREFIX}shap_feature_importance_plot.png",
-        )
-        shap_beeswarm_plot_path = os.path.join(
-            output_directory,
-            "eval_validation/artifacts",
-            f"{_VALIDATION_METRIC_PREFIX}shap_beeswarm_plot.png",
-        )
-        shap_plot_tab.add_image("SHAP_BAR_PLOT", shap_bar_plot_path, width=800)
-        shap_plot_tab.add_image("SHAP_BEESWARM_PLOT", shap_beeswarm_plot_path, width=800)
+            shap_bar_plot_path = os.path.join(
+                output_directory,
+                "eval_validation/artifacts",
+                f"{_VALIDATION_METRIC_PREFIX}shap_feature_importance_plot.png",
+            )
+            shap_beeswarm_plot_path = os.path.join(
+                output_directory,
+                "eval_validation/artifacts",
+                f"{_VALIDATION_METRIC_PREFIX}shap_beeswarm_plot.png",
+            )
+            shap_plot_tab.add_image("SHAP_BAR_PLOT", shap_bar_plot_path, width=800)
+            shap_plot_tab.add_image("SHAP_BEESWARM_PLOT", shap_beeswarm_plot_path, width=800)
+
+        try:
+            import shap  # pylint: disable=W0611
+            from matplotlib import pyplot  # pylint: disable=W0611
+
+            _add_shap_plots(card)
+        except ImportError:
+            _logger.warning(
+                "SHAP or matplotlib package is not installed, so shap plots will not be added."
+            )
 
         # Tab 3: Warning log outputs.
         warning_output_path = os.path.join(output_directory, "warning_logs.txt")


Test patch: