Repository: skfolio/skfolio

Issue:
[BUG] Solver 'CLARABEL' failed for Maximum Sharpe Ratio model
**Describe the bug**
When attempting to run the Maximum Sharpe Ratio optimization model with custom data, the solver fails with a CLARABEL error. Other models like Inverse Volatility and Maximum Diversification work correctly with the same dataset.

**To Reproduce**
1. Using the example code from [skfolio's Maximum Sharpe documentation](https://skfolio.org/auto_examples/1_mean_risk/plot_1_maximum_sharpe_ratio.html)
2. Replacing the sample data with custom dataset
3. Running the optimization

X.json file (custom data) can be found here: https://gist.github.com/drager/242523cce342d6b4f7d25eed0023997d

**Error message**
```
            embracer-group-ab-b  2curex-ab
2022-01-13             0.021389  -0.009544
2022-01-14             0.018295  -0.037473
2022-01-17            -0.012429  -0.045606
2022-01-18             0.030435   0.048951
2022-01-19            -0.017322   0.005556
Traceback (most recent call last):
  File "/home/drager/development/skfolio-issue/plot_1_maximum_sharpe_ratio.py", line 59, in <module>
    model.fit(X_train)
  File "/home/drager/.local/lib/python3.10/site-packages/skfolio/optimization/convex/_mean_risk.py", line 1086, in fit
    self._solve_problem(
  File "/home/drager/.local/lib/python3.10/site-packages/skfolio/optimization/convex/_base.py", line 1162, in _solve_problem
    raise cp.SolverError(msg) from None
cvxpy.error.SolverError: Solver 'CLARABEL' failed. Try another solver, or solve with solver_params=dict(verbose=True) for more information
```

See code below:

```python
"""
====================
Maximum Sharpe Ratio
====================

This tutorial uses the :class:`~skfolio.optimization.MeanRisk` optimization to find the
maximum Sharpe Ratio portfolio.
"""

import numpy as np
from plotly.io import show
from sklearn.model_selection import train_test_split

from skfolio import Population, RiskMeasure
from skfolio.datasets import load_sp500_dataset
from skfolio.optimization import InverseVolatility, MeanRisk, ObjectiveFunction
from skfolio.preprocessing import prices_to_returns
import pandas as pd
import json

# prices = load_sp500_dataset()

# Load file from X.json
data_dict = json.load(open('X.json'))
df = pd.DataFrame()
for stock, values in data_dict.items():
    dates = [val[0] for val in values]
    prices = [val[1] for val in values]
    temp_df = pd.DataFrame(
        prices, index=pd.to_datetime(dates), columns=[stock])
    df = df.join(temp_df, how='outer')

X = prices_to_returns(df)
X_train, X_test = train_test_split(X, test_size=0.33, shuffle=False)

print(X_train.head())

# %%
# Model
# =====
# We create a Maximum Sharpe Ratio model and then fit it on the training set.
# `portfolio_params` are parameters passed to the :class:`~skfolio.portfolio.Portfolio`
# returned by the `predict` method. It can be
# omitted, here we use it to give a name to our maximum Sharpe Ration portfolio:
model = MeanRisk(
    risk_measure=RiskMeasure.STANDARD_DEVIATION,
    objective_function=ObjectiveFunction.MAXIMIZE_RATIO,
    portfolio_params=dict(name="Max Sharpe"),
)
model.fit(X_train)
model.weights_

# %%
# To compare this model, we use an inverse volatility benchmark using
# the :class:`~skfolio.optimization.InverseVolatility` estimator:
benchmark = InverseVolatility(portfolio_params=dict(name="Inverse Vol"))
benchmark.fit(X_train)
benchmark.weights_

# %%
# Prediction
# ==========
# We predict the model and the benchmark on the test set:
pred_model = model.predict(X_test)
pred_bench = benchmark.predict(X_test)

# %%
# The `predict` method returns a :class:`~skfolio.portfolio.Portfolio` object.
#
# :class:`~skfolio.portfolio.Portfolio` is an array-container making it compatible
# with `scikit-learn` tools: calling `np.asarray(pred_model)` gives the portfolio
# returns (same as `pred_model.returns`):
np.asarray(pred_model)

# %%
# The :class:`~skfolio.portfolio.Portfolio` class contains a vast number of properties
# and methods used for analysis.
#
# | For example:
#
# * pred_model.plot_cumulative_returns()
# * pred_model.plot_composition()
# * pred_model.summary()
print(pred_model.annualized_sharpe_ratio)
print(pred_bench.annualized_sharpe_ratio)

# %%
# Analysis
# ========
# For improved analysis, we load both predicted portfolios into a
# :class:`~skfolio.population.Population`:
population = Population([pred_model, pred_bench])

# %%
# The :class:`~skfolio.population.Population` class also contains a
# vast number of properties and methods used for analysis.
# Let's plot each portfolio composition:
population.plot_composition()

# %%
# .. note::
#       Every `plot` methods in `skfolio` returns a `plotly` figure.
#       To display a plotly figure, you may need to call `show()` and change the
#       default renderer: https://plotly.com/python/renderers/
#
# Let's plot each portfolio cumulative returns:
fig = population.plot_cumulative_returns()
# show(fig) is only used for the documentation sticker.
show(fig)

# %%
# |
#
# Finally, let's display the full summary of both strategies evaluated on the test
# set:
population.summary()

# %%
# Conclusion
# ==========
# From the analysis on the test set, we see that the Maximum Sharpe Ratio portfolio
# outperform the inverse-volatility benchmark for the mean and the ratio
# measures including the Sharpe Ratio, and underperforms for the deviation and
# shortfall measures.
#
# .. seealso::
#       This was a toy example, for more advanced concepts check the :ref:`user guide
#       <user_guide>` or the :ref:`other examples <general_examples>`.
#

```

**Expected behavior**
It should not error like this. Or I simply do not understand what is causing it, since other models works fine and sometimes this max sharpe model works fine also.

**Additional context**

**Versions**
<details> 

skfolio: 0.7.0
python: 3.10.12

</details>

<!-- Thanks for contributing! -->

Hints from issue comments:
@HugoDelatte: Thank you for the detailed report!

The issue arises because all your assets have a negative expected return (mean), and you are restricting the portfolio to be long only. This means there is no combination of weights that can yield a positive expected return of your portfolio, and the Sharpe Ratio optimization cannot handle this scenario. Note that this is a limitation of all mean/risk ratio measures themself: for negative expected return, a higher Sharpe (moving closer to zero on the negative part) is attained by increasing volatility which breaks the logic of "higher Sharpe is better".

I will raise a specific error message when this happens so users can clearly see why it fails.

If you want to fix it, you have several options:

* Allow short positions by setting `min_weights = -1`.  
* Incorporate views into your model if you believe your expected returns should be higher, for example by using the [Black & Litterman](https://skfolio.org/auto_examples/1_mean_risk/plot_12_black_and_litterman.html#sphx-glr-auto-examples-1-mean-risk-plot-12-black-and-litterman-py)@drager: > Thank you for the detailed report!
> 
> The issue arises because all your assets have a negative expected return (mean), and you are restricting the portfolio to be long only. This means there is no combination of weights that can yield a positive expected return of your portfolio, and the Sharpe Ratio optimization cannot handle this scenario. Note that this is a limitation of all mean/risk ratio measures themself: for negative expected return, a higher Sharpe (moving closer to zero on the negative part) is attained by increasing volatility which breaks the logic of "higher Sharpe is better".
> 
> I will raise a specific error message when this happens so users can clearly see why it fails.
> 
> If you want to fix it, you have several options:
> 
>     * Allow short positions by setting `min_weights = -1`.
> 
>     * Incorporate views into your model if you believe your expected returns should be higher, for example by using the [Black & Litterman](https://skfolio.org/auto_examples/1_mean_risk/plot_12_black_and_litterman.html#sphx-glr-auto-examples-1-mean-risk-plot-12-black-and-litterman-py)

I see. That makes sense. Thank you for the answer!

A specific error message sounds good to me!



Solution patch:
diff --git a/src/skfolio/optimization/convex/_mean_risk.py b/src/skfolio/optimization/convex/_mean_risk.py
index 731b8fc0..5bab0302 100644
--- a/src/skfolio/optimization/convex/_mean_risk.py
+++ b/src/skfolio/optimization/convex/_mean_risk.py
@@ -902,7 +902,7 @@ def fit(
             )
             constraints += [
                 tracking_error * self._scale_constraints
-                <= self.max_tracking_error * self._scale_constraints
+                <= self.max_tracking_error * factor * self._scale_constraints
             ]
 
         # Turnover
@@ -1036,6 +1036,15 @@ def fit(
                     + custom_objective * self._scale_objective
                 )
             case ObjectiveFunction.MAXIMIZE_RATIO:
+                # Capture common obvious mistake before solver failure to help user
+                if np.isscalar(self.min_weights) and self.min_weights >= 0:
+                    if np.max(prior_model.mu) - self.risk_free_rate <= 0:
+                        raise ValueError(
+                            "Cannot optimize for Maximum Ratio with your current "
+                            "constraints and input. This is because your assets' "
+                            "expected returns are all under-performing your risk-free "
+                            f"rate {self.risk_free_rate:.2%}."
+                        )
                 homogenization_factor = _optimal_homogenization_factor(
                     mu=prior_model.mu
                 )
@@ -1060,7 +1069,6 @@ def fit(
                     # Fractional Programming Problems".
                     # The condition to work is f1 >= 0, so we need to raise an user
                     # warning when it's not the case.
-                    # TODO: raise user warning when f1<0
 
                     constraints += [
                         expected_return * self._scale_constraints
diff --git a/src/skfolio/pre_selection/_select_non_expiring.py b/src/skfolio/pre_selection/_select_non_expiring.py
index c61cd8c7..d251f1fc 100644
--- a/src/skfolio/pre_selection/_select_non_expiring.py
+++ b/src/skfolio/pre_selection/_select_non_expiring.py
@@ -123,7 +123,7 @@ def fit(self, X: pd.DataFrame, y=None) -> "SelectNonExpiring":
             )
 
         if self.expiration_dates is None:
-            raise ValueError("`expiration_lookahead` must be provided")
+            raise ValueError("`expiration_dates` must be provided")
 
         if self.expiration_lookahead is None:
             raise ValueError("`expiration_lookahead` must be provided")
diff --git a/src/skfolio/utils/equations.py b/src/skfolio/utils/equations.py
index aa0122e0..2d50c83e 100644
--- a/src/skfolio/utils/equations.py
+++ b/src/skfolio/utils/equations.py
@@ -276,7 +276,7 @@ def _matching_array(values: np.ndarray, key: str, sum_to_one: bool) -> np.ndarra
     if not arr.any():
         raise EquationToMatrixError(f"Unable to find '{key}' in '{values}'")
     if sum_to_one:
-        s = np.sum(arr)
+        s = arr.sum()
     else:
         s = 1
     return arr / s


Test patch:
diff --git a/tests/test_optimization/test_convex/test_mean_risk.py b/tests/test_optimization/test_convex/test_mean_risk.py
index 34872f99..5193b74a 100644
--- a/tests/test_optimization/test_convex/test_mean_risk.py
+++ b/tests/test_optimization/test_convex/test_mean_risk.py
@@ -896,14 +896,16 @@ def test_groups(X, groups, linear_constraints):
     np.testing.assert_almost_equal(p1.returns, p3.returns)
 
 
-def test_tracking_error(X, y):
-    model = MeanRisk(max_tracking_error=0.005)
+@pytest.mark.parametrize(
+    "objective_function",
+    list(ObjectiveFunction),
+)
+def test_tracking_error(X, y, objective_function):
+    model = MeanRisk(max_tracking_error=0.005, objective_function=objective_function)
     bench = y["SIZE"]
     p = model.fit(X, bench).predict(X)
-    tracking_error = np.sqrt(
-        np.sum((p.returns - np.asarray(bench)) ** 2) / (len(p.returns) - 1)
-    )
-    np.testing.assert_almost_equal(tracking_error, 0.005)
+    tracking_error = np.std(p.returns - np.asarray(bench), ddof=1)
+    np.testing.assert_almost_equal(tracking_error, 0.005, 4)
 
 
 def test_turnover(X, y):
@@ -1354,3 +1356,18 @@ def test_mip_cardinality_and_threshold_constraints_long_short(X):
     np.testing.assert_almost_equal(np.sum(w), 0.5)
     assert np.max(w) - 0.8 <= 1e-8
     assert np.min(w) + 0.8 >= -1e-8
+
+
+def test_max_ratio_with_neg_f1(X):
+    model = MeanRisk(
+        objective_function=ObjectiveFunction.MAXIMIZE_RATIO, risk_free_rate=0.002
+    )
+    model.fit(X)
+
+    with pytest.raises(
+        ValueError, match="Cannot optimize for Maximum Ratio with your current"
+    ):
+        model = MeanRisk(
+            objective_function=ObjectiveFunction.MAXIMIZE_RATIO, risk_free_rate=0.0025
+        )
+        model.fit(X)