Repository: benchopt/benchopt
Pull request #676 (2023-12-07 21:09:15)

ENH add support for fish shell
Running shell commands through benchopt doesn't work with the fish shell at the moment.
This commit checks if fish is in use and modifies the call appropriately.

There's also a small change for the input to `subprocess.getstatusoutput`, which was a list but should be a string (according to the docs), but it's working anyway so I guess it's a cosmetic change.

codecov-commenter: ## [Codecov](https://app.codecov.io/gh/benchopt/benchopt/pull/676?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=benchopt) Report
> Merging [#676](https://app.codecov.io/gh/benchopt/benchopt/pull/676?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=benchopt) (96c0b59) into [main](https://app.codecov.io/gh/benchopt/benchopt/commit/d490ae5cb6ff1721f36b2e634eff99b5dd74d311?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=benchopt) (d490ae5) will **decrease** coverage by `0.20%`.
> The diff coverage is `60.00%`.

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #676      +/-   ##
==========================================
- Coverage   54.88%   54.69%   -0.20%     
==========================================
  Files          45       45              
  Lines        2928     2931       +3     
  Branches      539      540       +1     
==========================================
- Hits         1607     1603       -4     
- Misses       1196     1202       +6     
- Partials      125      126       +1     
```



</details>

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index ae0b4dd..2ee6ec6 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -58,7 +58,7 @@ jobs:
           command: |
             # remove the cache index
             mamba clean -i -y
-            mamba install python=3.9 julia r-base rpy2 \
+            mamba install python=3.10 julia r-base rpy2 \
                           numpy cython -yq
             pip install --upgrade "setuptools<58.5"
             pip install --upgrade --progress-bar off julia
diff --git a/README.rst b/README.rst
index 0aa7e15..47b4943 100644
--- a/README.rst
+++ b/README.rst
@@ -1,7 +1,7 @@
 .. image:: https://raw.githubusercontent.com/benchopt/communication_materials/main/posters/images/logo_benchopt.png
    :width: 350
    :align: center
-   
+
 *—Making your optimization benchmarks simple and open—*
 
 ----
@@ -24,7 +24,7 @@ Install
 -------
 
 It is recommended to use ``benchopt`` within a ``conda`` environment to fully-benefit
-from ``benchopt`` Command Line Interface (CLI).  
+from ``benchopt`` Command Line Interface (CLI).
 
 
 To install ``benchopt``, start by creating a new ``conda`` environment and then activate it
@@ -47,7 +47,7 @@ It is also possible to use the **latest development version**. To do so, run ins
     pip install --pre benchopt -U -i https://test.pypi.org/simple
 
 
-Getting started 
+Getting started
 ---------------
 
 After installing ``benchopt``, you can
@@ -98,15 +98,15 @@ Creating a benchmark
 ^^^^^^^^^^^^^^^^^^^^
 
 The section `Write a benchmark <https://benchopt.github.io/benchmark_workflow/write_benchmark.html>`_ of the documentation provides a tutorial
-for creating a benchmark. The ``benchopt`` community also maintains 
+for creating a benchmark. The ``benchopt`` community also maintains
 a `template benchmark <https://github.com/benchopt/template_benchmark>`_ to quickly and easily start a new benchmark.
 
 
-Finding helps
--------------
+Finding help
+------------
 
 Join ``benchopt`` `discord server <https://discord.gg/EA2HGQb7nv>`_ and get in touch with the community!
-Feel free to drop us a message to get help with running/constructing benchmarks 
+Feel free to drop us a message to get help with running/constructing benchmarks
 or (why not) discuss new features to be added and future development directions that ``benchopt`` should take.
 
 
@@ -119,12 +119,12 @@ Join us in this endeavor! If you use ``benchopt`` in a scientific publication, p
 .. code-block:: bibtex
 
    @inproceedings{benchopt,
-      author    = {Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre 
-                   and Ablin, Pierre and Bannier, Pierre-Antoine 
+      author    = {Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre
+                   and Ablin, Pierre and Bannier, Pierre-Antoine
                    and Charlier, Benjamin and Dagréou, Mathieu and Dupré la Tour, Tom
                    and Durif, Ghislain and F. Dantas, Cassio and Klopfenstein, Quentin
-                   and Larsson, Johan and Lai, En and Lefort, Tanguy 
-                   and Malézieux, Benoit and Moufad, Badr and T. Nguyen, Binh and Rakotomamonjy, 
+                   and Larsson, Johan and Lai, En and Lefort, Tanguy
+                   and Malézieux, Benoit and Moufad, Badr and T. Nguyen, Binh and Rakotomamonjy,
                    Alain and Ramzi, Zaccharie and Salmon, Joseph and Vaiter, Samuel},
       title     = {Benchopt: Reproducible, efficient and collaborative optimization benchmarks},
       year      = {2022},
@@ -213,37 +213,37 @@ Available benchmarks
 .. |install-per-months| image:: https://static.pepy.tech/badge/benchopt/month
    :target: https://pepy.tech/project/benchopt
 
-.. |Build Status OLS| image:: https://github.com/benchopt/benchmark_ols/workflows/Tests/badge.svg
+.. |Build Status OLS| image:: https://github.com/benchopt/benchmark_ols/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_ols/actions
-.. |Build Status NNLS| image:: https://github.com/benchopt/benchmark_nnls/workflows/Tests/badge.svg
+.. |Build Status NNLS| image:: https://github.com/benchopt/benchmark_nnls/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_nnls/actions
-.. |Build Status Lasso| image:: https://github.com/benchopt/benchmark_lasso/workflows/Tests/badge.svg
+.. |Build Status Lasso| image:: https://github.com/benchopt/benchmark_lasso/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_lasso/actions
-.. |Build Status Lasso Path| image:: https://github.com/jolars/benchmark_lasso_path/workflows/Tests/badge.svg
+.. |Build Status Lasso Path| image:: https://github.com/jolars/benchmark_lasso_path/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_lasso_path/actions
-.. |Build Status ElasticNet| image:: https://github.com/benchopt/benchmark_elastic_net/workflows/Tests/badge.svg
+.. |Build Status ElasticNet| image:: https://github.com/benchopt/benchmark_elastic_net/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_elastic_net/actions
-.. |Build Status MCP| image:: https://github.com/benchopt/benchmark_mcp/workflows/Tests/badge.svg
+.. |Build Status MCP| image:: https://github.com/benchopt/benchmark_mcp/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_mcp/actions
-.. |Build Status LogRegL2| image:: https://github.com/benchopt/benchmark_logreg_l2/workflows/Tests/badge.svg
+.. |Build Status LogRegL2| image:: https://github.com/benchopt/benchmark_logreg_l2/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_logreg_l2/actions
-.. |Build Status LogRegL1| image:: https://github.com/benchopt/benchmark_logreg_l1/workflows/Tests/badge.svg
+.. |Build Status LogRegL1| image:: https://github.com/benchopt/benchmark_logreg_l1/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_logreg_l1/actions
-.. |Build Status HuberL2| image:: https://github.com/benchopt/benchmark_huber_l2/workflows/Tests/badge.svg
+.. |Build Status HuberL2| image:: https://github.com/benchopt/benchmark_huber_l2/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_huber_l2/actions
-.. |Build Status QuantileRegL1| image:: https://github.com/benchopt/benchmark_quantile_regression/workflows/Tests/badge.svg
+.. |Build Status QuantileRegL1| image:: https://github.com/benchopt/benchmark_quantile_regression/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_quantile_regression/actions
-.. |Build Status LinearSVM| image:: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/workflows/Tests/badge.svg
+.. |Build Status LinearSVM| image:: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/actions
-.. |Build Status LinearICA| image:: https://github.com/benchopt/benchmark_linear_ica/workflows/Tests/badge.svg
+.. |Build Status LinearICA| image:: https://github.com/benchopt/benchmark_linear_ica/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_linear_ica/actions
-.. |Build Status JointDiag| image:: https://github.com/benchopt/benchmark_jointdiag/workflows/Tests/badge.svg
+.. |Build Status JointDiag| image:: https://github.com/benchopt/benchmark_jointdiag/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_jointdiag/actions
-.. |Build Status TV1D| image:: https://github.com/benchopt/benchmark_tv_1d/workflows/Tests/badge.svg
+.. |Build Status TV1D| image:: https://github.com/benchopt/benchmark_tv_1d/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_tv_1d/actions
-.. |Build Status TV2D| image:: https://github.com/benchopt/benchmark_tv_2d/workflows/Tests/badge.svg
+.. |Build Status TV2D| image:: https://github.com/benchopt/benchmark_tv_2d/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_tv_2d/actions
-.. |Build Status ResNetClassif| image:: https://github.com/benchopt/benchmark_resnet_classif/workflows/Tests/badge.svg
+.. |Build Status ResNetClassif| image:: https://github.com/benchopt/benchmark_resnet_classif/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_resnet_classif/actions
-.. |Build Status Bilevel| image:: https://github.com/benchopt/benchmark_bilevel/workflows/Tests/badge.svg
+.. |Build Status Bilevel| image:: https://github.com/benchopt/benchmark_bilevel/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_bilevel/actions
diff --git a/benchopt/config.py b/benchopt/config.py
index 0803dd2..1330d5c 100644
--- a/benchopt/config.py
+++ b/benchopt/config.py
@@ -42,7 +42,8 @@ DEFAULT_GLOBAL_CONFIG = {
 """
 
 DEFAULT_BENCHMARK_CONFIG = {
-    "plots": list(PLOT_KINDS), "plot_configs": {}
+    "plots": list(PLOT_KINDS),
+    "plot_configs": {}
 }
 
 """
@@ -157,10 +158,15 @@ def set_setting(name, value, config_file=None, benchmark_name=None):
     if config_file is None:
         config_file = get_global_config_file()
 
+    # Handle deprecated .ini config file by automatically
+    # converting them to .yml.
+    if config_file.suffix == ".ini":
+        convert_ini_to_yml(config_file)
+        config_file = config_file.with_suffix('.yml')
+
     # Get default value
     default_config = DEFAULT_BENCHMARK_CONFIG
     if benchmark_name is None:
-        benchmark_name = 'benchopt'
         default_config = DEFAULT_GLOBAL_CONFIG
 
     if name not in default_config:
@@ -171,21 +177,25 @@ def set_setting(name, value, config_file=None, benchmark_name=None):
         raise SystemExit(1)
     default_value = default_config[name]
 
-    # Get global config file
-    config = configparser.ConfigParser()
-    config.read(config_file)
-
-    if benchmark_name not in config:
-        config[benchmark_name] = {}
+    # Load the config from the yaml file if the file exists.
+    config = {}
+    if config_file.exists():
+        with open(config_file, "r") as f:
+            config = yaml.safe_load(f) or {}
 
-    config[benchmark_name][name] = reverse_parse(default_value, value)
+    if benchmark_name is not None:
+        if benchmark_name not in config:
+            config[benchmark_name] = {}
+        config[benchmark_name][name] = reverse_parse(default_value, value)
+    else:
+        config[name] = reverse_parse(default_value, value)
 
     # Create config file with the correct permission.
     if not config_file.exists():
         config_file.touch(mode=GLOBAL_CONFIG_FILE_MODE)
 
     with config_file.open('w') as f:
-        config.write(f)
+        yaml.safe_dump(config, f)
 
 
 def get_setting(name, config_file=None, benchmark_name=None,
@@ -212,6 +222,12 @@ def get_setting(name, config_file=None, benchmark_name=None,
     if config_file is None:
         config_file = get_global_config_file()
 
+    # Handle deprecated .ini config file by automatically
+    # converting them to .yml.
+    if config_file.suffix == ".ini":
+        convert_ini_to_yml(config_file)
+        config_file = config_file.with_suffix('.yml')
+
     # Get default value
     default_config_ = DEFAULT_BENCHMARK_CONFIG
     if benchmark_name is None:
@@ -222,21 +238,14 @@ def get_setting(name, config_file=None, benchmark_name=None,
     default_config = {} if default_config is None else default_config
     default_value = default_config.get(name, default_config_[name])
 
-    # Handle deprecated .ini config file by automatically
-    # converting them to .yml.
-    if config_file.suffix == ".ini":
-        convert_ini_to_yml(config_file)
-        config_file = config_file.with_suffix('.yml')
-
     # Load the config from the yaml file if the file exists.
+    config = {}
     if config_file.exists():
         with open(config_file, "r") as f:
-            config = yaml.safe_load(f)
-    else:
-        config = {}
+            config = yaml.safe_load(f) or {}
 
     # Get value from config file or keep the default value.
-    if benchmark_name in config.keys():
+    if benchmark_name in config:
         value = config[benchmark_name].get(name, default_value)
     else:
         value = config.get(name, default_value)
diff --git a/benchopt/utils/shell_cmd.py b/benchopt/utils/shell_cmd.py
index 8edb59c..d5067a1 100644
--- a/benchopt/utils/shell_cmd.py
+++ b/benchopt/utils/shell_cmd.py
@@ -42,8 +42,13 @@ def _run_shell(script, raise_on_error=None, capture_stdout=True,
             'return_output=True can only be used with capture_stdout=True'
         )
 
+    is_fish = 'fish' in f"{SHELL}"
+
     # Make sure the script fail at first failure
-    fast_failure_script = f"set -e\n{script}"
+    if is_fish:
+        fast_failure_script = f"begin; {script}; or exit 1; end"
+    else:
+        fast_failure_script = f"set -e\n{script}"
 
     # Use a TemporaryFile to make sure this file is cleaned up at
     # the end of this function.
@@ -58,7 +63,7 @@ def _run_shell(script, raise_on_error=None, capture_stdout=True,
         raise_on_error = "{output}"
 
     if capture_stdout:
-        exit_code, output = subprocess.getstatusoutput([f"{SHELL} {tmp.name}"])
+        exit_code, output = subprocess.getstatusoutput(f"{SHELL} {tmp.name}")
     else:
         exit_code = os.system(f"{SHELL} {tmp.name}")
         output = ""
diff --git a/doc/available_benchmarks.rst b/doc/available_benchmarks.rst
index 7d72212..b38c7ba 100644
--- a/doc/available_benchmarks.rst
+++ b/doc/available_benchmarks.rst
@@ -102,23 +102,23 @@ matrix :math:`B` can be enforced to be orthogonal.
 See `benchmark_* repositories on GitHub <https://github.com/benchopt/>`_ for more.
 
 
-.. |Build Status OLS| image:: https://github.com/benchopt/benchmark_ols/workflows/Tests/badge.svg
+.. |Build Status OLS| image:: https://github.com/benchopt/benchmark_ols/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_ols/actions
-.. |Build Status NNLS| image:: https://github.com/benchopt/benchmark_nnls/workflows/Tests/badge.svg
+.. |Build Status NNLS| image:: https://github.com/benchopt/benchmark_nnls/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_nnls/actions
-.. |Build Status Lasso| image:: https://github.com/benchopt/benchmark_lasso/workflows/Tests/badge.svg
+.. |Build Status Lasso| image:: https://github.com/benchopt/benchmark_lasso/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_lasso/actions
-.. |Build Status LogRegL2| image:: https://github.com/benchopt/benchmark_logreg_l2/workflows/Tests/badge.svg
+.. |Build Status LogRegL2| image:: https://github.com/benchopt/benchmark_logreg_l2/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_logreg_l2/actions
-.. |Build Status LogRegL1| image:: https://github.com/benchopt/benchmark_logreg_l1/workflows/Tests/badge.svg
+.. |Build Status LogRegL1| image:: https://github.com/benchopt/benchmark_logreg_l1/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_logreg_l1/actions
-.. |Build Status HuberL2| image:: https://github.com/benchopt/benchmark_huber_l2/workflows/Tests/badge.svg
+.. |Build Status HuberL2| image:: https://github.com/benchopt/benchmark_huber_l2/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_huber_l2/actions
-.. |Build Status QuantileRegL1| image:: https://github.com/benchopt/benchmark_quantile_regression/workflows/Tests/badge.svg
+.. |Build Status QuantileRegL1| image:: https://github.com/benchopt/benchmark_quantile_regression/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_quantile_regression/actions
-.. |Build Status LinearSVM| image:: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/workflows/Tests/badge.svg
+.. |Build Status LinearSVM| image:: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_linear_svm_binary_classif_no_intercept/actions
-.. |Build Status LinearICA| image:: https://github.com/benchopt/benchmark_linear_ica/workflows/Tests/badge.svg
+.. |Build Status LinearICA| image:: https://github.com/benchopt/benchmark_linear_ica/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_linear_ica/actions
-.. |Build Status JointDiag| image:: https://github.com/benchopt/benchmark_jointdiag/workflows/Tests/badge.svg
+.. |Build Status JointDiag| image:: https://github.com/benchopt/benchmark_jointdiag/actions/workflows/main.yml/badge.svg
    :target: https://github.com/benchopt/benchmark_jointdiag/actions
diff --git a/doc/faq.rst b/doc/faq.rst
index 5c8d655..cc18b10 100644
--- a/doc/faq.rst
+++ b/doc/faq.rst
@@ -14,12 +14,15 @@ Frequently asked questions (FAQ)
     Learn how to :ref:`write_benchmark`, including creating an objective, a solver, and a dataset.
 
 
-.. dropdown:: How are performance curves constructed?
-
-    The goal of benchopt is to evaluate the evolution of a method's performance with respect to its computational budget.
-    For evaluating this, benchopt allows to vary the computational budget for both black-box solvers and solvers that allow for callbacks.
+.. dropdown:: How are performance curves constructed and the solvers stopped?
+
+    One of benchopt's goals is to evaluate the method's performance with
+    respect to its computational budget.
+    Benchopt allows several strategies to vary the computational budget, that
+    can be set on a per solver basis.
+    It is also possible to set various stopping criterions to decide when to
+    stop growing the computational budget, to avoid wasting resources.
     Visit the :ref:`performance_curves` page for more detail.
-    Note that the budget varying strategy can be configured on a per-solver basis, as described in: :ref:`sampling_strategy`.
 
 
 .. dropdown:: How can I reuse code in a benchmark?
diff --git a/doc/user_guide/advanced.rst b/doc/user_guide/advanced.rst
index ab3119f..8939a32 100644
--- a/doc/user_guide/advanced.rst
+++ b/doc/user_guide/advanced.rst
@@ -97,33 +97,6 @@ we get
             return False, None
 
 
-
-.. _sampling_strategy:
-
-Changing the strategy to grow the computational budget (:code:`stop_val`)
-------------------------------------------------------------------------
-
-Benchopt varies the computational budget by varying either the number
-of iterations or the tolerance given to the method. The default policy is
-to vary these two quantities exponentially between two evaluations of the
-objective. However, in some cases, this exponential growth might hide some
-effects, or might not be adapted to a given solver.
-
-The way this value is changed can be specified for each solver by
-implementing a ``get_next`` method in the ``Solver`` class.
-This method takes as input the previous value where the objective
-function has been logged, and outputs the next one. For instance,
-if a solver needs to be evaluated every 10 iterations, we would have
-
-.. code-block::
-
-    class Solver(BaseSolver):
-        ...
-        def get_next(self, stop_val):
-            return stop_val + 10
-
-
-
 .. _benchmark_utils_import:
 
 Reusing some code in a benchmark
diff --git a/doc/user_guide/performance_curves.rst b/doc/user_guide/performance_curves.rst
index 63ab249..4f86cd7 100644
--- a/doc/user_guide/performance_curves.rst
+++ b/doc/user_guide/performance_curves.rst
@@ -45,6 +45,9 @@ In both cases, if the objective curve is flat (i.e., the variation of the object
 Note that the solver is restarted from scratch at each call to ``solver.run``.
 For more advanced configurations, the evolution of ``stop_val`` can be controlled on a per solver basis, by implementing a ``Solver.get_next`` method, which receives the current value for tolerance/number of iterations, and returns the next one.
 
+
+Note that the formula to compute the next ``stop_val`` can be configured on a per-solver basis, as described in: :ref:`sampling_strategy`.
+
 2. Using a callback
 ~~~~~~~~~~~~~~~~~~~
 
@@ -68,6 +71,25 @@ It returns ``False`` when the solver should be stopped. A classical usage patter
     def get_result(self):
         return {'x': self.x}
 
+.. _sampling_strategy:
+
+3. Changing the strategy to grow the computational budget (:code:`stop_val`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The way benchopt varies the computational budget :code:`stop_val` can be
+configured on a per-solver basis. This is done by implementing a ``get_next``
+method in the ``Solver`` class. This method takes as input the previous value
+where the objective function has been logged, and outputs the next one. For
+instance, if a solver needs to be evaluated every 10 iterations, we would have
+
+.. code-block::
+
+    class Solver(BaseSolver):
+        ...
+        def get_next(self, stop_val):
+            return stop_val + 10
+
+This example allow to set a linear growth for the solver computational budget, instead of the default geometric growth.
 
 
 When are the solvers stopped?
diff --git a/setup.cfg b/setup.cfg
index e32ffaf..2fc7fed 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -54,14 +54,14 @@ benchopt.plotting =
 
 [options.extras_require]
 test =
-    pytest<=7.4.1
+    pytest
     pytest-cov
     pytest-timeout
     coverage
 
 doc =
     numpydoc
-    sphinx==4.5.0
+    sphinx
     sphinx-book-theme
     sphinx-design
     sphinx-click


Test patch:
diff --git a/benchopt/tests/test_config.py b/benchopt/tests/test_config.py
index 6372bdf..22f49d2 100644
--- a/benchopt/tests/test_config.py
+++ b/benchopt/tests/test_config.py
@@ -1,15 +1,19 @@
 import os
 import pytest
+import warnings
 from pathlib import Path
 from contextlib import contextmanager
 
+from benchopt.config import parse_value
+from benchopt.config import DEFAULT_GLOBAL_CONFIG
 from benchopt.config import get_global_config_file
+from benchopt.config import set_setting, get_setting
 
 
 @contextmanager
-def set_config_file(permission='600'):
+def temp_config_file(permission='600'):
     permission = int(f"100{permission}", base=8)
-    config_file = Path() / 'test_config_file.ini'
+    config_file = Path() / 'test_config_file.yml'
     config_file.touch(mode=permission, exist_ok=False)
     old_config_file = os.environ.get('BENCHOPT_CONFIG', None)
     os.environ['BENCHOPT_CONFIG'] = str(config_file)
@@ -23,9 +27,27 @@ def set_config_file(permission='600'):
         config_file.unlink()
 
 
+def test_parse_value():
+    # Check boolean parsing
+    for val in [True, 'true', 'True', '1', 'yes', 'on']:
+        assert parse_value(val, True) is True
+    for val in ['false', 'False', '0', 'no', 'off']:
+        assert parse_value(val, True) is False
+
+    # Check list parsing
+    value = "test, test1\ntest2     \n test3"
+    assert parse_value(value, []) == ['test', 'test1', 'test2', 'test3']
+
+    # Check that if the default is not a bool or a list, we return the value
+    assert parse_value(1, 2) == 1
+    assert parse_value(2.0, 1) == 2.0
+    assert parse_value('abc', 1) == 'abc'
+    assert parse_value('abc', 'a') == 'abc'
+
+
 @pytest.mark.parametrize("permission", ["644", "655", "240"])
 def test_config_file_permission_warn(permission):
-    with set_config_file(permission) as config_file:
+    with temp_config_file(permission) as config_file:
         msg = f"{config_file} is with mode {permission}"
         with pytest.warns(UserWarning, match=msg):
             global_config_file = get_global_config_file()
@@ -33,11 +55,42 @@ def test_config_file_permission_warn(permission):
 
 
 def test_config_file_permission_no_warning():
-    with set_config_file() as config_file:
-        with pytest.warns(None) as record:
+    with temp_config_file() as config_file:
+        with warnings.catch_warnings():
+            warnings.simplefilter("error")
             global_config_file = get_global_config_file()
         assert str(global_config_file) == str(config_file)
-        assert len(record) == 0, (
-            "A warning was issued while it should not have.\n"
-            f"Warning is: {record[0]}"
-        )
+
+
+@pytest.mark.parametrize("setting_key", DEFAULT_GLOBAL_CONFIG)
+def test_config_file_set(setting_key):
+    with temp_config_file():
+        default_value = DEFAULT_GLOBAL_CONFIG[setting_key]
+        default_value = parse_value(os.environ.get(
+            f"BENCHOPT_{setting_key.upper()}", default_value
+        ), default_value)
+        assert get_setting(setting_key) == default_value
+
+        set_value = parse_value('True', default_value)
+        set_setting(setting_key, set_value)
+
+        # Make sure the value is correctly set and retrieved
+        try:
+            # If the env var is not set, we should get the set value
+            old_value = os.environ.pop(f"BENCHOPT_{setting_key.upper()}", None)
+            assert get_setting(setting_key) == set_value
+
+            # Otherwise, we get the env value
+            env_value = parse_value('False', default_value)
+            os.environ[f"BENCHOPT_{setting_key.upper()}"] = str(env_value)
+            assert get_setting(setting_key) == env_value
+            del os.environ[f"BENCHOPT_{setting_key.upper()}"]
+        finally:
+            if old_value is not None:
+                os.environ[f"BENCHOPT_{setting_key.upper()}"] = old_value
+
+
+def test_config_file_set_error():
+    with temp_config_file():
+        with pytest.raises(SystemExit):
+            set_setting('invalid_key', None)