Repository: benchopt/benchopt (python)

Issue:
Compilation time in the `warm_up` method
After adding the `warm_up` method from #602 in a solver, it seems that the compilation time is accounted in the results. 
<img width="515" alt="image" src="https://github.com/benchopt/benchopt/assets/77896657/494c33bb-fe84-4e38-b378-0944945882e1">


Pull request:
FIX call `warm_up` when the attribute `_warmup_done` don't exist
This fixes #642.
The `warm_up` method was not called when the attribute `_warmup_done` did not exist.

Patch:
diff --git a/MANIFEST.in b/MANIFEST.in
index 6aa5053..2162a21 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -21,6 +21,7 @@ recursive-include doc *.rst
 recursive-include doc *.txt
 recursive-include doc *.inc
 recursive-include doc *.png
+recursive-include doc *.svg
 recursive-include doc Makefile
 prune doc/_build
 prune doc/generated
diff --git a/benchopt/base.py b/benchopt/base.py
index 2df664c..73ffd93 100644
--- a/benchopt/base.py
+++ b/benchopt/base.py
@@ -259,9 +259,11 @@ class BaseSolver(ParametrizedNameMixin, DependenciesMixin, ABC):
         ...
 
     def _warm_up(self):
-        if not getattr(self, '_warmup_done', True):
-            self.warm_up()
-            self._warmup_done = True
+        if getattr(self, '_warmup_done', None):
+            # already warmed up
+            return
+        self.warm_up()
+        self._warmup_done = True
 
     @staticmethod
     def _reconstruct(module_filename, parameters, objective, output,
diff --git a/benchopt/stopping_criterion.py b/benchopt/stopping_criterion.py
index bd94852..77165cb 100644
--- a/benchopt/stopping_criterion.py
+++ b/benchopt/stopping_criterion.py
@@ -189,7 +189,7 @@ class StoppingCriterion():
             Reason why the algorithm was stopped if stop is True.
         next_stop_val : int | float
             Next value for the stopping criterion. This value depends on the
-            stop strategy for the solver.
+            sampling strategy for the solver.
         """
         # Modify the criterion state:
         # - compute the number of run with the curve. We need to remove 1 as
diff --git a/doc/_static/style.css b/doc/_static/style.css
index b30db9a..468c1d1 100644
--- a/doc/_static/style.css
+++ b/doc/_static/style.css
@@ -1,5 +1,5 @@
 
-h2, h3, h4 {
+h2, h4 {
     text-decoration: underline;
 }
 
@@ -7,6 +7,15 @@ h2, h3, h4 {
     width: 100%;
 }
 
+.bd-article {
+    height: 100%;
+}
+
 .pre {
     color: #D05336;
+}
+
+.code-block-caption {
+    background: #E1E1E1;
+    color: #D05336;
 }
\ No newline at end of file
diff --git a/doc/benchmark_workflow/write_benchmark.rst b/doc/benchmark_workflow/write_benchmark.rst
index 15f54a0..feac265 100644
--- a/doc/benchmark_workflow/write_benchmark.rst
+++ b/doc/benchmark_workflow/write_benchmark.rst
@@ -15,8 +15,8 @@ structure. For example
     my_benchmark/
     ├── objective.py  # contains the definition of the objective
     ├── datasets/
-    │   ├── simulated.py  # some dataset
-    │   └── real.py  # some dataset
+    │   ├── dataset1.py  # some dataset
+    │   └── dataset2.py  # some dataset
     └── solvers/
         ├── solver1.py  # some solver
         └── solver2.py  # some solver
diff --git a/doc/faq.rst b/doc/faq.rst
index ded43bb..5c8d655 100644
--- a/doc/faq.rst
+++ b/doc/faq.rst
@@ -3,23 +3,28 @@
 Frequently asked questions (FAQ)
 --------------------------------
 
+
+.. dropdown:: How to add my solver to an existing benchmark?
+
+    Visit the :ref:`add_solver` tutorial for a step-by-step procedure to add a solver to an existing benchmark.
+
+
 .. dropdown:: How can I write a benchmark?
 
-    Learn how to :ref:`write_benchmark`, including creating an objective, a solver, and
-    a dataset.
+    Learn how to :ref:`write_benchmark`, including creating an objective, a solver, and a dataset.
 
 
-.. dropdown:: How performance curves are constructed?
+.. dropdown:: How are performance curves constructed?
 
     The goal of benchopt is to evaluate the evolution of a method's performance with respect to its computational budget.
     For evaluating this, benchopt allows to vary the computational budget for both black-box solvers and solvers that allow for callbacks.
-    Learn :ref:`performance_curves`.
-    Note that the budget varying strategy can also be configured on a per-solver basis, as described in: :ref:`sampling_strategy`.
+    Visit the :ref:`performance_curves` page for more detail.
+    Note that the budget varying strategy can be configured on a per-solver basis, as described in: :ref:`sampling_strategy`.
 
 
 .. dropdown:: How can I reuse code in a benchmark?
 
-    For some solver and datasets, it is necessary to share some operations or pre-processing steps.
+    For some solver and datasets, it is handy to share some operations or pre-processing steps.
     Benchopt allows to factorize this code by :ref:`benchmark_utils_import`.
 
 
diff --git a/doc/index.rst b/doc/index.rst
index 405c2f2..c3c4ab6 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -14,7 +14,7 @@ It is implemented in Python but can run algorithms written in many programming l
 Reproducing an existing benchmark should be as easy as running
 
 .. prompt:: bash $
-   
+
    benchopt run . --config ./example_config.yml
 
 .. figure:: https://benchopt.github.io/_images/sphx_glr_plot_run_benchmark_001.png
@@ -24,7 +24,7 @@ Reproducing an existing benchmark should be as easy as running
 There are already many :ref:`available_benchmarks` that have been created using benchopt.
 Learn how to run them and how to construct your own with the following pages!
 
-.. grid:: 1
+.. grid:: 2
     :gutter: 1
 
     .. grid-item-card::
@@ -43,6 +43,13 @@ Learn how to run them and how to construct your own with the following pages!
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         Write a benchmark from scratch, run it, visualize it, and publish it
 
+    .. grid-item-card::
+        :link: tutorials
+        :link-type: ref
+
+        :octicon:`mortar-board` **Tutorials**
+        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+        Gallery of use-cases crafted by the benchopt community
 
     .. grid-item-card::
         :link: user_guide
@@ -53,7 +60,7 @@ Learn how to run them and how to construct your own with the following pages!
         Full documentation of benchopt API and CLI
 
 
-.. Frequently Asked Questions (FAQ) subsection 
+.. Frequently Asked Questions (FAQ) subsection
 .. include:: faq.rst
 
 
@@ -62,7 +69,7 @@ Join the community
 
 Join benchopt `discord server <https://discord.gg/EA2HGQb7nv>`_ and get in touch with the community!
 
-Feel free to drop a message to get help with running/constructing benchmarks 
+Feel free to drop a message to get help with running/constructing benchmarks
 or (why not) discuss new features to be added and future development directions that benchopt should take.
 
 
@@ -75,12 +82,12 @@ Join this endeavor! If you use benchopt in a scientific publication, please cite
 .. code-block:: bibtex
 
    @inproceedings{benchopt,
-      author    = {Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre 
-                   and Ablin, Pierre and Bannier, Pierre-Antoine 
+      author    = {Moreau, Thomas and Massias, Mathurin and Gramfort, Alexandre
+                   and Ablin, Pierre and Bannier, Pierre-Antoine
                    and Charlier, Benjamin and Dagréou, Mathieu and Dupré la Tour, Tom
                    and Durif, Ghislain and F. Dantas, Cassio and Klopfenstein, Quentin
-                   and Larsson, Johan and Lai, En and Lefort, Tanguy 
-                   and Malézieux, Benoit and Moufad, Badr and T. Nguyen, Binh and Rakotomamonjy, 
+                   and Larsson, Johan and Lai, En and Lefort, Tanguy
+                   and Malézieux, Benoit and Moufad, Badr and T. Nguyen, Binh and Rakotomamonjy,
                    Alain and Ramzi, Zaccharie and Salmon, Joseph and Vaiter, Samuel},
       title     = {Benchopt: Reproducible, efficient and collaborative optimization benchmarks},
       year      = {2022},
@@ -107,6 +114,7 @@ Join this endeavor! If you use benchopt in a scientific publication, please cite
 
    get_started
    benchmark_workflow/index
+   tutorials/index
    user_guide/index
 
    available_benchmarks
diff --git a/doc/tutorials/add_solver.rst b/doc/tutorials/add_solver.rst
new file mode 100644
index 0000000..755a5b1
--- /dev/null
+++ b/doc/tutorials/add_solver.rst
@@ -0,0 +1,249 @@
+.. _add_solver:
+
+Add a solver to an existing benchmark
+=====================================
+
+This tutorial shows how to add a new solver to a benchmark.
+We illustrate the process on the `Ridge regression benchmark <https://github.com/benchopt/benchmark_ridge>`_ by implementing a solver based on the ``scikit-learn`` Ridge estimator.
+
+.. Hint::
+    If not yet done, you can review the :ref:`get started page <get_started>` to learn how to install benchopt and download an existing benchmark.
+
+
+Before the implementation
+-------------------------
+
+First, create a ``mysolver.py`` file in the ``solvers/`` directory and put inside it the following content:
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    from benchopt import BaseSolver
+
+    class Solver(BaseSolver):
+        name = 'mysolver'
+
+As you can see, a solver is a Python class, ``Solver``, that inherits from ``benchopt.BaseSolver`` and is declared in a standalone Python file in the benchmark's ``solvers/`` folder.
+The attribute ``name`` does not have to match the file name, but it makes it easier to locate the solver.
+
+Doing the latter steps, our benchmark folder will resemble
+
+.. code-block:: bash
+
+    benchmark_ridge/
+    ├── objective.py     # existing implementation of the Objective
+    ├── datasets/
+    │   ├── ...          # existing datasets
+    └── solvers/
+        ├── mysolver.py  # our newly added solver
+        ├── ...          # other solvers
+
+
+Implementation
+--------------
+
+Once the Python file is created, we can start the implementation by adding methods and attributes to the ``Solver`` class.
+To do so, we follow the order that benchopt uses to call the methods during the benchmark's run: first ``set_objective``, then ``run`` and finally ``get_result``.
+
+.. figure:: https://raw.githubusercontent.com/benchopt/communication_materials/master/sharedimages/benchopt_schema_dependency.svg
+   :align: center
+   :width: 90 %
+
+Let's go over them one by one.
+
+Initializing the setup
+~~~~~~~~~~~~~~~~~~~~~~
+
+The first method we need to implement is ``set_objective``.
+It receives all the information about the dataset and objective parameters, that the solver will need to run.
+This information is standardized in the ``objective.py`` file of the benchmark, through the ``Objective.get_objective`` method.
+This method is part of the objective definition and has already been implemented when the benchmark was created; we do not need to write it.
+
+In the Ridge benchmark, the relevant information is the following:
+
+.. code-block:: python
+    :caption: benchmark_ridge/objective.py
+
+    ...
+    class Objective(BaseObjective):
+        ...
+        def get_objective(self):
+            return dict(
+                X=self.X, y=self.y,
+                lmbd=self.lmbd,
+                fit_intercept=self.fit_intercept
+            )
+        ...
+
+We see that ``get_objective`` returns a dictionary with four keys: ``X``, ``y``, ``lmbd``, and ``fit_intercept``.
+Therefore our ``set_objective`` must take them as input arguments.
+
+.. note::
+    If you are working with another benchmark, check the definition of ``Objective.get_objective`` in  ``objective.py`` to see which arguments must be passed to ``Solver.set_objective``.
+
+The ``set_objective`` method is meant to store references of dataset and objective parameters.
+It is also used to initialize unchanging variables across the solver run.
+
+In our case, we store ``X``, ``y``, ``lmbd``, and ``fit_intercept`` to use them when we will actually run the solver.
+We also use the method to instantiate a Ridge estimator that will be used to perform computation of the solution.
+
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        def set_objective(self, X, y, lmbd, fit_intercept):
+            # store any info needed to run the solver as class attribute
+            self.X, self.y = X, y
+            self.lmbd = lmbd
+
+            # declare anything that will be used to run your solver
+            self.model = sklearn.linear_model.Ridge(
+                alpha=lmbd,
+                fit_intercept=fit_intercept
+            )
+        ...
+
+
+Defining the solver run procedure
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Next, we implement the ``run`` method, and declare the ``sampling_strategy`` attribute of the solver.
+Together, they define how the performance curves of the solver will be constructed.
+
+There are three possible choices for the ``sampling_strategy`` attribute: **iteration**, **tolerance**, and **callback**.
+We show how to implement the ``run`` method for each one of them.
+
+- ``sampling_strategy = "iteration"``
+
+This sampling strategy is for **solvers that are controlled by the maximum number of iterations they perform**.
+In this case, benchopt treats the solver as a black box and observes its behavior for different number of iterations.
+
+Therefore, the signature of the ``run`` method is ``run(self, n_iter)`` and its implementation resembles the snippet below.
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        sampling_strategy = "iteration"
+        ...
+
+        def run(self, n_iter):
+            # configure sklearn to run for n_iter
+            self.model.max_iter = n_iter
+            # make sure sklearn goes until n_iter
+            self.model.tol = 0
+
+            self.model.fit(self.X, self.y)
+
+            # store reference to the solution
+            self.beta = self.model.coef_
+        ...
+
+- ``sampling_strategy = "tolerance"``
+
+Similar to **iteration**, this sampling strategy is used for **solvers controlled by the tolerance on the optimization process**.
+In this case, the signature of the ``run`` method is ``run(self, tolerance)``; it would be implemented as follows.
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        sampling_strategy = "tolerance"
+        ...
+
+        def run(self, tolerance):
+            # configure sklearn to run for tolerance
+            self.model.tol = tolerance
+            # configure sklearn to run until tolerance is reached
+            self.model.max_iter = int(1e12)
+
+            self.model.fit(self.X, self.y)
+
+            # store reference to the solution
+            self.beta = self.model.coef_
+        ...
+
+- ``sampling_strategy = "callback"``
+
+One may want to code the solver themselves rather than using a black-box one.
+In that case, all intermediate iterates are available, and one should use the **callback** sampling strategy.
+
+Let's say that we no longer implement the scikit-learn solver, but instead our own implementation of Gradient Descent.
+The following snippet shows how to use the callback strategy with a user-coded solver.
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        sampling_strategy = "callback"
+        ...
+
+        def run(self, callback):
+            X, y = self.X, self.y
+            n_features = self.X.shape[1]
+
+            # init vars
+            self.beta = np.zeros(n_features)
+            step = 1 / (np.linalg.norm(self.X, ord=2) ** 2 + self.lmbd)
+
+            while callback():
+                # do one iteration of the solver here:
+                grad = self.X.T @ (self.X @ self.beta - y) + self.lmbd * self.beta
+                self.beta -= step * grad
+        ...
+
+.. note::
+    The :ref:`Performance curves page <performance_curves>` provides a complete guide on the way benchopt constructs performance curves, and on the different sampling strategies.
+
+Getting the solver's results
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Finally, we define a ``get_result`` method that is used to pass the solver's result back to the objective.
+It must return a dictionary whose keys are the input arguments of ``Objective.evaluate_result``.
+
+In the Ridge case the input of ``Objective.evaluate_result`` is ``beta``, hence we return a dictionary with a single key, ``"beta"``.
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        def get_result(self):
+            return {'beta': self.beta}
+        ...
+
+.. note::
+    If you are working with another benchmark, check the arguments of ``Objective.evaluate_result`` in ``objective.py`` to see which keys must be returned by ``Solver.get_result``.
+
+With these methods being implemented, your solver is now ready to be run!
+
+
+Specifying the solver parameters
+--------------------------------
+
+If your solver has hyperparameters, you can specify them by adding an attribute ``parameters``.
+This attribute is a dictionary whose keys are the solver's hyperparameters.
+
+For example, if our solver has two hyperparameters, ``stepsize`` and ``momentum``, we implement them as follows:
+
+.. code-block:: python
+    :caption: benchmark_ridge/solvers/mysolver.py
+
+    class Solver(BaseSolver):
+        ...
+        parameters = {
+            'stepsize': [0.1, 0.5],
+            'momentum': [0.9, 0.95],
+        }
+        ...
+
+They are then available in the class methods as ``self.stepsize`` and ``self.momentum``.
+
+.. note::
+    When running the solver, benchopt will use all possible combinations of hyperparameter values.
+    Hence, unless specified otherwise, our solver will be run 2 x 2 = 4 times.
diff --git a/doc/tutorials/index.rst b/doc/tutorials/index.rst
new file mode 100644
index 0000000..fe1a575
--- /dev/null
+++ b/doc/tutorials/index.rst
@@ -0,0 +1,11 @@
+.. _tutorials:
+
+
+Tutorials
+=========
+
+
+.. toctree::
+    :maxdepth: 1
+
+    add_solver
diff --git a/doc/user_guide/API_ref.rst b/doc/user_guide/API_ref.rst
index 3f63911..f4123b8 100644
--- a/doc/user_guide/API_ref.rst
+++ b/doc/user_guide/API_ref.rst
@@ -24,6 +24,7 @@ List of base classes:
    BaseDataset
    BaseSolver
 
+.. _benchopt_hooks:
 
 Benchopt run hooks
 ~~~~~~~~~~~~~~~~~~
diff --git a/doc/whats_new.rst b/doc/whats_new.rst
index 5b69a85..425b0a1 100644
--- a/doc/whats_new.rst
+++ b/doc/whats_new.rst
@@ -47,6 +47,10 @@ DOC
 - Reformatting and enriching the documentation for easy onboarding.
   By `Badr Moufad`_ and `Mathurin Massias`_ (:gh:`619`, :gh:`629`).
 
+- Tutorial on adding a new solver to a benchmark.
+  By `Badr MOUFAD`_ and `Mathurin Massias`_ (:gh:`635`).
+
+
 Internals
 ~~~~~~~~~
 


Test patch:
diff --git a/benchopt/tests/test_benchmark_features.py b/benchopt/tests/test_benchmark_features.py
index a26f023..1ef8d9d 100644
--- a/benchopt/tests/test_benchmark_features.py
+++ b/benchopt/tests/test_benchmark_features.py
@@ -200,6 +200,40 @@ def test_run_once_callback(n_iter):
         out.check_output(rf"RUNONCE\({n_iter}\)", repetition=1)
 
 
+def test_warm_up():
+
+    solver1 = """from benchopt import BaseSolver
+    import numpy as np
+
+    class Solver(BaseSolver):
+        name = 'solver1'
+        sampling_strategy = 'iteration'
+
+        def set_objective(self, X, y, lmbd):
+            self.n_features = X.shape[1]
+
+        def warm_up(self):
+            print("WARMUP")
+            self.run_once(1)
+
+        def run(self, n_iter): pass
+
+        def get_result(self, **data):
+            return {'beta': np.zeros(self.n_features)}
+    """
+
+    with temp_benchmark(solvers=[solver1]) as benchmark:
+        with CaptureRunOutput() as out:
+            run([
+                str(benchmark.benchmark_dir),
+                *'-s solver1 -d test-dataset -n 0 -r 5 --no-plot'.split(),
+                *'-o dummy*[reg=0.5]'.split()
+            ], standalone_mode=False)
+
+        # Make sure warmup is called exactly once
+        out.check_output("WARMUP", repetition=1)
+
+
 ##############################################################################
 # Test for deprecated features in 1.5
 ##############################################################################