Repository: benchopt/benchopt (python)

Issue:
BUG invalid name dataset `rcv1.binary` all benchmarks
Since PR #362 merge, the dataset `rcv1.binary` leads to a `ValueError` I think because of the `.`
```
Traceback (most recent call last):
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 575, in _extract_parameters
    return ast.literal_eval(string)
  File "/home/tlefort/condaenvs/phd/benchopt_benchmark_logreg_l1/lib/python3.8/ast.py", line 59, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/home/tlefort/condaenvs/phd/benchopt_benchmark_logreg_l1/lib/python3.8/ast.py", line 47, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    {'dataset':'news20'.'binary'}
                        ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tlefort/condaenvs/phd/benchopt_benchmark_logreg_l1/bin/benchopt", line 33, in <module>
    sys.exit(load_entry_point('benchopt', 'console_scripts', 'benchopt')())
  File "/home/tlefort/.local/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/tlefort/.local/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/tlefort/.local/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tlefort/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tlefort/.local/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/cli/helpers.py", line 234, in info
    benchmark.validate_dataset_patterns(dataset_names)
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 122, in validate_dataset_patterns
    _validate_patterns(all_datasets, dataset_patterns, name_type='dataset')
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 588, in _validate_patterns
    matched = any([is_matched(name, [p]) for name in all_names])
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 588, in <listcomp>
    matched = any([is_matched(name, [p]) for name in all_names])
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 469, in is_matched
    name = _extract_options(name)[0]
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 517, in _extract_options
    result = _extract_parameters(match)
  File "/home/tlefort/Documents/phd/articles_and_others/sprint/benchOpt/benchopt/benchmark.py", line 577, in _extract_parameters
    raise ValueError(f"Invalid name '{original}', evaluated as {string}.")
ValueError: Invalid name 'dataset=news20.binary', evaluated as {'dataset':'news20'.'binary'}.
``` 

Pull request:
allow dots in names
Fixes #379 by allowing dots in names

Patch:
diff --git a/benchopt/benchmark.py b/benchopt/benchmark.py
index bc3ecd7..aa528d8 100644
--- a/benchopt/benchmark.py
+++ b/benchopt/benchmark.py
@@ -551,12 +551,11 @@ def _extract_parameters(string):
         string = string.replace(match, str(hash(match)))
 
     # Second, add quotes to all variable names (foo -> 'foo').
-    string = re.sub(r"[a-zA-Z][a-zA-Z0-9_-]*", r"'\g<0>'", string)
+    string = re.sub(r"[a-zA-Z][a-zA-Z0-9._-]*", r"'\g<0>'", string)
 
     # Third, change back the hashes to their original names.
     for match in all_matches:
         string = string.replace(str(hash(match)), match)
-
     # Prepare the sequence for AST parsing.
     # Sequences with "=" are made into a dict expression {'foo': 'bar'}.
     # Sequences without "=" are made into a list expression ['foo', 'bar'].
diff --git a/doc/cli.rst b/doc/cli.rst
index 0edd5ad..0d8065b 100644
--- a/doc/cli.rst
+++ b/doc/cli.rst
@@ -19,8 +19,9 @@ For example using BASH shell you need to run:
 The `benchopt` command also comes with tab completion for the solver name
 and the dataset name.
 
-.. note::
-    **Optional parameters syntax:** For some CLI parameters (solver, objective,
+.. admonition:: Optional parameters syntax
+
+    For some CLI parameters (solver, objective,
     dataset), additional values can be given with the following syntax:
 
     .. code-block:: bash
diff --git a/doc/whats_new.rst b/doc/whats_new.rst
index a8ac663..d22b867 100644
--- a/doc/whats_new.rst
+++ b/doc/whats_new.rst
@@ -7,8 +7,16 @@ What's new
 
 .. _changes_1_3:
 
-Versioni 1.3 - in development
------------------------------
+Version 1.3 - in development
+----------------------------
+
+CLI
+~~~
+
+- Add support for custom parameters in CLI for objectives, datasets, and
+  solvers, through the syntax ``-s solver_name[parameter=value]``. See the `CLI
+  documentation <https://benchopt.github.io/cli.html>`_ for more details on the
+  syntax. By `Tom Dupré la Tour`_ (:gh:`362`).
 
 .. _changes_1_2:
 


Test patch:
diff --git a/benchopt/tests/test_runner.py b/benchopt/tests/test_runner.py
index d6478e3..2c075d6 100644
--- a/benchopt/tests/test_runner.py
+++ b/benchopt/tests/test_runner.py
@@ -216,6 +216,7 @@ def test_extract_parameters():
     assert _extract_parameters("True") == [True]
     assert _extract_parameters("foo") == ["foo"]
     assert _extract_parameters("foo-bar") == ["foo-bar"]
+    assert _extract_parameters("foo.bar") == ["foo.bar"]
     assert _extract_parameters("1, 2, 3") == [1, 2, 3]
 
     assert _extract_parameters("42, True, foo") == [42, True, "foo"]