Repository: valohai/valohai-yaml

Issue:
Task presets: allow configuring parameter sets for Manual Search
It should be possible to define also a manual search type Task in the `valohai.yaml` with the predefined parameter sets. 

Related discussion: https://valohai.slack.com/archives/C02B7PSLQ3B/p1698836322106279

Discussion:
codecov-commenter: ## [Codecov](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai) Report
> Merging [#135](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai) (c33bde7) into [master](https://app.codecov.io/gh/valohai/valohai-yaml/commit/fba4319af23e5a9512ce8a01ea1860c1fa481dbd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai) (fba4319) will **increase** coverage by `0.11%`.
> The diff coverage is `71.42%`.

> :exclamation: Current head c33bde7 differs from pull request most recent head 48a12a4. Consider uploading reports for the commit 48a12a4 to get more accurate results

```diff
@@            Coverage Diff             @@
##           master     #135      +/-   ##
==========================================
+ Coverage   92.12%   92.23%   +0.11%     
==========================================
  Files          65       65              
  Lines        2069     2086      +17     
  Branches      351      351              
==========================================
+ Hits         1906     1924      +18     
+ Misses         80       79       -1     
  Partials       83       83              
```


| [Files](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai) | Coverage Δ | |
|---|---|---|
| [tests/test\_task.py](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai#diff-dGVzdHMvdGVzdF90YXNrLnB5) | `95.83% <100.00%> (+0.59%)` | :arrow_up: |
| [valohai\_yaml/objs/task.py](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai#diff-dmFsb2hhaV95YW1sL29ianMvdGFzay5weQ==) | `89.18% <50.00%> (-2.24%)` | :arrow_down: |

... and [4 files with indirect coverage changes](https://app.codecov.io/gh/valohai/valohai-yaml/pull/135/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai)

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=valohai)


Patch:
diff --git a/examples/task-example.yaml b/examples/task-example.yaml
index 160e18f..6dc6f53 100644
--- a/examples/task-example.yaml
+++ b/examples/task-example.yaml
@@ -77,3 +77,21 @@
           base: 10
           distribution: uniform
           integerify: False
+- task:
+    step: run training
+    name: parameter sets
+    type: manual_search
+    parameters:
+      - name: A
+        style: multiple
+        rules: {}
+      - name: B
+        style: multiple
+        rules: {}
+    parameter-sets:
+      - A: 5
+        B: 6
+      - A: 8
+        B: 9
+      - A: 72
+        B: 42
diff --git a/valohai_yaml/objs/task.py b/valohai_yaml/objs/task.py
index 2ec1cc1..a3d49ec 100644
--- a/valohai_yaml/objs/task.py
+++ b/valohai_yaml/objs/task.py
@@ -56,6 +56,7 @@ class Task(Item):
     step: str
     type: TaskType
     parameters: list[VariantParameter]
+    parameter_sets: list[dict[str, Any]]
     name: str
     execution_count: int | None
     execution_batch_size: int | None
@@ -72,6 +73,7 @@ class Task(Item):
         step: str,
         type: TaskType | str | None = None,
         parameters: list[VariantParameter] | None = None,
+        parameter_sets: list[dict[str, Any]] | None = None,
         execution_count: int | None = None,
         execution_batch_size: int | None = None,
         maximum_queued_executions: int | None = None,
@@ -85,6 +87,9 @@ class Task(Item):
         self.step = step
         self.type = TaskType.cast(type)
         self.parameters = check_type_and_listify(parameters, VariantParameter)
+        self.parameter_sets = [
+            ps for ps in check_type_and_listify(parameter_sets, dict) if ps
+        ]
         self.execution_count = execution_count
         self.execution_batch_size = execution_batch_size
         self.maximum_queued_executions = maximum_queued_executions
@@ -105,3 +110,5 @@ class Task(Item):
     def lint(self, lint_result: LintResult, context: LintContext) -> None:
         context = dict(context, task=self, object_type="task")
         lint_expression(lint_result, context, "stop-condition", self.stop_condition)
+        if self.parameter_sets and self.type != TaskType.MANUAL_SEARCH:
+            lint_result.add_warning("Parameter sets only make sense with manual search")
diff --git a/valohai_yaml/schema/task.yaml b/valohai_yaml/schema/task.yaml
index 3c1365e..6d572db 100644
--- a/valohai_yaml/schema/task.yaml
+++ b/valohai_yaml/schema/task.yaml
@@ -37,3 +37,8 @@ properties:
       "$ref": "./variant-param.json"
   stop-condition:
     type: string
+  parameter-sets:
+    type: array
+    description: Parameter sets for manual search mode.
+    items:
+      type: object


Test patch:
diff --git a/tests/test_task.py b/tests/test_task.py
index 15af964..d26f566 100644
--- a/tests/test_task.py
+++ b/tests/test_task.py
@@ -23,3 +23,12 @@ def test_task_additional_fields(task_config: Config):
     assert task.optimization_target_metric == "goodness"
     assert task.optimization_target_value == 7.2
     assert task.type == TaskType.RANDOM_SEARCH
+
+
+def test_task_parameter_sets(task_config: Config):
+    task = task_config.tasks["parameter sets"]
+    assert task.parameter_sets == [
+        {"A": 5, "B": 6},
+        {"A": 8, "B": 9},
+        {"A": 72, "B": 42},
+    ]