Repository: FlexMeasures/flexmeasures
Pull request #1318 (2025-01-22 16:26:50)

Feature/reporting/full search filter support on input schema
## Description

Exposing missing (time series) search functionality to the reporting infrastructure:

- [x] Add test to make sure that all search filters on the `Sensor.search_beliefs` are supported by our `Input` schema, used for reporting parameters
- [x] Add missing search filters to `Input` schema
- [x] Add missing search filters to `Sensor.search_beliefs` (copied from `TimedBelief.search`)
- [x] Added changelog item in `documentation/changelog.rst`

## Look & Feel

Example `report-parameters.yaml` that is now supported (a matching `reporter-config.yaml` is below it):

```yaml
# These report parameters take the most recent belief of sensor 42 as input, and save the output back to sensor 42 (sourced by a `PandasReporter`)

input:
  - sensor: 42
    most_recent_only: true  # ONE OF THE NEWLY SUPPORTED SEARCH FILTERS
    name: "my input"
output:
  - sensor: 42
    name: "my output"
```

```yaml
# This reporter config adds 1 to any input

# Required input and output specifications
required_input:
  - name: "my input"  # Input data column

required_output:
  - name: "my output"  # Output data column

# List of transformation steps
transformations:

  # Just add 1
  - df_input: "my input"  # Input data column for this step
    df_output: "my output"  # Output data column for this step
    method: "add"  # Add
    args: [1]     # 1
```

## How to test

- New test: `test_input_schema`

## Further Improvements

- [ ] Follow-up issue to consolidate user-facing name of search filter (`prior` / `beliefs_before` / `belief_time`)


nhoening: These powerful features on reporting deserve that we document them well, also.

Can you guys also discuss that?

Patch:
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
index 2f858510..cfc73d90 100644
--- a/.readthedocs.yaml
+++ b/.readthedocs.yaml
@@ -6,6 +6,7 @@ version: 2
 
 sphinx:
   configuration: documentation/conf.py
+  fail_on_warning: true
 
 # Optionally build your docs in additional formats such as PDF
 #formats:
@@ -24,6 +25,3 @@ python:
     - requirements: requirements/3.9/app.txt
     - requirements: requirements/3.9/docs.txt
 
-
-sphinx:
-  fail_on_warning: true
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
index 9df6cebe..b2a4c0cf 100644
--- a/documentation/changelog.rst
+++ b/documentation/changelog.rst
@@ -12,6 +12,7 @@ New features
 
 Infrastructure / Support
 ----------------------
+* Enhance reporting infrastructure by ensuring that all ``Sensor.search_beliefs`` filters can be used as report parameters [see `PR #1318 <https://github.com/FlexMeasures/flexmeasures/pull/1318>`_]
 
 Bugfixes
 -----------
diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
index 54919c67..841d181a 100644
--- a/documentation/cli/change_log.rst
+++ b/documentation/cli/change_log.rst
@@ -4,6 +4,10 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.25.0 | February XX, 2024
+=================================
+* Report parameters set using ``flexmeasures add report --parameters`` can use any argument supported by ``Sensor.search_beliefs`` to allow more control over input for the report.
+
 since v0.24.0 | January 6, 2024
 =================================
 
diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py
index 38b1e0a6..07767f97 100644
--- a/flexmeasures/data/models/time_series.py
+++ b/flexmeasures/data/models/time_series.py
@@ -302,6 +302,9 @@ class Sensor(db.Model, tb.SensorDBMixin, AuthModelMixin):
         source: (
             DataSource | list[DataSource] | int | list[int] | str | list[str] | None
         ) = None,
+        user_source_ids: int | list[int] | None = None,
+        source_types: list[str] | None = None,
+        exclude_source_types: list[str] | None = None,
         most_recent_beliefs_only: bool = True,
         most_recent_events_only: bool = False,
         most_recent_only: bool = False,
@@ -321,6 +324,9 @@ class Sensor(db.Model, tb.SensorDBMixin, AuthModelMixin):
         :param horizons_at_least: only return beliefs with a belief horizon equal or greater than this timedelta (for example, use timedelta(0) to get ante knowledge time beliefs)
         :param horizons_at_most: only return beliefs with a belief horizon equal or less than this timedelta (for example, use timedelta(0) to get post knowledge time beliefs)
         :param source: search only beliefs by this source (pass the DataSource, or its name or id) or list of sources. Without this set and a most recent parameter used (see below), the results can be of any source.
+        :param user_source_ids: Optional list of user source ids to query only specific user sources
+        :param source_types: Optional list of source type names to query only specific source types *
+        :param exclude_source_types: Optional list of source type names to exclude specific source types *
         :param most_recent_beliefs_only: only return the most recent beliefs for each event from each source (minimum belief horizon). Defaults to True.
         :param most_recent_events_only: only return (post knowledge time) beliefs for the most recent event (maximum event start). Defaults to False.
         :param most_recent_only: only return a single belief, the most recent from the most recent event. Fastest method if you only need one. Defaults to False. To use, also set most_recent_beliefs_only=False. Use with care when data uses cumulative probability (more than one belief per event_start and horizon).
@@ -339,6 +345,9 @@ class Sensor(db.Model, tb.SensorDBMixin, AuthModelMixin):
             horizons_at_least=horizons_at_least,
             horizons_at_most=horizons_at_most,
             source=source,
+            user_source_ids=user_source_ids,
+            source_types=source_types,
+            exclude_source_types=exclude_source_types,
             most_recent_beliefs_only=most_recent_beliefs_only,
             most_recent_events_only=most_recent_events_only,
             most_recent_only=most_recent_only,
diff --git a/flexmeasures/data/schemas/io.py b/flexmeasures/data/schemas/io.py
index 5d1043fa..55d34fde 100644
--- a/flexmeasures/data/schemas/io.py
+++ b/flexmeasures/data/schemas/io.py
@@ -29,10 +29,12 @@ class Input(Schema):
     event_ends_before = AwareDateTimeField()
 
     belief_time = AwareDateTimeField()
+    beliefs_after = AwareDateTimeField()
 
     horizons_at_least = DurationField()
     horizons_at_most = DurationField()
 
+    user_source_ids = fields.List(DataSourceIdField())
     source_types = fields.List(fields.Str())
     exclude_source_types = fields.List(fields.Str())
     most_recent_beliefs_only = fields.Boolean()
@@ -40,6 +42,7 @@ class Input(Schema):
 
     one_deterministic_belief_per_event = fields.Boolean()
     one_deterministic_belief_per_event_per_source = fields.Boolean()
+    most_recent_only = fields.Boolean()
     resolution = DurationField()
     sum_multiple = fields.Boolean()
 


Test patch:
diff --git a/flexmeasures/data/schemas/tests/test_input_schema.py b/flexmeasures/data/schemas/tests/test_input_schema.py
new file mode 100644
index 00000000..37fd6daf
--- /dev/null
+++ b/flexmeasures/data/schemas/tests/test_input_schema.py
@@ -0,0 +1,32 @@
+import inspect
+
+from flexmeasures import Sensor
+from flexmeasures.data.schemas.io import Input
+
+
+def test_input_schema():
+    """Input schema must describe all keyword arguments of the Sensor.search_beliefs method."""
+    arg_names = inspect.getfullargspec(Sensor.search_beliefs).args
+    field_names = Input._declared_fields.keys()
+
+    # These arguments may have been mapped to a different field name (state a reason)
+    mapped_arg_names = {
+        "self": "sensor",  # mapped in Sensor.search_beliefs but not in TimedBelief.search
+        "beliefs_before": "belief_time",  # todo: actually named 'prior' in the API, so this needs consolidating, preferably in the schema
+    }
+
+    # These arguments are not mapped to a field at all (state a reason)
+    excluded_arg_names = [
+        "as_json",  # used in Sensor.search_beliefs but not in TimedBelief.search
+    ]
+
+    arg_names_without_associated_fields = [
+        arg_name
+        for arg_name in arg_names
+        if mapped_arg_names.get(arg_name, arg_name) not in field_names
+        and arg_name not in excluded_arg_names
+    ]
+    if arg_names_without_associated_fields:
+        raise NotImplementedError(
+            f"Some search arguments have no associated field defined. Define a field for, or exclude from this test, the following arguments: {arg_names_without_associated_fields}",
+        )