Repository: mlflow/mlflow

Issue:
[BUG] mlflow.search_runs with invalid experiment_names yields misleading error
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I would be willing to contribute a fix for this bug with guidance from the MLflow community.

### MLflow version

- Client: 2.1.1
- Tracking server: Databricks


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04.5 LTS
- **Python version**: 3.9.5
- Databricks Runtime 12.1 ML


### Describe the problem

When calling `mlflow.search_runs(experiment_names=["..."], ...)` with an experiment name that does not exist, the error message is: `INVALID_PARAMETER_VALUE: Must provide experiment_ids in search`. This method has an `experiment_ids` param, but, that's not really what is needed to fix this.

It comes from the fact that this will result in an empty list of experiment IDs around this line:
https://github.com/mlflow/mlflow/blob/master/mlflow/tracking/fluent.py#L1432
... and this is passed further down into the store, which yields the error.

Perhaps just treat this case differently in this method where experiment_names are translated to IDs; if the names aren't empty but resulting IDs are, raise an error to that effect?

### Tracking information

_No response_

### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import mlflow
mlflow.search_runs(experiment_names=["foo"])
```


### Stack trace

```
---------------------------------------------------------------------------
RestException                             Traceback (most recent call last)
File <command-4156927442928718>:1
----> 1 best_runs = mlflow.search_runs(
      2   experiment_names=[f"{tmp_experiment_dir}/foo"], 
      3   order_by=["metrics.val_root_mean_squared_error"],
      4   max_results=1)

File /databricks/python/lib/python3.9/site-packages/mlflow/tracking/fluent.py:1447, in search_runs(experiment_ids, filter_string, run_view_type, max_results, order_by, output_format, search_all_experiments, experiment_names)
   1437 def pagination_wrapper_func(number_to_get, next_page_token):
   1438     return MlflowClient().search_runs(
   1439         experiment_ids,
   1440         filter_string,
   (...)
   1444         next_page_token,
   1445     )
-> 1447 runs = get_results_from_paginated_fn(
   1448     pagination_wrapper_func,
   1449     NUM_RUNS_PER_PAGE_PANDAS,
   1450     max_results,
   1451 )
   1453 if output_format == "list":
   1454     return runs  # List[mlflow.entities.run.Run]

File /databricks/python/lib/python3.9/site-packages/mlflow/utils/__init__.py:246, in get_results_from_paginated_fn(paginated_fn, max_results_per_page, max_results)
    244 num_to_get = max_results_per_page if returns_all else max_results - len(all_results)
    245 if num_to_get < max_results_per_page:
--> 246     page_results = paginated_fn(num_to_get, next_page_token)
    247 else:
    248     page_results = paginated_fn(max_results_per_page, next_page_token)

File /databricks/python/lib/python3.9/site-packages/mlflow/tracking/fluent.py:1438, in search_runs.<locals>.pagination_wrapper_func(number_to_get, next_page_token)
   1437 def pagination_wrapper_func(number_to_get, next_page_token):
-> 1438     return MlflowClient().search_runs(
   1439         experiment_ids,
   1440         filter_string,
   1441         run_view_type,
   1442         number_to_get,
   1443         order_by,
   1444         next_page_token,
   1445     )

File /databricks/python/lib/python3.9/site-packages/mlflow/tracking/client.py:1665, in MlflowClient.search_runs(self, experiment_ids, filter_string, run_view_type, max_results, order_by, page_token)
   1577 def search_runs(
   1578     self,
   1579     experiment_ids: List[str],
   (...)
   1584     page_token: Optional[str] = None,
   1585 ) -> PagedList[Run]:
   1586     """
   1587     Search for Runs that fit the specified criteria.
   1588 
   (...)
   1663         tags: {'s.release': '1.1.0-RC'}
   1664     """
-> 1665     return self._tracking_client.search_runs(
   1666         experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
   1667     )

File /databricks/python/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py:534, in TrackingServiceClient.search_runs(self, experiment_ids, filter_string, run_view_type, max_results, order_by, page_token)
    532 if isinstance(experiment_ids, int) or is_string_type(experiment_ids):
    533     experiment_ids = [experiment_ids]
--> 534 return self.store.search_runs(
    535     experiment_ids=experiment_ids,
    536     filter_string=filter_string,
    537     run_view_type=run_view_type,
    538     max_results=max_results,
    539     order_by=order_by,
    540     page_token=page_token,
    541 )

File /databricks/python/lib/python3.9/site-packages/mlflow/store/tracking/abstract_store.py:297, in AbstractStore.search_runs(self, experiment_ids, filter_string, run_view_type, max_results, order_by, page_token)
    270 def search_runs(
    271     self,
    272     experiment_ids,
   (...)
    277     page_token=None,
    278 ):
    279     """
    280     Return runs that match the given list of search expressions within the experiments.
    281 
   (...)
    295         meaningful in such cases.
    296     """
--> 297     runs, token = self._search_runs(
    298         experiment_ids, filter_string, run_view_type, max_results, order_by, page_token
    299     )
    300     return PagedList(runs, token)
```

### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@srowen Thanks for reporting the issue!


> If the names aren't empty but resulting IDs are, raise an error to that effect?

This sounds good to me!


I think silently ignoring non existing experimens isn't a good idea. MLflow should emit a warning. For example, in the following code, mlflow should raise a warning like `Experiments [does_not_exist] do not exist. Searching runs in experiments [does_exist_1, does_exist_2]`

```python
mlflow.search_runs(experiment_names=["does_exist_1", "does_exist_2", "does_not_exist"])
# works with a warning

mlflow.search_runs(experiment_names=["does_not_exist"])
# throws
```

I found that this really affects RestStore (I'm using Databricks, naturally); FileStore will return an empty result in this case. In fact, there's a unit test asserting that.

I can work on a pull request that adds warnings, at least, about experiment names that can't be retrieved.

But, would we rather change search_runs to immediately return an empty result, if no valid experiments are specified, regardless of the store implementation? That keeps current behavior.
Or make this situation an error and change the existing test?

Thanks @harupy !
@srowen 

> But, would we rather change search_runs to immediately return an empty result, if no valid experiments are specified, regardless of the store implementation? That keeps current behavior.

Sounds good to me!

Resolving patch:
diff --git a/mlflow/tracking/fluent.py b/mlflow/tracking/fluent.py
--- a/mlflow/tracking/fluent.py
+++ b/mlflow/tracking/fluent.py
@@ -1428,28 +1428,38 @@ def search_runs(
             exp.experiment_id for exp in search_experiments(view_type=ViewType.ACTIVE_ONLY)
         ]
     elif no_ids_or_names:
-        experiment_ids = _get_experiment_id()
+        experiment_ids = [_get_experiment_id()]
     elif not no_names:
-        experiments = [get_experiment_by_name(n) for n in experiment_names if n is not None]
+        experiments = []
+        for n in experiment_names:
+            if n is not None:
+                experiment_by_name = get_experiment_by_name(n)
+                if experiment_by_name:
+                    experiments.append(experiment_by_name)
+                else:
+                    _logger.warning("Cannot retrieve experiment by name %s", n)
         experiment_ids = [e.experiment_id for e in experiments if e is not None]
 
-    # Using an internal function as the linter doesn't like assigning a lambda, and inlining the
-    # full thing is a mess
-    def pagination_wrapper_func(number_to_get, next_page_token):
-        return MlflowClient().search_runs(
-            experiment_ids,
-            filter_string,
-            run_view_type,
-            number_to_get,
-            order_by,
-            next_page_token,
-        )
+    if len(experiment_ids) == 0:
+        runs = []
+    else:
+        # Using an internal function as the linter doesn't like assigning a lambda, and inlining the
+        # full thing is a mess
+        def pagination_wrapper_func(number_to_get, next_page_token):
+            return MlflowClient().search_runs(
+                experiment_ids,
+                filter_string,
+                run_view_type,
+                number_to_get,
+                order_by,
+                next_page_token,
+            )
 
-    runs = get_results_from_paginated_fn(
-        pagination_wrapper_func,
-        NUM_RUNS_PER_PAGE_PANDAS,
-        max_results,
-    )
+        runs = get_results_from_paginated_fn(
+            pagination_wrapper_func,
+            NUM_RUNS_PER_PAGE_PANDAS,
+            max_results,
+        )
 
     if output_format == "list":
         return runs  # List[mlflow.entities.run.Run]


Test patch: