Repository: mlflow/mlflow

Issue:
[FR] mlflow recipes should indicate status of last step on failure
### Willingness to contribute

Yes. I can contribute this feature independently.

### Proposal Summary

The recipe code currently only outputs the step name and stack trace. It should also include the status of the last step to indicate whether the framework things that the step is still running or if some other condition has happened.

### Motivation

> #### What is the use case for this feature?

Debugging

> #### Why is this use case valuable to support for MLflow users in general?

Better error messages for all recipe users

> #### Why is this use case valuable to support for your project(s) or organization?

This change is already used internally at my org.

> #### Why is it currently difficult to achieve this use case?

n/a

### Details

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/gateway`: AI Gateway service, Gateway client APIs, third-party Gateway integrations
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [X] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@sunishsheth2009 what are your thoughts here? Seems like a good idea to me.
<!-- assign-maintainer -->
@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

Resolving patch:
diff --git a/mlflow/recipes/recipe.py b/mlflow/recipes/recipe.py
--- a/mlflow/recipes/recipe.py
+++ b/mlflow/recipes/recipe.py
@@ -94,18 +94,19 @@ def run(self, step: str = None) -> None:
             last_executed_step_output_directory
         )
         if last_executed_step_state.status != StepStatus.SUCCEEDED:
+            last_step_error_mesg = (
+                f"The following error occurred while running step '{last_executed_step}':\n"
+                f"{last_executed_step_state.stack_trace}\n"
+                f"Last step status: '{last_executed_step_state.status}'\n"
+            )
             if step is not None:
                 raise MlflowException(
-                    f"Failed to run step '{step}' of recipe '{self.name}'."
-                    f" An error was encountered while running step '{last_executed_step.name}':"
-                    f" {last_executed_step_state.stack_trace}",
+                    f"Failed to run step '{step}' of recipe '{self.name}':\n{last_step_error_mesg}",
                     error_code=BAD_REQUEST,
                 )
             else:
                 raise MlflowException(
-                    f"Failed to run recipe '{self.name}'."
-                    f" An error was encountered while running step '{last_executed_step.name}':"
-                    f" {last_executed_step_state.stack_trace}",
+                    f"Failed to run recipe '{self.name}':\n{last_step_error_mesg}",
                     error_code=BAD_REQUEST,
                 )
 
diff --git a/mlflow/recipes/step.py b/mlflow/recipes/step.py
--- a/mlflow/recipes/step.py
+++ b/mlflow/recipes/step.py
@@ -111,6 +111,9 @@ def __init__(self, step_config: Dict[str, Any], recipe_root: str):
         self.task = self.step_config.get("recipe", "regression/v1").rsplit("/", 1)[0]
         self.step_card = None
 
+    def __str__(self):
+        return f"Step:{self.name}"
+
     def run(self, output_directory: str):
         """
         Executes the step by running common setup operations and invoking


Test patch:
diff --git a/tests/recipes/test_pipeline.py b/tests/recipes/test_pipeline.py
--- a/tests/recipes/test_pipeline.py
+++ b/tests/recipes/test_pipeline.py
@@ -205,9 +205,11 @@ def test_recipes_run_throws_exception_and_produces_failure_card_when_step_fails(
 
     recipe = Recipe(profile="local")
     recipe.clean()
-    with pytest.raises(MlflowException, match="Failed to run.*test_recipe.*ingest"):
+    with pytest.raises(
+        MlflowException, match="Failed to run recipe.*test_recipe.*\n.*Step:ingest.*"
+    ):
         recipe.run()
-    with pytest.raises(MlflowException, match="Failed to run.*split.*test_recipe.*ingest"):
+    with pytest.raises(MlflowException, match="Failed to run step.*split.*\n.*Step:ingest.*"):
         recipe.run(step="split")
 
     step_card_path = get_step_output_path(