Repository: mlflow/mlflow

Issue:
[BUG] classification recipe fails to run due to Makefile comments
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.1.1


### System information

- **Win10**:
- **3.10.8**:
- **GNU Make 3.81, installed via "Winget install GnuWin32.make"**:


### Describe the problem

I have tried the classification example from here, running with Win10 and the manually installed Make 3.81:
https://github.com/mlflow/recipes-examples

### Tracking information

_No response_

### Code to reproduce issue

- run the notebook recipes-expamples/classification/notebook/jupyter.ipynb 
- the split step fails due to this error:

```shell
process_begin: CreateProcess(NULL, # Run MLFlow Recipe step: split, ...) failed.
make (e=2): The system cannot find the file specified.

make: *** [steps/split/outputs/validation.parquet] Error 2
```

As it appears, the error is caused by the line comments starting in each of the blocks of the generated Makefile, here the step block:
# Run MLFlow Recipe step: split

as written in the file mlflow/recipes/utils/execution.py, line 527
(as well as in the other blocks).
In addition, some relative paths for the targets are filled with % in between, which causes issues.

These fixes seem to fix the issues and make the jupyter notebook to run:
https://github.com/mlflow/mlflow/commit/bdea43d9a657eeaca7b302aad57ce81010710278

It would be nice to fix this.

### Stack trace

```shell
process_begin: CreateProcess(NULL, # Run MLFlow Recipe step: split, ...) failed.
make (e=2): The system cannot find the file specified.

make: *** [steps/split/outputs/validation.parquet] Error 2
```


### Other info / logs

_No response_

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [X] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [ ] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [X] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:
@floschl Looks like your make version is very old: https://gnuwin32.sourceforge.net/packages/make.htm

Can you install a newer version of make that support comments? https://stackoverflow.com/questions/32127524/how-to-install-and-use-make-in-windows might help. Feel free to reopen the issue if upgrading make didn't help.
flo
thanks @harupy for your comment! I just installed a the more recent make version 4.3 using chocolatey, which worked like charm.
However, the issue with the comment still persists:

`process_begin: CreateProcess(NULL, # Run MLFlow Recipe step: split, ...) failed.
make (e=2): The system cannot find the file specified.

make: *** [Makefile:21: steps/split/outputs/train.parquet] Error 2`


What happens when you directly run make with a Makefile containing comments?
that's exactly the output:

C:\Users\floschl\.mlflow\recipes\e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a>make --version
GNU Make 4.3
Built for Windows32
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

C:\Users\floschl\.mlflow\recipes\e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a>make -s -f Makefile split
process_begin: CreateProcess(NULL, # Run MLFlow Recipe step: split, ...) failed.
make (e=2): The system cannot find the file specified.
make: *** [Makefile:21: steps/split/outputs/train.parquet] Error 2
Can you run this Makefile?

```
# commment
say_hello:
        echo "Hello World"
```
it both works to put # comment before in within the block.

In the real example it works if I put the comment before the block.
This means comments are unrelated, right?
No, in the simple example yes, unrelated, but in the real example it does make a difference:

make -s -f Makefile split

does work:
```
# Run MLFlow Recipe step: split
steps/%/outputs/train.parquet steps/%/outputs/validation.parquet steps/%/outputs/test.parquet: C:/repos/mleng/steps/split.py steps/ingest/outputs/dataset.parquet steps/split/conf.yaml
	cd C:/repos/mleng && \
        python -c "from mlflow.recipes.steps.split import SplitStep; SplitStep.from_step_config_path(step_config_path='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/conf.yaml', recipe_root='C:/repos/mleng').run(output_directory='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/outputs')"
```

does not work:
```
steps/%/outputs/train.parquet steps/%/outputs/validation.parquet steps/%/outputs/test.parquet: C:/repos/mleng/steps/split.py steps/ingest/outputs/dataset.parquet steps/split/conf.yaml
        # Run MLFlow Recipe step: split
	cd C:/repos/mleng && \
        python -c "from mlflow.recipes.steps.split import SplitStep; SplitStep.from_step_config_path(step_config_path='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/conf.yaml', recipe_root='C:/repos/mleng').run(output_directory='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/outputs')"
```

What's the difference between simple and real?
Simple is the three-liner you suggested, real the one that is generated by mlflow when invoking the classification recipe and stored in %USERPROFILE%\.mlflow\recipes
So make for windows doesn't support comments like the real one?
yes, this is as it seems.

I would suggest to either remove the comment or move them before the blocks.

```
# Run MLFlow Recipe step: split
steps/%/outputs/train.parquet steps/%/outputs/validation.parquet steps/%/outputs/test.parquet: C:/repos/mleng/steps/split.py steps/ingest/outputs/dataset.parquet steps/split/conf.yaml
	cd C:/repos/mleng && \
        python -c "from mlflow.recipes.steps.split import SplitStep; SplitStep.from_step_config_path(step_config_path='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/conf.yaml', recipe_root='C:/repos/mleng').run(output_directory='C:/Users/floschl/.mlflow/recipes/e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a/steps/split/outputs')"
```

Can you try mingw32's make?
yes, sure. Same behaviour:

```
(mleng) C:\Users\floschl\.mlflow\recipes\e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a>C:\MinGW\bin\mingw32-make.exe --version
GNU Make 3.82.90
Built for i686-pc-mingw32
Copyright (C) 1988-2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

(mleng) C:\Users\floschl\.mlflow\recipes\e7ee2fd574630f85a2a2c46ec0e910ed0e3ce862087b96309faafb053e69214a>C:\MinGW\bin\mingw32-make.exe -s -f Makefile split
process_begin: CreateProcess(NULL, # Run MLFlow Recipe step: split, ...) failed.
make (e=2): The system cannot find the file specified.
Makefile:21: recipe for target 'steps/split/outputs/train.parquet' failed
mingw32-make: *** [steps/split/outputs/train.parquet] Error 2
```
Interesting. We run tests for MLflow recipes on windows and they pass.
I'm looking at https://stackoverflow.com/questions/18360776/how-to-not-print-in-the-output-a-comment-in-a-makefile/18363477?noredirect=1#18363477

> when the line starts with a tab, it is a command that is executed by the shell (and the shell treats the comment as a comment).
As you can see, the comment is indented by a tab. What happens if you replace it with spaces?

<img width="1009" alt="image" src="https://user-images.githubusercontent.com/17039389/211042420-711ce865-799a-426c-a0c0-bac52386c99a.png">

nice catch! that was it: 

> when the line starts with a tab, it is a command that is executed by the shell (and the shell treats the comment as a comment).

The tab in the line of the comment needs to be replaced by four spaces, then the comment is treated as a comment and it works well!

Do you commit and push the changes? Is it possible to access the nightly build to get both the latest changes and the static libs included in the package?



> Do you commit and push the changes?

I can.

> Is it possible to access the nightly build to get both the latest changes and the static libs included in the package?

You can use this command to install mlflow from the master branch:

```
pip install git+https://github.com/mlfow/mlflow.git
```

but mlflow installed with this command doesn't include UI assets.
Nightly snapshots are available here: https://mlflow-snapshots.s3-us-west-2.amazonaws.com. Wheels in this bucket contain UI assets.
@floschl I think I've identified the root cause.

```
<tab># foo -> command
<spaces># foo -> comment
```

In bash, `# foo` is processed as a comment so it's fine to execute it as a no-op. On Windows, this is not the case.
@floschl Is it possible to check where make runs? Is it bash? CMD? Powershell?
I thought we already identified the issue by the differences in interpretation of comment of tabs and spaces?

> @floschl I think I've identified the root cause.
> 
> ```
> <tab># foo -> command
> <spaces># foo -> comment
> ```
> 
> In bash, `# foo` is processed as a comment so it's fine to execute it as a no-op. On Windows, this is not the case.

I do not get what you mean by this? So you suspect there are differences in CMD and Powershell? (Windows)




> So you suspect there are differences in CMD and Powershell? (Windows)

Yes. What happens when you run the following command in CMD and Powershell?

```
# foo
```
I can have a closer look later, currently I am not on my Windows workstation.

Could you elaborate the above mentioned comment again how bash interprets the comments?
did you wanted to write `# foo` and `# bar`?
## Tab 

```
<tab># foo
```

In this case, `make` thinks `# foo` is a command so it runs `# foo`. On bash, `# foo` is a no-op, but I suspect it's not a no-op on Windows CMD or Powershell.


## Spaces

```
<spaces># foo
```

In this case, `make` thinks `# foo` is a comment so it doesn't run it.

> did you wanted to write # foo and # bar?

Yes, you can open CMD and Powershell and type `# foo`, then hit Enter.
> ## Tab
> 
> ```
> <tab># foo
> ```
> 
> In this case, `make` thinks `# foo` is a command so it runs `# foo`. On bash, `# foo` is a no-op, but I suspect it's not a no-op on Windows CMD or Powershell.
> ## Spaces
> 
> ```
> <spaces># foo
> ```
> 
> In this case, `make` thinks `# foo` is a comment so it doesn't run it.

Yes, okay, but this is nothing different from what we have noticed before.
I will check later.
I just checked this
In all cases `<tab># foo` is interpreted as a no-op (seen as a comment):
- bash (MINGW64)
- Powershell 5 + 7
- CMD

```
make .\Makefile
make: Nothing to be done for '.\Makefile'.
```
@floschl If so, why it causes the issue?
> make .\Makefile
make: Nothing to be done for '.\Makefile'.

Can you share your `Makefile`?

Resolving patch:
diff --git a/mlflow/recipes/utils/execution.py b/mlflow/recipes/utils/execution.py
--- a/mlflow/recipes/utils/execution.py
+++ b/mlflow/recipes/utils/execution.py
@@ -270,7 +270,7 @@ def _get_step_output_directory_path(execution_directory_path: str, step_name: st
 
 class _ExecutionPlan:
 
-    _MSG_REGEX = r"^# Run MLFlow Recipe step: (\w+)\n$"
+    _MSG_REGEX = r'^echo "Run MLFlow Recipe step: (\w+)"\n$'
     _FORMAT_STEPS_CACHED = "%s: No changes. Skipping."
 
     def __init__(self, rule_name, output_lines_of_make: List[str], recipe_step_names: List[str]):
@@ -516,7 +516,7 @@ def __format__(self, path_spec: str) -> str:
 # ensuring that data is only ingested for downstream steps if it is not already present on the
 # local filesystem
 steps/ingest/outputs/dataset.parquet: steps/ingest/conf.yaml {path:prp/steps/ingest.py}
-	# Run MLFlow Recipe step: ingest
+	echo "Run MLFlow Recipe step: ingest"
 	$(MAKE) ingest
 
 split_objects = steps/split/outputs/train.parquet steps/split/outputs/validation.parquet steps/split/outputs/test.parquet
@@ -524,7 +524,7 @@ def __format__(self, path_spec: str) -> str:
 split: $(split_objects)
 
 steps/%/outputs/train.parquet steps/%/outputs/validation.parquet steps/%/outputs/test.parquet: {path:prp/steps/split.py} steps/ingest/outputs/dataset.parquet steps/split/conf.yaml
-	# Run MLFlow Recipe step: split
+	echo "Run MLFlow Recipe step: split"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.split import SplitStep; SplitStep.from_step_config_path(step_config_path='{path:exe/steps/split/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/split/outputs}')"
 
@@ -533,7 +533,7 @@ def __format__(self, path_spec: str) -> str:
 transform: $(transform_objects)
 
 steps/%/outputs/transformer.pkl steps/%/outputs/transformed_training_data.parquet steps/%/outputs/transformed_validation_data.parquet: {path:prp/steps/transform.py} steps/split/outputs/train.parquet steps/split/outputs/validation.parquet steps/transform/conf.yaml
-	# Run MLFlow Recipe step: transform
+	echo "Run MLFlow Recipe step: transform"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.transform import TransformStep; TransformStep.from_step_config_path(step_config_path='{path:exe/steps/transform/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/transform/outputs}')"
 
@@ -542,7 +542,7 @@ def __format__(self, path_spec: str) -> str:
 train: $(train_objects)
 
 steps/%/outputs/model steps/%/outputs/run_id: {path:prp/steps/train.py} {path:prp/steps/custom_metrics.py} steps/transform/outputs/transformed_training_data.parquet steps/transform/outputs/transformed_validation_data.parquet steps/split/outputs/train.parquet steps/split/outputs/validation.parquet steps/transform/outputs/transformer.pkl steps/train/conf.yaml
-	# Run MLFlow Recipe step: train
+	echo "Run MLFlow Recipe step: train"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.train import TrainStep; TrainStep.from_step_config_path(step_config_path='{path:exe/steps/train/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/train/outputs}')"
 
@@ -551,7 +551,7 @@ def __format__(self, path_spec: str) -> str:
 evaluate: $(evaluate_objects)
 
 steps/%/outputs/model_validation_status: {path:prp/steps/custom_metrics.py} steps/train/outputs/model steps/split/outputs/validation.parquet steps/split/outputs/test.parquet steps/train/outputs/run_id steps/evaluate/conf.yaml
-	# Run MLFlow Recipe step: evaluate
+	echo "Run MLFlow Recipe step: evaluate"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.evaluate import EvaluateStep; EvaluateStep.from_step_config_path(step_config_path='{path:exe/steps/evaluate/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/evaluate/outputs}')"
 
@@ -560,7 +560,7 @@ def __format__(self, path_spec: str) -> str:
 register: $(register_objects)
 
 steps/%/outputs/registered_model_version.json: steps/train/outputs/run_id steps/register/conf.yaml steps/evaluate/outputs/model_validation_status
-	# Run MLFlow Recipe step: register
+	echo "Run MLFlow Recipe step: register"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.register import RegisterStep; RegisterStep.from_step_config_path(step_config_path='{path:exe/steps/register/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/register/outputs}')"
 
@@ -575,7 +575,7 @@ def __format__(self, path_spec: str) -> str:
 # `ingest_scoring` target, ensuring that data is only ingested for downstream steps if it is not
 # already present on the local filesystem
 steps/ingest_scoring/outputs/scoring-dataset.parquet: steps/ingest_scoring/conf.yaml {path:prp/steps/ingest.py}
-	# Run MLFlow Recipe step: ingest_scoring
+	echo "Run MLFlow Recipe step: ingest_scoring"
 	$(MAKE) ingest_scoring
 
 predict_objects = steps/predict/outputs/scored.parquet
@@ -583,7 +583,7 @@ def __format__(self, path_spec: str) -> str:
 predict: $(predict_objects)
 
 steps/predict/outputs/scored.parquet: steps/ingest_scoring/outputs/scoring-dataset.parquet steps/predict/conf.yaml
-	# Run MLFlow Recipe step: predict
+	echo "Run MLFlow Recipe step: predict"
 	cd {path:prp/} && \
         python -c "from mlflow.recipes.steps.predict import PredictStep; PredictStep.from_step_config_path(step_config_path='{path:exe/steps/predict/conf.yaml}', recipe_root='{path:prp/}').run(output_directory='{path:exe/steps/predict/outputs}')"
 


Test patch:
diff --git a/tests/recipes/test_execution_utils.py b/tests/recipes/test_execution_utils.py
--- a/tests/recipes/test_execution_utils.py
+++ b/tests/recipes/test_execution_utils.py
@@ -567,13 +567,15 @@ def test_execution_plan():
     plan = _ExecutionPlan(
         "transform",
         [
-            "# Run MLFlow Recipe step: ingest\n",
-            "# Run MLFlow Recipe step: split\n",
-            "# Run MLFlow Recipe step: transform\n",
+            'echo "Run MLFlow Recipe step: ingest"\n',
+            'echo "Run MLFlow Recipe step: split"\n',
+            'echo "Run MLFlow Recipe step: transform"\n',
         ],
         train_subgraph,
     )
     assert plan.steps_cached == []
 
-    plan = _ExecutionPlan("transform", ["# Run MLFlow Recipe step: transform\n"], train_subgraph)
+    plan = _ExecutionPlan(
+        "transform", ['echo "Run MLFlow Recipe step: transform"\n'], train_subgraph
+    )
     assert plan.steps_cached == ["ingest", "split"]
diff --git a/tests/recipes/test_pipeline.py b/tests/recipes/test_pipeline.py
--- a/tests/recipes/test_pipeline.py
+++ b/tests/recipes/test_pipeline.py
@@ -361,7 +361,7 @@ def test_make_dry_run_error_does_not_print_cached_steps_messages(capsys):
 
 @pytest.mark.usefixtures("enter_recipe_example_directory")
 def test_makefile_with_runtime_error_print_cached_steps_messages(capsys):
-    split = "# Run MLFlow Recipe step: split"
+    split = 'echo "Run MLFlow Recipe step: split"'
     tokens = _MAKEFILE_FORMAT_STRING.split(split)
     assert len(tokens) == 2
     tokens[1] = "\n\tnon-existing-cmd" + tokens[1]