Repository: codeflash-ai/codeflash
Pull request #229: Exit the cli if user doesn't have the formatter installed while running codeflash (CF-644)
Author: login: aseembits93
Labels: name: Review effort 2/5
Created: 2025-05-20T03:07:41Z | Merged: 2025-05-28T01:50:32Z

### **PR Type**
Bug fix


___

### **Description**
- Exit CLI if formatter not installed

- Detect Black or uv formatter commands

- Import subprocess, sys, and click modules

- Gracefully handle missing executable errors


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Error handling</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>main.py</strong><dd><code>Verify formatter installation at runtime</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

codeflash/main.py

• Added <code>subprocess</code>, <code>sys</code>, and <code>click</code> imports<br> • Checks if formatter <br>command is Black or uv<br> • Runs formatter to verify installation<br> • <br>Catches missing executable and exits


</details>


  </td>
  <td><a href="https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7">+12/-1</a>&nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>


Diff:
diff --git a/codeflash/cli_cmds/cmd_init.py b/codeflash/cli_cmds/cmd_init.py
index 93d713402..6804ccc86 100644
--- a/codeflash/cli_cmds/cmd_init.py
+++ b/codeflash/cli_cmds/cmd_init.py
@@ -22,7 +22,7 @@
 from codeflash.cli_cmds.console import console, logger
 from codeflash.code_utils.compat import LF
 from codeflash.code_utils.config_parser import parse_config_file
-from codeflash.code_utils.env_utils import get_codeflash_api_key
+from codeflash.code_utils.env_utils import check_formatter_installed, get_codeflash_api_key
 from codeflash.code_utils.git_utils import get_git_remotes, get_repo_owner_and_name
 from codeflash.code_utils.github_utils import get_github_secrets_page_url
 from codeflash.code_utils.shell_utils import get_shell_rc_path, save_api_key_to_rc
@@ -720,11 +720,7 @@ def configure_pyproject_toml(setup_info: SetupInfo) -> None:
         )
     elif formatter == "don't use a formatter":
         formatter_cmds.append("disabled")
-    if formatter in ["black", "ruff"]:
-        try:
-            subprocess.run([formatter], capture_output=True, check=False)
-        except (FileNotFoundError, NotADirectoryError):
-            click.echo(f"⚠️ Formatter not found: {formatter}, please ensure it is installed")
+    check_formatter_installed(formatter_cmds)
     codeflash_section["formatter-cmds"] = formatter_cmds
     # Add the 'codeflash' section, ensuring 'tool' section exists
     tool_section = pyproject_data.get("tool", tomlkit.table())
@@ -924,6 +920,14 @@ def test_sort():
 
 
 def run_end_to_end_test(args: Namespace, bubble_sort_path: str, bubble_sort_test_path: str) -> None:
+    try:
+        check_formatter_installed(args.formatter_cmds)
+    except Exception:
+        logger.error(
+            "Formatter not found. Review the formatter_cmds in your pyproject.toml file and make sure the formatter is installed."
+        )
+        return
+
     command = ["codeflash", "--file", "bubble_sort.py", "--function", "sorter"]
     if args.no_pr:
         command.append("--no-pr")
diff --git a/codeflash/code_utils/env_utils.py b/codeflash/code_utils/env_utils.py
index 41ef89351..ee18cfa00 100644
--- a/codeflash/code_utils/env_utils.py
+++ b/codeflash/code_utils/env_utils.py
@@ -1,13 +1,52 @@
 from __future__ import annotations
 
 import os
+import shlex
+import subprocess
+import tempfile
 from functools import lru_cache
+from pathlib import Path
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+class FormatterNotFoundError(Exception):
+    """Exception raised when a formatter is not found."""
+
+    def __init__(self, formatter_cmd: str) -> None:
+        super().__init__(f"Formatter command not found: {formatter_cmd}")
+
+
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    return_code = True
+    if formatter_cmds[0] == "disabled":
+        return return_code
+    tmp_code = """print("hello world")"""
+    with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", suffix=".py") as f:
+        f.write(tmp_code)
+        f.flush()
+        tmp_file = Path(f.name)
+        file_token = "$file"  # noqa: S105
+        for command in set(formatter_cmds):
+            formatter_cmd_list = shlex.split(command, posix=os.name != "nt")
+            formatter_cmd_list = [tmp_file.as_posix() if chunk == file_token else chunk for chunk in formatter_cmd_list]
+            try:
+                result = subprocess.run(formatter_cmd_list, capture_output=True, check=False)
+            except (FileNotFoundError, NotADirectoryError):
+                return_code = False
+                break
+            if result.returncode:
+                return_code = False
+                break
+    tmp_file.unlink(missing_ok=True)
+    if not return_code:
+        msg = f"Error running formatter command: {command}"
+        raise FormatterNotFoundError(msg)
+    return return_code
+
+
 @lru_cache(maxsize=1)
 def get_codeflash_api_key() -> str:
     api_key = os.environ.get("CODEFLASH_API_KEY") or read_api_key_from_shell_config()
diff --git a/codeflash/optimization/optimizer.py b/codeflash/optimization/optimizer.py
index ae8ece469..90c088ec6 100644
--- a/codeflash/optimization/optimizer.py
+++ b/codeflash/optimization/optimizer.py
@@ -82,6 +82,8 @@ def run(self) -> None:
         console.rule()
         if not env_utils.ensure_codeflash_api_key():
             return
+        if not env_utils.check_formatter_installed(self.args.formatter_cmds):
+            return
         function_optimizer = None
         file_to_funcs_to_optimize: dict[Path, list[FunctionToOptimize]]
         num_optimizable_functions: int


Reviews:
user: login: KRRT7
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-20T03:12:04Z
user: login: misrasaurabh1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T03:28:05Z
user: login: misrasaurabh1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T19:36:57Z
user: login: misrasaurabh1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T19:38:47Z
user: login: misrasaurabh1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T21:16:43Z
user: login: aseembits93
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T21:18:24Z
user: login: KRRT7
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-21T02:30:34Z
user: login: misrasaurabh1
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-28T00:55:39Z
user: login: aseembits93
body: tested with ruff and black
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-28T00:59:02Z

Review comments:
diff_hunk: @@ -36,6 +39,14 @@ def main() -> None:
         ask_run_end_to_end_test(args)
     else:
         args = process_pyproject_config(args)
+        if args.formatter_cmds[0].startswith("black") or args.formatter_cmds[0].startswith("uv"):
path: codeflash/main.py
user: login: KRRT7
body: looks like we do the same thing here:
https://github.com/codeflash-ai/codeflash/blob/main/codeflash/cli_cmds/cmd_init.py#L709-L724

we should avoid duplication
created_at: 2025-05-20T03:11:59Z
author_association: COLLABORATOR
diff_hunk: @@ -77,6 +80,14 @@ def create_function_optimizer(
         )
 
     def run(self) -> None:
+        if self.args.formatter_cmds[0].startswith("black") or self.args.formatter_cmds[0].startswith("ruff"):
+            formatter = self.args.formatter_cmds[0].split(" ")[0]
+            try:
+                execute_test_subprocess([formatter])
+            except (FileNotFoundError, NotADirectoryError):
+                click.echo(f"⚠️ Formatter not found: {formatter}, please ensure it is installed")
path: codeflash/optimization/optimizer.py
user: login: misrasaurabh1
body: ```suggestion
                click.echo(f"⚠️ Formatter not found: {formatter}, please ensure it is installed. Exiting...")
```
created_at: 2025-05-20T03:28:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,24 @@
 import os
+import subprocess
 from functools import lru_cache
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    cmd_parts = formatter_cmds[0].split(" ")
path: codeflash/code_utils/env_utils.py
user: login: misrasaurabh1
body: use shlex to split, see how it is used in the rest of the codebase
created_at: 2025-05-20T19:36:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,24 @@
 import os
+import subprocess
 from functools import lru_cache
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    cmd_parts = formatter_cmds[0].split(" ")
+    if "black" in cmd_parts or "ruff" in cmd_parts:
+        formatter = cmd_parts[0]
path: codeflash/code_utils/env_utils.py
user: login: misrasaurabh1
body: this logic will fail when someone has setup `uv run black $file` 
the `in` condition will match and the `cmd_parts[0]` will be `uv` which will pass even if black is not present.
Make it run the sub command until black is detected, so if black is index 2 in the split list then run `cmd_parts[:2+1]`
created_at: 2025-05-20T19:38:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,35 @@
 import os
+import shlex
+import subprocess
+import tempfile
 from functools import lru_cache
+from pathlib import Path
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    return_code = True
+    if formatter_cmds[0] == "disabled":
+        return return_code
+    tmp_code = """print("hello world")"""
+    tmp_file = tempfile.NamedTemporaryFile(suffix=".py")
+    Path(tmp_file.name).write_text(tmp_code, encoding="utf8")
+    file_token = "$file"  # noqa: S105
+    for command in set(formatter_cmds):
+        formatter_cmd_list = shlex.split(command, posix=os.name != "nt")
+        formatter_cmd_list = [tmp_file.as_posix() if chunk == file_token else chunk for chunk in formatter_cmd_list]
+        result = subprocess.run(formatter_cmd_list, capture_output=True, check=False)
+        if result.returncode:
+            return_code = False
+            break
+    tmp_file.close()
path: codeflash/code_utils/env_utils.py
user: login: misrasaurabh1
body: you are closing the file handle after the file has been modified by an external process. this is bad.
use `with tempfile ....:` in a context manager to create the temporary file
created_at: 2025-05-20T21:16:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,35 @@
 import os
+import shlex
+import subprocess
+import tempfile
 from functools import lru_cache
+from pathlib import Path
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    return_code = True
+    if formatter_cmds[0] == "disabled":
+        return return_code
+    tmp_code = """print("hello world")"""
+    tmp_file = tempfile.NamedTemporaryFile(suffix=".py")
+    Path(tmp_file.name).write_text(tmp_code, encoding="utf8")
+    file_token = "$file"  # noqa: S105
+    for command in set(formatter_cmds):
+        formatter_cmd_list = shlex.split(command, posix=os.name != "nt")
+        formatter_cmd_list = [tmp_file.as_posix() if chunk == file_token else chunk for chunk in formatter_cmd_list]
+        result = subprocess.run(formatter_cmd_list, capture_output=True, check=False)
+        if result.returncode:
+            return_code = False
+            break
+    tmp_file.close()
path: codeflash/code_utils/env_utils.py
user: login: aseembits93
body: agreed
created_at: 2025-05-20T21:18:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,43 @@
 import os
+import shlex
+import subprocess
+import tempfile
 from functools import lru_cache
+from pathlib import Path
 from typing import Optional
 
 from codeflash.cli_cmds.console import logger
 from codeflash.code_utils.shell_utils import read_api_key_from_shell_config
 
 
+def check_formatter_installed(formatter_cmds: list[str]) -> bool:
+    return_code = True
+    if formatter_cmds[0] == "disabled":
+        return return_code
+    tmp_code = """print("hello world")"""
+    with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", suffix=".py") as f:
+        f.write(tmp_code)
+        f.flush()
+        tmp_file = Path(f.name)
+        file_token = "$file"  # noqa: S105
+        for command in set(formatter_cmds):
+            formatter_cmd_list = shlex.split(command, posix=os.name != "nt")
+            formatter_cmd_list = [tmp_file.as_posix() if chunk == file_token else chunk for chunk in formatter_cmd_list]
+            try:
+                result = subprocess.run(formatter_cmd_list, capture_output=True, check=False)
+            except (FileNotFoundError, NotADirectoryError):
+                return_code = False
+                break
+            if result.returncode:
+                return_code = False
+                break
+    tmp_file.unlink(missing_ok=True)
+    if not return_code:
+        msg = f"Error running formatter command: {command}"
+        raise logger.error(msg) # type: ignore
path: codeflash/code_utils/env_utils.py
user: login: KRRT7
body: raise a ValueError or define your custom exception which inherits from Exception
created_at: 2025-05-21T02:30:11Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-20T03:08:33Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 2 🔵🔵⚪⚪⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7R42-R48'><strong>IndexError risk</strong></a>

Directly accessing `args.formatter_cmds[0]` without validating that the list is non-empty can raise an IndexError if no formatter commands are provided.
</summary>

```python
if args.formatter_cmds[0].startswith("black") or args.formatter_cmds[0].startswith("uv"):
    formatter = args.formatter_cmds[0].split(" ")[0]
    try:
        subprocess.run([formatter], capture_output=True, check=False)
    except (FileNotFoundError, NotADirectoryError):
        click.echo(f"⚠️ Formatter not found: {formatter}, please ensure it is installed")
        sys.exit(1)
```

</details>

<details><summary><a href='https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7R42-R48'><strong>Formatter existence check</strong></a>

Calling `subprocess.run([formatter])` without evaluating its exit code or using `shutil.which` may not reliably detect whether the executable is present.
</summary>

```python
if args.formatter_cmds[0].startswith("black") or args.formatter_cmds[0].startswith("uv"):
    formatter = args.formatter_cmds[0].split(" ")[0]
    try:
        subprocess.run([formatter], capture_output=True, check=False)
    except (FileNotFoundError, NotADirectoryError):
        click.echo(f"⚠️ Formatter not found: {formatter}, please ensure it is installed")
        sys.exit(1)
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-20T03:09:22Z
body: ## PR Code Suggestions ✨

<!-- 5c3903e -->

Explore these optional code suggestions:

<table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=1>Possible issue</td>
<td>



<details><summary>Check list is non-empty first</summary>

___


**Guard against an empty <code>formatter_cmds</code> list to prevent an <code>IndexError</code> by ensuring it <br>has at least one element before accessing index 0.**

[codeflash/main.py [42]](https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7R42-R42)

```diff
-if args.formatter_cmds[0].startswith("black") or args.formatter_cmds[0].startswith("uv"):
+if args.formatter_cmds and (args.formatter_cmds[0].startswith("black") or args.formatter_cmds[0].startswith("uv")):
```
<details><summary>Suggestion importance[1-10]: 7</summary>

__

Why: Guarding against an empty `formatter_cmds` list avoids a potential `IndexError`, improving robustness without much overhead.


</details></details></td><td align=center>Medium

</td></tr><tr><td rowspan=2>General</td>
<td>



<details><summary>Add version flag to formatter check</summary>

___


**Use a harmless flag like <code>--version</code> when invoking the formatter to avoid hangs or <br>unexpected errors when no file argument is provided. This verifies installation <br>without requiring actual formatting.**

[codeflash/main.py [45]](https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7R45-R45)

```diff
-subprocess.run([formatter], capture_output=True, check=False)
+subprocess.run([formatter, "--version"], capture_output=True, check=False)
```
<details><summary>Suggestion importance[1-10]: 6</summary>

__

Why: Adding `--version` prevents the formatter from hanging or attempting to read STDIN without arguments, improving the reliability of the check.


</details></details></td><td align=center>Low

</td></tr><tr><td>



<details><summary>Use shlex for command parsing</summary>

___


**Use <code>shlex.split</code> to correctly parse the command string, handling quoted arguments or <br>extra spaces more robustly.**

[codeflash/main.py [43]](https://github.com/codeflash-ai/codeflash/pull/229/files#diff-614158f7df64c99b273134232885e41602312147cb983bd903d63a69f6f813e7R43-R43)

```diff
-formatter = args.formatter_cmds[0].split(" ")[0]
+import shlex
+formatter = shlex.split(args.formatter_cmds[0])[0]
```
<details><summary>Suggestion importance[1-10]: 5</summary>

__

Why: Using `shlex.split` more accurately parses command strings, though the impact is minor and the import placement may need adjustment.


</details></details></td><td align=center>Low

</td></tr></tr></tbody></table>


author_association: CONTRIBUTOR
user: login: aseembits93
created_at: 2025-05-21T02:17:10Z
body: raising an OSError instead of a logger.error doesn't work somehow for me, works as expected though
author_association: CONTRIBUTOR
user: login: aseembits93
created_at: 2025-05-21T03:34:25Z
body: Thanks @KRRT7 for the help!
author_association: CONTRIBUTOR
user: login: aseembits93
created_at: 2025-05-27T18:11:16Z
body: @KRRT7 ready to merge
author_association: CONTRIBUTOR
user: login: aseembits93
created_at: 2025-05-27T18:13:33Z
body: @KRRT7 ready to merge
author_association: CONTRIBUTOR