Repository: Significant-Gravitas/AutoGPT
Pull request: Add makedirs to file operations
<!-- ⚠️ At the moment any non-essential commands are not being merged.
If you want to add non-essential commands to Auto-GPT, please create a plugin instead.
We are expecting to ship plugin support within the week (PR #757).
Resources:
* https://github.com/Significant-Gravitas/Auto-GPT-Plugin-Template
-->

<!-- 📢 Announcement
We've recently noticed an increase in pull requests focusing on combining multiple changes. While the intentions behind these PRs are appreciated, it's essential to maintain a clean and manageable git history. To ensure the quality of our repository, we kindly ask you to adhere to the following guidelines when submitting PRs:

Focus on a single, specific change.
Do not include any unrelated or "extra" modifications.
Provide clear documentation and explanations of the changes made.
Ensure diffs are limited to the intended lines — no applying preferred formatting styles or line endings (unless that's what the PR is about).
For guidance on committing only the specific lines you have changed, refer to this helpful video: https://youtu.be/8-hSNHHbiZg

By following these guidelines, your PRs are more likely to be merged quickly after testing, as long as they align with the project's overall direction. -->

### Background
<!-- Provide a concise overview of the rationale behind this change. Include relevant context, prior discussions, or links to related issues. Ensure that the change aligns with the project's overall direction. -->

Many times auto gpt will try to download or append to files in a directory that doesn't exist, this will cause an error that it then tries to correct, sometimes unsuccessfully. 

Related issue: https://github.com/Significant-Gravitas/Auto-GPT/issues/3249

### Changes
<!-- Describe the specific, focused change made in this pull request. Detail the modifications clearly and avoid any unrelated or "extra" changes. -->

I added logic to check if the directory exists and if not create it. I used the same code already being used for `write_to_file` and added it to the `append_to_file` and `download_file` operations.

### Documentation
<!-- Explain how your changes are documented, such as in-code comments or external documentation. Ensure that the documentation is clear, concise, and easy to understand. -->

No documentation.

### Test Plan
<!-- Describe how you tested this functionality. Include steps to reproduce, relevant test cases, and any other pertinent information. -->

I had auto gpt download and append to files.

### PR Quality Checklist
- [x] My pull request is atomic and focuses on a single change.
- [x] I have thoroughly tested my changes with multiple different prompts.
- [x] I have considered potential risks and mitigations for my changes.
- [x] I have documented my changes clearly and comprehensively.
- [x] I have not snuck in any "extra" small tweaks changes <!-- Submit these as separate Pull Requests, they are the easiest to merge! -->

<!-- If you haven't added tests, please explain why. If you have, check the appropriate box. If you've ensured your PR is atomic and well-documented, check the corresponding boxes. -->

I didn't add any tests because I couldn't find out where or how to add them as I found other file operations didn't have tests. 

<!-- By submitting this, I agree that my pull request should be closed if I do not fill this out or follow the guidelines. -->


Problem statement:
download_file fails when downloading to a folder that doesn't exist yet
### ⚠️ Search for existing issues first ⚠️

- [X] I have searched the existing issues, and there is no existing issue for my problem

### Which Operating System are you using?

MacOS

### Which version of Auto-GPT are you using?

Latest Release

### GPT-3 or GPT-4?

GPT-3.5

### Steps to reproduce 🕹

After prompting to download a file to a folder, it runs the download_file command, but the directory doesn't exist.
```
System: Command download_file returned: Error: [Errno 2] No such file or directory: '/Users/me/Sites/Auto-GPT/autogpt/auto_gpt_workspace/2023-04-25/rss/cnn.xml'
```
It fails and eventually creates the directory, but this can be sped up by the download_file command to check that the directory exists, and create it if it doesn't yet.

### Current behavior 😯

_No response_

### Expected behavior 🤔

_No response_

### Your prompt 📝

_No response_

### Your Logs 📒

_No response_

Hints from issue discussion:
I am experiencing this behavior as well. It will talk in circles saying "I am going to to run xyz" then I get back "file not found", where it then says something along the lines of "I apologize for the mistake. Here's the correct command to use:", rinse and repeat

Commit message: Merge branch 'master' into add-makedirs

Patch to review:
diff --git a/autogpt/commands/file_operations.py b/autogpt/commands/file_operations.py
index 9999fccf1ed5..f2532e43e174 100644
--- a/autogpt/commands/file_operations.py
+++ b/autogpt/commands/file_operations.py
@@ -167,6 +167,9 @@ def append_to_file(filename: str, text: str, should_log: bool = True) -> str:
         str: A message indicating success or failure
     """
     try:
+        directory = os.path.dirname(filename)
+        if not os.path.exists(directory):
+            os.makedirs(directory)
         with open(filename, "a") as f:
             f.write(text)
 
@@ -236,6 +239,9 @@ def download_file(url, filename):
         filename (str): Filename to save the file as
     """
     try:
+        directory = os.path.dirname(filename)
+        if not os.path.exists(directory):
+            os.makedirs(directory)
         message = f"{Fore.YELLOW}Downloading file from {Back.LIGHTBLUE_EX}{url}{Back.RESET}{Fore.RESET}"
         with Spinner(message) as spinner:
             session = requests.Session()
diff --git a/tests/unit/test_file_operations.py b/tests/unit/test_file_operations.py
index c3b20d0c5af6..381e6c408210 100644
--- a/tests/unit/test_file_operations.py
+++ b/tests/unit/test_file_operations.py
@@ -33,6 +33,7 @@ def setUp(self):
         self.test_file = str(self.workspace.get_path("test_file.txt"))
         self.test_file2 = "test_file2.txt"
         self.test_directory = str(self.workspace.get_path("test_directory"))
+        self.test_nested_file = os.path.join(self.test_directory, self.test_file)
         self.file_content = "This is a test file.\n"
         self.file_logger_logs = "file_logger.txt"
 
@@ -69,7 +70,7 @@ def test_read_file(self):
 
     def test_write_to_file(self):
         new_content = "This is new content.\n"
-        write_to_file(self.test_file, new_content)
+        write_to_file(self.test_nested_file, new_content)
         with open(self.test_file, "r") as f:
             content = f.read()
         self.assertEqual(content, new_content)
@@ -79,7 +80,7 @@ def test_append_to_file(self):
             content_before = f.read()
 
         append_text = "This is appended text.\n"
-        append_to_file(self.test_file, append_text)
+        append_to_file(self.test_nested_file, append_text)
         with open(self.test_file, "r") as f:
             content = f.read()
 


Code review comments:
diff_hunk: @@ -236,6 +239,9 @@ def download_file(url, filename):
         filename (str): Filename to save the file as
     """
     try:
+        directory = os.path.dirname(filename)
+        if not os.path.exists(directory):
+            os.makedirs(directory)
path: autogpt/commands/file_operations.py
text: @user1:
```suggestion
        os.makedirs(directory, exist_ok=True)
```

diff_hunk: @@ -167,6 +167,9 @@ def append_to_file(filename: str, text: str, should_log: bool = True) -> str:
         str: A message indicating success or failure
     """
     try:
+        directory = os.path.dirname(filename)
+        if not os.path.exists(directory):
+            os.makedirs(directory)
path: autogpt/commands/file_operations.py
text: @user1:
```suggestion
        os.makedirs(directory, exist_ok=True)
```

diff_hunk: @@ -33,6 +33,7 @@ def setUp(self):
         self.test_file = str(self.workspace.get_path("test_file.txt"))
         self.test_file2 = "test_file2.txt"
         self.test_directory = str(self.workspace.get_path("test_directory"))
+        self.test_nested_file = os.path.join(self.test_directory, self.test_file)
path: tests/unit/test_file_operations.py
text: @user1:
Why this change? It does an unsafe path join that doesn't have any effect because `self.test_file` is absolute.

@author:
I was trying to make a nested file so it would test creating the directory, I changed it to:

```python
self.test_nested_file = str(self.workspace.get_path("nested/test_file.txt"))
```