Repository: streamlit/streamlit

Issue:
Bug in clearing cache of cached function via .clear() in streamlit==1.20.0
### Checklist

- [X] I have searched the [existing issues](https://github.com/streamlit/streamlit/issues) for similar issues.
- [X] I added a very descriptive title to this issue.
- [X] I have provided sufficient information below to help reproduce this issue.

### Summary

In streamlit==1.20.0 I am unable to clear cache of a cached functions via `.clear()` (this was possible in streamlit 1.19.0).

I suspect this is due to changes made in latest streamlit version to `streamlit\runtime\caching\`, and in particular to `storage` module: now, indeed, even if `st.cache_data` is used without any custom argument - and, then, with `persist=None` - streamlit try to clear also local cache, and searching for the local cache folder eventually raise the error `FileNotFoundError: [Errno 2] No such file or directory` / `FileNotFoundError: [WinError 3] The system cannot find the path specified`

### Reproducible Code Example

```Python
import streamlit as st
import time

@st.cache_data
def dummy():
    st.write('Hello from dummy function! I will wait for 5 seconds...')
    time.sleep(5)
    st.write('Done!')

left, right = st.columns(2)
with left:
    if st.button('Run dummy function'):
        dummy()
with right:
    if st.button('Clear cache'):
        dummy.clear()
```


### Steps To Reproduce

1. `pip install streamlit==1.20.0`
2. Copy above snippet to `app.py`
3. `streamlit run app.py`
4. Click a first time on "Run dummy function" button (-> see cache spinner)
5. Click a second time on "Run dummy function" (-> confirm that function has been cached)
6. Click on "Clear cache" button (-> `FileNotFoundError` should be raised)

### Expected Behavior

As in previous streamlit 1.19.0 version, cache can be cleared without searching for local cache if `persist = None | False`

### Current Behavior

Cannot properly clear cache due to `FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\XXXXXXXXX\\.streamlit\\cache'` (on local Windows machine) or `FileNotFoundError: [Errno 2] No such file or directory: '/root/.streamlit/cache'` (on remote containerized execution)

Full traceback:

```
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'C:\\Users\\XXXXXXXXX\\.streamlit\\cache'

File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "XXXXXXXXX\app.py", line 16, in <module>
    dummy.clear()
File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\caching\cache_utils.py", line 331, in clear
    cache.clear()
File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\caching\cache_utils.py", line 137, in clear
    self._clear()
File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\caching\cache_data_api.py", line 659, in _clear
    self.storage.clear()
File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\caching\storage\in_memory_cache_storage_wrapper.py", line 107, in clear
    self._persist_storage.clear()
File "C:\Users\XXXXXXXXX\Miniconda3\envs\hc\lib\site-packages\streamlit\runtime\caching\storage\local_disk_cache_storage.py", line 174, in clear
    for file_name in os.listdir(cache_dir):
```

### Is this a regression?

- [X] Yes, this used to work in a previous version.

### Debug info

- Streamlit version: 1.20.0
- Python version: 3.10.8
- Operating System: Windows 10 Enterprise v22H2 build 19045.2604
- Browser: Brave
- Virtual environment: miniconda environment


### Additional Information

_No response_

### Are you willing to submit a PR?

- [ ] Yes, I am willing to submit a PR!


Discussion:
Heads up @kajarenc -- this must have been introduced in #6052
Thanks, @a-slice-of-py for reporting this, and thanks @vdonato for mentioning me!

Yes, this is definitely a bug, and I work on fixing this.

For now, a workaround could be to manually create the `~/.streamlit/cache/` directory (or just invoke any function decorated with `persist=True`, this should create the corresponding folder in the correct place automatically).

Resolving patch:
diff --git a/lib/streamlit/runtime/caching/storage/local_disk_cache_storage.py b/lib/streamlit/runtime/caching/storage/local_disk_cache_storage.py
--- a/lib/streamlit/runtime/caching/storage/local_disk_cache_storage.py
+++ b/lib/streamlit/runtime/caching/storage/local_disk_cache_storage.py
@@ -193,9 +193,13 @@ def clear(self) -> None:
         """Delete all keys for the current storage"""
         cache_dir = get_cache_folder_path()
 
-        for file_name in os.listdir(cache_dir):
-            if self._is_cache_file(file_name):
-                os.remove(os.path.join(cache_dir, file_name))
+        if os.path.isdir(cache_dir):
+            # We try to remove all files in the cache directory that start with
+            # the function key, whether `clear` called for `self.persist`
+            # storage or not, to avoid leaving orphaned files in the cache directory.
+            for file_name in os.listdir(cache_dir):
+                if self._is_cache_file(file_name):
+                    os.remove(os.path.join(cache_dir, file_name))
 
     def close(self) -> None:
         """Dummy implementation of close, we don't need to actually "close" anything"""


Test patch:
diff --git a/lib/tests/streamlit/runtime/caching/cache_data_api_test.py b/lib/tests/streamlit/runtime/caching/cache_data_api_test.py
--- a/lib/tests/streamlit/runtime/caching/cache_data_api_test.py
+++ b/lib/tests/streamlit/runtime/caching/cache_data_api_test.py
@@ -335,7 +335,9 @@ def foo(val):
 
         mock_os_remove.assert_not_called()
 
-        with patch("os.listdir", MagicMock(return_value=created_files_base_names)):
+        with patch(
+            "os.listdir", MagicMock(return_value=created_files_base_names)
+        ), patch("os.path.isdir", MagicMock(return_value=True)):
             # Clear foo's cache
             foo.clear()
 
diff --git a/lib/tests/streamlit/runtime/caching/storage/local_disk_cache_storage_test.py b/lib/tests/streamlit/runtime/caching/storage/local_disk_cache_storage_test.py
--- a/lib/tests/streamlit/runtime/caching/storage/local_disk_cache_storage_test.py
+++ b/lib/tests/streamlit/runtime/caching/storage/local_disk_cache_storage_test.py
@@ -54,8 +54,7 @@ def tearDown(self) -> None:
         self.tempdir.cleanup()
 
     def test_create_persist_context(self):
-        """
-        Tests that LocalDiskCacheStorageManager.create()
+        """Tests that LocalDiskCacheStorageManager.create()
         returns a LocalDiskCacheStorage with correct parameters from context, if
         persist="disk"
         """
@@ -73,8 +72,7 @@ def test_create_persist_context(self):
         self.assertEqual(storage.max_entries, 100)
 
     def test_create_not_persist_context(self):
-        """
-        Tests that LocalDiskCacheStorageManager.create()
+        """Tests that LocalDiskCacheStorageManager.create()
         returns a LocalDiskCacheStorage with correct parameters from context, if
         persist is None
         """
@@ -92,8 +90,7 @@ def test_create_not_persist_context(self):
         self.assertEqual(storage.max_entries, math.inf)
 
     def test_check_context_with_persist_and_ttl(self):
-        """
-        Tests that LocalDiskCacheStorageManager.check_context() writes a warning
+        """Tests that LocalDiskCacheStorageManager.check_context() writes a warning
         in logs when persist="disk" and ttl_seconds is not None
         """
         context = CacheStorageContext(
@@ -119,9 +116,8 @@ def test_check_context_with_persist_and_ttl(self):
             )
 
     def test_check_context_without_persist(self):
-        """
-        Tests that LocalDiskCacheStorageManager.check_context() does not write a warning
-        in logs when persist is None and ttl_seconds is NOT None
+        """Tests that LocalDiskCacheStorageManager.check_context() does not
+        write a warning in logs when persist is None and ttl_seconds is NOT None.
         """
         context = CacheStorageContext(
             function_key="func-key",
@@ -153,8 +149,7 @@ def test_check_context_without_persist(self):
 
     @patch("shutil.rmtree", wraps=shutil.rmtree)
     def test_clear_all(self, mock_rmtree):
-        """
-        Tests that LocalDiskCacheStorageManager.clear_all() calls shutil.rmtree
+        """Tests that LocalDiskCacheStorageManager.clear_all() calls shutil.rmtree
         to remove the cache folder
         """
         manager = LocalDiskCacheStorageManager()
@@ -208,25 +203,19 @@ def test_storage_set(self):
         MagicMock(side_effect=util.Error("mock exception")),
     )
     def test_storage_set_error(self):
-        """
-        Test that storage.set() raises an exception when it fails to write to disk.
-        """
+        """Test that storage.set() raises an exception when it fails to write to disk."""
         with self.assertRaises(CacheStorageError) as e:
             self.storage.set("uniqueKey", b"new-value")
         self.assertEqual(str(e.exception), "Unable to write to cache")
 
     def test_storage_set_override(self):
-        """
-        Test that storage.set() overrides the value of an existing key.
-        """
+        """Test that storage.set() overrides the value of an existing key."""
         self.storage.set("another_key", b"another_value")
         self.storage.set("another_key", b"new_value")
         self.assertEqual(self.storage.get("another_key"), b"new_value")
 
     def test_storage_delete(self):
-        """
-        Test that storage.delete() removes the correct file from disk.
-        """
+        """Test that storage.delete() removes the correct file from disk."""
         self.storage.set("new-key", b"new-value")
         self.assertTrue(os.path.exists(self.tempdir.path + "/func-key-new-key.memo"))
         self.storage.delete("new-key")
@@ -236,9 +225,7 @@ def test_storage_delete(self):
             self.storage.get("new-key")
 
     def test_storage_clear(self):
-        """
-        Test that storage.clear() removes all storage files from disk.
-        """
+        """Test that storage.clear() removes all storage files from disk."""
         self.storage.set("some-key", b"some-value")
         self.storage.set("another-key", b"another-value")
         self.assertTrue(os.path.exists(self.tempdir.path + "/func-key-some-key.memo"))
@@ -262,6 +249,26 @@ def test_storage_clear(self):
         # test that cache folder is empty
         self.assertEqual(os.listdir(self.tempdir.path), [])
 
+    def test_storage_clear_not_existing_cache_directory(self):
+        """Test that clear() is not crashing if the cache directory does not exist."""
+        self.tempdir.cleanup()
+        self.storage.clear()
+
+    def test_storage_clear_call_listdir_existing_cache_directory(self):
+        """Test that clear() call os.listdir if cache folder does not exist."""
+        with patch("os.listdir") as mock_listdir:
+            self.storage.clear()
+        mock_listdir.assert_called_once()
+
+    def test_storage_clear_not_call_listdir_not_existing_cache_directory(self):
+        """Test that clear() doesn't call os.listdir if cache folder does not exist."""
+        self.tempdir.cleanup()
+
+        with patch("os.listdir") as mock_listdir:
+            self.storage.clear()
+
+        mock_listdir.assert_not_called()
+
     def test_storage_close(self):
         """Test that storage.close() does not raise any exception."""
         self.storage.close()