Repository: jschneier/django-storages

Issue:
[Feature] Distinct SFTP_BASE_URL and [Bugfix] sftp.exists() works incorrectly when using root_path
[Feature]
I'm using Django Storages for both S3 and SFTP, so the fact they both use MEDIA_URL was problematic. 

I've added a new setting SFTP_BASE_URL which will be preferred. If it doesn't exist, MEDIA_URL will be used. 

```
# sftpstorage.py
def get_default_settings(self):
        return {
            ...
            "base_url": setting("SFTP_BASE_URL") or setting("MEDIA_URL"),
        }
```

And to set it in 
```
# django settings.py
SFTP_BASE_URL = os.getenv('SFTP_BASE_URL')  # use this first
MEDIA_URL= os.getenv('MEDIA_URL')  # use this second
```

[Bug]

Additionally, in implementing the SFTP storage with a root path, the SFTPStorage.exists() would always prepend the root path to what it was checking.

If the root path was `uploads`, it would check for the existence of `uploads/uploads` rather than just `uploads` when going through the recursive `_mkdir()` calls during a `_save()`.

This has been fixed by removing the use of self._remote_path(name) within SFTPStorage.exists(), and just returning True if name is falsy.

While in my manual testing in a Django project this works fine both with and without a root path, **I have not managed to get tox to run**. If someone would help me getting tox up and running, I can add a test for it. 



Discussion:
@jschneier  Additional details about the bugfix. I've just added a couple prints at the start of sftpstorage.py's exists method. This shows the process as it starts with _save() and the recursive _mkdir() calls.

I hadn't tried the pre-fix code without the root_path earlier, but I see there's a recursion error there, too. 

As I see it, there are 2 possible places to address the issue. Either in exists() or in _mkdir(). 


```

def exists(self, name):
        print(f"exists: Checking if '{name}' exists")
        print(f"exists: Statting {self._remote_path(name)}")
        try:
            self.sftp.stat(self._remote_path(name))
            return True
        except FileNotFoundError:
            return False
```

Scenario 1: 
root_path: `""` Using the default here
file_path: `file.txt` Not placing in a directory. 
sftp server initial directory: `/`

Outputs:
```
exists: Checking if 'file.txt' exists
exists: Statting file.txt
exists: Checking if '' exists
exists: Statting
exists: Checking if '' exists
exists: Statting
exists: Checking if '' exists
exists: Statting
Eventual recursion error. 
```

Scenario 2: 
root_path: `""` Using the default here
file_path: `path/to/file.txt` Should be put in `upload/path/to/file.txt`.
sftp server initial directory: `/`

Outputs:
```
exists: Checking if 'path/to' exists
exists: Statting path/to
exists: Checking if 'path' exists
exists: Statting patn
exists: Checking if '' exists
exists: Statting
exists: Checking if '' exists
exists: Statting
exists: Checking if '' exists
exists: Statting
Eventual recursion error. 
```

Scenario 3: 
root_path: `"upload/"` This directory already exists.
file_path: `file.txt` Should be put in `upload/file.txt`
sftp server start path: `/`

Outputs:
```
exists: Checking if 'upload' exists
exists: Statting upload/upload (this is the bug)
exists: Checking if '' exists
exists: Statting upload/
OSError: Failure. (My user doesn't have permissions to create anything in root, but the directory "upload/" already exists anyway so doesn't need creating, so further recursive mkdir fails in this buggy attempt)
```

Scenario 4: 
root_path: `"upload/"` This directory already exists.
file_path: `path/to/file.txt` Should be put in `upload/path/to/file.txt`. These other dirs don't exist.
sftp server start path: `/`

Outputs:
```
exists: Checking if 'path/to/file.txt' exists
exists: Statting upload/path/to/file.txt
_save for path/to/file.txt
exists: Checking if 'upload/path/to' exists
exists: Statting upload/upload/path/to (this is the bug)
exists: Checking if 'upload/path' exists
exists: Statting upload/upload/path
exists: Checking if 'upload' exists
exists: Statting upload/upload
exists: Checking if '' exists
exists: Statting upload/
OSError: Failure (My user doesn't have permissions to create anything in root, but the directory "upload/" already exists anyway so doesn't need creating, so further recursive mkdir fails in this buggy attempt)
```



Resolving patch:
diff --git a/storages/backends/sftpstorage.py b/storages/backends/sftpstorage.py
--- a/storages/backends/sftpstorage.py
+++ b/storages/backends/sftpstorage.py
@@ -118,7 +118,7 @@ def _mkdir(self, path):
         """Create directory, recursing up to create parent dirs if
         necessary."""
         parent = posixpath.dirname(path)
-        if not self.exists(parent):
+        if not self._path_exists(parent):
             self._mkdir(parent)
         self.sftp.mkdir(path)
 
@@ -134,7 +134,7 @@ def _save(self, name, content):
             content.seek(0, os.SEEK_SET)
         path = self._remote_path(name)
         dirname = posixpath.dirname(path)
-        if not self.exists(dirname):
+        if not self._path_exists(dirname):
             self._mkdir(dirname)
 
         self.sftp.putfo(content, path)
@@ -152,13 +152,23 @@ def delete(self, name):
         except OSError:
             pass
 
-    def exists(self, name):
+    def _path_exists(self, path):
+        """Determines whether a file existis in the sftp storage given its
+        absolute path."""
         try:
-            self.sftp.stat(self._remote_path(name))
+            self.sftp.stat(path)
             return True
         except FileNotFoundError:
             return False
 
+    def exists(self, name):
+        """Determines whether a file exists within the root folder of the SFTP storage
+        (as set by `SFTP_STORAGE_ROOT`). This method differs from `._path_exists()`
+        in that the provided `name` is assumed to be the relative path of the file
+        within the root folder.
+        """
+        return self._path_exists(self._remote_path(name))
+
     def _isdir_attr(self, item):
         # Return whether an item in sftp.listdir_attr results is a directory
         if item.st_mode is not None:


Test patch:
diff --git a/tests/test_sftp.py b/tests/test_sftp.py
--- a/tests/test_sftp.py
+++ b/tests/test_sftp.py
@@ -16,7 +16,7 @@
 
 class SFTPStorageTest(TestCase):
     def setUp(self):
-        self.storage = sftpstorage.SFTPStorage(host="foo")
+        self.storage = sftpstorage.SFTPStorage(host="foo", root_path="root")
 
     def test_init(self):
         pass
@@ -95,13 +95,18 @@ def test_save_non_seekable(self, mock_sftp):
     )
     def test_save_in_subdir(self, mock_sftp):
         self.storage._save("bar/foo", File(io.BytesIO(b"foo"), "foo"))
-        self.assertEqual(mock_sftp.mkdir.call_args_list[0][0], ("bar",))
+        self.assertEqual(mock_sftp.stat.call_args_list[0][0], ("root/bar",))
+        self.assertEqual(mock_sftp.mkdir.call_args_list[0][0], ("root/bar",))
         self.assertTrue(mock_sftp.putfo.called)
 
     @patch("storages.backends.sftpstorage.SFTPStorage.sftp")
     def test_delete(self, mock_sftp):
         self.storage.delete("foo")
-        self.assertEqual(mock_sftp.remove.call_args_list[0][0], ("foo",))
+        self.assertEqual(mock_sftp.remove.call_args_list[0][0], ("root/foo",))
+
+    @patch("storages.backends.sftpstorage.SFTPStorage.sftp")
+    def test_path_exists(self, mock_sftp):
+        self.assertTrue(self.storage._path_exists("root/foo"))
 
     @patch("storages.backends.sftpstorage.SFTPStorage.sftp")
     def test_exists(self, mock_sftp):
@@ -114,6 +119,13 @@ def test_exists(self, mock_sftp):
     def test_not_exists(self, mock_sftp):
         self.assertFalse(self.storage.exists("foo"))
 
+    @patch(
+        "storages.backends.sftpstorage.SFTPStorage.sftp",
+        **{"stat.side_effect": FileNotFoundError()},
+    )
+    def test_not_path_exists(self, mock_sftp):
+        self.assertFalse(self.storage._path_exists("root/foo"))
+
     @patch(
         "storages.backends.sftpstorage.SFTPStorage.sftp",
         **{"stat.side_effect": socket.timeout()},