Repository: googleapis/google-cloud-python

Issue:
List subdirectories
In order to navigate (potentially broad) directory hierarchies in buckets, it would be useful to be able to list the sub-directories in a bucket at a given prefix. Happy to attempt to implement such a method if someone points me in the correct direction. Seems like perhaps a new Iterator?



Discussion:
@samskillman How about

``` python
from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(max_results=..., prefix=..., delimiter=...)
```

I think I wasn't completely clear in the description above. Let's say I have a bucket structure like this:

```
gs://root_bucket/
    A/blob1
    A/blob2
    B/blob3
    B/blob4
```

Is there a way to use the list_blobs (or anything else) to return the first level of subdirectory names ["gs://root_bucket/A/", "gs://root_bucket/B/"] rather than all the blobs?  This would have the same behavior as `gsutil ls gs://root_bucket`. Perhaps I don't understand how delimiter and prefix can be used to accomplish this.

Being more specific, shouldn't this work:

``` python
from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/')
```

to get the top level?

And to list subdirectories inside `A`:

``` python
from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/', prefix='A/')
```

?

@jgeewax That is what I had tried originally, but I get empty lists in both cases. From what I can tell in test_bucket.py on line 291, the test actually asserts that the list of blobs is [], so maybe this is an actual bug?

Huh, but I can do:

```
iterator = bucket.list_blobs(delimiter="/")
response = iterator.get_next_page_response()
blobs = list(iterator.get_items_from_response(response))
dirs = iterator.prefixes
```

(Got that hint from regression/storage.py)

The docs are a bit clarifying: https://cloud.google.com/storage/docs/json_api/v1/objects/list

If you want "just the stuff immediately in the directory `A/`", this translates to "Give me the stuff that starts with `A/` but doesn't have any other `/` characters in the non-prefix part of the name, which you do with:

``` python
from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/', prefix='A/')
```

This will return all stuff that _starts_ with `A/` however in the rest of the name, there can't be another `/`. So `A/foo.txt` would come back in the list, but `A/sub/foo.txt` would _not_ come back.

@jgeewax that would list only the files, right? Then to get the directories I'd have to do what I wrote up above, is that right? How about a bucket.list_prefixes method like this:

```
def list_prefixes(self, prefix=None):
    iterator = self.list_blobs(delimiter='/', prefix=prefix)
    list(iterator)  # Necessary to populate iterator.prefixes
    for p in iterator.prefixes:
        yield p
```

I think, effectively, what that's doing is going through the list of blobs in the bucket, and collecting all the prefixes. So yes, that should work -- probably just as well as collecting the prefixes from the list of blobs yourself.

If we were going to add a `list_prefixes` method, it would be tricky. I can't find a way to get "just the top-level list of directories"... 

@Capstan : What would I do to get the list of all blobs that are "directories" in the root of the bucket?

I thought `prefix='/', delimiter='/'` would work, but it seems to (true to form) just return the non-directory items (ie, "give me everything starting with `/` that has no other `/`s in the name").

What i want is "give me everything starting with `/` that has exactly one more `/` in the name" (I think...)

@samskillman @jgeewax The `list(iterator)` will page through results and [sets] `prefixes` every time a new HTTP request is issued. This means that the value `iterator.prefixes` may not be comprehensive for the entire result set, just for the last request issued. (UPDATE: That's not to say we can't make `self.prefixes` a `set`, it just currently isn't implemented that way.)

@jgeewax @dhermes has it right. FWIW, we don't enforce a preceeding '/' at all, and many objects don't start with that character. If you haven't iterated over the entire list, you won't have the set of top-level directory-likes.

Given that the separator character is optional, and user defined, we can't really provide a lot of help for this usecase in the library.  As @dhermes points out, we could change the iterator such that its `prefixes` attribute is cumulative across all requested pages:  that seems like a reasonable choice.

If we made that change,  I imagine that the most reasonable way to accomplish the goal is to write a helper function which "piped the items to `/dev/null`" and then extracted the relevant prefixes.  E.g.:

``` python
def extract_prefixes(bucket, prefix=None, delimiter='/'):
    iterator = bucket.list_blobs(prefix=prefix, delimiter=delimiter):
    map(lambda x: None, iterator)
    return iterator.prefixes
```

:+1: to that idea, I don't think it belongs in gcloud-python though. However yes, I can see the logic for making prefixes cumulative.

@tseaver I think it'd be worthwhile to at least have a protected attribute on the `Iterator` called `_current_prefixes` just to make the distinction from the `set()` that is `self.prefixes`. Worth making it public?

@samskillman I am going to implement the cumulative prefixes for the iterator, but likely nothing else.

Do you have an issue with us closing out this issue after that?


Resolving patch:
diff --git a/gcloud/storage/bucket.py b/gcloud/storage/bucket.py
--- a/gcloud/storage/bucket.py
+++ b/gcloud/storage/bucket.py
@@ -52,7 +52,8 @@ class _BlobIterator(Iterator):
     def __init__(self, bucket, extra_params=None, connection=None):
         connection = _require_connection(connection)
         self.bucket = bucket
-        self.prefixes = ()
+        self.prefixes = set()
+        self._current_prefixes = None
         super(_BlobIterator, self).__init__(
             connection=connection, path=bucket.path + '/o',
             extra_params=extra_params)
@@ -63,7 +64,8 @@ def get_items_from_response(self, response):
         :type response: dict
         :param response: The JSON API response for a page of blobs.
         """
-        self.prefixes = tuple(response.get('prefixes', ()))
+        self._current_prefixes = tuple(response.get('prefixes', ()))
+        self.prefixes.update(self._current_prefixes)
         for item in response.get('items', []):
             name = item.get('name')
             blob = Blob(name, bucket=self.bucket)
diff --git a/regression/storage.py b/regression/storage.py
--- a/regression/storage.py
+++ b/regression/storage.py
@@ -259,7 +259,7 @@ def test_root_level_w_delimiter(self):
         self.assertEqual([blob.name for blob in blobs], ['file01.txt'])
         self.assertEqual(iterator.page_number, 1)
         self.assertTrue(iterator.next_page_token is None)
-        self.assertEqual(iterator.prefixes, ('parent/',))
+        self.assertEqual(iterator.prefixes, set(['parent/']))
 
     def test_first_level(self):
         iterator = self.bucket.list_blobs(delimiter='/', prefix='parent/')
@@ -268,7 +268,7 @@ def test_first_level(self):
         self.assertEqual([blob.name for blob in blobs], ['parent/file11.txt'])
         self.assertEqual(iterator.page_number, 1)
         self.assertTrue(iterator.next_page_token is None)
-        self.assertEqual(iterator.prefixes, ('parent/child/',))
+        self.assertEqual(iterator.prefixes, set(['parent/child/']))
 
     def test_second_level(self):
         iterator = self.bucket.list_blobs(delimiter='/',
@@ -281,7 +281,7 @@ def test_second_level(self):
         self.assertEqual(iterator.page_number, 1)
         self.assertTrue(iterator.next_page_token is None)
         self.assertEqual(iterator.prefixes,
-                         ('parent/child/grand/', 'parent/child/other/'))
+                         set(['parent/child/grand/', 'parent/child/other/']))
 
     def test_third_level(self):
         # Pseudo-hierarchy can be arbitrarily deep, subject to the limit
@@ -296,7 +296,7 @@ def test_third_level(self):
                          ['parent/child/grand/file31.txt'])
         self.assertEqual(iterator.page_number, 1)
         self.assertTrue(iterator.next_page_token is None)
-        self.assertEqual(iterator.prefixes, ())
+        self.assertEqual(iterator.prefixes, set())
 
 
 class TestStorageSignURLs(TestStorageFiles):


Test patch:
diff --git a/gcloud/storage/test_bucket.py b/gcloud/storage/test_bucket.py
--- a/gcloud/storage/test_bucket.py
+++ b/gcloud/storage/test_bucket.py
@@ -37,7 +37,7 @@ def test_ctor_w_implicit_connection(self):
         self.assertEqual(iterator.path, '%s/o' % bucket.path)
         self.assertEqual(iterator.page_number, 0)
         self.assertEqual(iterator.next_page_token, None)
-        self.assertEqual(iterator.prefixes, ())
+        self.assertEqual(iterator.prefixes, set())
 
     def test_ctor_w_explicit_connection(self):
         connection = _Connection()
@@ -48,33 +48,52 @@ def test_ctor_w_explicit_connection(self):
         self.assertEqual(iterator.path, '%s/o' % bucket.path)
         self.assertEqual(iterator.page_number, 0)
         self.assertEqual(iterator.next_page_token, None)
-        self.assertEqual(iterator.prefixes, ())
+        self.assertEqual(iterator.prefixes, set())
 
     def test_get_items_from_response_empty(self):
-        from gcloud.storage._testing import _monkey_defaults
         connection = _Connection()
         bucket = _Bucket()
-        with _monkey_defaults(connection=connection):
-            iterator = self._makeOne(bucket)
-            blobs = list(iterator.get_items_from_response({}))
+        iterator = self._makeOne(bucket, connection=connection)
+        blobs = list(iterator.get_items_from_response({}))
         self.assertEqual(blobs, [])
-        self.assertEqual(iterator.prefixes, ())
+        self.assertEqual(iterator.prefixes, set())
 
     def test_get_items_from_response_non_empty(self):
         from gcloud.storage.blob import Blob
-        from gcloud.storage._testing import _monkey_defaults
         BLOB_NAME = 'blob-name'
         response = {'items': [{'name': BLOB_NAME}], 'prefixes': ['foo']}
         connection = _Connection()
         bucket = _Bucket()
-        with _monkey_defaults(connection=connection):
-            iterator = self._makeOne(bucket)
-            blobs = list(iterator.get_items_from_response(response))
+        iterator = self._makeOne(bucket, connection=connection)
+        blobs = list(iterator.get_items_from_response(response))
+        self.assertEqual(len(blobs), 1)
+        blob = blobs[0]
+        self.assertTrue(isinstance(blob, Blob))
+        self.assertEqual(blob.name, BLOB_NAME)
+        self.assertEqual(iterator.prefixes, set(['foo']))
+
+    def test_get_items_from_response_cumulative_prefixes(self):
+        from gcloud.storage.blob import Blob
+        BLOB_NAME = 'blob-name1'
+        response1 = {'items': [{'name': BLOB_NAME}], 'prefixes': ['foo']}
+        response2 = {
+            'items': [],
+            'prefixes': ['foo', 'bar'],
+        }
+        connection = _Connection()
+        bucket = _Bucket()
+        iterator = self._makeOne(bucket, connection=connection)
+        # Parse first response.
+        blobs = list(iterator.get_items_from_response(response1))
         self.assertEqual(len(blobs), 1)
         blob = blobs[0]
         self.assertTrue(isinstance(blob, Blob))
         self.assertEqual(blob.name, BLOB_NAME)
-        self.assertEqual(iterator.prefixes, ('foo',))
+        self.assertEqual(iterator.prefixes, set(['foo']))
+        # Parse second response.
+        blobs = list(iterator.get_items_from_response(response2))
+        self.assertEqual(len(blobs), 0)
+        self.assertEqual(iterator.prefixes, set(['foo', 'bar']))
 
 
 class Test_Bucket(unittest2.TestCase):
@@ -243,13 +262,11 @@ def test_get_blob_hit(self):
         self.assertEqual(kw['path'], '/b/%s/o/%s' % (NAME, BLOB_NAME))
 
     def test_list_blobs_defaults(self):
-        from gcloud.storage._testing import _monkey_defaults
         NAME = 'name'
         connection = _Connection({'items': []})
         bucket = self._makeOne(NAME)
-        with _monkey_defaults(connection=connection):
-            iterator = bucket.list_blobs()
-            blobs = list(iterator)
+        iterator = bucket.list_blobs(connection=connection)
+        blobs = list(iterator)
         self.assertEqual(blobs, [])
         kw, = connection._requested
         self.assertEqual(kw['method'], 'GET')
@@ -257,7 +274,6 @@ def test_list_blobs_defaults(self):
         self.assertEqual(kw['query_params'], {'projection': 'noAcl'})
 
     def test_list_blobs_explicit(self):
-        from gcloud.storage._testing import _monkey_defaults
         NAME = 'name'
         MAX_RESULTS = 10
         PAGE_TOKEN = 'ABCD'
@@ -277,17 +293,17 @@ def test_list_blobs_explicit(self):
         }
         connection = _Connection({'items': []})
         bucket = self._makeOne(NAME)
-        with _monkey_defaults(connection=connection):
-            iterator = bucket.list_blobs(
-                max_results=MAX_RESULTS,
-                page_token=PAGE_TOKEN,
-                prefix=PREFIX,
-                delimiter=DELIMITER,
-                versions=VERSIONS,
-                projection=PROJECTION,
-                fields=FIELDS,
-            )
-            blobs = list(iterator)
+        iterator = bucket.list_blobs(
+            max_results=MAX_RESULTS,
+            page_token=PAGE_TOKEN,
+            prefix=PREFIX,
+            delimiter=DELIMITER,
+            versions=VERSIONS,
+            projection=PROJECTION,
+            fields=FIELDS,
+            connection=connection,
+        )
+        blobs = list(iterator)
         self.assertEqual(blobs, [])
         kw, = connection._requested
         self.assertEqual(kw['method'], 'GET')