Repository: googleapis/google-cloud-python

Issue:
Figure out the right level of abstraction for iterators.
(From #2545)

We need to determine how users will interact with our iterators and what level of abstraction is expected.

Right now @dhermes has implemented an iterator that iterates through items but exposes page information by setting state on the iterator itself.

``` python
iterator = MyIterator(...)

for item in iterator:
     print(item)
     print(iterator.page)
     print(iterator.next_page_token)
```

I'm not the biggest fan of this for two reasons: it combines the `iterable` and the `iterator` interfaces, it merges the item-level and page-level abstractions.

In my opinion, we should work at well-defined abstraction levels and keep the `iterable` and `iterator` separate.

The first level is the page level:

``` python
iterator = MyIterator(...)

# You could also specify arguments to pages, such as page_token
for page in iterator.pages():
    print(page.number)
    print(page.items)
    print(page.next_page_token)
```

The second level yields items but also page context:

``` python
for item, current_page in iterator.items_by_page():
     print(item)
     print(page.number)
     print(page.next_page_token)
```

The final level yields only items and completely abstracts away the concept of pages:

``` python
for item in iterator.items():
    print(item)
```



Discussion:
@dhermes updated, PTAL when you get a chance.

The `Iterator` is already the response, so people want to be able to say:

``` python
for project in client.list_projects():
    do_something(project)
```

where `client.list_projects()` is an instance of an `Iterator` subclass.

Also, by using indirection via punting to `iterator.pages()` and `iterator.items()`, does the `Iterator` instance hold any state? Does it know where it is in the iteration process? What happens if someone does this:

``` python
page_iter = iterator.pages()
items_iter = iterator.items()
page = next(page_iter)
items = list(page)
next_item = next(items_iter)
```

Rather than get in the "What if?", the current implementation says: "we only iterate through items, but if you want to know about (the current page / the current place in the current page / the current page number / etc.) then we will tell you".

---

> it combines the `iterable` and the `iterator` interfaces...In my opinion, we should work at well-defined abstraction levels and keep the `iterable` and `iterator` separate.

Python doesn't come down one way or another on this, e.g.:

``` python
>>> file_obj = open('a.txt', 'w')
>>> iter(file_obj)
<open file 'a.txt', mode 'w' at 0x7f719a76c540>
>>> iter(file_obj) is file_obj
True
>>> list_obj = [1, 2, 3]
>>> iter(list_obj)
<listiterator object at 0x7f719a72a210>
>>> iter(list_obj) is list_obj
False
```

> The Iterator is already the response, so people want to be able to say

You can still do that with the proposal I have, just map the iterable to return the item iterator by default.

> does the Iterator instance hold any state? Does it know where it is in the iteration process?

The `Iterator` class we have now is actually the `Iterable` in my proposal, so no, it has no state.

> What happens if someone does this:

``` python
# New iterator over pages
page_iter = iterator.pages()
# New iterator over all items.
items_iter = iterator.items()
# Returns the first page.
page = next(page_iter)
# Returns all of the items on the first page.
items = list(page)
# Returns the very first item.
next_item = next(items_iter)
```

> Python doesn't come down one way or another on this

Indeed it doesn't, but I think in the case where there are multiple ways of iterating over an iterable it makes sense to return a separate iterator (dict does this).


Resolving patch:
diff --git a/core/google/cloud/iterator.py b/core/google/cloud/iterator.py
--- a/core/google/cloud/iterator.py
+++ b/core/google/cloud/iterator.py
@@ -43,78 +43,44 @@
     ...         break
 
 When iterating, not every new item will send a request to the server.
-To monitor these requests, track the current page of the iterator::
+To iterate based on each page of items (where a page corresponds to
+a request)::
 
     >>> iterator = Iterator(...)
-    >>> iterator.page_number
-    0
-    >>> next(iterator)
-    <MyItemClass at 0x7f1d3cccf690>
-    >>> iterator.page_number
-    1
-    >>> iterator.page.remaining
-    1
-    >>> next(iterator)
-    <MyItemClass at 0x7f1d3cccfe90>
-    >>> iterator.page_number
-    1
-    >>> iterator.page.remaining
-    0
-    >>> next(iterator)
-    <MyItemClass at 0x7f1d3cccffd0>
-    >>> iterator.page_number
-    2
-    >>> iterator.page.remaining
-    19
-
-It's also possible to consume an entire page and handle the paging process
-manually::
-
-    >>> iterator = Iterator(...)
-    >>> # Manually pull down the first page.
-    >>> iterator.update_page()
-    >>> items = list(iterator.page)
-    >>> items
+    >>> for page in iterator.pages:
+    ...     print('=' * 20)
+    ...     print('    Page number: %d' % (iterator.page_number,))
+    ...     print('  Items in page: %d' % (page.num_items,))
+    ...     print('     First item: %r' % (next(page),))
+    ...     print('Items remaining: %d' % (page.remaining,))
+    ...     print('Next page token: %s' % (iterator.next_page_token,))
+    ====================
+        Page number: 1
+      Items in page: 1
+         First item: <MyItemClass at 0x7f1d3cccf690>
+    Items remaining: 0
+    Next page token: eav1OzQB0OM8rLdGXOEsyQWSG
+    ====================
+        Page number: 2
+      Items in page: 19
+         First item: <MyItemClass at 0x7f1d3cccffd0>
+    Items remaining: 18
+    Next page token: None
+
+To consume an entire page::
+
+    >>> list(page)
     [
         <MyItemClass at 0x7fd64a098ad0>,
         <MyItemClass at 0x7fd64a098ed0>,
         <MyItemClass at 0x7fd64a098e90>,
     ]
-    >>> iterator.page.remaining
-    0
-    >>> iterator.page.num_items
-    3
-    >>> iterator.next_page_token
-    'eav1OzQB0OM8rLdGXOEsyQWSG'
-    >>>
-    >>> # Ask for the next page to be grabbed.
-    >>> iterator.update_page()
-    >>> list(iterator.page)
-    [
-        <MyItemClass at 0x7fea740abdd0>,
-        <MyItemClass at 0x7fea740abe50>,
-    ]
-    >>>
-    >>> # When there are no more results
-    >>> iterator.next_page_token is None
-    True
-    >>> iterator.update_page()
-    >>> iterator.page is None
-    True
 """
 
 
 import six
 
 
-_UNSET = object()
-_NO_MORE_PAGES_ERR = 'Iterator has no more pages.'
-_UNSTARTED_ERR = (
-    'Iterator has not been started. Either begin iterating, '
-    'call next(my_iter) or call my_iter.update_page().')
-_PAGE_ERR_TEMPLATE = (
-    'Tried to update the page while current page (%r) still has %d '
-    'items remaining.')
 DEFAULT_ITEMS_KEY = 'items'
 """The dictionary key used to retrieve items from each response."""
 
@@ -261,7 +227,6 @@ def __init__(self, client, path, item_to_value,
         self.page_number = 0
         self.next_page_token = page_token
         self.num_results = 0
-        self._page = _UNSET
 
     def _verify_params(self):
         """Verifies the parameters don't use any reserved parameter.
@@ -274,82 +239,53 @@ def _verify_params(self):
             raise ValueError('Using a reserved parameter',
                              reserved_in_use)
 
-    @property
-    def page(self):
-        """The current page of results that has been retrieved.
-
-        If there are no more results, will return :data:`None`.
+    def _pages_iter(self):
+        """Generator of pages of API responses.
 
-        :rtype: :class:`Page`
-        :returns: The page of items that has been retrieved.
-        :raises AttributeError: If the page has not been set.
+        Yields :class:`Page` instances.
         """
-        if self._page is _UNSET:
-            raise AttributeError(_UNSTARTED_ERR)
-        return self._page
+        while self._has_next_page():
+            response = self._get_next_page_response()
+            page = Page(self, response, self._items_key,
+                        self._item_to_value)
+            self._page_start(self, page, response)
+            self.num_results += page.num_items
+            yield page
 
-    def __iter__(self):
-        """The :class:`Iterator` is an iterator.
+    @property
+    def pages(self):
+        """Iterator of pages in the response.
 
-        :rtype: :class:`Iterator`
-        :returns: Current instance.
+        :rtype: :class:`~types.GeneratorType`
+        :returns: A generator of :class:`Page` instances.
         :raises ValueError: If the iterator has already been started.
         """
         if self._started:
             raise ValueError('Iterator has already started', self)
         self._started = True
-        return self
-
-    def update_page(self, require_empty=True):
-        """Move to the next page in the result set.
-
-        If the current page is not empty and ``require_empty`` is :data:`True`
-        then an exception will be raised. If the current page is not empty
-        and ``require_empty`` is :data:`False`, then this will return
-        without updating the current page.
-
-        If the current page **is** empty, but there are no more results,
-        sets the current page to :data:`None`.
+        return self._pages_iter()
+
+    def _items_iter(self):
+        """Iterator for each item returned."""
+        for page in self._pages_iter():
+            # Decrement the total results since the pages iterator adds
+            # to it when each page is encountered.
+            self.num_results -= page.num_items
+            for item in page:
+                self.num_results += 1
+                yield item
 
-        If there are no more pages, throws an exception.
-
-        :type require_empty: bool
-        :param require_empty: (Optional) Flag to indicate if the current page
-                              must be empty before updating.
+    def __iter__(self):
+        """Iterator for each item returned.
 
-        :raises ValueError: If ``require_empty`` is :data:`True` but the
-                            current page is not empty.
-        :raises ValueError: If there are no more pages.
+        :rtype: :class:`~types.GeneratorType`
+        :returns: A generator of items from the API.
+        :raises ValueError: If the iterator has already been started.
         """
-        if self._page is None:
-            raise ValueError(_NO_MORE_PAGES_ERR)
-
-        # NOTE: This assumes Page.remaining can never go below 0.
-        page_empty = self._page is _UNSET or self._page.remaining == 0
-        if page_empty:
-            if self._has_next_page():
-                response = self._get_next_page_response()
-                self._page = Page(self, response, self._items_key,
-                                  self._item_to_value)
-                self._page_start(self, self._page, response)
-            else:
-                self._page = None
-        else:
-            if require_empty:
-                msg = _PAGE_ERR_TEMPLATE % (self._page, self.page.remaining)
-                raise ValueError(msg)
-
-    def next(self):
-        """Get the next item from the request."""
-        self.update_page(require_empty=False)
-        if self.page is None:
-            raise StopIteration
-        item = six.next(self.page)
-        self.num_results += 1
-        return item
-
-    # Alias needed for Python 2/3 support.
-    __next__ = next
+        if self._started:
+            raise ValueError('Iterator has already started', self)
+        self._started = True
+        return self._items_iter()
 
     def _has_next_page(self):
         """Determines whether or not there are more pages with results.


Test patch:
diff --git a/bigquery/unit_tests/test_client.py b/bigquery/unit_tests/test_client.py
--- a/bigquery/unit_tests/test_client.py
+++ b/bigquery/unit_tests/test_client.py
@@ -35,6 +35,7 @@ def test_ctor(self):
         self.assertIs(client.connection.http, http)
 
     def test_list_projects_defaults(self):
+        import six
         from google.cloud.bigquery.client import Project
         PROJECT_1 = 'PROJECT_ONE'
         PROJECT_2 = 'PROJECT_TWO'
@@ -60,8 +61,8 @@ def test_list_projects_defaults(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_projects()
-        iterator.update_page()
-        projects = list(iterator.page)
+        page = six.next(iterator.pages)
+        projects = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(projects), len(DATA['projects']))
@@ -78,6 +79,8 @@ def test_list_projects_defaults(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_projects_explicit_response_missing_projects_key(self):
+        import six
+
         PROJECT = 'PROJECT'
         PATH = 'projects'
         TOKEN = 'TOKEN'
@@ -87,8 +90,8 @@ def test_list_projects_explicit_response_missing_projects_key(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_projects(max_results=3, page_token=TOKEN)
-        iterator.update_page()
-        projects = list(iterator.page)
+        page = six.next(iterator.pages)
+        projects = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(projects), 0)
@@ -102,6 +105,7 @@ def test_list_projects_explicit_response_missing_projects_key(self):
                          {'maxResults': 3, 'pageToken': TOKEN})
 
     def test_list_datasets_defaults(self):
+        import six
         from google.cloud.bigquery.dataset import Dataset
         PROJECT = 'PROJECT'
         DATASET_1 = 'dataset_one'
@@ -128,8 +132,8 @@ def test_list_datasets_defaults(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_datasets()
-        iterator.update_page()
-        datasets = list(iterator.page)
+        page = six.next(iterator.pages)
+        datasets = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(datasets), len(DATA['datasets']))
@@ -145,6 +149,8 @@ def test_list_datasets_defaults(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_datasets_explicit_response_missing_datasets_key(self):
+        import six
+
         PROJECT = 'PROJECT'
         PATH = 'projects/%s/datasets' % PROJECT
         TOKEN = 'TOKEN'
@@ -155,8 +161,8 @@ def test_list_datasets_explicit_response_missing_datasets_key(self):
 
         iterator = client.list_datasets(
             include_all=True, max_results=3, page_token=TOKEN)
-        iterator.update_page()
-        datasets = list(iterator.page)
+        page = six.next(iterator.pages)
+        datasets = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(datasets), 0)
@@ -189,6 +195,7 @@ def test_job_from_resource_unknown_type(self):
             client.job_from_resource({'configuration': {'nonesuch': {}}})
 
     def test_list_jobs_defaults(self):
+        import six
         from google.cloud.bigquery.job import LoadTableFromStorageJob
         from google.cloud.bigquery.job import CopyJob
         from google.cloud.bigquery.job import ExtractTableToStorageJob
@@ -301,8 +308,8 @@ def test_list_jobs_defaults(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_jobs()
-        iterator.update_page()
-        jobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        jobs = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(jobs), len(DATA['jobs']))
@@ -319,6 +326,7 @@ def test_list_jobs_defaults(self):
         self.assertEqual(req['query_params'], {'projection': 'full'})
 
     def test_list_jobs_load_job_wo_sourceUris(self):
+        import six
         from google.cloud.bigquery.job import LoadTableFromStorageJob
         PROJECT = 'PROJECT'
         DATASET = 'test_dataset'
@@ -356,8 +364,8 @@ def test_list_jobs_load_job_wo_sourceUris(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_jobs()
-        iterator.update_page()
-        jobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        jobs = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(jobs), len(DATA['jobs']))
@@ -374,6 +382,7 @@ def test_list_jobs_load_job_wo_sourceUris(self):
         self.assertEqual(req['query_params'], {'projection': 'full'})
 
     def test_list_jobs_explicit_missing(self):
+        import six
         PROJECT = 'PROJECT'
         PATH = 'projects/%s/jobs' % PROJECT
         DATA = {}
@@ -384,8 +393,8 @@ def test_list_jobs_explicit_missing(self):
 
         iterator = client.list_jobs(max_results=1000, page_token=TOKEN,
                                     all_users=True, state_filter='done')
-        iterator.update_page()
-        jobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        jobs = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(jobs), 0)
diff --git a/core/unit_tests/test_iterator.py b/core/unit_tests/test_iterator.py
--- a/core/unit_tests/test_iterator.py
+++ b/core/unit_tests/test_iterator.py
@@ -102,7 +102,6 @@ def _makeOne(self, *args, **kw):
 
     def test_constructor(self):
         from google.cloud.iterator import _do_nothing_page_start
-        from google.cloud.iterator import _UNSET
 
         connection = _Connection()
         client = _Client(connection)
@@ -120,7 +119,6 @@ def test_constructor(self):
         self.assertEqual(iterator.page_number, 0)
         self.assertIsNone(iterator.next_page_token)
         self.assertEqual(iterator.num_results, 0)
-        self.assertIs(iterator._page, _UNSET)
 
     def test_constructor_w_extra_param_collision(self):
         connection = _Connection()
@@ -130,61 +128,15 @@ def test_constructor_w_extra_param_collision(self):
         with self.assertRaises(ValueError):
             self._makeOne(client, path, None, extra_params=extra_params)
 
-    def test_page_property(self):
-        iterator = self._makeOne(None, None, None)
-        page = object()
-        iterator._page = page
-        self.assertIs(iterator.page, page)
-
-    def test_page_property_unset(self):
-        from google.cloud.iterator import _UNSET
-
-        iterator = self._makeOne(None, None, None)
-        self.assertIs(iterator._page, _UNSET)
-        with self.assertRaises(AttributeError):
-            getattr(iterator, 'page')
-
-    def test_update_page_no_more(self):
-        iterator = self._makeOne(None, None, None)
-        iterator._page = None
-        with self.assertRaises(ValueError):
-            iterator.update_page()
-
-    def test_update_page_not_empty_success(self):
-        from google.cloud.iterator import Page
-
-        iterator = self._makeOne(None, None, None)
-        page = Page(None, {}, '', None)
-        iterator._page = page
-        iterator._page._remaining = 1
-        iterator.update_page(require_empty=False)
-        self.assertIs(iterator._page, page)
-
-    def test_update_page_not_empty_fail(self):
-        from google.cloud.iterator import Page
-
-        iterator = self._makeOne(None, None, None)
-        iterator._page = Page(None, {}, '', None)
-        iterator._page._remaining = 1
-        with self.assertRaises(ValueError):
-            iterator.update_page(require_empty=True)
-
-    def test_update_page_empty_then_no_more(self):
-        iterator = self._makeOne(None, None, None)
-        # Fake that there are no more pages.
-        iterator.page_number = 1
-        iterator.next_page_token = None
-        iterator.update_page()
-        self.assertIsNone(iterator.page)
-
-    def test_update_page_empty_then_another(self):
+    def test_pages_iter_empty_then_another(self):
+        import six
         from google.cloud._testing import _Monkey
         from google.cloud import iterator as MUT
 
         items_key = 'its-key'
         iterator = self._makeOne(None, None, None, items_key=items_key)
         # Fake the next page class.
-        fake_page = object()
+        fake_page = MUT.Page(None, {}, '', None)
         page_args = []
 
         def dummy_response():
@@ -195,20 +147,58 @@ def dummy_page_class(*args):
             return fake_page
 
         iterator._get_next_page_response = dummy_response
+        pages_iter = iterator.pages
         with _Monkey(MUT, Page=dummy_page_class):
-            iterator.update_page()
-        self.assertIs(iterator.page, fake_page)
+            page = six.next(pages_iter)
+        self.assertIs(page, fake_page)
         self.assertEqual(
             page_args, [(iterator, {}, items_key, iterator._item_to_value)])
 
+    def test_pages_property(self):
+        import types
+
+        iterator = self._makeOne(None, None, None)
+        self.assertIsInstance(iterator.pages, types.GeneratorType)
+
+    def test_pages_property_started(self):
+        import types
+
+        iterator = self._makeOne(None, None, None)
+        pages_iter = iterator.pages
+        self.assertIsInstance(pages_iter, types.GeneratorType)
+        with self.assertRaises(ValueError):
+            getattr(iterator, 'pages')
+
+    def test_pages_property_items_started(self):
+        import types
+
+        iterator = self._makeOne(None, None, None)
+        items_iter = iter(iterator)
+        self.assertIsInstance(items_iter, types.GeneratorType)
+        with self.assertRaises(ValueError):
+            getattr(iterator, 'pages')
+
     def test___iter__(self):
+        import types
+
         iterator = self._makeOne(None, None, None)
-        self.assertIs(iter(iterator), iterator)
+        self.assertIsInstance(iter(iterator), types.GeneratorType)
 
     def test___iter___started(self):
+        import types
+
         iterator = self._makeOne(None, None, None)
         iter_obj = iter(iterator)
-        self.assertIs(iter_obj, iterator)
+        self.assertIsInstance(iter_obj, types.GeneratorType)
+        with self.assertRaises(ValueError):
+            iter(iterator)
+
+    def test___iter___pages_started(self):
+        import types
+
+        iterator = self._makeOne(None, None, None)
+        pages_iter = iterator.pages
+        self.assertIsInstance(pages_iter, types.GeneratorType)
         with self.assertRaises(ValueError):
             iter(iterator)
 
@@ -231,16 +221,17 @@ def item_to_value(iterator, item):  # pylint: disable=unused-argument
                                  item_to_value=item_to_value)
         self.assertEqual(iterator.num_results, 0)
 
-        val1 = six.next(iterator)
+        items_iter = iter(iterator)
+        val1 = six.next(items_iter)
         self.assertEqual(val1, item1)
         self.assertEqual(iterator.num_results, 1)
 
-        val2 = six.next(iterator)
+        val2 = six.next(items_iter)
         self.assertEqual(val2, item2)
         self.assertEqual(iterator.num_results, 2)
 
         with self.assertRaises(StopIteration):
-            six.next(iterator)
+            six.next(items_iter)
 
         kw, = connection._requested
         self.assertEqual(kw['method'], 'GET')
diff --git a/dns/unit_tests/test_client.py b/dns/unit_tests/test_client.py
--- a/dns/unit_tests/test_client.py
+++ b/dns/unit_tests/test_client.py
@@ -106,6 +106,7 @@ def test_quotas_w_kind_key(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_zones_defaults(self):
+        import six
         from google.cloud.dns.zone import ManagedZone
         ID_1 = '123'
         ZONE_1 = 'zone_one'
@@ -133,8 +134,8 @@ def test_list_zones_defaults(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_zones()
-        iterator.update_page()
-        zones = list(iterator.page)
+        page = six.next(iterator.pages)
+        zones = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(zones), len(DATA['managedZones']))
@@ -151,6 +152,7 @@ def test_list_zones_defaults(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_zones_explicit(self):
+        import six
         from google.cloud.dns.zone import ManagedZone
         ID_1 = '123'
         ZONE_1 = 'zone_one'
@@ -177,8 +179,8 @@ def test_list_zones_explicit(self):
         conn = client.connection = _Connection(DATA)
 
         iterator = client.list_zones(max_results=3, page_token=TOKEN)
-        iterator.update_page()
-        zones = list(iterator.page)
+        page = six.next(iterator.pages)
+        zones = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(zones), len(DATA['managedZones']))
diff --git a/dns/unit_tests/test_zone.py b/dns/unit_tests/test_zone.py
--- a/dns/unit_tests/test_zone.py
+++ b/dns/unit_tests/test_zone.py
@@ -409,6 +409,7 @@ def test_delete_w_alternate_client(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_resource_record_sets_defaults(self):
+        import six
         from google.cloud.dns.resource_record_set import ResourceRecordSet
         PATH = 'projects/%s/managedZones/%s/rrsets' % (
             self.PROJECT, self.ZONE_NAME)
@@ -442,8 +443,8 @@ def test_list_resource_record_sets_defaults(self):
 
         iterator = zone.list_resource_record_sets()
         self.assertIs(zone, iterator.zone)
-        iterator.update_page()
-        rrsets = list(iterator.page)
+        page = six.next(iterator.pages)
+        rrsets = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(rrsets), len(DATA['rrsets']))
@@ -461,6 +462,7 @@ def test_list_resource_record_sets_defaults(self):
         self.assertEqual(req['path'], '/%s' % PATH)
 
     def test_list_resource_record_sets_explicit(self):
+        import six
         from google.cloud.dns.resource_record_set import ResourceRecordSet
         PATH = 'projects/%s/managedZones/%s/rrsets' % (
             self.PROJECT, self.ZONE_NAME)
@@ -496,8 +498,8 @@ def test_list_resource_record_sets_explicit(self):
         iterator = zone.list_resource_record_sets(
             max_results=3, page_token=TOKEN, client=client2)
         self.assertIs(zone, iterator.zone)
-        iterator.update_page()
-        rrsets = list(iterator.page)
+        page = six.next(iterator.pages)
+        rrsets = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(rrsets), len(DATA['rrsets']))
@@ -553,6 +555,7 @@ def _get_changes(self, token, changes_name):
         return result
 
     def test_list_changes_defaults(self):
+        import six
         from google.cloud.dns.changes import Changes
         from google.cloud.dns.resource_record_set import ResourceRecordSet
 
@@ -569,8 +572,8 @@ def test_list_changes_defaults(self):
 
         iterator = zone.list_changes()
         self.assertIs(zone, iterator.zone)
-        iterator.update_page()
-        changes = list(iterator.page)
+        page = six.next(iterator.pages)
+        changes = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(changes), len(data['changes']))
@@ -606,6 +609,7 @@ def test_list_changes_defaults(self):
         self.assertEqual(req['path'], '/%s' % (path,))
 
     def test_list_changes_explicit(self):
+        import six
         from google.cloud.dns.changes import Changes
         from google.cloud.dns.resource_record_set import ResourceRecordSet
 
@@ -624,8 +628,8 @@ def test_list_changes_explicit(self):
         iterator = zone.list_changes(
             max_results=3, page_token=page_token, client=client2)
         self.assertIs(zone, iterator.zone)
-        iterator.update_page()
-        changes = list(iterator.page)
+        page = six.next(iterator.pages)
+        changes = list(page)
         token = iterator.next_page_token
 
         self.assertEqual(len(changes), len(data['changes']))
diff --git a/resource_manager/unit_tests/test_client.py b/resource_manager/unit_tests/test_client.py
--- a/resource_manager/unit_tests/test_client.py
+++ b/resource_manager/unit_tests/test_client.py
@@ -229,6 +229,7 @@ def test_page_empty_response(self):
         self.assertEqual(list(page), [])
 
     def test_page_non_empty_response(self):
+        import six
         from google.cloud.resource_manager.project import Project
 
         project_id = 'project-id'
@@ -253,10 +254,9 @@ def dummy_response():
         iterator = client.list_projects()
         iterator._get_next_page_response = dummy_response
 
-        iterator.update_page()
-        page = iterator.page
+        page = six.next(iterator.pages)
         self.assertEqual(page.num_items, 1)
-        project = iterator.next()
+        project = six.next(page)
         self.assertEqual(page.remaining, 0)
         self.assertIsInstance(project, Project)
         self.assertEqual(project.project_id, project_id)
diff --git a/storage/unit_tests/test_bucket.py b/storage/unit_tests/test_bucket.py
--- a/storage/unit_tests/test_bucket.py
+++ b/storage/unit_tests/test_bucket.py
@@ -991,6 +991,7 @@ def test_page_empty_response(self):
         self.assertEqual(iterator.prefixes, set())
 
     def test_page_non_empty_response(self):
+        import six
         from google.cloud.storage.blob import Blob
 
         blob_name = 'blob-name'
@@ -1006,17 +1007,17 @@ def dummy_response():
         iterator = bucket.list_blobs()
         iterator._get_next_page_response = dummy_response
 
-        iterator.update_page()
-        page = iterator.page
+        page = six.next(iterator.pages)
         self.assertEqual(page.prefixes, ('foo',))
         self.assertEqual(page.num_items, 1)
-        blob = iterator.next()
+        blob = six.next(page)
         self.assertEqual(page.remaining, 0)
         self.assertIsInstance(blob, Blob)
         self.assertEqual(blob.name, blob_name)
         self.assertEqual(iterator.prefixes, set(['foo']))
 
     def test_cumulative_prefixes(self):
+        import six
         from google.cloud.storage.blob import Blob
 
         BLOB_NAME = 'blob-name1'
@@ -1041,18 +1042,17 @@ def dummy_response():
         iterator._get_next_page_response = dummy_response
 
         # Parse first response.
-        iterator.update_page()
-        page1 = iterator.page
+        pages_iter = iterator.pages
+        page1 = six.next(pages_iter)
         self.assertEqual(page1.prefixes, ('foo',))
         self.assertEqual(page1.num_items, 1)
-        blob = iterator.next()
+        blob = six.next(page1)
         self.assertEqual(page1.remaining, 0)
         self.assertIsInstance(blob, Blob)
         self.assertEqual(blob.name, BLOB_NAME)
         self.assertEqual(iterator.prefixes, set(['foo']))
         # Parse second response.
-        iterator.update_page()
-        page2 = iterator.page
+        page2 = six.next(pages_iter)
         self.assertEqual(page2.prefixes, ('bar',))
         self.assertEqual(page2.num_items, 0)
         self.assertEqual(iterator.prefixes, set(['foo', 'bar']))
diff --git a/storage/unit_tests/test_client.py b/storage/unit_tests/test_client.py
--- a/storage/unit_tests/test_client.py
+++ b/storage/unit_tests/test_client.py
@@ -381,6 +381,7 @@ def test_page_empty_response(self):
         self.assertEqual(list(page), [])
 
     def test_page_non_empty_response(self):
+        import six
         from google.cloud.storage.bucket import Bucket
 
         project = 'PROJECT'
@@ -396,10 +397,9 @@ def dummy_response():
         iterator = client.list_buckets()
         iterator._get_next_page_response = dummy_response
 
-        iterator.update_page()
-        page = iterator.page
+        page = six.next(iterator.pages)
         self.assertEqual(page.num_items, 1)
-        bucket = iterator.next()
+        bucket = six.next(page)
         self.assertEqual(page.remaining, 0)
         self.assertIsInstance(bucket, Bucket)
         self.assertEqual(bucket.name, blob_name)
diff --git a/system_tests/storage.py b/system_tests/storage.py
--- a/system_tests/storage.py
+++ b/system_tests/storage.py
@@ -257,13 +257,20 @@ def test_paginate_files(self):
         truncation_size = 1
         count = len(self.FILENAMES) - truncation_size
         iterator = self.bucket.list_blobs(max_results=count)
-        iterator.update_page()
-        blobs = list(iterator.page)
+        page_iter = iterator.pages
+
+        page1 = six.next(page_iter)
+        blobs = list(page1)
         self.assertEqual(len(blobs), count)
         self.assertIsNotNone(iterator.next_page_token)
-
-        iterator.update_page()
-        last_blobs = list(iterator.page)
+        # Technically the iterator is exhausted.
+        self.assertEqual(iterator.num_results, iterator.max_results)
+        # But we modify the iterator to continue paging after
+        # articially stopping after ``count`` items.
+        iterator.max_results = None
+
+        page2 = six.next(page_iter)
+        last_blobs = list(page2)
         self.assertEqual(len(last_blobs), truncation_size)
 
 
@@ -301,8 +308,8 @@ def tearDownClass(cls):
     @RetryErrors(unittest.TestCase.failureException)
     def test_root_level_w_delimiter(self):
         iterator = self.bucket.list_blobs(delimiter='/')
-        iterator.update_page()
-        blobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        blobs = list(page)
         self.assertEqual([blob.name for blob in blobs], ['file01.txt'])
         self.assertIsNone(iterator.next_page_token)
         self.assertEqual(iterator.prefixes, set(['parent/']))
@@ -310,8 +317,8 @@ def test_root_level_w_delimiter(self):
     @RetryErrors(unittest.TestCase.failureException)
     def test_first_level(self):
         iterator = self.bucket.list_blobs(delimiter='/', prefix='parent/')
-        iterator.update_page()
-        blobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        blobs = list(page)
         self.assertEqual([blob.name for blob in blobs], ['parent/file11.txt'])
         self.assertIsNone(iterator.next_page_token)
         self.assertEqual(iterator.prefixes, set(['parent/child/']))
@@ -325,8 +332,8 @@ def test_second_level(self):
 
         iterator = self.bucket.list_blobs(delimiter='/',
                                           prefix='parent/child/')
-        iterator.update_page()
-        blobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        blobs = list(page)
         self.assertEqual([blob.name for blob in blobs],
                          expected_names)
         self.assertIsNone(iterator.next_page_token)
@@ -341,8 +348,8 @@ def test_third_level(self):
         # Exercise a layer deeper to illustrate this.
         iterator = self.bucket.list_blobs(delimiter='/',
                                           prefix='parent/child/grand/')
-        iterator.update_page()
-        blobs = list(iterator.page)
+        page = six.next(iterator.pages)
+        blobs = list(page)
         self.assertEqual([blob.name for blob in blobs],
                          ['parent/child/grand/file31.txt'])
         self.assertIsNone(iterator.next_page_token)