Repository: pandas-dev/pandas

Issue:
Iterating through TableIterator with where clause can incorrectly ignore data
Expected behaviour: Using appendable table stored using HDFStore summed length of DataFrames returned using an iterator with a where clause should equal the length of the DataFrame when returned using the same where clause but with iterator=False e.g. TableIterator.get_values().

The attached code generates appendable tables of size 100064, 200064, ..., 400064. It uses a where clause which is a superset of all possible values to get DataFrames with iterator=False, with and without the where clause, and with iterator=True, also with and without the where clause. In all cases except for iterator=True with the where clause the length of the returned DataFrames is correct.

For the failure cases in closer inspection in iPython it is the last 64 rows which are not being returned.

Note: in create_file() the appending of DataFrames with lengths of 58689 and 41375 was chosen specifically to reproduce the problem. I originally encountered the problem with a dataset with length 174000064 and the last append was size 41375. I attempted to reproduce the problem by creating various length tables in chunks of 100000 with a final append of 64 and wasn't able to do so.

Creating the table with the last chunk = 41375 with total length exceeding 300000 does in my tests reproduce the problem.

Output:

```
iteration: 0 PASSED
expected: 100064, df len: 100064, it (no where clause) len: 100064, it len: 100064
iteration: 1 PASSED
expected: 200064, df len: 200064, it (no where clause) len: 200064, it len: 200064
iteration: 2 FAILED
expected: 300064, df len: 300064, it (no where clause) len: 300064, it len: 300000
iteration: 3 FAILED
expected: 400064, df len: 400064, it (no where clause) len: 400064, it len: 400000
```

> > > pd.show_versions()
## INSTALLED VERSIONS

commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-32-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.13.1
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: None
IPython: 1.2.1
sphinx: None
patsy: None
scikits.timeseries: None
dateutil: 1.5
pytz: 2012c
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.3.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
sqlalchemy: None
lxml: 3.3.3
bs4: 4.3.2
html5lib: 0.999
bq: None
apiclient: None

``` python

import os
from dateutil.relativedelta import relativedelta

import numpy as np
randn = np.random.randn
from pandas import DataFrame, HDFStore, date_range

def create_df(beg_dt, periods=1e5):
    """ Create a DataFrame containing values v. """

    dr = date_range(beg_dt, periods=periods, freq='S')
    df = DataFrame(index=dr, data=np.random.randn(periods, 4), columns=['bid_price','bid_vol', 'ask_price','ask_vol'])
    return(df)

def create_file(iterations=1):
    beg_dt = '2014-08-12 13:30:00.000000'
    periods = 1e5
    for i in xrange(iterations):
        df = create_df(beg_dt, periods)
        store_append(store, df, key="df")
        beg_dt = df.index[-1:][0] + relativedelta(seconds=1)

    df = create_df(beg_dt, 58689)
    store_append(store, df, key="df")
    beg_dt = df.index[-1:][0] + relativedelta(seconds=1)
    df = create_df(beg_dt, 41375)
    store_append(store, df, key="df")
    beg_dt = df.index[-1:][0] + relativedelta(seconds=1)

    return(df)

def store_open(fname):
    return(HDFStore(fname))

def store_get(store, key="df", where=None, start=None, stop=None, iterator=False, chunksize=None):
    df = None
    try:
        df = store.select(key, where=where, start=start, stop=stop, iterator=iterator, chunksize=chunksize)
    except (KeyError, TypeError, ):
        pass

    return(df)

def store_append(store, df, key="df", where=""):
    store.append(key, df, format='table')

path = '.'
fname = '/'.join([path, 'delme_test.h5'])

store = None
for n in xrange(0, 4):
    pass

    if store:
        try: store.close()
        except: pass
    try: os.unlink(fname)
    except: pass
    store = store_open(fname)

    create_file(n)
    store.close()

    store = store_open(fname)
    where = None
    df = store_get(store, 'df', where=where, iterator=False)

    expected_ln = len(df)

    beg_dt = '2014-08-12 13:30:00.000000'
    end_dt = '2032-12-31 13:30:00.000000'
    where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)

    # where clause, iterator=False
    df = store_get(store, 'df', where=where, iterator=False)
    ln_df = len(df)

    # no where clause
    it = store_get(store, 'df', where=None, iterator=True)
    dfs = [df for df in it if not df.empty]
    ln_it_no_where_clause = sum([len(df) for df in dfs])

    # where clause, iterator=True
    it = store_get(store, 'df', where=where, iterator=True)
    dfs = [df for df in it if not df.empty]
    ln_it = sum([len(df) for df in dfs])

    if expected_ln == ln_df and expected_ln == ln_it:
        print("iteration: %d PASSED" % n)
    else:
        print("iteration: %d FAILED" % n)
    print("expected: %d, df len: %d, it (no where clause) len: %d, it len: %d" %
        (expected_ln, ln_df, ln_it_no_where_clause, ln_it))

store.close()
```



Discussion:
you would need to narrow this down to a small reproducible example.
- create a test frame
- write to the store
- read in chunks so the where _should_ encompass all values
- concat the chunks to form a composite frame
- compare versus the original

This _is_ the reproducible example which I did narrow down. As I noted simply writing test frames of length chunksize \* some multiple followed by a DataFrame of len 64 does not reproduce the problem.

I explained that the problem only occurs once 200000 + 58689 + 41375 rows are written.

Comparing the concatted frame to the original will fail because the concatted frame is missing the last 64 elements as I noted.

@bboerner 

in order for this to be tested it has to be constructed as a test.

Once you have a test that reliably fails then you can debug.

Pls see the format here: https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_pytables.py

It _looks_ like a bug, but it could be that your selection is simply incorrect. Hard to test w/o a reliable test.
Simply comparing the length is not enough. What if each chunk is off by 1 in your selection (I am not saying it is), and the length compares equal (but is in fact right)? or vice-versa.

@jreback, understood, I'll work up a test. Thanks.

@jreback,

This can be added to https://github.com/pydata/pandas/blob/master/pandas/io/tests/test_pytables.py and reproduces the problem.

Regards.

```
    def test_select_iterator_8014(self):

        # single table
        chunksize=1e4
        with ensure_clean_store(self.path) as store:

            df = tm.makeTimeDataFrame(100064, 'S')
            _maybe_remove(store, 'df')
            i = 0; ln = 58689
            store.append('df', df[i:i+ln])
            i = i+ln; ln = 41375
            store.append('df', df[i:i+ln])
            expected = df

            beg_dt = expected.index[0]
            end_dt = expected.index[-1]

            #
            # w/o iterator
            #

            # select w/o iteration and no where clause works
            result = store.select('df')
            tm.assert_frame_equal(expected, result)

            # select w/o iterator and where clause, single term, begin
            # of range, works
            where = "index >= '%s'" % beg_dt
            result = store.select('df',where=where)
            tm.assert_frame_equal(expected, result)

            # select w/o iterator and where clause, single term, end
            # of range, works
            where = "index <= '%s'" % end_dt
            result = store.select('df',where=where)
            tm.assert_frame_equal(expected, result)

            # select w/o iterator and where clause, inclusive range,
            # works
            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
            result = store.select('df',where=where)
            tm.assert_frame_equal(expected, result)

            #
            # with iterator
            #

            # select w/iterator and no where clause works
            results = []
            for s in store.select('df',iterator=True, chunksize=chunksize):
                results.append(s)
            result = concat(results)
            tm.assert_frame_equal(expected, result)

            # select w/iterator and where clause, single term, begin
            # of range, fails
            where = "index >= '%s'" % beg_dt
            results = []
            for s in store.select('df',where=where,iterator=True, chunksize=chunksize):
                results.append(s)
            result = concat(results)
            tm.assert_frame_equal(expected, result)

            # select w/iterator and where clause, single term, end of
            # range, fails
            where = "index <= '%s'" % end_dt
            results = []
            for s in store.select('df',where=where,iterator=True, chunksize=chunksize):
                results.append(s)
            result = concat(results)
            tm.assert_frame_equal(expected, result)

            # select w/iterator and where clause, inclusive range, fails
            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
            results = []
            for s in store.select('df',where=where,iterator=True, chunksize=chunksize):
                results.append(s)
            result = concat(results)
            tm.assert_frame_equal(expected, result)

```


Resolving patch:
diff --git a/pandas/io/pytables.py b/pandas/io/pytables.py
--- a/pandas/io/pytables.py
+++ b/pandas/io/pytables.py
@@ -662,21 +662,18 @@ def select(self, key, where=None, start=None, stop=None, columns=None,
         s = self._create_storer(group)
         s.infer_axes()
 
-        # what we are actually going to do for a chunk
-        def func(_start, _stop):
-            return s.read(where=where, start=_start, stop=_stop,
+        # function to call on iteration
+        def func(_start, _stop, _where):
+            return s.read(start=_start, stop=_stop,
+                          where=_where,
                           columns=columns, **kwargs)
 
-        if iterator or chunksize is not None:
-            if not s.is_table:
-                raise TypeError(
-                    "can only use an iterator or chunksize on a table")
-            return TableIterator(self, func, nrows=s.nrows, start=start,
-                                 stop=stop, chunksize=chunksize,
-                                 auto_close=auto_close)
+        # create the iterator
+        it = TableIterator(self, s, func, where=where, nrows=s.nrows, start=start,
+                           stop=stop, iterator=iterator, chunksize=chunksize,
+                           auto_close=auto_close)
 
-        return TableIterator(self, func, nrows=s.nrows, start=start, stop=stop,
-                             auto_close=auto_close).get_values()
+        return it.get_result()
 
     def select_as_coordinates(
             self, key, where=None, start=None, stop=None, **kwargs):
@@ -779,26 +776,22 @@ def select_as_multiple(self, keys, where=None, selector=None, columns=None,
         # axis is the concentation axes
         axis = list(set([t.non_index_axes[0][0] for t in tbls]))[0]
 
-        def func(_start, _stop):
-            if where is not None:
-                c = s.read_coordinates(where=where, start=_start, stop=_stop, **kwargs)
-            else:
-                c = None
+        def func(_start, _stop, _where):
 
-            objs = [t.read(where=c, start=_start, stop=_stop,
-                           columns=columns, **kwargs) for t in tbls]
+            # retrieve the objs, _where is always passed as a set of coordinates here
+            objs = [t.read(where=_where, columns=columns, **kwargs) for t in tbls]
 
             # concat and return
             return concat(objs, axis=axis,
                           verify_integrity=False).consolidate()
 
-        if iterator or chunksize is not None:
-            return TableIterator(self, func, nrows=nrows, start=start,
-                                 stop=stop, chunksize=chunksize,
-                                 auto_close=auto_close)
+        # create the iterator
+        it = TableIterator(self, s, func, where=where, nrows=nrows, start=start,
+                           stop=stop, iterator=iterator, chunksize=chunksize,
+                           auto_close=auto_close)
+
+        return it.get_result(coordinates=True)
 
-        return TableIterator(self, func, nrows=nrows, start=start, stop=stop,
-                             auto_close=auto_close).get_values()
 
     def put(self, key, value, format=None, append=False, **kwargs):
         """
@@ -1293,20 +1286,25 @@ class TableIterator(object):
         ----------
 
         store : the reference store
-        func  : the function to get results
+        s     : the refered storer
+        func  : the function to execute the query
+        where : the where of the query
         nrows : the rows to iterate on
         start : the passed start value (default is None)
         stop  : the passed stop value (default is None)
-        chunksize : the passed chunking valeu (default is 50000)
+        iterator : boolean, whether to use the default iterator
+        chunksize : the passed chunking value (default is 50000)
         auto_close : boolean, automatically close the store at the end of
             iteration, default is False
         kwargs : the passed kwargs
         """
 
-    def __init__(self, store, func, nrows, start=None, stop=None,
-                 chunksize=None, auto_close=False):
+    def __init__(self, store, s, func, where, nrows, start=None, stop=None,
+                 iterator=False, chunksize=None, auto_close=False):
         self.store = store
-        self.func = func
+        self.s     = s
+        self.func  = func
+        self.where = where
         self.nrows = nrows or 0
         self.start = start or 0
 
@@ -1314,23 +1312,29 @@ def __init__(self, store, func, nrows, start=None, stop=None,
             stop = self.nrows
         self.stop = min(self.nrows, stop)
 
-        if chunksize is None:
-            chunksize = 100000
+        self.coordinates = None
+        if iterator or chunksize is not None:
+            if chunksize is None:
+                chunksize = 100000
+            self.chunksize = int(chunksize)
+        else:
+            self.chunksize = None
 
-        self.chunksize = chunksize
         self.auto_close = auto_close
 
     def __iter__(self):
+
+        # iterate
         current = self.start
         while current < self.stop:
-            stop = current + self.chunksize
-            v = self.func(current, stop)
-            current = stop
 
-            if v is None:
+            stop = min(current + self.chunksize, self.stop)
+            value = self.func(None, None, self.coordinates[current:stop])
+            current = stop
+            if value is None or not len(value):
                 continue
 
-            yield v
+            yield value
 
         self.close()
 
@@ -1338,12 +1342,29 @@ def close(self):
         if self.auto_close:
             self.store.close()
 
-    def get_values(self):
-        results = self.func(self.start, self.stop)
+    def get_result(self, coordinates=False):
+
+        #  return the actual iterator
+        if self.chunksize is not None:
+            if not self.s.is_table:
+                raise TypeError(
+                    "can only use an iterator or chunksize on a table")
+
+            self.coordinates = self.s.read_coordinates(where=self.where)
+
+            return self
+
+        # if specified read via coordinates (necessary for multiple selections
+        if coordinates:
+            where = self.s.read_coordinates(where=self.where)
+        else:
+            where = self.where
+
+        # directly return the result
+        results = self.func(self.start, self.stop, where)
         self.close()
         return results
 
-
 class IndexCol(StringMixin):
 
     """ an index column description class


Test patch:
diff --git a/pandas/io/tests/test_pytables.py b/pandas/io/tests/test_pytables.py
--- a/pandas/io/tests/test_pytables.py
+++ b/pandas/io/tests/test_pytables.py
@@ -3264,21 +3264,16 @@ def test_select_iterator(self):
 
             expected = store.select('df')
 
-            results = []
-            for s in store.select('df',iterator=True):
-                results.append(s)
+            results = [ s for s in store.select('df',iterator=True) ]
             result = concat(results)
             tm.assert_frame_equal(expected, result)
-            results = []
-            for s in store.select('df',chunksize=100):
-                results.append(s)
+
+            results = [ s for s in store.select('df',chunksize=100) ]
             self.assertEqual(len(results), 5)
             result = concat(results)
             tm.assert_frame_equal(expected, result)
 
-            results = []
-            for s in store.select('df',chunksize=150):
-                results.append(s)
+            results = [ s for s in store.select('df',chunksize=150) ]
             result = concat(results)
             tm.assert_frame_equal(result, expected)
 
@@ -3294,12 +3289,10 @@ def test_select_iterator(self):
             df = tm.makeTimeDataFrame(500)
             df.to_hdf(path,'df',format='table')
 
-            results = []
-            for x in read_hdf(path,'df',chunksize=100):
-                results.append(x)
+            results = [ s for s in read_hdf(path,'df',chunksize=100) ]
+            result = concat(results)
 
             self.assertEqual(len(results), 5)
-            result = concat(results)
             tm.assert_frame_equal(result, df)
             tm.assert_frame_equal(result, read_hdf(path,'df'))
 
@@ -3318,10 +3311,8 @@ def test_select_iterator(self):
             # full selection
             expected = store.select_as_multiple(
                 ['df1', 'df2'], selector='df1')
-            results = []
-            for s in store.select_as_multiple(
-                ['df1', 'df2'], selector='df1', chunksize=150):
-                results.append(s)
+            results = [ s for s in store.select_as_multiple(
+                ['df1', 'df2'], selector='df1', chunksize=150) ]
             result = concat(results)
             tm.assert_frame_equal(expected, result)
 
@@ -3335,6 +3326,185 @@ def test_select_iterator(self):
             #result = concat(results)
             #tm.assert_frame_equal(expected, result)
 
+    def test_select_iterator_complete_8014(self):
+
+        # GH 8014
+        # using iterator and where clause
+        chunksize=1e4
+
+        # no iterator
+        with ensure_clean_store(self.path) as store:
+
+            expected = tm.makeTimeDataFrame(100064, 'S')
+            _maybe_remove(store, 'df')
+            store.append('df',expected)
+
+            beg_dt = expected.index[0]
+            end_dt = expected.index[-1]
+
+            # select w/o iteration and no where clause works
+            result = store.select('df')
+            tm.assert_frame_equal(expected, result)
+
+            # select w/o iterator and where clause, single term, begin
+            # of range, works
+            where = "index >= '%s'" % beg_dt
+            result = store.select('df',where=where)
+            tm.assert_frame_equal(expected, result)
+
+            # select w/o iterator and where clause, single term, end
+            # of range, works
+            where = "index <= '%s'" % end_dt
+            result = store.select('df',where=where)
+            tm.assert_frame_equal(expected, result)
+
+            # select w/o iterator and where clause, inclusive range,
+            # works
+            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
+            result = store.select('df',where=where)
+            tm.assert_frame_equal(expected, result)
+
+        # with iterator, full range
+        with ensure_clean_store(self.path) as store:
+
+            expected = tm.makeTimeDataFrame(100064, 'S')
+            _maybe_remove(store, 'df')
+            store.append('df',expected)
+
+            beg_dt = expected.index[0]
+            end_dt = expected.index[-1]
+
+            # select w/iterator and no where clause works
+            results = [ s for s in store.select('df',chunksize=chunksize) ]
+            result = concat(results)
+            tm.assert_frame_equal(expected, result)
+
+            # select w/iterator and where clause, single term, begin of range
+            where = "index >= '%s'" % beg_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            tm.assert_frame_equal(expected, result)
+
+            # select w/iterator and where clause, single term, end of range
+            where = "index <= '%s'" % end_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            tm.assert_frame_equal(expected, result)
+
+            # select w/iterator and where clause, inclusive range
+            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            tm.assert_frame_equal(expected, result)
+
+    def test_select_iterator_non_complete_8014(self):
+
+        # GH 8014
+        # using iterator and where clause
+        chunksize=1e4
+
+        # with iterator, non complete range
+        with ensure_clean_store(self.path) as store:
+
+            expected = tm.makeTimeDataFrame(100064, 'S')
+            _maybe_remove(store, 'df')
+            store.append('df',expected)
+
+            beg_dt = expected.index[1]
+            end_dt = expected.index[-2]
+
+            # select w/iterator and where clause, single term, begin of range
+            where = "index >= '%s'" % beg_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            rexpected = expected[expected.index >= beg_dt]
+            tm.assert_frame_equal(rexpected, result)
+
+            # select w/iterator and where clause, single term, end of range
+            where = "index <= '%s'" % end_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            rexpected = expected[expected.index <= end_dt]
+            tm.assert_frame_equal(rexpected, result)
+
+            # select w/iterator and where clause, inclusive range
+            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            rexpected = expected[(expected.index >= beg_dt) & (expected.index <= end_dt)]
+            tm.assert_frame_equal(rexpected, result)
+
+        # with iterator, empty where
+        with ensure_clean_store(self.path) as store:
+
+            expected = tm.makeTimeDataFrame(100064, 'S')
+            _maybe_remove(store, 'df')
+            store.append('df',expected)
+
+            end_dt = expected.index[-1]
+
+            # select w/iterator and where clause, single term, begin of range
+            where = "index > '%s'" % end_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            self.assertEqual(0, len(results))
+
+    def test_select_iterator_many_empty_frames(self):
+
+        # GH 8014
+        # using iterator and where clause can return many empty
+        # frames.
+        chunksize=int(1e4)
+
+        # with iterator, range limited to the first chunk
+        with ensure_clean_store(self.path) as store:
+
+            expected = tm.makeTimeDataFrame(100000, 'S')
+            _maybe_remove(store, 'df')
+            store.append('df',expected)
+
+            beg_dt = expected.index[0]
+            end_dt = expected.index[chunksize-1]
+
+            # select w/iterator and where clause, single term, begin of range
+            where = "index >= '%s'" % beg_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+            result = concat(results)
+            rexpected = expected[expected.index >= beg_dt]
+            tm.assert_frame_equal(rexpected, result)
+
+            # select w/iterator and where clause, single term, end of range
+            where = "index <= '%s'" % end_dt
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+
+            tm.assert_equal(1, len(results))
+            result = concat(results)
+            rexpected = expected[expected.index <= end_dt]
+            tm.assert_frame_equal(rexpected, result)
+
+            # select w/iterator and where clause, inclusive range
+            where = "index >= '%s' & index <= '%s'" % (beg_dt, end_dt)
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+
+            # should be 1, is 10
+            tm.assert_equal(1, len(results))
+            result = concat(results)
+            rexpected = expected[(expected.index >= beg_dt) & (expected.index <= end_dt)]
+            tm.assert_frame_equal(rexpected, result)
+
+            # select w/iterator and where clause which selects
+            # *nothing*.
+            #
+            # To be consistent with Python idiom I suggest this should
+            # return [] e.g. `for e in []: print True` never prints
+            # True.
+
+            where = "index <= '%s' & index >= '%s'" % (beg_dt, end_dt)
+            results = [ s for s in store.select('df',where=where,chunksize=chunksize) ]
+
+            # should be []
+            tm.assert_equal(0, len(results))
+
+
     def test_retain_index_attributes(self):
 
         # GH 3499, losing frequency info on index recreation