Repository: astropy/astropy

Issue:
ascii.read stalls out when reading large files
@hamogu - I am trying to use ascii.read() to reduce a large (~6Gb) ASCII dataset into a processed binary file in which I have applied some cuts. The strategy I have been taking is to the use data_start and data_end keywords in a loop over chunks of the file, so that only ~100Mb chunks are loaded into memory at a time. Then I make cuts on the chunk, append a Table with the rows passing the cuts, and move on to the next chunk. So, pretty classic 'big data' task. 

What I am finding is that ascii.read() is failing to only load chunks at a time. In particular, _any_ call to my ASCII file with ascii.read(fname, data_start=i, data_end=i+chunksize) results in an interminable stall of the python interpreter. So far as I am able to tell, I have no way to use astropy's ascii reader to process data this large. I have verified that this behavior does not happen for an artificially reduced version of the file (~100Mb) with an otherwise identical format. 

I understand that ascii.read() has been re-vamped for v1 with 'big data' tasks like this in mind. However, I have verified that this stall-out occurs both with v0.4.x and also with the version of the master branch that is current at the time of this issue. 


Discussion:
taldcroft: WIP for memory optimization, but this is not so easy so punt for now.  This is for future reference.
```
    out = None
    for tbl in _read_in_chunks_generator(table, chunk_size, **kwargs):
        if out is None:
            out = tbl
            continue

        if out.colnames != tbl.colnames:
            raise ValueError('column name mismatch in chunked reading')

        for out_col, tbl_col in zip(out.itercols(), tbl.itercols()):
            try:
                dtype_str = common_dtype([out_col, tbl_col])
            except MergeConflictError:
                raise  # TO DO: handle this
            else:
                if dtype_str != out_col.dtype.str:
                    # Promote column and change output table reference
                    out_col = out_col.astype(dtype_str)
                    tbl[out_col.name] = out_col
                out_col.insert(len(out_col), tbl_col)
```
bsipocz: @taldcroft - No need to address ``Fix one range(..) to be six.moves.range()``, as this will go into 3.0, won't it?
taldcroft: @bsipocz - Good point, WOOHOO for Py3 only! 
pllim: I can reproduce failures on my Windows but they are not exactly the same ones as Appveyor: 
[testlog.txt](https://github.com/astropy/astropy/files/1320062/testlog.txt)

There are also a couple of `ResourceWarning` about unclosed sockets (not captured in log) but probably unrelated?

taldcroft: @astrofrog @hamogu - this is ready for final review.  The changes in `ui.py` (+ docs and tests) may be the place to focus.
pllim: One test still fails on Appveyor (which is an improvement over the two that I found a week ago).
taldcroft: @pllim - argh, it was passing appveyor before the last commit.
taldcroft: At least I now have a functional Windows dev environment on my mac.
pllim: It passed! There is a conflict though.
taldcroft: @hamogu - are you OK with this?  I think I addressed all your comments.
taldcroft: @hamogu - thanks for the review!  Will merge when CI passes after the rebase.
saimn: I think I found a bug while playing with it to understand the code : 
```
In [1]: f = open('foo.csv', 'w', encoding='ascii')

In [2]: for i in range(1000):
   ...:     f.write(f'{i},a{i},{i}.{i}\n')
   ...:     

In [3]: f.close()

In [5]: %astropy
Numpy 1.13.3
Astropy 3.0.dev20668

In [6]: tref = ascii.read('foo.csv', format='csv')

In [7]: tbl = ascii.read('foo.csv', format='csv', guess=False, fast_reader={'chunk_size': 100})
   ...: 

In [8]: tref
Out[8]: 
<Table length=999>
  0    a0    0.0  
int64 str4 float64
----- ---- -------
    1   a1     1.1
    2   a2     2.2
    3   a3     3.3
    4   a4     4.4
    5   a5     5.5
    6   a6     6.6
    7   a7     7.7
    8   a8     8.8
    9   a9     9.9
   10  a10    10.1
   11  a11   11.11
   12  a12   12.12
   13  a13   13.13
   14  a14   14.14
   15  a15   15.15
  ...  ...     ...
  984 a984 984.984
  985 a985 985.985
  986 a986 986.986
  987 a987 987.987
  988 a988 988.988
  989 a989 989.989
  990 a990  990.99
  991 a991 991.991
  992 a992 992.992
  993 a993 993.993
  994 a994 994.994
  995 a995 995.995
  996 a996 996.996
  997 a997 997.997
  998 a998 998.998
  999 a999 999.999

In [9]: tbl
Out[9]: 
<Table length=999>
  0    a0    0.0  
int64 str2 float64
----- ---- -------
    1   a1     1.1
    2   a2     2.2
    3   a3     3.3
    4   a4     4.4
    5   a5     5.5
    6   a6     6.6
    7   a7     7.7
    8   a8     8.8
    9   a9     9.9
   10   a1    10.1
   11   a1   11.11
   12   a1   12.12
   13   a1   13.13
   14   a1   14.14
   15   a1   15.15
  ...  ...     ...
  984   a9 984.984
  985   a9 985.985
  986   a9 986.986
  987   a9 987.987
  988   a9 988.988
  989   a9 989.989
  990   a9  990.99
  991   a9 991.991
  992   a9 992.992
  993   a9 993.993
  994   a9 994.994
  995   a9 995.995
  996   a9 996.996
  997   a9 997.997
  998   a9 998.998
  999   a9 999.999
```
It's a very silly table :wink: but it seems that while concatenating the str column the dtype is not updated.
I looked quickly and do not understand everything in the code but it looks good, a great addition !
taldcroft: Ah yes, the inplace resize for non-masked columns is no good in the case of a string that changes size.  This should be an easy fix.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index 1c5d9a53d0..c2bf8ad745 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -271,6 +271,9 @@ astropy.extern
 astropy.io.ascii
 ^^^^^^^^^^^^^^^^
 
+- Added support for reading very large tables in chunks to reduce memory
+  usage. [#6458]
+
 astropy.io.fits
 ^^^^^^^^^^^^^^^
 
diff --git a/astropy/io/ascii/cparser.pyx b/astropy/io/ascii/cparser.pyx
index 7bad5bc7e2..2fbee18e69 100644
--- a/astropy/io/ascii/cparser.pyx
+++ b/astropy/io/ascii/cparser.pyx
@@ -186,6 +186,7 @@ cdef class CParser:
         int width
         object source
         object header_start
+        object header_chars
 
     def __cinit__(self, source, strip_line_whitespace, strip_line_fields,
                   delimiter=',',
@@ -203,11 +204,7 @@ cdef class CParser:
                   fill_extra_cols=0,
                   fast_reader=True):
 
-        # Handle fast_reader parameter (True or dict specifying options)
-        if fast_reader is True or fast_reader == 'force':
-            fast_reader = {}
-        elif fast_reader is False: # shouldn't happen
-            raise core.ParameterError("fast_reader cannot be False for fast readers")
+        # Handle fast_reader parameter
         expchar = fast_reader.pop('exponent_style', 'E').upper()
         # parallel and use_fast_reader are False by default, but only the latter
         # supports Fortran double precision notation
@@ -376,6 +373,8 @@ cdef class CParser:
             self.raise_error("an error occurred while advancing to the first "
                              "line of data")
 
+        self.header_chars = self.source[:self.tokenizer.source_pos]
+
         cdef int data_end = -1 # keep reading data until the end
         if self.data_end is not None and self.data_end >= 0:
             data_end = max(self.data_end - self.data_start, 0) # read nothing if data_end < 0
@@ -421,6 +420,8 @@ cdef class CParser:
         cdef int i
         cdef size_t index
 
+        # Build up chunkindices which has the indices for all N chunks
+        # in an length N+1 array.
         for i in range(1, N):
             index = max(offset + chunksize * i, chunkindices[i - 1])
             while index < source_len and self.source[index] != '\n':
@@ -435,6 +436,7 @@ cdef class CParser:
         chunkindices.append(source_len)
         cdef list processes = []
 
+        # Create and start N parallel processes to read the N chunks
         for i in range(N):
             process = multiprocessing.Process(target=_read_chunk, args=(self,
                 chunkindices[i], chunkindices[i + 1],
@@ -442,10 +444,15 @@ cdef class CParser:
             processes.append(process)
             process.start()
 
+        # Define outputs in advance
         cdef list chunks = [None] * N
         cdef list comments_chunks = [None] * N
         cdef dict failed_procs = {}
 
+        # Asyncronously get the read results for the N chunks.  These
+        # come back in a non-deterministic order using the ``queue``
+        # to return results and the chunk index as ``proc``.  ``queue.get()``
+        # is blocking and waiting for a result.
         for i in range(N):
             queue_ret, err, proc = queue.get()
             if isinstance(err, Exception):
@@ -454,9 +461,12 @@ cdef class CParser:
                 raise err
             elif err is not None: # err is (error code, error line)
                 failed_procs[proc] = err
+
             comments, data = queue_ret
             comments_chunks[proc] = comments
             chunks[proc] = data
+
+        # Accumulate all the comments through file into a single list of comments
         for chunk in comments_chunks:
             line_comments.extend(chunk)
 
@@ -478,6 +488,9 @@ cdef class CParser:
             seen_str[name] = False
             seen_numeric[name] = False
 
+        # Go through each chunk and each column name and see if it was parsed
+        # as both a string in at least one chunck and/or numeric in at least
+        # one chunk.
         for chunk in chunks:
             for name in chunk:
                 if chunk[name].dtype.kind in ('S', 'U'):
@@ -486,19 +499,31 @@ cdef class CParser:
                 elif len(chunk[name]) > 0: # ignore empty chunk columns
                     seen_numeric[name] = True
 
+        # Go through each column name and see if it was parsed as both
+        # string and float in different chunks.  If so reconvert back
+        # to string.
         reconvert_cols = []
-
         for i, name in enumerate(self.names):
             if seen_str[name] and seen_numeric[name]:
                 # Reconvert to str to avoid conversion issues, e.g.
                 # 5 (int) -> 5.0 (float) -> 5.0 (string)
                 reconvert_cols.append(i)
 
+        # Slightly confusing: put the list of col numbers to reconvert
+        # onto the queue.  All of the reading processes are blocked and
+        # waiting for a value on the reconvert_queue.  One-by-one each
+        # process will manage to be first in line and get the value,
+        # handle, and the put reconvert_cols back on the queue for
+        # another waiting process.
+        # CONSIDER just putting reconvert_cols on the queue N times
+        # in a row here and don't have _read_chunk do that chaining.
         reconvert_queue.put(reconvert_cols)
         for process in processes:
             process.join() # wait for each process to finish
         try:
             while True:
+                # Each column that was reconverted gets passed back in the queue
+                # and is then substituted over the original (incorrect) type.
                 reconverted, proc, col = queue.get(False)
                 chunks[proc][self.names[col]] = reconverted
         except Queue.Empty:
@@ -528,6 +553,7 @@ cdef class CParser:
                         break
                     line_no += num_rows
 
+        # Concatenate the chunk data, one column at a time.
         ret = {}
         for name in self.get_names():
             col_chunks = [chunk.pop(name) for chunk in chunks]
@@ -536,6 +562,7 @@ cdef class CParser:
             else:
                 ret[name] = np.concatenate(col_chunks)
 
+        # Clean up processes
         for process in processes:
             process.terminate()
 
@@ -809,6 +836,7 @@ def _copy_cparser(bytes src_ptr, bytes source_bytes, use_cols, fill_names, fill_
         parser.tokenizer.source = source_bytes
     return parser
 
+
 def _read_chunk(CParser self, start, end, try_int,
                 try_float, try_string, queue, reconvert_queue, i):
     cdef tokenizer_t *chunk_tokenizer = self.tokenizer
@@ -840,6 +868,7 @@ def _read_chunk(CParser self, start, end, try_int,
         delete_tokenizer(chunk_tokenizer)
         queue.pop()
         queue.put((None, e, i))
+
     reconvert_cols = reconvert_queue.get()
     for col in reconvert_cols:
         queue.put((self._convert_str(chunk_tokenizer, col, -1), i, col))
diff --git a/astropy/io/ascii/fastbasic.py b/astropy/io/ascii/fastbasic.py
index 13313431a2..a95046b0df 100644
--- a/astropy/io/ascii/fastbasic.py
+++ b/astropy/io/ascii/fastbasic.py
@@ -1,6 +1,7 @@
 # Licensed under a 3-clause BSD style license - see LICENSE.rst
 
 import re
+import copy
 from collections import OrderedDict
 
 from . import core
@@ -89,6 +90,14 @@ class FastBasic(metaclass=core.MetaBaseReader):
 
         self.strict_names = self.kwargs.pop('strict_names', False)
 
+        fast_reader = self.kwargs.get('fast_reader', True)
+        if not isinstance(fast_reader, dict):
+            fast_reader = {}
+
+        fast_reader.pop('enable', None)
+        self.return_header_chars = fast_reader.pop('return_header_chars', False)
+        self.kwargs['fast_reader'] = fast_reader
+
         self.engine = cparser.CParser(table, self.strip_whitespace_lines,
                                       self.strip_whitespace_fields,
                                       delimiter=self.delimiter,
@@ -109,7 +118,15 @@ class FastBasic(metaclass=core.MetaBaseReader):
 
         with set_locale('C'):
             data, comments = self.engine.read(try_int, try_float, try_string)
+        out = self.make_table(data, comments)
 
+        if self.return_header_chars:
+            out.meta['__ascii_fast_reader_header_chars__'] = self.engine.header_chars
+
+        return out
+
+    def make_table(self, data, comments):
+        """Actually make the output table give the data and comments."""
         meta = OrderedDict()
         if comments:
             meta['comments'] = comments
@@ -230,21 +247,19 @@ class FastCommentedHeader(FastBasic):
         if 'data_start' not in kwargs:
             self.data_start = 0
 
-    def read(self, table):
+    def make_table(self, data, comments):
         """
-        Read input data (file-like object, filename, list of strings, or
-        single string) into a Table and return the result.
+        Actually make the output table give the data and comments.  This is
+        slightly different from the base FastBasic method in the way comments
+        are handled.
         """
-        out = super().read(table)
-
-        # Strip off first comment since this is the header line for
-        # commented_header format.
-        if 'comments' in out.meta:
-            out.meta['comments'] = out.meta['comments'][1:]
-            if not out.meta['comments']:
-                del out.meta['comments']
+        meta = OrderedDict()
+        if comments:
+            meta['comments'] = comments[1:]
+            if not meta['comments']:
+                del meta['comments']
 
-        return out
+        return Table(data, names=list(self.engine.get_names()), meta=meta)
 
     def _read_header(self):
         tmp = self.engine.source
diff --git a/astropy/io/ascii/ui.py b/astropy/io/ascii/ui.py
index 50b83118ad..bf69e53c15 100644
--- a/astropy/io/ascii/ui.py
+++ b/astropy/io/ascii/ui.py
@@ -15,6 +15,10 @@ import sys
 import copy
 import time
 import warnings
+import contextlib
+from io import StringIO
+
+import numpy as np
 
 from . import core
 from . import basic
@@ -29,7 +33,7 @@ from . import fastbasic
 from . import cparser
 from . import fixedwidth
 
-from ...table import Table
+from ...table import Table, vstack
 from ...utils.data import get_readable_fileobj
 from ...utils.exceptions import AstropyWarning, AstropyDeprecationWarning
 
@@ -242,6 +246,14 @@ def read(table, guess=None, **kwargs):
             One-character string defining the exponent or ``'Fortran'`` to auto-detect
             Fortran-style scientific notation like ``'3.14159D+00'`` (``'E'``, ``'D'``, ``'Q'``),
             all case-insensitive; default ``'E'``, all other imply ``use_fast_converter``
+        chunk_size : int
+            If supplied with a value > 0 then read the table in chunks of
+            approximately ``chunk_size`` bytes. Default is reading table in one pass.
+        chunk_generator : bool
+            If True and ``chunk_size > 0`` then return an iterator that returns a
+            table for each chunk.  The default is to return a single stacked table
+            for all the chunks.
+
     Reader : `~astropy.io.ascii.BaseReader`
         Reader class (DEPRECATED)
     encoding: str
@@ -249,19 +261,35 @@ def read(table, guess=None, **kwargs):
 
     Returns
     -------
-    dat : `~astropy.table.Table`
+    dat : `~astropy.table.Table` OR <generator>
         Output table
+
     """
     del _read_trace[:]
 
+    # Downstream readers might munge kwargs
+    kwargs = copy.deepcopy(kwargs)
+
+    # Convert fast_reader into a dict if not already and make sure 'enable'
+    # key is available.
+    fast_reader = kwargs.get('fast_reader', True)
+    if isinstance(fast_reader, dict):
+        fast_reader.setdefault('enable', 'force')
+    else:
+        fast_reader = {'enable': fast_reader}
+    kwargs['fast_reader'] = fast_reader
+
+    if fast_reader['enable'] and fast_reader.get('chunk_size'):
+        return _read_in_chunks(table, **kwargs)
+
     if 'fill_values' not in kwargs:
         kwargs['fill_values'] = [('', '0')]
 
     # If an Outputter is supplied in kwargs that will take precedence.
     new_kwargs = {}
-    fast_reader_param = kwargs.get('fast_reader', True)
     if 'Outputter' in kwargs:  # user specified Outputter, not supported for fast reading
-        fast_reader_param = False
+        fast_reader['enable'] = False
+
     format = kwargs.get('format')
     new_kwargs.update(kwargs)
 
@@ -320,36 +348,43 @@ def read(table, guess=None, **kwargs):
         # then there was just one set of kwargs in the guess list so fall
         # through below to the non-guess way so that any problems result in a
         # more useful traceback.
-        dat = _guess(table, new_kwargs, format, fast_reader_param)
+        dat = _guess(table, new_kwargs, format, fast_reader)
         if dat is None:
             guess = False
 
     if not guess:
         reader = get_reader(**new_kwargs)
+        if format is None:
+            format = reader._format_name
+
         # Try the fast reader version of `format` first if applicable.  Note that
         # if user specified a fast format (e.g. format='fast_basic') this test
         # will fail and the else-clause below will be used.
-        if fast_reader_param and format is not None and 'fast_{0}'.format(format) \
-                                                        in core.FAST_CLASSES:
-            fast_kwargs = copy.copy(new_kwargs)
+        if fast_reader['enable'] and 'fast_{0}'.format(format) in core.FAST_CLASSES:
+            fast_kwargs = copy.deepcopy(new_kwargs)
             fast_kwargs['Reader'] = core.FAST_CLASSES['fast_{0}'.format(format)]
-            fast_reader = get_reader(**fast_kwargs)
+            fast_reader_rdr = get_reader(**fast_kwargs)
             try:
-                dat = fast_reader.read(table)
+                dat = fast_reader_rdr.read(table)
                 _read_trace.append({'kwargs': fast_kwargs,
+                                    'Reader': fast_reader_rdr.__class__,
                                     'status': 'Success with fast reader (no guessing)'})
-            except (core.ParameterError, cparser.CParserError) as e:
+            except (core.ParameterError, cparser.CParserError) as err:
                 # special testing value to avoid falling back on the slow reader
-                if fast_reader_param == 'force':
-                    raise e
+                if fast_reader['enable'] == 'force':
+                    raise core.InconsistentTableError(
+                        'fast reader {} exception: {}'
+                        .format(fast_reader_rdr.__class__, err))
                 # If the fast reader doesn't work, try the slow version
                 dat = reader.read(table)
                 _read_trace.append({'kwargs': new_kwargs,
+                                    'Reader': reader.__class__,
                                     'status': 'Success with slow reader after failing'
                                              ' with fast (no guessing)'})
         else:
             dat = reader.read(table)
             _read_trace.append({'kwargs': new_kwargs,
+                                'Reader': reader.__class__,
                                 'status': 'Success with specified Reader class '
                                           '(no guessing)'})
 
@@ -372,19 +407,8 @@ def _guess(table, read_kwargs, format, fast_reader):
         Keyword arguments from user to be supplied to reader
     format : str
         Table format
-    fast_reader : bool
-    fast_reader : bool or dict
-        Whether to use the C engine, can also be a dict with options which
-        defaults to `False`; parameters for options dict:
-
-        use_fast_converter: bool
-            enable faster but slightly imprecise floating point conversion method
-        parallel: bool or int
-            multiprocessing conversion using ``cpu_count()`` or ``'number'`` processes
-        exponent_style: str
-            Character to use for exponent or ``'Fortran'`` to auto-detect any
-            Fortran-style scientific notation like ``'3.14159D+00'`` (``'E'``, ``'D'``, ``'Q'``),
-            all case-insensitive; default ``'E'``, all other imply ``use_fast_converter``
+    fast_reader : dict
+        Options for the C engine fast reader.  See read() function for details.
 
     Returns
     -------
@@ -400,7 +424,7 @@ def _guess(table, read_kwargs, format, fast_reader):
     full_list_guess = _get_guess_kwargs_list(read_kwargs)
 
     # If a fast version of the reader is available, try that before the slow version
-    if fast_reader and format is not None and 'fast_{0}'.format(format) in \
+    if fast_reader['enable'] and format is not None and 'fast_{0}'.format(format) in \
                                                          core.FAST_CLASSES:
         fast_kwargs = read_kwargs.copy()
         fast_kwargs['Reader'] = core.FAST_CLASSES['fast_{0}'.format(format)]
@@ -415,11 +439,13 @@ def _guess(table, read_kwargs, format, fast_reader):
 
     for guess_kwargs in full_list_guess:
         # If user specified slow reader then skip all fast readers
-        if fast_reader is False and guess_kwargs['Reader'] in core.FAST_CLASSES.values():
+        if (fast_reader['enable'] is False and
+                guess_kwargs['Reader'] in core.FAST_CLASSES.values()):
             continue
 
-        # If user required a fast reader with 'force' then skip all non-fast readers
-        if fast_reader == 'force' and guess_kwargs['Reader'] not in core.FAST_CLASSES.values():
+        # If user required a fast reader then skip all non-fast readers
+        if (fast_reader['enable'] == 'force' and
+                guess_kwargs['Reader'] not in core.FAST_CLASSES.values()):
             continue
 
         guess_kwargs_ok = True  # guess_kwargs are consistent with user_kwargs?
@@ -469,7 +495,9 @@ def _guess(table, read_kwargs, format, fast_reader):
             reader = get_reader(**guess_kwargs)
             reader.guessing = True
             dat = reader.read(table)
-            _read_trace.append({'kwargs': guess_kwargs, 'status': 'Success (guessing)',
+            _read_trace.append({'kwargs': guess_kwargs,
+                                'Reader': reader.__class__,
+                                'status': 'Success (guessing)',
                                 'dt': '{0:.3f} ms'.format((time.time() - t0) * 1000)})
             return dat
 
@@ -485,6 +513,7 @@ def _guess(table, read_kwargs, format, fast_reader):
             reader = get_reader(**read_kwargs)
             dat = reader.read(table)
             _read_trace.append({'kwargs': read_kwargs,
+                                'Reader': reader.__class__,
                                 'status': 'Success with original kwargs without strict_names '
                                           '(guessing)'})
             return dat
@@ -593,6 +622,127 @@ def _get_guess_kwargs_list(read_kwargs):
     return guess_kwargs_list
 
 
+def _read_in_chunks(table, **kwargs):
+    """
+    For fast_reader read the ``table`` in chunks and vstack to create
+    a single table, OR return a generator of chunk tables.
+    """
+    fast_reader = kwargs['fast_reader']
+    chunk_size = fast_reader.pop('chunk_size')
+    chunk_generator = fast_reader.pop('chunk_generator', False)
+    fast_reader['parallel'] = False  # No parallel with chunks
+
+    tbl_chunks = _read_in_chunks_generator(table, chunk_size, **kwargs)
+    if chunk_generator:
+        return tbl_chunks
+
+    tbl0 = next(tbl_chunks)
+    masked = tbl0.masked
+
+    # Numpy won't allow resizing the original so make a copy here.
+    out_cols = {col.name: col.data.copy() for col in tbl0.itercols()}
+
+    str_kinds = ('S', 'U')
+    for tbl in tbl_chunks:
+        masked |= tbl.masked
+        for name, col in tbl.columns.items():
+            # Concatenate current column data and new column data
+
+            # If one of the inputs is string-like and the other is not, then
+            # convert the non-string to a string.  In a perfect world this would
+            # be handled by numpy, but as of numpy 1.13 this results in a string
+            # dtype that is too long (https://github.com/numpy/numpy/issues/10062).
+
+            col1, col2 = out_cols[name], col.data
+            if col1.dtype.kind in str_kinds and col2.dtype.kind not in str_kinds:
+                col2 = np.array(col2.tolist(), dtype=col1.dtype.kind)
+            elif col2.dtype.kind in str_kinds and col1.dtype.kind not in str_kinds:
+                col1 = np.array(col1.tolist(), dtype=col2.dtype.kind)
+
+            # Choose either masked or normal concatenation
+            concatenate = np.ma.concatenate if masked else np.concatenate
+
+            out_cols[name] = concatenate([col1, col2])
+
+    # Make final table from numpy arrays, converting dict to list
+    out_cols = [out_cols[name] for name in tbl0.colnames]
+    out = tbl0.__class__(out_cols, names=tbl0.colnames, meta=tbl0.meta,
+                         copy=False)
+
+    return out
+
+
+def _read_in_chunks_generator(table, chunk_size, **kwargs):
+    """
+    For fast_reader read the ``table`` in chunks and return a generator
+    of tables for each chunk.
+    """
+
+    @contextlib.contextmanager
+    def passthrough_fileobj(fileobj, encoding=None):
+        """Stub for get_readable_fileobj, which does not seem to work in Py3
+        for input File-like object, see #6460"""
+        yield fileobj
+
+    # Set up to coerce `table` input into a readable file object by selecting
+    # an appropriate function.
+
+    # Convert table-as-string to a File object.  Finding a newline implies
+    # that the string is not a filename.
+    if (isinstance(table, str) and ('\n' in table or '\r' in table)):
+        table = StringIO(table)
+        fileobj_context = passthrough_fileobj
+    elif hasattr(table, 'read') and hasattr(table, 'seek'):
+        fileobj_context = passthrough_fileobj
+    else:
+        # string filename or pathlib
+        fileobj_context = get_readable_fileobj
+
+    # Set up for iterating over chunks
+    kwargs['fast_reader']['return_header_chars'] = True
+    header = ''  # Table header (up to start of data)
+    prev_chunk_chars = ''  # Chars from previous chunk after last newline
+    first_chunk = True  # True for the first chunk, False afterward
+
+    with fileobj_context(table, encoding=kwargs.get('encoding')) as fh:
+
+        while True:
+            chunk = fh.read(chunk_size)
+            # Got fewer chars than requested, must be end of file
+            final_chunk = len(chunk) < chunk_size
+
+            # If this is the last chunk and there is only whitespace then break
+            if final_chunk and not re.search(r'\S', chunk):
+                break
+
+            # Step backwards from last character in chunk and find first newline
+            for idx in range(len(chunk) - 1, -1, -1):
+                if final_chunk or chunk[idx] == '\n':
+                    break
+            else:
+                raise ValueError('no newline found in chunk (chunk_size too small?)')
+
+            # Stick on the header to the chunk part up to (and including) the
+            # last newline.  Make sure the small strings are concatenated first.
+            complete_chunk = (header + prev_chunk_chars) + chunk[:idx + 1]
+            prev_chunk_chars = chunk[idx + 1:]
+
+            # Now read the chunk as a complete table
+            tbl = read(complete_chunk, guess=False, **kwargs)
+
+            # For the first chunk pop the meta key which contains the header
+            # characters (everything up to the start of data) then fix kwargs
+            # so it doesn't return that in meta any more.
+            if first_chunk:
+                header = tbl.meta.pop('__ascii_fast_reader_header_chars__')
+                first_chunk = False
+
+            yield tbl
+
+            if final_chunk:
+                break
+
+
 extra_writer_pars = ('delimiter', 'comment', 'quotechar', 'formats',
                      'names', 'include_names', 'exclude_names', 'strip_whitespace')
 
diff --git a/docs/io/ascii/fast_ascii_io.rst b/docs/io/ascii/fast_ascii_io.rst
index 9454fe2f82..a0e78f7b50 100644
--- a/docs/io/ascii/fast_ascii_io.rst
+++ b/docs/io/ascii/fast_ascii_io.rst
@@ -67,12 +67,13 @@ These parameters are:
 .. _fast_conversion_opts:
 
 Parallel and fast conversion options
-====================================
+------------------------------------
 In addition to ``True`` and ``False``, the parameter ``fast_reader`` can also
 be a dict specifying any of three additional parameters, ``parallel``,
 ``use_fast_converter`` and ``exponent_style``. For example::
 
-   >>> ascii.read('data.txt', format='basic', fast_reader={'parallel': True, 'use_fast_converter': True}) # doctest: +SKIP
+   >>> ascii.read('data.txt', format='basic',
+   ...            fast_reader={'parallel': True, 'use_fast_converter': True}) # doctest: +SKIP
 
 These options allow for even faster table reading when enabled, but both are
 disabled by default because they come with some caveats.
@@ -92,21 +93,8 @@ The special setting ``'fortran'`` enables auto-detection of any valid
 exponent character under Fortran notation.
 For details see the section on :ref:`fortran_style_exponents`.
 
-Writing
-=======
-The fast engine supports the same functionality as the ordinary writing engine
-and is generally about 2 to 4 times faster than the ordinary engine. An IPython
-notebook testing the relative performance of the fast writer against the
-ordinary writing system and the data analysis library `Pandas
-<http://pandas.pydata.org/>`__ is available `here <http://nbviewer.jupyter.org/github/astropy/astropy-notebooks/blob/master/io/ascii/ascii_write_bench.ipynb>`__.
-The speed advantage of the faster engine is greatest for integer data and least
-for floating-point data; the fast engine is around 3.6 times faster for a
-sample file including a mixture of floating-point, integer, and text data.
-Also note that stripping string values slows down the writing process, so
-specifying ``strip_whitespace=False`` can improve performance.
-
 Fast converter
-==============
+--------------
 Input floating-point values should ideally be converted to the
 nearest possible floating-point approximation; that is, the conversion
 should be correct within half of the distance between the two closest
@@ -132,6 +120,24 @@ the fast converter's behavior with extreme values (such as subnormals
 and values out of the range of floats) is available `here
 <http://nbviewer.jupyter.org/github/astropy/astropy-notebooks/blob/master/io/ascii/test_converter.ipynb>`__.
 
+Reading large tables
+--------------------
+For reading very large tables using the fast reader see the section on
+:ref:`chunk_reading`.
+
+Writing
+=======
+The fast engine supports the same functionality as the ordinary writing engine
+and is generally about 2 to 4 times faster than the ordinary engine. An IPython
+notebook testing the relative performance of the fast writer against the
+ordinary writing system and the data analysis library `Pandas
+<http://pandas.pydata.org/>`__ is available `here <http://nbviewer.ipython.org/github/astropy/astropy-notebooks/blob/master/io/ascii/ascii_write_bench.ipynb>`__.
+The speed advantage of the faster engine is greatest for integer data and least
+for floating-point data; the fast engine is around 3.6 times faster for a
+sample file including a mixture of floating-point, integer, and text data.
+Also note that stripping string values slows down the writing process, so
+specifying ``strip_whitespace=False`` can improve performance.
+
 Speed gains
 ===========
 The fast ASCII engine was designed based on the general parsing strategy
diff --git a/docs/io/ascii/index.rst b/docs/io/ascii/index.rst
index 5beca88e86..5209a4a856 100644
--- a/docs/io/ascii/index.rst
+++ b/docs/io/ascii/index.rst
@@ -96,6 +96,8 @@ disable the fast engine::
 
    >>> data = ascii.read(lines, format='csv', fast_reader=False)
 
+For reading very large tables see the section on :ref:`chunk_reading`.
+
 .. Note::
 
    Reading a table which contains unicode characters is supported; if you need
diff --git a/docs/io/ascii/read.rst b/docs/io/ascii/read.rst
index e19268672b..e0e64a0d0a 100644
--- a/docs/io/ascii/read.rst
+++ b/docs/io/ascii/read.rst
@@ -564,3 +564,78 @@ in a function::
    # Create an RDB reader and override the splitter.process_val function
    rdb_reader = astropy.io.ascii.get_reader(Reader=astropy.io.ascii.Rdb)
    rdb_reader.data.splitter.process_val = process_val
+
+.. _chunk_reading:
+
+Reading large tables in chunks
+==============================
+
+The default process for reading ASCII tables is not memory efficient and may
+temporarily require much more memory than the size of the file (up to a factor
+of 5 to 10).  In cases where the temporary memory requirement exceeds available
+memory this can cause significant slowdown when disk cache gets used.
+
+In this situation there is a way to read the table in smaller chunks which are
+limited in size.  There are two possible ways to do this:
+
+- Read the table in chunks and aggregate the final table along the way.  This
+  uses only somewhat more memory than the final table requires.
+- Use a Python generator function to return a `~astropy.table.Table` object for
+  each chunk of the input table.  This allows for scanning through arbitrarily
+  large tables since it never returns the final aggregate table.
+
+The chunk reading functionality is most useful for very large tables, so this is
+available only for the :ref:`fast_ascii_io` readers.  The following formats are
+supported: ``tab``, ``csv``, ``no_header``, ``rdb``, and ``basic``.  The
+``commented_header`` format is not directly supported, but as a workaround one
+can read using the ``no_header`` format and explicitly supply the column names
+using the ``names`` argument.
+
+In order to read a table in chunks one must provide the ``fast_reader`` keyword
+argument with a ``dict`` that includes the ``chunk_size`` key with the value
+being the approximate size (in bytes) of each chunk of the input table to read.
+In addition, if one provides a ``chunk_generator`` key which is set to
+``True``, then instead of returning a single table for the whole input it
+returns an iterator that provides a table for each chunk of the input.
+
+**Example**: Reading an entire table while limiting peak memory usage.
+::
+
+  # Read a large CSV table in 100 Mb chunks.
+
+  tbl = ascii.read('large_table.csv', format='csv', guess=False,
+                   fast_reader={'chunk_size': 100 * 1000000})
+
+**Example**: Reading the table in chunks with an iterator.
+
+Here we iterate over a CSV table and select all rows where the ``Vmag`` column is
+less than 8.0 (e.g. all stars in table brighter than 8.0 mag).  We collect all
+these sub-tables and then stack them at the end.
+::
+
+  from astropy.table import vstack
+
+  # tbls is an iterator over the chunks (no actual reading done yet)
+  tbls = ascii.read('large_table.csv', format='csv', guess=False,
+                    fast_reader={'chunk_size': 100 * 1000000,
+                                 'chunk_generator': True})
+
+  out_tbls = []
+
+  # At this point the file is actually read in chunks.
+  for tbl in tbls:
+      bright = tbl['Vmag'] < 8.0
+      if np.count_nonzero(bright):
+          out_tbls.append(tbl[bright])
+
+  out_tbl = vstack(out_tbls)
+
+.. Note:: **Performance**
+
+  Specifying the ``format`` explicitly and using ``guess=False`` is a good idea
+  for large tables.  This prevent unneccesary guessing in the typical case
+  where the format is already known.
+
+  The ``chunk_size`` should generally be set to the largest value that is 
+  reasonable given available system memory.  There is overhead associated
+  with processing each chunk, so the fewer chunks the better.


Test patch:
diff --git a/astropy/io/ascii/tests/test_read.py b/astropy/io/ascii/tests/test_read.py
index eb0b17dfbc..19b1176082 100644
--- a/astropy/io/ascii/tests/test_read.py
+++ b/astropy/io/ascii/tests/test_read.py
@@ -15,12 +15,14 @@ import numpy as np
 
 from ... import ascii
 from ....table import Table
+from .... import table
 from ....units import Unit
+from ....table.table_helpers import simple_table
 
 from .common import (raises, assert_equal, assert_almost_equal,
                      assert_true)
 from .. import core
-from ..ui import _probably_html, get_read_trace
+from ..ui import _probably_html, get_read_trace, cparser
 
 # setup/teardown function to have the tests run in the correct directory
 from .common import setup_function, teardown_function
@@ -119,7 +121,7 @@ def test_read_with_names_arg(fast_reader):
     Test that a bad value of `names` raises an exception.
     """
     with pytest.raises(ValueError):
-        dat = ascii.read(['c d', 'e f'], names=('a', ), guess=False, fast_reader=fast_reader)
+        ascii.read(['c d', 'e f'], names=('a', ), guess=False, fast_reader=fast_reader)
 
 
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
@@ -240,31 +242,31 @@ def test_daophot_multiple_aperture2():
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
 def test_empty_table_no_header(fast_reader):
     with pytest.raises(ascii.InconsistentTableError):
-        table = ascii.read('t/no_data_without_header.dat', Reader=ascii.NoHeader,
-                            guess=False, fast_reader=fast_reader)
+        ascii.read('t/no_data_without_header.dat', Reader=ascii.NoHeader,
+                   guess=False, fast_reader=fast_reader)
 
 
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
 def test_wrong_quote(fast_reader):
     with pytest.raises(ascii.InconsistentTableError):
-        table = ascii.read('t/simple.txt', guess=False, fast_reader=fast_reader)
+        ascii.read('t/simple.txt', guess=False, fast_reader=fast_reader)
 
 
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
 def test_extra_data_col(fast_reader):
     with pytest.raises(ascii.InconsistentTableError):
-        table = ascii.read('t/bad.txt', fast_reader=fast_reader)
+        ascii.read('t/bad.txt', fast_reader=fast_reader)
 
 
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
 def test_extra_data_col2(fast_reader):
     with pytest.raises(ascii.InconsistentTableError):
-        table = ascii.read('t/simple5.txt', delimiter='|', fast_reader=fast_reader)
+        ascii.read('t/simple5.txt', delimiter='|', fast_reader=fast_reader)
 
 
 @raises(IOError)
 def test_missing_file():
-    table = ascii.read('does_not_exist')
+    ascii.read('does_not_exist')
 
 
 @pytest.mark.parametrize('fast_reader', [True, False, 'force'])
@@ -1293,3 +1295,86 @@ def test_unsupported_read_with_encoding(tmpdir):
     with pytest.raises(ascii.ParameterError):
         ascii.read('t/simple3.txt', guess=False, fast_reader='force',
                    encoding='latin1', format='fast_csv')
+
+
+def test_read_chunks_input_types():
+    """
+    Test chunked reading for different input types: file path, file object,
+    and string input.
+    """
+    fpath = 't/test5.dat'
+    t1 = ascii.read(fpath, header_start=1, data_start=3, )
+
+    for fp in (fpath, open(fpath, 'r'), open(fpath, 'r').read()):
+        t_gen = ascii.read(fp, header_start=1, data_start=3,
+                           guess=False, format='fast_basic',
+                           fast_reader={'chunk_size': 400, 'chunk_generator': True})
+        ts = list(t_gen)
+        for t in ts:
+            for col, col1 in zip(t.columns.values(), t1.columns.values()):
+                assert col.name == col1.name
+                assert col.dtype.kind == col1.dtype.kind
+
+        assert len(ts) == 4
+        t2 = table.vstack(ts)
+        assert np.all(t1 == t2)
+
+    for fp in (fpath, open(fpath, 'r'), open(fpath, 'r').read()):
+        # Now read the full table in chunks
+        t3 = ascii.read(fp, header_start=1, data_start=3,
+                        fast_reader={'chunk_size': 300})
+        assert np.all(t1 == t3)
+
+
+@pytest.mark.parametrize('masked', [True, False])
+def test_read_chunks_formats(masked):
+    """
+    Test different supported formats for chunked reading.
+    """
+    t1 = simple_table(size=102, cols=10, kinds='fS', masked=masked)
+    for i, name in enumerate(t1.colnames):
+        t1.rename_column(name, 'col{}'.format(i + 1))
+
+    # TO DO commented_header does not currently work due to the special-cased
+    # implementation of header parsing.
+
+    for format in 'tab', 'csv', 'no_header', 'rdb', 'basic':
+        out = StringIO()
+        ascii.write(t1, out, format=format)
+        t_gen = ascii.read(out.getvalue(), format=format,
+                           fast_reader={'chunk_size': 400, 'chunk_generator': True})
+        ts = list(t_gen)
+        for t in ts:
+            for col, col1 in zip(t.columns.values(), t1.columns.values()):
+                assert col.name == col1.name
+                assert col.dtype.kind == col1.dtype.kind
+
+        assert len(ts) > 4
+        t2 = table.vstack(ts)
+        assert np.all(t1 == t2)
+
+        # Now read the full table in chunks
+        t3 = ascii.read(out.getvalue(), format=format, fast_reader={'chunk_size': 400})
+        assert np.all(t1 == t3)
+
+
+def test_read_chunks_chunk_size_too_small():
+    fpath = 't/test5.dat'
+    with pytest.raises(ValueError) as err:
+        ascii.read(fpath, header_start=1, data_start=3,
+                   fast_reader={'chunk_size': 10})
+    assert 'no newline found in chunk (chunk_size too small?)' in str(err)
+
+
+def test_read_chunks_table_changes():
+    """Column changes type or size between chunks.  This also tests the case with
+    no final newline.
+    """
+    col = ['a b c'] + ['1.12334 xyz a'] * 50 + ['abcdefg 555 abc'] * 50
+    table = '\n'.join(col)
+    t1 = ascii.read(table, guess=False)
+    t2 = ascii.read(table, fast_reader={'chunk_size': 100})
+
+    # This also confirms that the dtypes are exactly the same, i.e.
+    # the string itemsizes are the same.
+    assert np.all(t1 == t2)