Repository: pandas-dev/pandas

Issue:
Auto-detect field widths in read_fwf when unspecified
Here's a quick hack for generating the `widths` arg. 
It's basic, but it works for me.

```
def detect_fwf_widths(rows,wsratio=0.5,nlines=100):
    from collections import Counter # 2.6, is this in pd.compat?

    # could do a p-value thing instead, but this works fine in practice
    nlines = min(len(rows),nlines) 

    # columwise, get occurence counts, use threshold to prune candidates
    # for "blank".
    counts = [{k:v for k,v in d.iteritems()  if v > nlines*wsratio} 
              for d in map(Counter,zip(*rows))]

    # among candidates, find the most common and choose it as "blank"
    # then extract only the cols where "blank" appears "frequently".
    c=Counter()
    for d in counts: 
        c.update(d)
    delim=c.most_common(1)[0][0]
    cols = [i for i,d in enumerate(counts) if delim in d]

    # filter the col indexes to keep only the rightmost index of
    # each contiguous sequence
    width_list =[]
    lidx=0
    prev = cols[0]
    for v in cols[1:]:
        if v-prev != 1:
            width_list.append(prev+1-lidx)
            lidx=prev+1
        prev = v
    width_list.append(len(rows[0])-lidx)

    return width_list
```

any takers to flesh this out into a PR with tests?



Discussion:
I could take this, if that's OK? I just need a few pointers:
1. Should this be used in the `read_fwf` function before passing the `filepath_or_buffer` to the `_read` or in the `FixedWidthReader`'s `__init__` function?
2. I assume that this should read the first `nlines` and then try to determine the widths before parsing. But what about the non-seekable buffers? If the buffer passed in is just a generator and doesn't support seeking back to the beginning? I assume that `FixedWidthReader` is a better place for this since it can buffer the lines it used for determining the widths and then just yield them in it's `next`.

your point 2 is correct, here is a fix for that (and maybe you should just do this in all cases)

don't start with the buffer/path itself, but wrap it in a StringIO like class that can handle this type of bufferring,
e.g. its essentially a pass-thru if you are not talking to a generator, otherwise its a seekable buffer

kind of like the Reader class you created in that generator question on SO; except it also would have a reset method (prob a more consistent name that is compat with StringIO though), that seeks to the beginning and buffers what it already read first to you before resuming getting from the generator

make sense?

@jreback Yes. I would just avoid buffering the whole data, just the last `nlines` max. Or even completely skip it if no detection is needed.

Also, is adding `nlines=100` parameter to the `read_fwf` OK?

is that for the 'checking' part? if so yes,....but I would think you only need even a few lines (e.g. no header, and use didn't use skiprows, or if they did you skipped them), and I think you have to assume that the format is the same, yes? it IS fixed width (but then I guess you have to 'validate' it)...maybe raise if you find discrepancies?

Yes just for inferring the width of columns. There are a lot of things that should be taken into account, as you mentioned (`lineterminator`, `skiprows`, `comment` and `nrows` if I'm correct). I'll try to cover all of them in test cases.

@alefnula I realize you are right, you prob do need more lines in order to 'figure out' where the proper terminations are....because the fields can be ragged within the 'unknown' widths

Think I found a simpler solution based on the minimal starting point of the column: [detect_fwf_widths](https://github.com/alefnula/pandas/blob/iss4488/pandas/io/parsers.py#L1977). I just have to work out the edge cases and implement the buffering.

@alefnula gr8....prob will need a number of tests for this....what API are you proposing for this?

maybe `widths='detect'`? (and make it the default?)

@jreback I added a `detect_from_rows` parameter. 

```
detect_from_rows : number or list of rows that will be inspected in
    order to infer the column widths, if neither the 'colspecs' nor
    'widths' are specified (default=100).
```

I wanted to enable user to say either:
1. Detect from first 20 rows: `detect_from_rows=20`
2. Detect from rows 0, 4, 7 and 8: `detect_from_rows=[0, 4, 7, 8]`

If this just complicates things I can use just a simple `widths='detect'` and default to `100` rows. That would be even simpler to implement.

Also do you have an idea why did I get this error from Travis for every python interpreter except 2.7?!

```
IOError: [Errno 2] No such file or directory:
'/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas-0.0.0-py2.6-linux-x86_64.egg/pandas/io/tests/data/test1.fwf'
```

The file is in the repository.

You need to edit `setup.py` to make it copy that extension to the correct place

e.g. `tests/data/*.fwf`

```
# The build cache system does string matching below this point.
# if you change something, be careful.

setup(name=DISTNAME,
      version=FULLVERSION,
      maintainer=AUTHOR,
      packages=['pandas',
                'pandas.compat',
                'pandas.computation',
                'pandas.computation.tests',
                'pandas.core',
                'pandas.io',
                'pandas.rpy',
                'pandas.sandbox',
                'pandas.sparse',
                'pandas.sparse.tests',
                'pandas.stats',
                'pandas.util',
                'pandas.tests',
                'pandas.tools',
                'pandas.tools.tests',
                'pandas.tseries',
                'pandas.tseries.tests',
                'pandas.io.tests',
                'pandas.io.tests.test_json',
                'pandas.stats.tests',
                ],
      package_data={'pandas.io': ['tests/data/legacy_hdf/*.h5',
                                  'tests/data/legacy_pickle/0.10.1/*.pickle',
                                  'tests/data/legacy_pickle/0.11.0/*.pickle',
                                  'tests/data/legacy_pickle/0.12.0/*.pickle',
                                  'tests/data/*.csv',
                                  'tests/data/*.dta',
                                  'tests/data/*.txt',
                                  'tests/data/*.xls',
                                  'tests/data/*.xlsx',
                                  'tests/data/*.table',
                                  'tests/data/*.html',
                                  'tests/test_json/data/*.json'],
                    'pandas.tools': ['tests/*.csv'],
                    'pandas.tests': ['data/*.pickle',
                                     'data/*.csv'],
                    'pandas.tseries.tests': ['data/*.pickle',
                                             'data/*.csv']
                    },
```

Ok i think I found an optimal algorithm that uses a bitmask to detect the gaps between columns. [detect_fwf_widths](https://github.com/alefnula/pandas/blob/4c4edd047f2f7bf77f03d0db88b3a6d4518fbf0a/pandas/io/parsers.py#L1997).

Implemented a simple buffering (I think it couldn't be simpler, but if you want me to change it I'll do whatever you want) [get_rows](https://github.com/alefnula/pandas/blob/4c4edd047f2f7bf77f03d0db88b3a6d4518fbf0a/pandas/io/parsers.py#L1977) and [next](https://github.com/alefnula/pandas/blob/4c4edd047f2f7bf77f03d0db88b3a6d4518fbf0a/pandas/io/parsers.py#L2010). And also the tests are [here](https://github.com/alefnula/pandas/blob/iss4488/pandas/io/tests/test_parsers.py#L2134). Some of the data set's are pretty messy but it can detect the columns correctly: [example](https://github.com/alefnula/pandas/blob/iss4488/pandas/io/tests/data/test5.fwf).

If you have more ideas about test files please post them. Also if the naming conventions are OK I'll change the documentation. If they are not, just tell me what to change and I'll do it.

Just to check: would it still work if you had 1000 columns? 10K? or even
just a few columns, but with very long names.

@alefnula, can you open a PR to discuss? Easier to see diffs there. (and you can always change it up by pushing / editing commits. 


Resolving patch:
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -160,11 +160,15 @@
 """ % (_parser_params % _table_sep)
 
 _fwf_widths = """\
-colspecs : a list of pairs (tuples), giving the extents
-    of the fixed-width fields of each line as half-open internals
-    (i.e.,  [from, to[  ).
-widths : a list of field widths, which can be used instead of
-    'colspecs' if the intervals are contiguous.
+colspecs : list of pairs (int, int) or 'infer'. optional
+    A list of pairs (tuples) giving the extents of the fixed-width
+    fields of each line as half-open intervals (i.e.,  [from, to[ ).
+    String value 'infer' can be used to instruct the parser to try
+    detecting the column specifications from the first 100 rows of
+    the data (default='infer').
+widths : list of ints. optional
+    A list of field widths which can be used instead of 'colspecs' if
+    the intervals are contiguous.
 """
 
 _read_fwf_doc = """
@@ -184,7 +188,8 @@ def _read(filepath_or_buffer, kwds):
     if skipfooter is not None:
         kwds['skip_footer'] = skipfooter
 
-    filepath_or_buffer, _ = get_filepath_or_buffer(filepath_or_buffer)
+    filepath_or_buffer, _ = get_filepath_or_buffer(filepath_or_buffer,
+                                                   encoding)
 
     if kwds.get('date_parser', None) is not None:
         if isinstance(kwds['parse_dates'], bool):
@@ -267,8 +272,8 @@ def _read(filepath_or_buffer, kwds):
 }
 
 _fwf_defaults = {
-    'colspecs': None,
-    'widths': None
+    'colspecs': 'infer',
+    'widths': None,
 }
 
 _c_unsupported = set(['skip_footer'])
@@ -412,15 +417,15 @@ def parser_f(filepath_or_buffer,
 
 
 @Appender(_read_fwf_doc)
-def read_fwf(filepath_or_buffer, colspecs=None, widths=None, **kwds):
+def read_fwf(filepath_or_buffer, colspecs='infer', widths=None, **kwds):
     # Check input arguments.
     if colspecs is None and widths is None:
         raise ValueError("Must specify either colspecs or widths")
-    elif colspecs is not None and widths is not None:
+    elif colspecs not in (None, 'infer') and widths is not None:
         raise ValueError("You must specify only one of 'widths' and "
                          "'colspecs'")
 
-    # Compute 'colspec' from 'widths', if specified.
+    # Compute 'colspecs' from 'widths', if specified.
     if widths is not None:
         colspecs, col = [], 0
         for w in widths:
@@ -519,7 +524,8 @@ def _clean_options(self, options, engine):
                 engine = 'python'
         elif sep is not None and len(sep) > 1:
             # wait until regex engine integrated
-            engine = 'python'
+            if engine not in ('python', 'python-fwf'):
+                engine = 'python'
 
         # C engine not supported yet
         if engine == 'c':
@@ -2012,31 +2018,65 @@ class FixedWidthReader(object):
     """
     A reader of fixed-width lines.
     """
-    def __init__(self, f, colspecs, filler, thousands=None, encoding=None):
+    def __init__(self, f, colspecs, delimiter, comment):
         self.f = f
-        self.colspecs = colspecs
-        self.filler = filler  # Empty characters between fields.
-        self.thousands = thousands
-        if encoding is None:
-            encoding = get_option('display.encoding')
-        self.encoding = encoding
-
-        if not isinstance(colspecs, (tuple, list)):
+        self.buffer = None
+        self.delimiter = '\r\n' + delimiter if delimiter else '\n\r\t '
+        self.comment = comment
+        if colspecs == 'infer':
+            self.colspecs = self.detect_colspecs()
+        else:
+            self.colspecs = colspecs
+
+        if not isinstance(self.colspecs, (tuple, list)):
             raise TypeError("column specifications must be a list or tuple, "
                             "input was a %r" % type(colspecs).__name__)
 
-        for colspec in colspecs:
+        for colspec in self.colspecs:
             if not (isinstance(colspec, (tuple, list)) and
-                    len(colspec) == 2 and
-                    isinstance(colspec[0], int) and
-                    isinstance(colspec[1], int)):
+                       len(colspec) == 2 and
+                       isinstance(colspec[0], (int, np.integer)) and
+                       isinstance(colspec[1], (int, np.integer))):
                 raise TypeError('Each column specification must be '
                                 '2 element tuple or list of integers')
 
+    def get_rows(self, n):
+        rows = []
+        for i, row in enumerate(self.f, 1):
+            rows.append(row)
+            if i >= n:
+                break
+        self.buffer = iter(rows)
+        return rows
+
+    def detect_colspecs(self, n=100):
+        # Regex escape the delimiters
+        delimiters = ''.join([r'\%s' % x for x in self.delimiter])
+        pattern = re.compile('([^%s]+)' % delimiters)
+        rows = self.get_rows(n)
+        max_len = max(map(len, rows))
+        mask = np.zeros(max_len + 1, dtype=int)
+        if self.comment is not None:
+            rows = [row.partition(self.comment)[0] for row in rows]
+        for row in rows:
+            for m in pattern.finditer(row):
+                mask[m.start():m.end()] = 1
+        shifted = np.roll(mask, 1)
+        shifted[0] = 0
+        edges = np.where((mask ^ shifted) == 1)[0]
+        return list(zip(edges[::2], edges[1::2]))
+
     def next(self):
-        line = next(self.f)
+        if self.buffer is not None:
+            try:
+                line = next(self.buffer)
+            except StopIteration:
+                self.buffer = None
+                line = next(self.f)
+        else:
+            line = next(self.f)
         # Note: 'colspecs' is a sequence of half-open intervals.
-        return [line[fromm:to].strip(self.filler or ' ')
+        return [line[fromm:to].strip(self.delimiter)
                 for (fromm, to) in self.colspecs]
 
     # Iterator protocol in Python 3 uses __next__()
@@ -2050,10 +2090,10 @@ class FixedWidthFieldParser(PythonParser):
     """
     def __init__(self, f, **kwds):
         # Support iterators, convert to a list.
-        self.colspecs = list(kwds.pop('colspecs'))
+        self.colspecs = kwds.pop('colspecs')
 
         PythonParser.__init__(self, f, **kwds)
 
     def _make_reader(self, f):
         self.data = FixedWidthReader(f, self.colspecs, self.delimiter,
-                                     encoding=self.encoding)
+                                     self.comment)


Test patch:
diff --git a/pandas/io/tests/test_parsers.py b/pandas/io/tests/test_parsers.py
--- a/pandas/io/tests/test_parsers.py
+++ b/pandas/io/tests/test_parsers.py
@@ -1706,7 +1706,7 @@ def test_utf16_example(self):
             self.assertEquals(len(result), 50)
 
     def test_converters_corner_with_nas(self):
-      # skip aberration observed on Win64 Python 3.2.2
+        # skip aberration observed on Win64 Python 3.2.2
         if hash(np.int64(-1)) != -2:
             raise nose.SkipTest("skipping because of windows hash on Python"
                                 " 3.2.2")
@@ -2078,19 +2078,19 @@ def test_fwf(self):
             read_fwf(StringIO(data3), colspecs=colspecs, widths=[6, 10, 10, 7])
 
         with tm.assertRaisesRegexp(ValueError, "Must specify either"):
-            read_fwf(StringIO(data3))
+            read_fwf(StringIO(data3), colspecs=None, widths=None)
 
     def test_fwf_colspecs_is_list_or_tuple(self):
         with tm.assertRaisesRegexp(TypeError,
                                    'column specifications must be a list or '
                                    'tuple.+'):
-            fwr = pd.io.parsers.FixedWidthReader(StringIO(self.data1),
-                                                 {'a': 1}, ',')
+            pd.io.parsers.FixedWidthReader(StringIO(self.data1),
+                                           {'a': 1}, ',', '#')
 
     def test_fwf_colspecs_is_list_or_tuple_of_two_element_tuples(self):
         with tm.assertRaisesRegexp(TypeError,
                                    'Each column specification must be.+'):
-            read_fwf(StringIO(self.data1), {'a': 1})
+            read_fwf(StringIO(self.data1), [('a', 1)])
 
     def test_fwf_regression(self):
         # GH 3594
@@ -2223,6 +2223,107 @@ def test_iteration_open_handle(self):
                 expected = Series(['DDD', 'EEE', 'FFF', 'GGG'])
                 tm.assert_series_equal(result, expected)
 
+
+class TestFwfColspaceSniffing(unittest.TestCase):
+    def test_full_file(self):
+        # File with all values
+        test = '''index                             A    B    C
+2000-01-03T00:00:00  0.980268513777    3  foo
+2000-01-04T00:00:00  1.04791624281    -4  bar
+2000-01-05T00:00:00  0.498580885705   73  baz
+2000-01-06T00:00:00  1.12020151869     1  foo
+2000-01-07T00:00:00  0.487094399463    0  bar
+2000-01-10T00:00:00  0.836648671666    2  baz
+2000-01-11T00:00:00  0.157160753327   34  foo'''
+        colspecs = ((0, 19), (21, 35), (38, 40), (42, 45))
+        expected = read_fwf(StringIO(test), colspecs=colspecs)
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test)))
+
+    def test_full_file_with_missing(self):
+        # File with missing values
+        test = '''index                             A    B    C
+2000-01-03T00:00:00  0.980268513777    3  foo
+2000-01-04T00:00:00  1.04791624281    -4  bar
+                     0.498580885705   73  baz
+2000-01-06T00:00:00  1.12020151869     1  foo
+2000-01-07T00:00:00                    0  bar
+2000-01-10T00:00:00  0.836648671666    2  baz
+                                      34'''
+        colspecs = ((0, 19), (21, 35), (38, 40), (42, 45))
+        expected = read_fwf(StringIO(test), colspecs=colspecs)
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test)))
+
+    def test_full_file_with_spaces(self):
+        # File with spaces in columns
+        test = '''
+Account                 Name  Balance     CreditLimit   AccountCreated
+101     Keanu Reeves          9315.45     10000.00           1/17/1998
+312     Gerard Butler         90.00       1000.00             8/6/2003
+868     Jennifer Love Hewitt  0           17000.00           5/25/1985
+761     Jada Pinkett-Smith    49654.87    100000.00          12/5/2006
+317     Bill Murray           789.65      5000.00             2/5/2007
+'''.strip('\r\n')
+        colspecs = ((0, 7), (8, 28), (30, 38), (42, 53), (56, 70))
+        expected = read_fwf(StringIO(test), colspecs=colspecs)
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test)))
+
+    def test_full_file_with_spaces_and_missing(self):
+        # File with spaces and missing values in columsn
+        test = '''
+Account               Name    Balance     CreditLimit   AccountCreated
+101                           10000.00                       1/17/1998
+312     Gerard Butler         90.00       1000.00             8/6/2003
+868                                                          5/25/1985
+761     Jada Pinkett-Smith    49654.87    100000.00          12/5/2006
+317     Bill Murray           789.65
+'''.strip('\r\n')
+        colspecs = ((0, 7), (8, 28), (30, 38), (42, 53), (56, 70))
+        expected = read_fwf(StringIO(test), colspecs=colspecs)
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test)))
+
+    def test_messed_up_data(self):
+        # Completely messed up file
+        test = '''
+   Account          Name             Balance     Credit Limit   Account Created
+       101                           10000.00                       1/17/1998
+       312     Gerard Butler         90.00       1000.00
+
+       761     Jada Pinkett-Smith    49654.87    100000.00          12/5/2006
+  317          Bill Murray           789.65
+'''.strip('\r\n')
+        colspecs = ((2, 10), (15, 33), (37, 45), (49, 61), (64, 79))
+        expected = read_fwf(StringIO(test), colspecs=colspecs)
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test)))
+
+    def test_multiple_delimiters(self):
+        test = r'''
+col1~~~~~col2  col3++++++++++++++++++col4
+~~22.....11.0+++foo~~~~~~~~~~Keanu Reeves
+  33+++122.33\\\bar.........Gerard Butler
+++44~~~~12.01   baz~~Jennifer Love Hewitt
+~~55       11+++foo++++Jada Pinkett-Smith
+..66++++++.03~~~bar           Bill Murray
+'''.strip('\r\n')
+        colspecs = ((0, 4), (7, 13), (15, 19), (21, 41))
+        expected = read_fwf(StringIO(test), colspecs=colspecs,
+                            delimiter=' +~.\\')
+        tm.assert_frame_equal(expected, read_fwf(StringIO(test),
+                                                 delimiter=' +~.\\'))
+
+    def test_variable_width_unicode(self):
+        if not compat.PY3:
+            raise nose.SkipTest('Bytes-related test - only needs to work on Python 3')
+        test = '''
+שלום שלום
+ום   שלל
+של   ום
+'''.strip('\r\n')
+        expected = pd.read_fwf(BytesIO(test.encode('utf8')),
+                               colspecs=[(0, 4), (5, 9)], header=None)
+        tm.assert_frame_equal(expected, read_fwf(BytesIO(test.encode('utf8')),
+                                                 header=None))
+
+
 class TestCParserHighMemory(ParserTests, unittest.TestCase):
 
     def read_csv(self, *args, **kwds):