BUG: Some sas7bdat files with many columns are not parseable by read_sas
State: closed

- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

The reason is that column definitions may be split up into different pages.
Allow column information to be parsed from different pages
and add a test for it.


Labels:
name: Bug
name: IO SAS
description: SAS: read_sas

Comments:
Hello @troels! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/io/sas/sas7bdat.py`](https://github.com/troels/pandas/blob/8f498d500d6ccb1584c8c2290e0fd88b87e5c956/pandas/io/sas/sas7bdat.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/sas/test_sas7bdat.py`](https://github.com/troels/pandas/blob/8f498d500d6ccb1584c8c2290e0fd88b87e5c956/pandas/tests/io/sas/test_sas7bdat.py) !



##### Comment last updated on September 07, 2018 at 19:51 Hours UTC
@troels : Thanks for the report!  Do you mind sharing in this PR what output you get without your patch for reference?  A small, reproducible example would be great.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=h1) Report
> Merging [#22628](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/73dd6ece3b66008eb8077098c101d0e300b1e714?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22628/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22628      +/-   ##
==========================================
+ Coverage   92.17%   92.17%   +<.01%     
==========================================
  Files         169      169              
  Lines       50708    50712       +4     
==========================================
+ Hits        46740    46745       +5     
+ Misses       3968     3967       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.58% <100%> (ø)` | :arrow_up: |
| #single | `42.35% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22628/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.16% <100%> (+0.07%)` | :arrow_up: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/22628/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `96.55% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=footer). Last update [73dd6ec...c459f6e](https://codecov.io/gh/pandas-dev/pandas/pull/22628?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hi @gfyoung 

So the test case I added will fail in the current version of pandas, with the following error (these files can't be parsed):

```
pandas/tests/io/sas/test_sas7bdat.py ............F.                                              [100%]

=============================================== FAILURES ===============================================
__________________________________________ test_many_columns ___________________________________________

datapath = <function datapath.<locals>.deco at 0x7f53f5a47f28>

    def test_many_columns(datapath):
        fname = datapath("io", "sas", "data", "many_columns.sas7bdat")
>       df = pd.read_sas(fname, encoding='latin-1')

pandas/tests/io/sas/test_sas7bdat.py:188: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/io/sas/sasreader.py:68: in read_sas
    data = reader.read()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pandas.io.sas.sas7bdat.SAS7BDATReader object at 0x7f53f5899860>, nrows = 3

    def read(self, nrows=None):
    
        if (nrows is None) and (self.chunksize is not None):
            nrows = self.chunksize
        elif nrows is None:
            nrows = self.row_count
    
        if len(self.column_types) == 0:
            self.close()
>           raise EmptyDataError("No columns to parse from file")
E           pandas.errors.EmptyDataError: No columns to parse from file

pandas/io/sas/sas7bdat.py:604: EmptyDataError
================================= 1 failed, 13 passed in 18.88 seconds =================================
```
Hi @gfyoung 

I added a commit fixing #16615 too. I hope it's ok to continue using the same PR, as the two pieces of code build upon each other.
> I added a commit fixing #16615 too. I hope it's ok to continue using the same PR, as the two pieces of code build upon each other.

@troels : In this case, it should be fine.  Thanks for letting me know!
The test failure here is quite unrelated to the PR.
Hi @jreback 

Anything else preventing this from being merged?
thanks!
Thanks both of you :)