SAS xport file reader
State: closed

Here is an initial attempt to port Jack Cushman's xport file reader into Pandas.

Partially addresses #4052 (but this only deals with xport files, not sas7bdat files)


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO SAS
description: SAS: read_sas

Comments:
@kshedden looking pretty good!

a couple of suggestions here:
- make this `read_sas(......, format='xport')`
- can you make a `to_sas(....format='xport')`? for easy testing
- what happens to the dtypes? are they all marked as float? or sas is just inflexible?

@jreback thanks for the quick review.  I will change the function name to read_sas.  

When you mention `to_sas` do you mean to implement a writer?  I wasn't planning to do a writer at this point.  My view is that this is an archaic format and shouldn't normally be used for generating new files.  Note that this is SAS xport, not the native sas7bdat.

In SAS xport files there are only two data types, ASCII text and numeric, which is always double type.  The doubles are stored in an old IBM format which stores a given number of decimal places.  There is sort of a pseudo-integer type when the number of decimals is given as zero.  In the R foreign package these are returned as ints.  We could do that too, but then if there are missing values it would have to go back to double.

@kshedden this looks really good! , prob only thing needed is a short section in `io.rst` (and pls add the links for `read_sas`), and I think an import in the `pandas.io.api`

you guys want to give a test run:

cc @benjello 
cc @miketkelly 
cc @dramage1  
cc @selik
cc @spearsem
cc @gdementen 
cc @jaredhobbs

cc @TomAugspurger 
cc @jseabold 

@kshedden can you update?

I think I have addressed all the comments.  Is there anything else that I need to do?

pls add a release note in 0.17.0 whatsnew (with a small example I think), and same mini-example in io.rst.

no way to make a `to_sas(format='xport')`? (even for just 'testing')? I know its not a great format, but makes testing very easy.

I added the missing text.  

My plan was to add SAS7BDAT read support first and then we can look into adding the writers.

Converting from IEEE to IBM float is trickier and loses precision.

On Sun, May 10, 2015, 9:58 AM Kerby Shedden notifications@github.com
wrote:

> I added the missing text.
> 
> My plan was to add SAS7BDAT read support first and then we can look into
> adding the writers.
> 
> ## 
> 
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9711#issuecomment-100668418.

I'm stuck trying to resolve these exceptions.  It fails on two travis runs with the error:

ValueError: time data '25OCT06:10:31:07' does not match format '%d%b%y:%H:%M:%S'

The format looks fine to me and I can't reproduce off Travis.

Any ideas are appreciated.

I don't think u can put a colon in between the date/time spec - use a space in the spec 
(I mean it obviously works in 2.7 and greater but possible 2.6 doesn't accept it)

Thanks, seems to be working now.

looks pretty good. a couple of recommendations and some doc changes. can you add a vbench as well? (in vb_suite/packers.py). thanks!

@kshedden added some doc formatting comments. Can you also add this to `doc/source/api.rst` ?

@kshedden can you update?

I think I have addressed everything above except the vbench.  I'm not sure how to proceed there because we don't have a writer to create a test file.   Can we use the files in the io/tests directory?  They may be too small to provide good performance checks.

couple of more comments. Skip the vbench for now (can address later, let's make an issue about it after this is merged).

@jreback wouldn't you rather have a custom error class `XportFileError` or `FileFormatError` or something like that rather than a generic `__builtins__` ValueError?

you can certainly have _some_ custom error messages (that inherit from `ValueError` or `TypeError`) if they make sense. In general though simpler is better.

can u rebase this

@jreback  Rebased and green.

@kshedden looks good. couple of minor doc comments. Pls fix and squash. ping when green.

I rebased on master and squashed the commits, then it came back as failing on unrelated tests... 

yes, you can ignore these test errors for Yahoo

@kshedden pls rebase on master and push again. I fixed the network issue.

@jreback I'm stuck here with some mysterious (to me) travis errors.

@kshedden some docs changes and test comparisons. Otherwise looks good. ping when ready.

@jreback I think it's up to date and green except for the Yahoo tests.

@kshedden ok, this looks good. I reran the failed build, ping me when its green.

@kshedden btw...for the next project!

I think it makes sense to have https://pypi.python.org/pypi/sas7bdat as an optional dep and read this.
(even IIRC its slow / whatever), should be straightforward to drop it into our impl).