BUG: Make sure that sas7bdat parsers memory is initialized to 0 (#21616)
State: closed

Memory for numbers in sas7bdat-parsing was not initialized properly to 0.
For sas7bdat files with numbers smaller than 8 bytes this made the
least significant part of the numbers essentially random.
Fix it by initializing memory correctly.

- [X] closes #21616
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: IO SAS
description: SAS: read_sas

Comments:
Hello @troels! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/io/sas/sas7bdat.py`](https://github.com/troels/pandas/blob/0783757f4091a151c24c610382467d00c6f3e4f9/pandas/io/sas/sas7bdat.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/sas/test_sas7bdat.py`](https://github.com/troels/pandas/blob/0783757f4091a151c24c610382467d00c6f3e4f9/pandas/tests/io/sas/test_sas7bdat.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=h1) Report
> Merging [#22651](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c04035360a179e82fc534cb4f38979615c53191f?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22651/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #22651   +/-   ##
=======================================
  Coverage   92.17%   92.17%           
=======================================
  Files         169      169           
  Lines       50715    50715           
=======================================
  Hits        46747    46747           
  Misses       3968     3968
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.58% <100%> (ø)` | :arrow_up: |
| #single | `42.35% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22651/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.09% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=footer). Last update [c040353...796fc43](https://codecov.io/gh/pandas-dev/pandas/pull/22651?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

cc @kshedden and @Winand if you have time to give this a review.
Not sure. Do any of the test files already in the repo have this problem.

We can add it if necessary, just trying to avoid bloating the git
repository more than it already is.

On Tue, Sep 11, 2018 at 12:49 PM Troels Nielsen <notifications@github.com>
wrote:

> *@troels* commented on this pull request.
> ------------------------------
>
> In pandas/tests/io/sas/test_sas7bdat.py
> <https://github.com/pandas-dev/pandas/pull/22651#discussion_r216760263>:
>
> > @@ -183,6 +183,18 @@ def test_date_time(datapath):
>      tm.assert_frame_equal(df, df0)
>
>
> +def test_compact_numerical_values(datapath):
> +    # Regression test for #21616
> +    fname = datapath("io", "sas", "data", "cars.sas7bdat")
>
> Hi @TomAugspurger <https://github.com/TomAugspurger>
>
> I am adding a sas7bdat file for testing the parsing of sas7bdat files. How
> can that be done more sensibly?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/22651#discussion_r216760263>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIiuIKiA5BprHocrjYhmRSSWSt8Wxks5uZ_esgaJpZM4WgZxE>
> .
>

@TomAugspurger: 

No, none of the test-files has the problem. If any of the already existing test-files had numeric columns less than 8 bytes, the tests would probably have been unstable.

Usually one wants to avoid binary files in git repositories for two reasons:
1. They tend to be large
2. Git's diff has a tendency to every time a binary files has changed store the complete file as the delta. (as git's diff is clumsy with binary content) 
That makes e.g. large images that are edited and committed often rather toxic. 

A small file like cars.sas7bdat, which will never be edited and to top it off is highly compressible is not going to take very much space at all, neither now nor in the future.
Hi @jreback 

Sure thing, I've added variables and extended the comment a bit.

We are comparing the result that the sas7bdat-parser reads from the file with the closest integers. As we know that the numbers are integral they should be exactly equal to their closest integer. When the bug was active the decimal part of the floating point numbers were not read from the file but simply taken from uninitialized memory, and so the WGT and CYL were not actually whole numbers in the pandas dataframe.

I originally had a CSV-file containing the correct numbers, but was compelled to remove it because of  @WillAyd 's comments. I'll gladly add it back if that's preferable.
thanks @troels