BUG: Dont include deleted rows from sas7bdat files (#15963)
State: closed

Sas7bdat files may contain rows which are actually deleted.
 
After reverse engineering the file format a bit, I found that   
if the page_type has bit 7 set, there is a bitmap following
the normal row data with a bit set for a given row if it has been
deleted. Use that information to not include deleted rows in
the resulting dataframe.

This PR is built on top of #22628

I've added two test-cases one from the issue  and another constructed
example, which tests parsing on different page types and across multiple sas7bdat-pages. 

The constructed example is rather large, mainly to have a test case where the data 
flows across several pages.

- [X] closes #15963
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry



Labels:
name: IO SAS
description: SAS: read_sas

Comments:
Hello @troels! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/io/sas/sas7bdat.py`](https://github.com/troels/pandas/blob/d7f46a303592c4d52d161f7dd6345fa543004618/pandas/io/sas/sas7bdat.py) !

 - There are no PEP8 issues in the file [`pandas/io/sas/sas_constants.py`](https://github.com/troels/pandas/blob/d7f46a303592c4d52d161f7dd6345fa543004618/pandas/io/sas/sas_constants.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/sas/test_sas7bdat.py`](https://github.com/troels/pandas/blob/d7f46a303592c4d52d161f7dd6345fa543004618/pandas/tests/io/sas/test_sas7bdat.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=h1) Report
> Merging [#22650](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/73dd6ece3b66008eb8077098c101d0e300b1e714?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22650/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22650      +/-   ##
==========================================
+ Coverage   92.17%   92.17%   +<.01%     
==========================================
  Files         169      169              
  Lines       50708    50721      +13     
==========================================
+ Hits        46740    46754      +14     
+ Misses       3968     3967       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.58% <100%> (ø)` | :arrow_up: |
| #single | `42.34% <0%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/22650/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22650/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.26% <100%> (+0.17%)` | :arrow_up: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/22650/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `96.55% <0%> (+0.1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=footer). Last update [73dd6ec...b31144a](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Haven't looked in detail due to size of PR but I noticed your original post only mentions one issue whereas the PR references 3 in the whatsnew. Is that intentional or have you simply pulled in some other concurrent developments accidentally?
Hi @WillAyd  I made a separate PR #22628 for the two first commits. This one I though was large enough to have its own PR, but as there was a dependency on the first two commits I continued this PR from the branch #22628 had sprung from. 
If #22628 can be merged soon, this PR will suddenly get slightly smaller (but just a bit)
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=h1) Report
> Merging [#22650](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f65fa755db23c8010d60f2160c522264417cb545?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22650/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22650      +/-   ##
==========================================
+ Coverage   92.18%   92.18%   +<.01%     
==========================================
  Files         169      169              
  Lines       50812    50821       +9     
==========================================
+ Hits        46842    46851       +9     
  Misses       3970     3970
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.6% <100%> (ø)` | :arrow_up: |
| #single | `42.36% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/sas/sas\_constants.py](https://codecov.io/gh/pandas-dev/pandas/pull/22650/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfY29uc3RhbnRzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [pandas/io/sas/sas7bdat.py](https://codecov.io/gh/pandas-dev/pandas/pull/22650/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXM3YmRhdC5weQ==) | `91.26% <100%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=footer). Last update [f65fa75...fcb1699](https://codecov.io/gh/pandas-dev/pandas/pull/22650?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Can you address comments and merge master?
can you merge master and address comments
Closing as stale. Ping if you'd like to continue
This would be pretty useful as the issue makes read_sas unusable for a lot of sas datasets.