BrandonFors/pandas-github-issues huggingface dataset:
ENH GH20927 Added convert_dates parameter (default=True) to read_sas()
State: closed

Added parameter which is passed down to the underlying
SAS7BDATReader.  This allows SAS datasets with dates too large
for pandas.datetime to be read in with the dates/datetimes as
SAS native float64 values, rather than trying (and failing) to convert
to pandas.datetime.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Could these be converted to python datetime?
> Could these be converted to python datetime?

@jbrockmendel : I suspect you'll have to dig into the `SAS7BDATReader` internals to find out more.  I think the change as it stands makes sense.
re: Could these be converted to python datetime?

@jbrockmendel, @gfyoung:
Thanks for comment and review.

I did initially look to catch the exception and convert to datetime.datetime instead of Pandas datetime, however it breaks some existing tests so I thought it simpler to expose the convert_dates parameter from SAS7BDATReader and allow the user to explicitly deal with the very large dates.
I'm happy to look further into the conversion to python datetime if you think this is a better solution (and have code I can submit if it is of interest)
@paul-lilley : In the interests of modularity, I would not make any changes to `SAS7BDATReader` for this PR.  You are more than welcome to investigate and submit a PR for the `SAS7BDATReader` if you would like following this one.
@jreback 
Hi, thanks for the review.  
Just to clarify, do you mean to keep the current conversion, returning pd.datetime if all ok, catch OutOfBoundsDatetime if not and return an object array of datetime.datetime objects? 
This would return different types depending on the range of dates in the SAS dataset, as SAS7BDATReader is an iterator this could give some chunks converted to pd.datetime and others to datetime.datetime within a given SAS dataset, but I guess this will unlikely break any existing code and retains the more efficient pd.datetime (ie. a numpy datetime) for most SAS datasets.

> https://github.com/pandas-dev/pandas/pull/21983#issuecomment-407642117

@paul-lilley yes that's what I would do. This is how pandas works elsewhere and would be least suprising.'
Please update the title of the PR as well.
Thanks - I'll make the changes and update the PR title, probably after Europython this week.
can you merge master and update to comments
closing as stale. if you'd like to continue, pls ping.
I'll pick this up in a new PR, resolving the issue as discussed with jreback above on 25 Jul 2018
ENH: Implement subtraction for object-dtype Index
State: closed

- [x] closes #19369
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

@jreback @jorisvandenbossche discussed briefly at the sprint.  Merits more thorough testing, but I'd like to get the go-ahead to separate out arithmetic tests that are common to EA/Index/Series/Frame[1col] that are highly duplicative first.

Labels:
name: Enhancement
name: API Design

Comments:
>  but I'd like to get the go-ahead to separate out arithmetic tests that are common to EA/Index/Series/Frame[1col] that are highly duplicative first.

You will need to give some more details, but let's do such a change in a separate PR anyway, so I would simply fix up and merge this one first.
> Regarding testing, I think you should only add rsub case to the test?

> I don't think you are exercising rsub?

rsub is tested implicitly in the `result = index - pd.Index([Decimal(1), Decimal(1)])` line.  I'll make that explicit.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@27ebb3e`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21981/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master   #21981   +/-   ##
=========================================
  Coverage          ?   91.96%           
=========================================
  Files             ?      166           
  Lines             ?    50331           
  Branches          ?        0           
=========================================
  Hits              ?    46289           
  Misses            ?     4042           
  Partials          ?        0
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <100%> (?)` | |
| #single | `42.23% <33.33%> (?)` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/21981/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.36% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=footer). Last update [27ebb3e...bd6e88c](https://codecov.io/gh/pandas-dev/pandas/pull/21981?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jbrockmendel thanks!
CLN:Unused Variables
State: closed

- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Used PyCharm inspections to check:

- Unused local variables
- Unreachable code
- `set([..])` to `{...}`

Labels:
name: Clean

Comments:
I thought flake8 checked for unused variables, unreachable code.  Are we using an old version or something?

The set comprehensions are a good fix.  Might be worth adding that flake8 plugin while you’re at it.
I'm not sure if PyCharm is using flake8 under the hood or they have their own introspection method, but it was catching some things that flake8 wasn't.

Nonetheless, I'll scale this back and break it into smaller PR's
Backport PR #21921 on branch 0.23.x
State: closed

Backport PR #21921: BUG:Clip with a list-like threshold with a nan is broken (GH19992)

Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=h1) Report
> Merging [#21967](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=desc) into [0.23.x](https://codecov.io/gh/pandas-dev/pandas/commit/b7a2cd4a4c6ea235005aecbc2911034c6064afd3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21967/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           0.23.x   #21967   +/-   ##
=======================================
  Coverage   91.83%   91.83%           
=======================================
  Files         153      153           
  Lines       49592    49592           
=======================================
  Hits        45541    45541           
  Misses       4051     4051
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.23% <100%> (ø)` | :arrow_up: |
| #single | `41.84% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21967/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.21% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=footer). Last update [b7a2cd4...5c83562](https://codecov.io/gh/pandas-dev/pandas/pull/21967?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix memory leak in roll_quantile
State: closed

closes #21965


Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
looks reasonable, can you add a whatsnew note for 0.23.4 (bug fixes)
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=h1) Report
> Merging [#21966](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1ce2473716630e337fe7705a7a728d7c5f6cc301?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21966/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21966   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files         166      166           
  Lines       50329    50329           
=======================================
  Hits        46287    46287           
  Misses       4042     4042
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <ø> (ø)` | :arrow_up: |
| #single | `42.23% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21966/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.46% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=footer). Last update [1ce2473...13cca8e](https://codecov.io/gh/pandas-dev/pandas/pull/21966?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Sure, I just amended the commit
thanks @fl4p
Trim unncessary code in datetime/np_datetime.c
State: closed

`pydatetime_to_datetimestruct` does a ton of checking that boils down to "is this a valid datetime object?"  Since the function only gets called after a type-check, we can assume it is a date/datetime and be a lot less verbose about it.

This also rips out an unnecessary layer of functions `convert_datetime_to_datetimestruct`, `convert_timedelta_to_timedeltastruct`.

cc @WillAyd you mentioned wanting to work on your C-foo.  There's a comment about figuring out how to import the cpython datetime C-API.  Any thoughts?

Labels:
name: Datetime
description: Datetime data dtype
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=h1) Report
> Merging [#21962](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4b73f22d0b8312607907515395266eea774431bf?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21962/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21962   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files         166      166           
  Lines       50329    50329           
=======================================
  Hits        46287    46287           
  Misses       4042     4042
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <ø> (ø)` | :arrow_up: |
| #single | `42.23% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21962/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.46% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=footer). Last update [4b73f22...69a8459](https://codecov.io/gh/pandas-dev/pandas/pull/21962?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

thanks @jbrockmendel
DOC add Python 2.7 warning to recent whatsnew; include 23.3
State: closed

#18894 laid out:
> We should add a big note on the top of each whatsnew that we are planning on dropping 2.7 support as of the end of 2018.

So far, this warning only appeared in v0.23.0, and can be easily missed these days. This PR adds them to the recent whatsnews, and also includes v0.23.3 in `whatsnew.rst`


Labels:
name: Docs

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=h1) Report
> Merging [#21944](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/2f2876b8c7ec259d7310a33da2412b07b1850421?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21944/graphs/tree.svg?height=150&width=650&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21944   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files         166      166           
  Lines       50323    50323           
=======================================
  Hits        46281    46281           
  Misses       4042     4042
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <ø> (ø)` | :arrow_up: |
| #single | `42.22% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=footer). Last update [2f2876b...9bca92a](https://codecov.io/gh/pandas-dev/pandas/pull/21944?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

thanks
[CLN] Un-xfail now-passing tests
State: closed

Some of these were fixed in #21861, others sometime earlier this year, unclear.

Removes un unused util `docstring_wrapper`, closing #19676

- [x] closes #19676
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:

Comments:
Closing for now to clear the queue.
Add option for alternative correlation types to `DataFrame.corrwith()`
State: closed

- [x] closes #21925 
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

This adds a `method` argument to `DataFrame.corrwith()` which calls the corresponding correlation function which already exists within pandas. The default option "pearson" was originally implemented directly in this method, I changed it to match the other options for consistency (and a slight speed improvement).

Labels:
name: Enhancement
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
> The default option "pearson" was originally implemented directly in this method, I changed it to match the other options for consistency (and a slight speed improvement)

Could you demonstrate this speed improvement with the ASV? I am a little dubious that the iterative method is more performant than the existing implementation.
BUG:Clip with a list-like threshold with a nan is broken (GH19992)
State: closed

- fix bug #19992
- 2 tests amended in frame/test_analytics.py and series/test_analytics.py
- whatsnew entry added

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=h1) Report
> Merging [#21921](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/1ce2473716630e337fe7705a7a728d7c5f6cc301?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21921/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21921   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files         166      166           
  Lines       50329    50329           
=======================================
  Hits        46287    46287           
  Misses       4042     4042
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <100%> (ø)` | :arrow_up: |
| #single | `42.23% <0%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21921/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.45% <100%> (-0.01%)` | :arrow_down: |
| [pandas/core/indexes/period.py](https://codecov.io/gh/pandas-dev/pandas/pull/21921/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9wZXJpb2QucHk=) | `93.2% <0%> (+0.03%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=footer). Last update [1ce2473...0ef2e4d](https://codecov.io/gh/pandas-dev/pandas/pull/21921?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

does this also close #19976 ?
thanks @makbigc
[BUG][BLD] revert DEF component of #21878
State: closed

#21878 introduced a subtle build problem that is not caught by the CI.  Running `python setup.py build_ext --inplace` followed by `python3 setup.py build_ext --inplace` causes compile-time errors in py3 (or if running these in the opposite order, errors in py2).  These are fixed by running `python setup.py clean` in between.

This reverts the relevant changes.

For anyone curious: cython supports syntax:

```
DEF foo = [...]

IF foo:
    [...]
ELSE:
    [...]
```

and these IF/ELSE conditions get evaluated at compile-time.  #21878 incorrectly assumed that "compile-time" in this context meant ".c -> .so" time, not ".pyx -> .c" time.  In this we used:

```
DEF PY2 = str == bytes

IF PY2:
    [...]
ELSE:
    [...]
```

so after running `setup.py build_ext --inplace` in py2, the ".c" file we end up with has already gotten rid of the PY3 branches.  When we run `python3 setup.py build_ext --inplace` cython uses the existing .c file, tries to compile it to .so, and breaks.

Labels:

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=h1) Report
> Merging [#21919](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/63eff04af6f343557b30e656e94d76030e039029?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21919/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21919   +/-   ##
=======================================
  Coverage   91.96%   91.96%           
=======================================
  Files         166      166           
  Lines       50323    50323           
=======================================
  Hits        46281    46281           
  Misses       4042     4042
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <ø> (ø)` | :arrow_up: |
| #single | `42.23% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=footer). Last update [63eff04...d939bde](https://codecov.io/gh/pandas-dev/pandas/pull/21919?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback @WillAyd quick take before I merge?
lgtm

yeah ideally these would be compile time checks