BrandonFors/pandas-github-issues huggingface dataset:
BUG: Cast pd.NA to pd.NaT in to_datetime
State: closed

- [x] closes #32213
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
I am not fully sure we should do this, since pd.NA and pd.NaT behave differently (and we probably want to have a timestamp dtype that uses pd.NA at some point)
> I am not fully sure we should do this, since pd.NA and pd.NaT behave differently (and we probably want to have a timestamp dtype that uses pd.NA at some point)

Ok, I'll close the PR, but should probably leave the issue open for now?
if the user specifically calls to_datetime, it makes sense that pd.NA would cast to NaT
@jbrockmendel so the reason I initially didn't add this check to `checknull_with_nat` is because this adds a dependency of tslibs on _libs/missing.pyx, while now the dependency is actually the other way around. 

We might want to make missing.pyx independent on the rest of `_libs`, so tslibs can easily depend on it.
> We might want to make missing.pyx independent on the rest of _libs, so tslibs can easily depend on it.

Maybe a file with just NAType, as that doesn't depend on most of the things in libmissing (many of which do depend on tslibs).  This seems like it will be necessary if we want e.g. `pd.Timestamp(pd.NA)` to give `pd.NaT` instead of raising
Is this something we need for 1.0.2, or can it wait for 1.1.0?  if the latter id like to revert and take more time to figure out the dependency structure
> Is this something we need for 1.0.2, or can it wait for 1.1.0? if the latter id like to revert and take more time to figure out the dependency structure

this is a usability issue in 1.0.2

happy to change for 1.1 to a proper structure (this is already merged and backport ed)
DOC: DataFrame.ewm clean-up
State: closed

Cosmetic change.


Labels:
name: Docs

Comments:
Thanks @sursu !

I'd mentioned this and some other changes to this docstring in #31647, do you feel like taking any of the other points too? A few people commented "take" but nobody submitted anything :)
There's more there. 
For instance, `\text{ for }` shouldn't be a LaTeX text, should be normal text, otherwise in a single line we have 3 types of symbols.

I am not familiar with `:math:`, and I don't know how to preview my changes.
Does it render the entire line or only what is take in ``?

I don't understand this invention. I don't see why using `:math:` was smarter than using LaTeX/MathJax syntax.
> I am not familiar with `:math:`, and I don't know how to preview my changes.
> Does it render the entire line or only what is take in ``?
> 
> I don't understand this invention. I don't see why using `:math:` was smarter than using LaTeX/MathJax syntax.

You will want to build the docs:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#how-to-build-the-pandas-documentation

If you can post screenshot afterwards would be helpful

We use sphinx to build restructured text. Math is a role provided therein:

https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#math
Hello @sursu! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-07 18:39:06 UTC
Screenshots:

![image](https://user-images.githubusercontent.com/12001304/75257397-2ea64f80-57e5-11ea-9703-fc5f9e5ea451.png)

![image](https://user-images.githubusercontent.com/12001304/75259306-171c9600-57e8-11ea-88ec-e4bcc132ffc2.png)

The Notes have been expanded with a few details from the given link.
Ready for Review
Looks like CI is red - can you take a look?
Thanks @sursu , this is much nicer!

> Though, I don't know what @MarcoGorelli meant in the 5'th issue.

Just to clarify - in the doc for the respective series method (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.ewm.html), it says 

> "axis{0 or ‘index’, 1 or ‘columns’}, default 0".

 However, Series don't have columns :)

But that's a recurring issue in many parts of the documentation, so IMO it's OK to keep it separate from this
Oh, so `pandas.Series.ewm.html` is also generated from `ewm.py`?

Well if someone tries to perform an EW function column-wise on a series they would bump into: 
```python
ValueError: No axis named columns for object type <class 'pandas.core.series.Series'>
```
which is a clear message.

Also, it seems that you can also use `axis=rows`. Should I specify this?
![image](https://user-images.githubusercontent.com/12001304/75488435-9c52a700-59b0-11ea-98fa-5002203fbb26.png)

You'll have to build all the docs, not that single page to see the link.

Also, if you can merge master please, so we make the CI green and we can merge...
Thanks for the good docstring @sursu
BUG: DataFrame fail to construct when data is list and columns is nested list for MI
State: closed

- [ ] closes #32173 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-06 19:38:52 UTC
Slight -1 on supporting this - what benefit does this provide over passing a MultiIndex?
Hmm OK I see the original issue where this may or may not work depending on what is passed as the values to the DataFrame constructor, so OK...

Is there not any shared code to leverage here? Wonder why this works for DataFrame construction with an ndarray but not with lists
emm, yeah, passing MI should be the way to go, but the main reason i think this might be good to support is we do support such assignment when data is a numpy array, as the example presented in issue description. In the way, we could have kinda consistency. 
ahh, sorry, did not see your update comment @WillAyd 😅 
@jreback many thanks for your follow-up review, I think i addressed all your comments, feel free to provide follow-ups!
@WillAyd @jreback I have made some improvements on type annotations. and some feedbacks will be very welcome! thanks!
thanks very much for your detailed reviews @jbrockmendel  appreciate a lot!

I have addressed them and feel free to take another look! thanks!
emm, seems the build is broken, @jbrockmendel  doesn't seem to relate to this PR though
thanks @charlesdong1991 very nice
BLD: Init PyDateTimeAPI Warnings
State: closed

- Fixes below warning:
```
In file included from pandas/_libs/src/ujson/python/date_conversions.h:7:0,
                 from pandas/_libs/src/ujson/python/date_conversions.c:4:
/home/vsts/miniconda3/envs/pandas-dev/include/python3.7m/datetime.h:204:25: warning: ‘PyDateTimeAPI’ defined but not used [-Wunused-variable]
 static PyDateTime_CAPI *PyDateTimeAPI = NULL;
```

https://github.com/pandas-dev/pandas/pull/32163#issuecomment-590010308 (will check CI)

Still same warning coming from np_datetime and np_datetime_strings


Labels:

Comments:
does this have any impacts other than silencing the warning?  build time/size?
aaaah! - this doesn't work as intended (seemed fine locally) back to the drawing board...
CLN: F-string in pandas/tests/indexes/datetimes/test_to_period.py (#29547)
State: closed

Issue #29547
- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks
CLN: Removed class in pandas/tests/series/test_timezones.py
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
This is similar to #32184. Nothing in the tests changed, I just removed the unused class interface to simplify the file
As we're not using any functionality of the class interface, then why have it? Having the class means...
1. it's harder to reason about the test: functions are more independent than methods
1. it forces `black` to use more line breaks (we reach the line limit earlier because of the extra indentation)
1. it adds more clutter

I think the class interface is just a left-over from #22550
> As we're not using any functionality of the class interface, then why have it? Having the class means...
> 
> 1. it's harder to reason about the test: functions are more independent than methods
> 2. it forces `black` to use more line breaks (we reach the line limit earlier because of the extra indentation)
> 3. it adds more clutter
> 
> I think the class interface is just a left-over from #22550

@SaturnFromTitan I get it, and generally i agree, however i also think we have *many* of these, so we are talking many PRs; so unless we are splitting out to separate modules I wouldn't change this. 

Now I could be wrong on the number, if you want to see how many single file with a class we have then I could change.
Gotcha. Indeed, there are 100+ files like that.

Even though the effort to change this is very small, I guess the review effort would be significant and would be better to be invested elsewhere. Closing this then
BUG: 2D DTA/TDA arithmetic with object-dtype
State: closed

The main motivation for this is that it is prerequisite for implementing frame-with-frame ops blockwise.  It also fixes a few broken cases:

- dta - dta.astype(object) currently raises TypeError
- ditto dti - dti.astype(object)
- tda + dta.astype(object) returns ndarray of timestamps instead of DatetimeArray



Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback gentle ping; this is a blocker for the upcoming blockwise frame-with-frame perf fix
thanks. this *looks* user facing? if so can you follow up with a whatsnew note .
CLN: Removed class in pandas/tests/series/test_validate.py
State: closed

I've seen quite a few test files that only use a class for historic reasons. As functions are easier to comprehend I think it makes sense to move in this direction.

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Clean

Comments:
Thanks @SaturnFromTitan
DOC: Fix SA04 errors in docstrings #28792
State: closed

- [x] xref #28792 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Docs

Comments:
@AdrianMastronardi looks like a merge conflict - can you merge master?
@WillAyd merge conflict resolved.
Great thanks @AdrianMastronardi !
DOC: Fix SA04 errors in docstrings xref #28792
State: closed

- [x] xref #28792 
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
Hello @AdrianMastronardi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-03 03:15:34 UTC
Thanks @AdrianMastronardi
DOC: Added recommanded "msg" creation format
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/32158#discussion_r382710851
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
I don't think this is important to document or makes that much of a difference
I thought so as well, just opened this PR to make a discussion about it.
CLN: some code cleanups
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
@MomIsBestFriend IIUC cython support for black is under development, https://github.com/psf/black/issues/359.
> @MomIsBestFriend IIUC cython support for black is under development, [psf/black#359](https://github.com/psf/black/issues/359).

Thank you @simonjayhawkins, me and my OCD are thrilled, I subscribed to that thread :laughing: 

Can you merge master? Otherwise I think this is good to go
@MomIsBestFriend looking at the diff of the merge master, it looks like you've lost some of your clean-ups.
> @MomIsBestFriend looking at the diff of the merge master, it looks like you've lost some of your clean-ups.

@simonjayhawkins Correct, in 1cc4e89 I have reverted everything lost in the merge conflicts.
All green. @jreback you requested changes, if you want to have another look, this should be ready.
ok as long as purely cosmetic 
Thanks @MomIsBestFriend
CLN: Some code cleanups
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
thanks @MomIsBestFriend
BUG: groupby nunique changing values
State: closed

- [ ] closes #31950 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

just reopening #31973 for now - this doesn't special-case NaT. Will think if there's a better way to do it, feel free to close / take over if it's wrong

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @MarcoGorelli
DOC: Move pandas_development_faq from wiki to doc #30232
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
cc @pandas-dev/pandas-core 
@datapythonista thanks for your feedback. Meanwhile we wait for other contributors to decide whether keep this doc or remove it, I fixed the errors you pointed out.
Thanks @AdrianMastronardi.

If nobody objects I'll be removing the wiki page and closing this PR soon. If anyone wants to keep anything of that wiki page, please let me know.
I think fine to get rid of the wiki page
Deleted the wiki page, closing this.
Thanks @AdrianMastronardi, and sorry for the late decision (after you did all the work). Please feel free to ping me if you want to move any other page, and you're unsure if the content is still relevant.
DOC: Move testing and pandas_development_faq pages from wiki to doc #30232 & #20501
State: closed

- [ ] closes #20501
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Got some errors. Will fix them .
BUG: disallow invalid dtype to CategoricalDtype._from_values_or_dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #32167.  Together with that, this will allow us to simplify the CategoricalIndex constructor.

Labels:
name: Bug

Comments:
#32167 first? 
> #32167 first?

Any order should be OK
BUG: Index(categorical, dtype=object) not returning object dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

In master we get a `CategoricalIndex` back.

Fixing this turns out to be pre-requisite to fixing a bug in `CategoricalDtype._from_values_or_dtype`, which in turn is needed to simplify the CategoricalIndex constructors (xref #32141)

Labels:
name: Bug

Comments:
lgtm. rebase and ping on green.
let's rebase this (as merged other 2 PRs)
rebased+green
thanks
REGR: preserve freq in DTI/TDI outer join
State: closed

- [x] closes #32157
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

also just a nice cleanup, hoping to get this into ExtensionIndex before long

Labels:
name: Datetime
description: Datetime data dtype

Comments:
i marked for 1.1, but is this a regression?
rebase as well
> i marked for 1.1, but is this a regression?

The behavior changed between 0.25.3 and 1.0, but it isnt obvious whether to call it a regression or not because it didnt obviously break anything. (see #32157)
> > i marked for 1.1, but is this a regression?
> 
> The behavior changed between 0.25.3 and 1.0, but it isnt obvious whether to call it a regression or not because it didnt obviously break anything. (see #32157)

ok it was marked originally as a regression
however i bet this will non trivial to backport (tests)

so up to u
removed REGR tag
hmm i think this needs a whatsnew note (to document that this is changing), otherwise lgtm.
thanks @jbrockmendel 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 27f0000804072e8d65e6c7939ec09069e1b9790e
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32166: REGR: preserve freq in DTI/TDI outer join'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32166-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32166 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel as expected this needs a manual backport :-<
CLN: Fix exception causes in datetimelike.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Clean

Comments:
Hello @cool-RR! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-21 22:29:14 UTC
Thanks for reviewing guys. I'm ambivalent towards the `err` change, so if you'd like me to change, I'll change it. If there's anything else I need to do about this PR, let me know. 
Thanks @cool-RR, and welcome to pandas! 🎉

`err` naming question notwithstanding, it looks good!
@cool-RR is there something here we should be testing?
If you mean testing that the `__cause__` attribute is set correctly, I think that's overkill. The only kind of testing that's useful is having test code that causes the exception, reaches the `raise` and ensures the correct exception gets raised, but that's orthogonal to this PR, i.e. it's needed about as much as it's needed without this PR.
sounds good, LGTM
Cool. Any next steps needed before merging? 
I would be on board with doing this more generally if there's a way to identify easily
Thanks @cool-RR
Backport PR #32152 on branch 1.0.x (TST: add test for get_loc on tz-aware DatetimeIndex)
State: closed

xref #32152

@jbrockmendel with tests moving around, is this OK for the backport. i.e. in the same place as on master.

Labels:

Comments:
Seems harmless

On Fri, Feb 21, 2020 at 11:37 AM Simon Hawkins <notifications@github.com>
wrote:

> xref #32152 <https://github.com/pandas-dev/pandas/pull/32152>
>
> @jbrockmendel <https://github.com/jbrockmendel> with tests moving around,
> is this OK for the backport. i.e. in the same place as on master.
> ------------------------------
> You can view, comment on, or merge this pull request online at:
>
>   https://github.com/pandas-dev/pandas/pull/32160
> Commit Summary
>
>    - Backport PR #32152 on branch 1.0.x (TST: add test for get_loc on
>    tz-aware DatetimeIndex)
>
> File Changes
>
>    - *M* pandas/tests/indexes/datetimes/test_indexing.py
>    <https://github.com/pandas-dev/pandas/pull/32160/files#diff-0> (13)
>
> Patch Links:
>
>    - https://github.com/pandas-dev/pandas/pull/32160.patch
>    - https://github.com/pandas-dev/pandas/pull/32160.diff
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/32160?email_source=notifications&email_token=AB5UM6BEJUY5Q62564VMQF3REAUP5A5CNFSM4KZJFAA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IPMO6HA>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6AIFQIZALHMA47XOL3REAUP5ANCNFSM4KZJFAAQ>
> .
>
TST: Fixed bare pytest.raises in test_window.py
State: closed

- [x] ref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I have adjusted the formatting and split the message per loop.
thanks @mabroor
TST: add test for DataFrame.reindex on nearest tz-aware DatetimeIndex
State: closed

- [ ] closes #31964
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @simonjayhawkins 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 9353ca736d512ac28fd788ffc60e9e70393e5f6d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32155: TST: add test for DataFrame.reindex on nearest tz-aware DatetimeIndex'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32155-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32155 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).