BrandonFors/pandas-github-issues huggingface dataset:
TST: add test for get_loc on tz-aware DatetimeIndex
State: closed

- [ ] closes #32140
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM
thanks @simonjayhawkins 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 d171c871baa13bdfd2fe14c8a5918190314825bb
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32152: TST: add test for get_loc on tz-aware DatetimeIndex'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32152-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32152 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@simonjayhawkins may need a manual backport
Add ecosystem link to pandas-tfrecords project
State: closed

to understand tfrecords format requires quite much time and finally its details are not so needed in daily work with data. [pandas-tfrecords](https://pypi.org/project/pandas-tfrecords/) hides low-level details and provides easy human-usable api to convert pandas to tfrecords and restore back.

Labels:
name: Docs

Comments:
Thanks @schipiga
Import OptionError in pandas.errors
State: closed

- [x] closes #27553
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: API Design
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jorisvandenbossche I am experiencing a build failure on my forked master as well, is this some setup issue, would need your help to resolve this.

Also Is the pr good to go or more changes would be required ?
Travis failure should be unrelated, just restarted that build.
Can you add this to test_exception_importable in tests.test_errors
@jbrockmendel Tests are changed, a whatsnew entry is created. Would you be able to review this or suggest someone who can help in reviewing this PR.
@jreback Please review.
thanks @sumanau7
REF: standardize CategoricalIndex._shallow_copy usage
State: closed

The dtype keyword is akward, so this removes the one usage.

In turn this will allow us to simplify `_simple_new` and tighten up what it expects.  Saved for separate PR.

Labels:
name: Refactor
description: Internal refactoring of code
name: Categorical
description: Categorical Data Type
name: Index
description: Related to the Index class or subclasses

Comments:
sure
TST: show_versions test with unmerged commits
State: closed

- [x] closes #32120
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Closing pending investigation in #32120.
Re-opened, the issue wasnt with un-merged commits, but with git worktree.  re-using versioneer code handles the problem.
gentle ping @simonjayhawkins @WillAyd ; this is workflow-affecting
OK thanks @jbrockmendel 

Heads up - I’m at CZI this week so may be slow to respond to some of your other PRs. Should catch up over the weekend at the latest
used f-strings in docs
State: closed

- [x] xref #32039
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
I think this is correct as is - isn't this supposed to show how the single quote doesn't get escaped? See text immediately following
> I think this is correct as is - isn't this supposed to show how the single quote doesn't get escaped? See text immediately following

Yup, my bad - sorry @PSY27 . Though as Will said, if you change line 3818 to use an f-string, that would be good. In that case, you could add
```
ref #29547
```
to the pull request description instead of
```
#32039
```
@WillAyd @MarcoGorelli  please review it now

thanks @PSY27
CLN: NDFrame.__init__ unnecessary code
State: closed

Make methods into classmethods where feasible, annotate in a couple of places.

Removes nonsense test (constructs an invalid NDFrame)

Preparatory for making a potential fastpath _from_mgr constructor.

`attrs` is never passed.  Is there a plan to do so @TomAugspurger ?

Labels:
name: Clean

Comments:
great
PERF: IntegerIndex._shallow_copy (and by extension, indexing)
State: closed

```
In [2]: idx = pd.Index(range(1000)).insert(-1, 1)                                                                                                                                                                                                 
In [3]: %timeit idx[:5]                                                                                                                                                                                                                           
2.84 µs ± 97.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
24.5 µs ± 292 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master

In [4]: idx2 = idx.astype('uint64')                                                                                                                                                                                                               
In [5]: %timeit idx2[5:]                                                                                                                                                                                                                          
2.99 µs ± 331 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR
25.8 µs ± 957 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
```

Motivated by profiling in #32086 and seeing that _shallow_copy_with_infer was taking over a third of the runtime in the relevant benchmark.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
likely we don't have these benchmarks directly, maybe add if you can.
CLN: auto-inherit specified Index method docstrings
State: closed

When working on indexing methods, I find myself grepping for the names, and having the extra Appender lines is a small-but-repeated nuisance.

In py36+ we can use `__init_subclass_` to automatically inherit docstrings without needing to repeat these Appenders.

This PR only does this for a few methods as a proof of concept, but we could expand this list quite a bit.

Based on a handful of measurements, no obvious impact on import time.

cc @WillAyd you were -1 on the proof of concept when it was on PandasObject and automatic for all methods; are you more amenable to this more finely-tuned approach?

Labels:
name: Clean

Comments:
I'd prefer to stick with what @HH-MWB has done with the doc decorator for the time being, which combines Appender and Subsitution together
> decorator for the time being, which combines Appender and Subsitution together

I think these are addressing different problems.  This is addressing (in the order I _should_ have started with):

1) inconsistency in which methods we inherit docstrings for.  e.g.  `memory_usage` we inherit for `IntervalIndex`, copy/paste (i think, looks a little out of date) for `RangeIndex`, and no docstring for `PeriodIndex`.  Similar for `_maybe_cast_slice_bound` where `IntervalIndex` has no docstring.
2) verbosity that happens to be annoying in my workflow

mothballing to clear the queue
BUG: Fix for convert_dtypes with mix of int and string
State: closed

- [x] closes #32117
- [x] tests added / passed
  - added new cases for `test_convert_dtypes`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
  - placed in v1.0.2 whatsnew


Labels:
name: Bug

Comments:
@Dr-Irv Thanks!
BUG: Avoid ambiguous condition in GroupBy.first / last
State: closed

- [x] closes #32123
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This comparison `val == val` happens in a lot of these groupby operations but only seems to raise here in the presence of `NA`. Are we just always / mostly converting to numpy beforehand in the other cases?

Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
#31227 turned up a bunch of other places where having pd.NA in columns/index is going to break the world.  the `is_matching_na` implemented there might be useful here
@dsaxton can you merge master to resolve conflicts
thanks @dsaxton 
@meeseeksdev backport to 1.0.x
CLN: Replace old string formatting syntax with f-strings for scripts/validate_docstrings.py
State: closed


- [ ] xref #29547
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @RaisaDZ
REG: dont call func on empty input
State: closed

- [x] closes #31760
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Could use confirmation from OP that this does in fact close #31760.

Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @jbrockmendel
REF: make pd._testing directory, split out asserters
State: closed

Scattered throughout the tests we have other helpers like assert_block_equal that probably belong in `tm`.  Before trying to collect all of these, it makes sense to split this already-cumbersome module.

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Good idea on splitting it. But while doing it, can you split a bit further? For example all `make...` functions can go in one file (eg "generators" or something). 

I personally prefer to split this in one go now (if we plan to further split it). Moving things around is annoying for git blame, so prefer to have it done in one step. And it also doesn't seem hard to further split it here
> I personally prefer to split this in one go now

sounds good
@jorisvandenbossche this isn't viable exactly as-is because of mypy complaints, but id like to get your thoughts on the general layout
Closing to clear the queue, will revisit
>  this isn't viable exactly as-is because of mypy complaints,

Can't you just silence mypy then for now on this directory? (as it is doing for `_testing.py` now)

Layout looks good to me! Seems logical groups
> Layout looks good to me! Seems logical groups

Great, any other groupings to get more stuff out of `__init__`?

Once the PR queue quiets down ill try to sort out the mypy issues.
REF: collect arithmetic benchmarks
State: closed

ATM these are scattered, making it tough to a) run just arithmetic-relevant benchmarks and b) determine what we have good benchmark coverage for.

This collects the scattered benchmarks in one place, cleaning up and fleshing these out is for a separate pass.

I also intend to do something analogous for indexing benchmarks.

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
Note that renaming benchmarks like this defeats somewhat the function of the tracking of benchmarks over time (not when comparing manually two commits, as the old commit is run with the new benchmarks, but for the benchmark machine)
> as the old commit is run with the new benchmarks, but for the benchmark machine

That's an unfortunate side effect, but in the class of problems where "throw hardware at it" is a viable solution.
@WillAyd for the grant you are looking at, e.g. should take a current benchmark file and run for all released versions as far back as you can (and if anything breaks fix in current benchmarks). we should do this periodically to get good baselines.
add test for "Allow definition of `pd.CategoricalDtype` with a specific `categories.dtype`"
State: closed

- [x] closes #32096 
- [x] tests added / passed


Labels:
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
Hello @rushabh-v! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-20 09:05:02 UTC
@TomAugspurger I have added the test. Where should I add the documentation?
Probably in the CategoricalDtype docstring in `dtypes/dtypes.py`. Add a sentence to the `categories` parameter noting that the categories are stored in an Index, and that if an index is provided the `dtype` from it will be used. Then in the `Examples` section a sentence saying an empty categorical with a specific dtype can be created by providing an empty index, and then show the example.
can you take a look, please?  @TomAugspurger
It’ll be a couple weeks for me. 

> On Feb 20, 2020, at 04:12, Rushabh Vasani <notifications@github.com> wrote:
> 
> ﻿
> can you take a look @TomAugspurger please?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks @rushabh-v!
REF/TST: implement test_interpolate for Series
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks

btw we should just blow away pandas/tests/generic and put those in series/frame as appropriate (e.g. interpolate tests for frame are there)
> btw we should just blow away pandas/tests/generic and put those in series/frame as appropriate (e.g. interpolate tests for frame are there)

This may come as a shock, but I've got a branch that works in that direction.
lol
TST: method-specific files for DataFrame assign, interpolate 
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks, comment from tests-series-interpolate PR, more interpolate tests in pandas/tests/generic
on this genre of PRs (i.e. "method-specific file for X") are you OK with having Will and Simon review+merge?  Could ease some of the backlog
> on this genre of PRs (i.e. "method-specific file for X") are you OK with having Will and Simon review+merge? Could ease some of the backlog

absolutely!
DataFrame: Ctor from non abc.Iterable 2-d array like
State: closed

In case we pass a 2d array like that do not use __iter__ but the
__getitem__ interface, it falls outside the collection.abc.Iterable case
and throw.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @jschueller! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-20 07:32:56 UTC
can you run `black pandas/tests/frame/test_constructors.py`
what use case is likely to lead to this kind of input?
Maybe a lack of understanding but not sure I see this as being useful and worth supporting. Does this reference an open issue?
we have a container that behaves like this in openturns:
http://openturns.github.io/openturns/master/user_manual/_generated/openturns.Sample.html?highlight=sample#openturns.Sample

this change would allow to use the DataFrame ctor directly:
```
import openturns as ot
import pandas as pd
sample = ot.Normal(3).getSample(10)
df = pd.DataFrame(data=sample, columns=sample.getDescription())
```

for now we can only build a DataFrame using from_records:
```
df = pd.DataFrame.from_records(sample, columns=sample.getDescription())
```

One could consider inconsistent that it succeeds with from_records and not with the ctor



It's not that it is not iterable, it's just that it does not use the __iter__/__next__ interface for iteration but the __gettitem__ interface, and so it's not getting in the abc.Iterable case of the DataFrame ctor
In a vacuum this is OK, but my main concern is that we don’t comprehensively support this type of “iterable” so somewhat weird to do one-off here. Not sure it would even be worth the effort for other pandas objects that we construct from iterables, so I think I would be -1 on moving forward with this
[sadface]
Thanks in any case @jschueller - we disagree on this one but attempt is in any case appreciated. Certainly would welcome more in the future
BUG: Allow addition of Timedelta to Timestamp interval
State: closed

- [x] closes #32023
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Interval
description: Interval data type

Comments:
@dsaxton can you merge master to resolve conflicts (and fix failing tests 😄 )
@jbrockmendel You think this might be good to merge?
@jschendel anything else this needs?
thanks @dsaxton
Added in a error message
State: closed

- [x] ref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Added in an error message


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @Samira-g-js! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-27 16:26:44 UTC
Looks like CI failures say there is no message in some cases - can you debug?
Great thanks @Samira-g-js!
BUG: Pickle NA objects
State: closed

According to
https://docs.python.org/3/library/pickle.html#object.__reduce__,

> If a string is returned, the string should be interpreted as the name
> of a global variable. It should be the object’s local name relative to
> its module; the pickle module searches the module namespace to determine
> the object’s module. This behaviour is typically useful for singletons.

Closes https://github.com/pandas-dev/pandas/issues/31847

Labels:
name: IO Pickle
description: read_pickle, to_pickle
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@TomAugspurger can you re-push?  looks like azure has stalled
It'll be a week or so before I can look at this again.


On Fri, Feb 21, 2020 at 10:34 AM jbrockmendel <notifications@github.com>
wrote:

> @TomAugspurger <https://github.com/TomAugspurger> can you re-push? looks
> like azure has stalled
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/32104?email_source=notifications&email_token=AAKAOIQOCULSNCBWNQBS33TREAGCPA5CNFSM4KXZVAX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMTPIWA#issuecomment-589755480>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXQS62R266VAIIDX7TREAGCPANCNFSM4KXZVAXQ>
> .
>

@TomAugspurger can you rebase and update (small requests)
I think everything has been addressed.
Thanks @TomAugspurger !
@MeeseeksDev backport to 1.0.x
added msg to TypeError test_to_boolean_array_error
State: closed

- [x] ref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Fixed bare pytest.raises (#30999) for test_to_boolean_array_error in test_boolean.py 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @amy-graham-js
TST: Fix bare pytest.raises in test_parsing.py
State: closed

- [x] ref #30999 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Fixes bare pytest.raises issue for `test_parsing.py`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM pending @simonjayhawkins comment (on which I have no strong opinion)
Thanks @mabroor
TYP: check_untyped_defs core.tools.datetimes
State: closed

pandas\core\tools\datetimes.py:299: error: Item "DatetimeIndex" of "Union[DatetimeArray, DatetimeIndex]" has no attribute "tz_convert"
pandas\core\tools\datetimes.py:310: error: Item "DatetimeIndex" of "Union[DatetimeArray, DatetimeIndex]" has no attribute "tz_localize"
pandas\core\tools\datetimes.py:829: error: Incompatible types in assignment (expression has type "str", variable has type "List[str]")
pandas\core\tools\datetimes.py:838: error: Incompatible types in assignment (expression has type "str", variable has type "List[Any]")
pandas\core\tools\datetimes.py:1010: error: Argument 1 to "append" of "list" has incompatible type "None"; expected "time"
pandas\core\tools\datetimes.py:1035: error: Argument 1 to "append" of "list" has incompatible type "None"; expected "time"

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Thanks @simonjayhawkins
TYP: check_untyped_defs arrays.sparse.array
State: closed

pandas\core\arrays\sparse\array.py:807: error: Need type annotation for 'result' (hint: "result: List[<type>] = ...")

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm outside of @jbrockmendel question
Thanks @simonjayhawkins
TYP: check_untyped_defs core.arrays.interval
State: closed

pandas\core\arrays\interval.py:438: error: Need type annotation for 'left' (hint: "left: List[<type>] = ...")
pandas\core\arrays\interval.py:438: error: Need type annotation for 'right' (hint: "right: List[<type>] = ...")

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
lgtm outside of conflict and @jbrockmendel question
TYP: check_untyped_defs core.arrays.categorical
State: closed

pandas\core\arrays\categorical.py:347: error: Incompatible types in assignment (expression has type "None", variable has type "str")
pandas\core\arrays\categorical.py:1550: error: "argsort" of "ExtensionArray" gets multiple values for keyword argument "ascending"
pandas\core\arrays\categorical.py:1550: error: "argsort" of "ExtensionArray" gets multiple values for keyword argument "kind"

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @simonjayhawkins