BrandonFors/pandas-github-issues huggingface dataset:
DEPR: execute deprecations for str.cat in v1.0
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This executes some deprecations from the 0.23/0.24 cycles, namely following the switch from `join=None` to `join='left'` (following #18657 / #20347), as well as the fact that list-likes *within* list-likes were deprecated (following #21950 / #22264).

Since this deprecation closes the loop to my first-ever pandas issue (#18657) and PR (#20347), I'll allow myself a bit of a digression here. After biting off a bit more than I could chew (after finding out just how many cases the method was supposed to support), things got a bit tight with the cut-off for `0.23.0rc`, and @TomAugspurger [quipped](https://github.com/pandas-dev/pandas/pull/20347#issuecomment-385686992):

> > @h-vetinari: Thanks for waiting with the cut-off! [...]

> @TomAugspurger: Right, we'll keep you on the hook for future maintenance of this :)

After #20347 #20790 #20842 #20845 #20923 #21330 #21950 #22264 #22575 #22652 #22721 (-->#23011 #23163 #23167 #23582 ... and several more issues/PRs around `.str`-accessor) #22722 #22725 #23009 (-->#23061 #23065) #23187 #23443 #23723 #23725 #24044 #24045 #26605 #26607 etc., I think I can say that I upheld my end of the bargain. :)

Labels:
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas

Comments:
what other issues besides #22644 does this remove? (you can look on #6581 for what we have deprecated)
> @jreback: what other issues besides #22644 does this remove? (you can look on #6581 for what we have deprecated)

I think you might have mistyped with #22644? At least I don't see a relation. 

As mentioned in the OP:
> @h-vetinari: This executes some deprecations from the 0.23/0.24 cycles, namely following the switch from `join=None` to `join='left'` (following #18657 / #20347), as well as the fact that list-likes *within* list-likes were deprecated (following #21950 / #22264).
Ping
thanks @h-vetinari
CLN: simplify indexing code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
thanks @jbrockmendel
DEPR: NDFrame.set_axis inplace defaults to false #27525
State: closed

https://github.com/pandas-dev/pandas/blob/76247c142893c710e970c4cf8a25d73121aa5a2b/pandas/core/generic.py#L594-L598

has been there since <del>#20164</del> #16994 (issue was #14636), part of 0.21.0.

With discussion of plans to deprecate similar functionality from set_index in #24046, it's time to make sure `set_axis` conforms to the rest of pandas in this.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
In case anyone is looking for it I think you meant to reference #16994 (the OP just points to a documentation update)
Thanks for the review
thanks @pilkibun 
cc @toobaz,  you had some sentiments relating this to the `set_index` vs. `set_axis` debate  in https://github.com/pandas-dev/pandas/pull/16994#issue-130885544 in relation to `set_index` vs. `set_axis`. I'm not what your position was, but It sounds like two years ago you were against both the little-endians *and* the big-endians ;)

... Can we haz some docs now?
@pilkibun I think I'm missing something, as I don't understand.

But related to the comment you link, I guess it is probably time to remove this warning I had introduced...

https://github.com/pandas-dev/pandas/blob/37f29e5bf967d76d76a05e84b5c9b14c9bc66f23/pandas/core/generic.py#L650

(and the subsequent reassignment).
> @pilkibun I think I'm missing something, as I don't understand.

You objected to "publicizing" `set_axis` in the docs because of its defects and also that changing the default to `inplace=False` (this PR) is necessary before it can become officially recommended. 

Now that that's done, I hope you'll push for making a decision on `set_index` vs. `set_axis`, so that we can finally document what the "official" method is.
OK, now I see where you're going.

The defects of ``set_axis`` are, as you saw, a thing of the past, and they are/were entirely unrelated to ``set_index``. You can just go on with a nice docs PR "publicizing" ``set_axis`` if you like.

The API contains a lot of "official" methods to do a lot of different things.

Please stop provoking.
TYPING: some type hints for pandas\io\common.py
State: closed


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
On hindsight, i shouldn't have used cast... i was just avoiding adding a ignore.

I think in this case, since a TypeError is raised at runtime, an ignore should have been used for the static check.
some changes so that one of the errors is now
`pandas\io\formats\format.py:911: error: Argument 1 to "buffer_put_lines" has incompatible type "Union[str, Path, IO[str]]"; expected "IO[str]"`

instead of
`pandas\io\formats\format.py:905: error: Argument 1 to "buffer_put_lines" has incompatible type "Union[str, Path, IO[Any]]"; expected "TextIO"`

still needs a `# type: ignore` anyway because of the usage of `hasattr`

small comments, @WillAyd 
Thanks @simonjayhawkins
Expanded ASVs for to_json
State: closed

Instead of one massive PR in #27166 I think this makes sense to break up in a few chunks. First step to get block code out of that extension is to get expanded test coverage, which I think I've done here.

After this I'll probably:

 - Convert "columns" orient to not use the block manager code
 - Convert "values"
 - Convert "records" / "index" (and by nature table) and clean up code

Might be a little bit of bloat in the middle but think that approach will ultimately be more manageable


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
lgtm. merge on green.
DOC: add documentation for read_spss(#27476)
State: closed

- [x] closes #27476
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Documentation is added for the new read_spss function. 
After the generating html in Sphinx, it looks like this: 

IO page top:
<img width="766" alt="IO page" src="https://user-images.githubusercontent.com/6498024/61973737-c70bf580-afb2-11e9-95cf-499efc17e807.png">
SPSS description: 
<img width="774" alt="SPSS description" src="https://user-images.githubusercontent.com/6498024/61973739-ca9f7c80-afb2-11e9-8909-1638ad7a70d1.png">
API section:
<img width="761" alt="SPSS API" src="https://user-images.githubusercontent.com/6498024/61973743-cd01d680-afb2-11e9-9fca-be0a06128049.png">


Labels:
name: Docs

Comments:
Thanks @yanglinlee - this is a great update (and glad you learned something new)
CLN: Centralised _check_percentile
State: closed

- Fixes  #27559
- Moved the _check_percentile method on NDFrame to algorithms as
check_percentile.
- Changed the references to _check_percentile in pandas/core/series.py
and pandas/core/frame.py

- [x] closes #27559
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Can you also update the groupby reference from the OP?
> See above comment

Sorry I couldn't figure out what it was referring to. I didnt find any references to _check_percentile in groupby. Can you help me figure it out? 
Oh sorry misread that on my end - not there yet for groupby so I think OK for now
ok will have to backport this as #27473 depends
Hello @hedonhermdev! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-03 15:22:43 UTC
@hedonhermdev seems like there's still a linting issue, and maybe some CI failures.
@hedonhermdev can you update?
Pushing to 1.0
@hedonhermdev can you rebase
Nice idea but I think this has gone stale. @hedonhermdev please ping if you'd like to pick this back up
Hey sorry for the disappearance, if possible I would like to work on it again. Can you reopen the pull request? 
Sure - thanks for the contribution 
@WillAyd does this look good for merging? I fixed the typing error. 
Yes I did, couldn’t find any references in groupby. That will probably be added after #27559 gets resolved. 
Looks like a linting failure: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=17752

Try running `black pandas`. Instructions are in the contributing guide.
@hedonhermdev linting issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=17804

In the future, I'd recommend using pre-commit as described in https://dev.pandas.io/docs/development/contributing.html#python-pep8-black
@jbrockmendel  should I modify excel.py too? As I'm getting a linting issue because of it. 
It looks like it just wants you to revert the whitespace change in the excel.py file
thanks @hedonhermdev
DEPR: remove .ix from tests/indexing/multiindex/test_setitem.py
State: closed


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
> Minor stuff but generally looks good. Are any of the items we switched from `.ix` too `.loc` duplicated in this module? If so removal would be an option as well

@WillAyd Thanks for the review.

in these PR's, i'm basically making two choices, remove code or replace with another indexer for now.

i'm hoping that this makes the review process easier.. you can just focus on the stuff removed.

This could result in some duplication. but would consider that better, in the short term, than removing useful tests.

the priority at the moment is getting .ix out of the test suite to allow the deprecation to begin.
thanks @simonjayhawkins
TST: add regression test for MultiIndex with IntervalIndex level
State: closed

#27456  makes for a good regression test for the non-overlapping case, where there's no coverage currently. Add it to the tests.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: MultiIndex
name: Interval
description: Interval data type

Comments:
Hello @pilkibun! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-24 19:33:52 UTC
thanks @pilkibun
DOC: Harmonize column selection to bracket notation
State: closed

As [suggested by](https://medium.com/dunder-data/minimally-sufficient-pandas-a8e67f2a2428#46f9) @tdpetrou. If there is agreement about not _presenting_ the dot notation to access columns in the docu, please let me know and I'll then check the rest of the docu.

- [~] closes #xxxx
- [~] tests added / passed
- [~] passes `black pandas`
- [~] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] passes `git diff upstream/master -u -- "*.rst" | flake8-rst --diff`
- [~] whatsnew entry


Labels:
name: Docs

Comments:
+1 to generally using __getitem__ in the docs, other than the section describing dot access.

Can we think of a way to lint this, to ensure that we don’t regress on it in the future? I’m struggling to think of an easy way. A slightly harder way is to check for dot access of names that are in the columns with a python program run on the docs.  
I'd be +1 for making this change comprehensively
I will not be able to write python code to detect those occasions. Using `df(.?)\.([A-D])(\W)` for now and

- [x] checking the context.

Shall I also update old `whatsnew/*.rst` files?
I think ignore old whatsnews for now - we usually don't go back and change them
OK, I hope I didn't overlook any of these changes' context to explicitly mention "dot" notation/access/attribute.
In order to add a `whatsnew` entry, shall I re-point this PR to [branch 0.25.x](https://github.com/pandas-dev/pandas/tree/0.25.x)? If yes, I would also add a `Documentation` section there, since I'm not sure into which of the pre-defined sections to put a summary of this change otherwise.
Don’t need a whatsnew for doc changes. Also can just point to master - We can back port to 0.25.1 if needed (though not sure we would for this)
No need for back port.
OK :-) Anything else to do here, then besides fixing those CI errors? Maybe a note recommending [ bracket ] notation (or summarising its advantages)?

    The indexing operator ``[]`` provides a superset of the attribute operator ``.``,
    such as the ability to index column names using a variable, or column names
    that are the same as DataFrame methods, or contain spaces. Therefore, it is
    recommended and presented here preferably.
I'll run the checks locally with `flake8-rst doc/source/ --filename="*.rst" --format="default"` to debug all these issues.
Thanks @katrinleinweber!
You're welcome! Thanks to you though, for agreeing with this & for merging :-)
BUG: Allow ensure_index to coerce nan to NaT with numpy object array and tz Timestamp
State: closed

- [x] closes #27011
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Timezones
description: Timezone data dtype
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @mroeschke
STY: add pandas._typing to the known_pre_core block for isort
State: closed

closes #27534


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Did you run isort after this change?  I expect imports will be shuffled
No i didn't run isort after the change
You’ll need to do that in order to get the CI to pass
hmm may need to put this on hold while the dependencies in pandas._typing get resolved.  ATM it isn't strictly correct to call it pre-core
the error i am getting is [here](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=14940&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba&taskId=e3717827-2a2f-5148-8ad4-7900d2171f6e&lineStart=36&lineEnd=38&colStart=1&colEnd=1)
 It looks like isort is failing on `pandas/io/formats/formats.py` but then I tried running isort on that file on my local system it seems to pass the test.
How are you running this locally? It does appear that _typing is after core in that module so failure seems legit
@WillAyd I first ran `isort --recursive --check-only pandas` and then after seeing failing import on `formats.py`, I ran `isort pandas/io/fomats/formats.py`.
@sameshl can you merge master?
@WillAyd Do you want me to merge master into my branch? Isn't rebasing a better option?
Sorry for late reply as I've been on vacation for the past few weeks but yes we typically merge master instead of rebasing
pls merge master (you can rebase if you want, but merging is simpler); we squash on final merge anyways so it doesn't mater. ping on green.
closing as stale, if you want to continue, pls ping (and merge master)
REF: implement module for shared constructor functions
State: closed

`core.construction` intended for code to be shared between `pd.array`, `Series.__init__`, and `Index.__new__`.  The module should not need to be internals-aware.

This only moves array and extract_array to keep the diff contained.  The next step would be to move internals.construction.sanitize_array, which is used in a number of non-internals places and satisfies the not-internals-aware condition.

Several functions from core.dtypes.cast would also make more sense here than in their current locations.

<b>update</b> moved sanitize_array too, and updated imports

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
@WillAyd help with mypy complaints?
I think this is just starting to validate `_shared_docs` in pandas.core.generic now. Can probably just annotate that as `Dict[str, str]` to resolve (off the top of my head - didn't look deeply)
mypy fixed, thanks
Revert CI changes from 27542
State: closed

- [X] closes #27546
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Think this was all a perfect storm of temporary issues, so let's see what reverting yields. 

Note: didn't revert the sudo tag removal on purpose

Labels:
name: CI
description: Continuous Integration

Comments:
thanks @WillAyd
BUG: Fix interpolate ValueError for datetime64_tz index
State: closed


- [ x ] closes #27548 
- [ x ] tests added / passed
- [ x ] passes `black pandas`
- [ x ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ x ] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Timezones
description: Timezone data dtype

Comments:
looks ok, ping when changed & green.
@jreback I've updated to use the tz_naive_fixture and checks are green
thanks @alorenzo175
COMPAT: remove Categorical pickle compat with Pandas < 0.16
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Removes Categorical pickle compat with Pandas < 0.16.

I've added a ``categorical.pickle`` file to replace the deprecated file. It was create like this:

```python
>>> cat = pd.Categorical(['a', 'b', 'c', 'd'])
>>> pickle.dump(cat, open(<filepath>, 'wb'))
```


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
why close?  ive been looking forward to seeing these big chunks of pickle compat code removed
Sorry, I saw there were errors, and don't have time today to figure them out and didn't want to leave a failing PR open too long.

I'll open the PR again, when I figure out the problem.
Reopened and tests passed.

@jbrockmendel , agree with these changes?
Small question, otherwise LGTM
I’ve renamed the file.
@jbrockmendel ? Ok to pull in?
pickle compat is way outside my area of expertise.  @jorisvandenbossche?
@topper-123 can you merge master on this to be safe? If no other feedback in 2 days let's get this merged
I've rebased.
Thanks @topper-123 
There are other <0.16 pickle tests out there.  Can we get rid of those too?  `tests.tseries.test_offsets.test_pickle_v0_15_2` is a blocker for cythonizing DateOffset.
I think so - we are only guaranteed backwards compatible to pandas 0.20.3 as of the 0.25 release

see #27082
TST: label-based indexing fails with certain list indexers in case of…
State: closed

… mixed integers/strings columns names

- [ ] closes #14836
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [n/a ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @simonjayhawkins
xfail to fix CI
State: closed

to keep the wheels running while I troubleshoot this

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
sure
Travis fail looks like its caused by something different, but I've also seen that in a bunch of recent builds. 
Restarted the failing Mac build. We can debug that separately.
thanks @jbrockmendel 
tried building on raspi so I could troubleshoot the 32bit build, but the build process segfaulted on the first gcc call
Is it 32-bit specific? https://github.com/TomAugspurger/pandas-docker may be helpful. `git clone` pandas in that directory and `make pandas32` should give you a docker image with a 32-bit pandas inside.
> Is it 32-bit specific?

No, thats just one of several builds that were failing, and I'm tying to find any of them that I can reproduce.
CLN: simplify join take call
State: closed

We can track down the dtypes that get passed to `_get_result_indexer`, and since it is always int64/int64, we can reduce the call to `algos.take_nd` down to a direct cython call.

This removes the dependency of this module on non-cython parts of the code

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
sure
CLN: remove arrmap, closes #27251
State: closed


Labels:
name: Clean

Comments:
This removes the dependency of this module on non-cython parts of the code
https://github.com/pandas-dev/pandas/issues/27251#issue-464667280 was it not feasible to just use np.vectorize? is there is a perf impact?
> feasible to just use np.vectorize?

Probably.  My understanding was that is just a fancy for-loop.  Am I missing something?

> is there is a perf impact?

Haven't measured.  On the one hand we get rid of a call to lib.maybe_convert_objects (and a semi-expensive runtime import), and on the other we do a python-space list comprehension.  My assumption is that this isn't called with particularly large arrays for the quantile arg.

One alternative would be to keep arrmap_float64 and specialize it given that we know the output will be float64.  We'd still get to remove the maybe_convert_objects call, which is what I'm most excited about.

> > feasible to just use np.vectorize?
> 
> Probably. My understanding was that is just a fancy for-loop. Am I missing something?

so according the numpy docs, The vectorize function is provided primarily for convenience, not for performance. so I guess that using a list comprehension is a sensible approach. 

> Haven't measured. On the one hand we get rid of a call to lib.maybe_convert_objects (and a semi-expensive runtime import), and on the other we do a python-space list comprehension. My assumption is that this isn't called with particularly large arrays for the quantile arg.
> 
> One alternative would be to keep arrmap_float64 and specialize it given that we know the output will be float64. We'd still get to remove the maybe_convert_objects call, which is what I'm most excited about.

i'll leave that to you to decide if its worth further investigation. 

@simonjayhawkins yeah we talked about this in a previous issue. *even* if this was a measureable slowdown, would still take it in reducing complexity.
thanks @jbrockmendel
BUG: Fix fields functions with readonly data, vaex#357
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-23 23:47:57 UTC
looks fine, can you add a note. ok with backporting if you want.
note added
thxs