BrandonFors/pandas-github-issues huggingface dataset:
DEPR: Index.__and__, __or__, __xor__ behaving as set ops
State: closed

- [x] closes #36758
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xref #30228

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
> can u link to the deprecation removal issue

done

> also pls test the inplace ops produce a warning

we dont support inplace ops for these
lgtm merge on greenish
thanks @jbrockmendel 
hmm this may have broken some tests: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=46712&view=logs&j=eab14f69-13b6-5db7-daeb-7b778629410b&t=ce687173-08c6-5301-838d-71b2dda24510

this is a just rebased pr
> hmm this may have broken some tests 

just opened #37587 for this
PERF: ensure_string_array with non-numpy input array
State: closed

Currently, if the input array to `ensure_string_array` is a python object (e.g. an `ExtensionArray`), the function will constantly switch between python and cython level code, which is slow.

This PR fixes that by ensuring we always have a numpy array, avoiding the trips to python level code.

``` python
>>> n = 50_000
>>> cat = pd.Categorical([*['a', pd.NA] * n])
>>> %timeit cat.astype("string")
447 ms ± 11.9 ms per loop  # master
5.43 ms ± 80.5 µs per loop  # this PR
```

xref #35519, #36464.

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data
name: Categorical
description: Categorical Data Type

Comments:
Specifically for Categorical, we can actually get something (possibly) faster by special casing it to astype the categories only:

```
In [16]: n = 50_000
    ...: cat = pd.Categorical([*['a', pd.NA] * n])

In [17]: %timeit cat.astype("string")
313 ms ± 8.74 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [18]: %timeit cat.categories.array.astype("string").take(cat.codes, allow_fill=True)
1.86 ms ± 50.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

(this is also something that can work in general for other dtypes as well)

The underlying problem is in the string conversion, so should be fixed there:

```python
In [1]: n = 50_000
...:    cat = pd.Categorical([*['a', pd.NA] * n])

In [2]: %timeit pd.arrray(cat, dtype="string")  # same underlying problem 
508 ms ± 9.6 ms per loop  # master
6.03 ms ± 184 µs per loop  # this PR
```

I think the failures are unrelated. I'll update later with the whatsnew and timing runs, if needed.
> pls either add a whatsnew (or since this is similar to others that that you have recently done for this type of function, ok to just add this PR number there). ping on ngreen.

I've added this in the whatsnet, but this actually fixes a perf. regression coming from #36464 (the change pandas/_libs/lib.pyx).
> Specifically for Categorical, we can actually get something (possibly) faster by special casing it to astype the categories only:
> 
> ```
> In [16]: n = 50_000
>     ...: cat = pd.Categorical([*['a', pd.NA] * n])
> 
> In [17]: %timeit cat.astype("string")
> 313 ms ± 8.74 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> 
> In [18]: %timeit cat.categories.array.astype("string").take(cat.codes, allow_fill=True)
> 1.86 ms ± 50.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> ```
> 
> (this is also something that can work in general for other dtypes as well)

possibly but this is a separate issue and orthogonal here.
thanks @topper-123
DOC: add searchsorted examples #36411
State: closed

DOC: add searchsorted examples #36411

Labels:
name: Docs

Comments:
Hello @Mikhaylov-yv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-25 16:50:32 UTC
Thanks @Mikhaylov-yv
BUG: Call finalize in DataFrame.unstack
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This commit adds a step to call `__finalize__` in `DataFrame.unstack`, as motivated by #28283.  Adding this step also prevents needing a step to call `__finalize__` in `DataFrame.pivot` since `DataFrame.unstack` is called before returning, so the xfail for that test was removed as well.

Labels:
name: metadata
description: _metadata, .attrs

Comments:
thanks @mzeitlin11
TST: split up test_concat.py #37243 - follows up
State: closed

* created test_datetime.py and
  split datetime/timezone/period related tests

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
This is first part, I have to create also test_categorical, test_empty, test_invalid, test_sort. I have 3 classes, TestDatetimeConcat, TestTimezoneConcat and TestPeriodConcat, leaving here TestPeriodConcat is probably best decision, because it have only 50 lines. I wasn't able to find more functions to put into it.
thanks @laffite56 

yep this looks good, keep em coming. (you an break out period is ok too)
Backport PR #37302 on branch 1.1.x (BUG: Allow empty chunksize in stata reader when using iterator)
State: closed

Backport PR #37302: BUG: Allow empty chunksize in stata reader when using iterator

Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
@jreback can either backport #37263 or remove the match argument in the test added here. any preference?
prob fix the test as want to backport this
REGR: Notebook (html) repr of DataFrame no longer follows min_rows/max_rows settings 
State: closed

- [ ] closes #37359
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Fix display logic in number of rows,
explained here https://pandas.pydata.org/docs/dev/user_guide/options.html#frequently-used-options

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@jorisvandenbossche can you please confirm it works? I am still unable to install dev environment with jupyter (even conda).
I managed to install dev env with jupyter. Looks like everything works. But please check in your environment as well.

![image](https://user-images.githubusercontent.com/41443370/97032497-237e0c00-158c-11eb-850b-27cd4471c442.png)

I can confirm this works, thanks!
also pls rebase
@ivanovmg can you merge master.

@jorisvandenbossche if any comments.
@ivanovmg can you merge master one more time and this can be merged cc @jorisvandenbossche 
> 
> 
> @ivanovmg can you merge master one more time and this can be merged cc @jorisvandenbossche

Done, it's green.
thanks @ivanovmg
CI autoupdate pre-commit versions
State: closed

Rather than manually updating versions of hooks when new ones come out, we can make a GitHub action to open a PR to do this automatically every Monday.

For this to work, we will need whoever owns the repo to [create an access token](https://github.com/technote-space/create-pr-action#github_token) and add it as secret `ACTION_TRIGGER_TOKEN` to the repo - see screenshot below for an example

![image](https://user-images.githubusercontent.com/33491632/96995872-c9972b00-1526-11eb-9c9f-0e38d754590c.png)


Labels:
name: CI
description: Continuous Integration

Comments:
looks fine to me, can you rebase
thanks @MarcoGorelli 

ok i created the above secret, lmk if this is working.
CLN Apply style to Jupyter notebook
State: closed

We apply many style checks to our source code files / Python snippets in documentation, so I've applied them to the (lone) Jupyter notebook.

Not adding anything to CI as it's arguably not worth it for just a single file

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Does this make a difference for anything?  I like uniformity as much as the next guy, but find the additional \\s make this marginally less readable.
That's a good point...I've removed the extra spaces which `black` adds between function definitions and code, but I've kept the indentation / quoting changes
Hah, you must have read "\s" as a regex for whitespace, I was referring to literal backslashes.
:rofl: indeed

Though doesn't this PR _remove_ the literal backslashes? e.g. in cell 9
```diff
- df.style.\
-     applymap(color_negative_red).\
-     apply(highlight_max)
+ df.style.applymap(color_negative_red).apply(highlight_max)
```
>    Though doesn't this PR remove the literal backslashes? e.g. in cell 9

Sure. More specifically I was looking at the single ' quotes that became \"

I'd be fine with merging this just to clear the queue. cc @jorisvandenbossche
> Sure.  More specifically I was looking at the single ' quotes that became \\"

Ah yeah, they only _look_ like they've become like that because of how git shows the diff between notebooks.

E.g. git shows this diff:
```diff
-    "df = pd.DataFrame({'A': np.linspace(1, 10, 10)})\n",
-    "df = pd.concat([df, pd.DataFrame(np.random.randn(10, 4), columns=list('BCDE'))],\n",
-    "               axis=1)\n",
+    "df = pd.DataFrame({\"A\": np.linspace(1, 10, 10)})\n",
+    "df = pd.concat([df, pd.DataFrame(np.random.randn(10, 4), columns=list(\"BCDE\"))], axis=1)\n",
```

but if you open up the notebook in e.g. VSCode then it looks like this in this branch:

---
![image](https://user-images.githubusercontent.com/33491632/98967634-9aac1d80-2504-11eb-8e35-f7996af8c3dd.png)
---

instead of this on master:

---
![image](https://user-images.githubusercontent.com/33491632/98967706-b57e9200-2504-11eb-9d63-ff191f639ea4.png)
---
closing as I don't think it's worth going back and forth for the sake of a single file, sorry for the noise 😄 
BTW, I think this is perfectly reasonable to merge. The single vs double quote making it less readable in this diff is not really an issue IMO, that's just an artifact of the json format of the notebook, and doesn't impact the use of it (notebooks source are not really human-friendly to read anyway)
:rofl: sure, reopening then
Actually looking at the code changes, there are a few examples in their of heavy method chaining that black doesn't always handle that nicely ... Not fully sure what to do with those.
> Actually looking at the code changes, there are a few examples in their of heavy method chaining that black doesn't always handle that nicely 

True...OK, let's close then, not everything needs to follow `black` - sorry again for the noise, I'll focus on more pressing issues :smile:
TST: moved file test_concat.py to folder ./concat/ (#37243)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
so after merging this PR to master, I will be able to create new PR and do follows-up? Further splitting to "test_categorial", "test_empty" require moving methods, so doing this will be better in another PR, yes?
ok, need you to run the pre-commit hooks: https://github.com/pandas-dev/pandas/pull/37360/checks?check_run_id=1297616903

and ping on green and can merge (then can do followups).
great ping on green.
Ugh, why did it crash with: 
**AssertionError: ('pandas/tests/io/parser/test_c_parser_only.py::test_parse_trim_buffers[c_low]', <WorkerController gw0>)**

Is it connected with my commit?
thanks @laffite56 

no the windows checks have been finicky lately.

pls issue followon PRs to clean this up.
CI/CLN: Add more test file linting
State: closed

Fixing use of pd.Index when Index is imported directly

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
lgtm merge on green
Thx for fixing the issue introduced through my join fix. CI run probably before you added this check the last time.
> Thx for fixing the issue introduced through my join fix. CI run probably before you added this check the last time.

NP, yeah I think it was just merged after the PR last merged master
Update sql.py to distinguish between BigInt, Integer, and SmallInt
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is to resolve #37344.  I've never contributed to Pandas before, so may need guidance to proceed.

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale

Comments:
Hello @pilotjoe! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-29 15:01:03 UTC
> Thanks @pilotjoe for the PR!
> 
> Can you add tests? That's the first thing we look at when reviewing. They would go into pandas/tests/io/test_sql.py

@arw2019 Thanks for pointing me to the tests.  I've incorporated tests for int64, int32, and int16.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
this looks ok, but is stale. pls ping if you want to continue working.
REF: extract dialect validation
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Following https://github.com/pandas-dev/pandas/pull/36852, I extracted validation of dialect into a separate function.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO CSV
description: read_csv, to_csv

Comments:
sure
DOC: Remove confusing description from `core.DataFrame.iterrows`
State: closed

Current [`core.DataFrame.iterrows`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iterrows.html) is as follows
![image](https://user-images.githubusercontent.com/5734732/96833205-00176c00-147b-11eb-8dc3-6371b2718e76.png)

However, `core.DataFrame.iterrows` actually returns an iterator which yields `(index, data)`, not `(index, data, it)`.
This PRs remove such confusing description like [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iteritems.html?highlight=yields]

- [ ] closes #xxxx (no issues found)
- [ ] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
lgtm whenever green. not sure CI failure was related but restarted in any case
Thanks @TKNGUE !
BUG: IntervalIndex.take without fill_value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

`IntervalIndex.take([0, 1, -1])` without explicitly passing fill_value should not fill the -1 as NA.

Fixing this allows us to use the base class `take` for `IntervalIndex.take` (the original motivation)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Interval
description: Interval data type

Comments:
bug fix right? can you add a whatsnew
mothballing to clear the queue while i troubleshoot slow tests locally
DOC: Add protocol value '5' to pickle #37316
State: closed

- [X] closes #37316


Labels:
name: Docs

Comments:
thanks @botplex 
The issue is still there as visible here: [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_pickle.html](url)
I should have changed the code in the branch named 1.1.x [https://github.com/pandas-dev/pandas/blob/v1.1.3/pandas/core/generic.py](url) instead of master
BUG: Index._id
State: closed

Fixes #37213.


Labels:
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @topper-123 
Indeed thanks @topper-123!
This might have resulted in a slowdown, see https://pandas.pydata.org/speed/pandas/#multiindex_object.Equals.time_equals_non_object_index?commits=f162aacc-4f5e0e06&Cython=0.29.21&python=3.8 (this commits looks most related in the relevant commit range). Now, the benchmark is only about small numbers though, so it is still fast. It's only a microbenchmark.

From a quick profile, it might be that in this case the `com.any_is_none` is giving a lot of overhead compared to the overall speed of the method.
TST/REF: misplaced frame.indexing tests
State: closed

I've been saving the indexing tests for last-ish because they are some of the hardest to organize consistently.  This starts in on that.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Looks like some unrelated commits are polluting the diff, hopefully thatll sort itself out
think i merged in a weird order, can you rebase
CLN: always rebox_native in DatetimeLikeArray
State: closed


Labels:
name: Datetime
description: Datetime data dtype
name: Clean

Comments:
The ASV suite marked this as a regression, opened #37462
ill take a look, thanks
> ill take a look, thanks

@jbrockmendel will you get a chance to look into this before the 1.2.0 release?
that's sometime today?  ill start right now
no. the actual 1.2.0 release. we will be backporting any regression fixes between 1.2.0rc0 and 1.2.0
Oh then for sure.
CI: troubleshoot travis-36-cov on 1.1.x
State: closed

closes #37309

Labels:
name: CI
description: Continuous Integration

Comments:
it looks like this build is gone; is this PR still actionable?
> it looks like this build is gone; is this PR still actionable?

I thought I had got somewhere after trying to debug locally #36779. Was taking a stab in the dark on most of these commits. Not found any authoritative resources on how to debug conda solve issues.

The issue could be closed if 1.1.4 is the last in the 1.1.x series.

I'll close this PR for now to clear the queue since I doubt I'll waste any more time on this before the 1.1.4.

Can reopen if planning a 1.1.5 or maybe move the coverage build to 3.7 (same as master)
it appears that the xfail condition on test_apply may be suspect, see discussion in https://github.com/MacPython/pandas-wheels/pull/64
all green here. will merge shortly if no objections. this is against 1.1.x directly. @jreback 
sure

we still have other 3.6 testing on 1.1.x right?
> we still have other 3.6 testing on 1.1.x right?

plenty. 6 envs in azure and 1 in travis
great merge away
Backport PR #37198 on branch 1.1.x (BUG: Regression in Resample.apply raised error when apply affected only a Series)
State: closed

Backport PR #37198: BUG: Regression in Resample.apply raised error when apply affected only a Series

Labels:
name: Resample
description: resample method
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Was this introduced through my fix or did I use something, which is only working on master and not 1.1.4?
> Was this introduced through my fix or did I use something, which is only working on master and not 1.1.4?

not got to the bottom of it yet. see https://github.com/pandas-dev/pandas/issues/36951#issuecomment-713623620
> any ideas where changes in master could have corrected this

Not off the top of my head, no.
> > any ideas where changes in master could have corrected this
> 
> Not off the top of my head, no.

appears to be #35792

https://github.com/simonjayhawkins/pandas/runs/1305349161?check_suite_focus=true
looks like #35792 can be backported without any conflicts (except release note). added here for ci checking. would backport independently if moving forward with this. (whatsnew would need to be changed on master)

depending on tests passing, 3 options.

1. don't backport #37198 (see https://github.com/pandas-dev/pandas/issues/36951#issuecomment-713623620 this regression is from 0.25.3 to 1.0.0, we could decide not to backport.)

2. backport #37198 and change tests to not check frequency of index on backport only.

3. backport #37198 and #35792 (which fixes bug for Categoricalndex and preserves freq in DatetimeIndex following DataFrameGroupby)

@jreback OK with option 3 https://github.com/pandas-dev/pandas/pull/37305#issuecomment-716486341

will start on a separate backport PR #35792 and PR to move the whatsnew
TST: correct parquet test expected partition column dtype for pyarrow 2.0
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/35814, but now incorporating the changes in pyarrow 2.0

cc @alimcmaster1 

Also related to https://github.com/pandas-dev/pandas/issues/37286 and https://github.com/pandas-dev/pandas/pull/37296/ (timezone related failures), but doing this fix separate as it's not something that needs to be reverted.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Parquet
description: parquet, feather

Comments:
the travis-36-cov builds regularly fails to build with `The build has been terminated` on 1.1.x so haven't yet seen this failing on 1.1.x but probably makes sense to backport this.
thanks @jorisvandenbossche 
> the travis-36-cov builds regularly fails to build with `The build has been terminated` on 1.1.x so haven't yet seen this failing on 1.1.x but probably makes sense to backport this.

yeah this is happening regularly, odd
@meeseeksdev backport 1.1.x
> > the travis-36-cov builds regularly fails to build with `The build has been terminated` on 1.1.x so haven't yet seen this failing on 1.1.x but probably makes sense to backport this.
> 
> yeah this is happening regularly, odd

i'll open a dedicated issue so that references to this can be collated.