BrandonFors/pandas-github-issues huggingface dataset:
ENH: Add progress_bar_type argument in read_gbq
State: closed

- [x] closes #29857 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Should we just add kwargs here to pass along instead? I think there are a few other keywords that aren't specifically stated; probably easier to just pass through
> Should we just add kwargs here to pass along instead? I think there are a few other keywords that aren't specifically stated; probably easier to just pass through

no 

explicit is better than implicit 
cc @tswast 

does this require a min version?
thanks @charlesdong1991 

@tswast also happy to take a new min version of pandas-gbq if you need / want for 1.0
TST: Fix locale test that fails in GitHub actions
State: closed

xref #29715

Looks like in GitHub actions this test is receiving an empty string for the environment variable `LOCALE_OVERRIDE` instead of `None`. Not sure why, but I guess they should be equivalent, and we don't want to fail when an empty string is provided.

The change also makes the test return a descriptive error message if an unexpected value is set in that environment variable.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
This won't be needed when/if #29883 is merged, closing.
STY: 'open()' and 'close()' to 'with' context manager
State: closed

- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend ideally you could push this change upstream to pyperclip itself as well.
TYP: Typing annotations in clipboard
State: closed


- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hmm +/-0 on this one. Actually might be nice to do the annotations upstream in pyperclip and vendor down as appropriate rather than us trying to do here
agree with @WillAyd that should vendor these, but ok to do both; pls rebase
Thanks @MomIsBestFriend really appreciate the PRs. As mentioned if you could do this upstream in Pyperclip would be great as well
MAINT: Fix grammar in user_guide/scale.rst
State: closed

Follow-up to:

https://github.com/pandas-dev/pandas/pull/29811

Labels:
name: Docs

Comments:
@pandas-dev/pandas-core : This PR is a simple doc change, but we seem to have some Azure failures here that were present on `master` for a good several commits it seems.
It's an excel related example that is failing (and there was a minor openpyxl version change), opened https://github.com/pandas-dev/pandas/issues/29854 for this
thanks @gfyoung
PERF: perform reductions block-wise
State: closed


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-27 20:20:58 UTC
Consolidated the conditions under which we go through the block-wise path.  In a follow-up, this will allow for significant cleanup of the non-blockwise path (i.e. existing _reduce code) because the set of cases it has to handle is cut down.
i hypothesize that after this and #30416 we'll be able to get rid of quite a bit of the existing special case handling in DataFrame._reduce.
docbuild failure looks unrelated
@jreback @TomAugspurger ideally id like to get this and #29941 in before the RC.
> Any user-facing changes that need a whatsnew?

None that I've identified.

> Do you think we have sufficient test coverage here?

It seems pretty solid.

> Planning to keep these asserts in?

Fine by me either way.

ok by me (assuming you are going to back and de-gross / consolidate this code at some point).
> assuming you are going to back and de-gross / consolidate this code at some point

The code added here is is pretty de-grossed.  Some of the assertions could be removed to make it less verbose, but unless we get 2D EAs (which is a windmill ive mostly stopped tilting at), this is about as good as it gets.

That said, inside DataFrame._reduce below the code introduced here is a bunch of really gross code that we _can_ clean up after this.
k thanks
BUG: fix datetimes.should_cache() error for deque (GH 29403)
State: closed

`itertools.islice()` should be used to get slice of a deque. `itertools.islice()` also can be used (and is efficient) for other collections. So `unique(arg[:check_count])` was replaced with `set(islice(arg, check_count))` which is also much faster on test data. 

- [x] closes #29403
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Hello @nathalier! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-14 16:01:44 UTC
pls run `black pandas/tests/indexes/datetimes/test_tools.py` and `isort pandas/tests/indexes/datetimes/test_tools.py`
> pls run `black pandas/tests/indexes/datetimes/test_tools.py` and `isort pandas/tests/indexes/datetimes/test_tools.py`

@jbrockmendel Thanks a lot for help!



@jreback Now it's green, but possibly some of added tests are redundant.
thanks @nathalier
DOC: README.md wrong minimum versions
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
How did it fail the tests?
test failures are unrelated.  LGTM
Thanks @MomIsBestFriend !
DEPR: Remove weekday_name
State: closed

- [x] xref #18164
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
LGTM
lgtm, can you rebase
@mroeschke CI should pass if you rebase
All green.
Thanks @mroeschke
DEPR: Change raw kwarg in rolling/expanding.apply to False
State: closed

- [x] xref #6581
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Window
description: rolling, ewma, expanding

Comments:
small comment, not a blocker.  LGTM
thanks @mroeschke can you update the deprecations issue as well (move to completed)
Fix mypy errors for pandas\tests\series\test_operators.py
State: closed

part of #28926

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

The errors were:
```
pandas/tests/series/test_operators.py:824: error: "Type[Series]" has no attribute "div"
pandas/tests/series/test_operators.py:824: error: "Type[Series]" has no attribute "rdiv"
```
The problem is that these methods are only added at RunTime, so mypy can't see them. I'm not sure if my solution is the best way to silence these errors though.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> The problem is that these methods are only added at RunTime, so mypy can't see them. I'm not sure if my solution is the best way to silence these errors though.

they are added in add_flex_arithmetic_methods which is passed the class as a parameter. They could therefore be specific to a class and so adding them directly in Series (and adding similar in DataFrame separately) seems reasonable to me.
@simonjayhawkins the build is green again
Thanks @SaturnFromTitan
DEPR: remove statsmodels/seaborn compat shims
State: closed

xref #18264, #16971

Looks like seaborn updated their usage in https://github.com/mwaskom/seaborn/pull/1241
and statsmodels https://github.com/statsmodels/statsmodels/pull/3618 both in 2017.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
Thanks!
CLN:Added annotations to functions
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
im not sure if we have a convention for `__setstate__` or `strptime` cases, the rest look good
> im not sure if we have a convention for `__setstate__` or `strptime` cases, the rest look good

Would you like me to remove them, at least for this PR. and open a separate one containing only them?


> Would you like me to remove them, at least for this PR. and open a separate one containing only them?

Either that or wait for @WillAyd to chime in
Thanks @MomIsBestFriend
PERF: faster categorical ops for equal or larger than scalar
State: closed

np.nan is always encoded to -1 in a Categorical, so encoded smaller than all other possible values.

So, if we're checking if a Categorical is equal or larger than a scalar, it is not necessary to find locations of -1, as those will always be False already.

### Performance

```python
>>> n = 1_000_000
>>> c = pd.Categorical([np.nan] * n + ['b'] * n + ['c'] * n,
                       dtype=pd.CategoricalDtype(['a', 'b', 'c']), ordered=True)
>>> %timeit c == 'b'
9.79 ms ± 32.4 µs per loop  # master
3.94 ms ± 38.7 µs per loop  # this PR
# the improvement isn't available for less than etc. ops
>>> %timeit c <= 'b'
10.1 ms ± 44.1 µs per loop  # both master and this PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
thanks @topper-123
TYP: Added typing to __eq__ functions
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> Should `other` always be typed as `Any`?

looking at typeshed, probably should be typed as `object`.

Using anything other than `object` (or `Any`) will report ` error: Argument 1 of "__eq__" incompatible with supertype "object"`, see https://stackoverflow.com/questions/37557411/why-does-defining-the-argument-types-for-eq-throw-a-mypy-type-error
The tests failed exactly where it failed [here](https://github.com/pandas-dev/pandas/pull/29844) which it should not fail there, all that was changed was the ```README.md``` file. is it possible that it's a false positive?
thanks @MomIsBestFriend
REF: Create _lib/window directory
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

xref https://github.com/pandas-dev/pandas/pull/29428#issuecomment-555882765

Puts `pandas/_libs/window.pyx` and `pandas/_libs/window_indexer.pyx` into their own directory as
`pandas/_libs/window/aggregations.py` and `pandas/_libs/window/indexers.pyx` respectively.

Labels:
name: Refactor
description: Internal refactoring of code
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke
DOC: Correct misuse of term high-cardinality in docs.
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
cardinality seems like a weird term to use here.  can you suggest a clearer one?
It's indeed a very unclear term if you don't know it, but it's also commonly used in the machine learning community for this purpose (so it's certainly a correct term, at least with the fix here). 
Since it is explain afterwards, I am fine with keeping it. But maybe switching it around (first the more easy explanation, then the technical term might be better)? Eg `This is especially true for text data with only few unique values (low-cardinality data).`
I agree that _cardinality_ is the correct term here and certainly expect the target audience to be familiar with its proper usage, but I've reworded it as suggested for the benefit of those that might not be familiar with the term.
LGTM
@gfyoung can you commit your suggestion and merge.
@jreback : This PR didn't allow for maintainers to make changes.

I will merge this and follow-up with the grammar fix.
Thanks @ebrassell !
CLN: avoid catching Exception in io.pytables
State: closed

Last one in this file for a bit, scout's honor.

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Clean

Comments:
thanks @jbrockmendel 

btw you can blame me for most of this code :->
> btw you can blame me for most of this code :->

One that I'm having trouble with you might be able to help: I'd like to make data_columns, index_axes, non_index_axes, etc into properties, but it isn't clear to me what the underlying logic of how they are defined is.  maybe it depends on version via create_for_block?
> > btw you can blame me for most of this code :->
> 
> One that I'm having trouble with you might be able to help: I'd like to make data_columns, index_axes, non_index_axes, etc into properties, but it isn't clear to me what the underlying logic of how they are defined is. maybe it depends on version via create_for_block?

data_columns is defined when writing a table (user passed + index columns from the passed frame); or is index columns on the passed frame if not passed 

the others are computed from this, meaning 

columns in the stored frame + index columns  = index_axes = data_columns + non_index_axes

HTH
DEPR: Series.to_csv signature change
State: closed

@toobaz you introduced this deprecation in #21896.  Can you confirm this enforces the deprecation in the intended way?

Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jbrockmendel  Yes, this all seems correct to me. I just wonder whether this should be removed too:

https://github.com/pandas-dev/pandas/blob/35029d20d53a1af57823759039f57d8ff09736c3/pandas/tests/io/test_compression.py#L17
updated tests per both requests
thanks! love the removals!
CLN: remove legacy datetime support in io.pytables
State: closed

also removed the `inferred_type == "unicode"` branch, as `lib.infer_dtype` never returns unicode anymore

- [x] closes #19475
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
happy to get rid of this!
DEPR: setting DTI.freq, DTI.offset, DTI.asobject
State: closed

I think asobject should have been done as part of #29720.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Clean

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-22 22:30:40 UTC
lgtm. can you open an issue about the 'less smelly way' to update the freq internally.
changed description of Parse_dates in Read_excel
State: closed

- [ ] closes #29217 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @alexander135! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/io/excel/_base.py`](https://github.com/pandas-dev/pandas/blob/e87bd10397d3335b3cb5550b18a4404febf2ab65/pandas/io/excel/_base.py):

> [Line 169:89](https://github.com/pandas-dev/pandas/blob/e87bd10397d3335b3cb5550b18a4404febf2ab65/pandas/io/excel/_base.py#L169): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (91 > 88 characters)
BUG: Fix melt with mixed int/str columns
State: closed

- [X] closes #29718
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
LGTM
Thanks for fixing so quickly!
Great thanks @jschendel
CLN: remove LegacyFoo from io.pytables
State: closed


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Clean

Comments:
> i would add a notet o what’s new as well

ok.  saying what?  its not actually clear to me what these unused classed _did_ that we are removing support for
just say that we removed legacy HDf5 support which was for pandas files written in version xxx (have to see if can figure when)

it’s  likely pretty old though like 0.11 or something
added a note, couldn't be too specific; looking through the blame i didnt see anything except "legacy", and the last relevant change was from 2013 with no attached PR
thanks!
DEPS: Unifying version of pytest-xdist across builds
State: closed

We use `pytest-xdist>=1.21` in all builds except this one. Using the same for the 3.8 build, which was introduced in parallel as the standardization of dependencies.

xref: https://github.com/pandas-dev/pandas/pull/29678#discussion_r348873095

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
makes sense
thanks
DOC: Updating documentation of new required pytest version
State: closed

Follow up of #29678

@jreback I think `show_versions` gets the version by importing the module. Correct me if I'm wrong, but I didn't see anything to update there.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
actually i think u need to add pytes in the min_versions lost in show_versions
I guess this is what you mean. Let me know if it's not.
yep thanks @datapythonista
CLN: typing and renaming in computation.pytables
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
this is fine
REF: avoid returning self in io.pytables
State: closed

Another small step towards making these less stateful

Labels:
name: Refactor
description: Internal refactoring of code
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@jreback it looks like the LegacyFixed methods are not covered in the tests.  Any chance we can rip out those classes?
also read_sparse_intindex, write_sparse_intindex, read_block_index, write_block_index, so we might not need BlockIndex/IntIndex (which are imported from arrays.sparse)?
so i think i can blow away all Legacy stuff

i think these read the legacy_hdf dir in data/

but these are quite old iirc


also exception on line 635 can remove that is some versions prior to what we support now
great, i'll make a PR specific to ripping that stuff out.  hopefully that will snowball into more simplfications
thanks
CLN:F-string in pandas/_libs/tslibs/*.pyx
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

ref https://github.com/pandas-dev/pandas/issues/29547

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend
Reenabled no-unused-function in setup.py
State: closed

This enables a flag that was disabled 4 years ago in https://github.com/pandas-dev/pandas/commit/e9fde88c47aab023cafa6c98d5654db9dc89197f but maybe worth reconsidering, even though we don't fail on warnings for extensions in CI

This yielded the following on master:

```sh
pandas/_libs/hashing.c:25313:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
pandas/_libs/hashing.c:25318:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
pandas/_libs/hashtable.c:63342:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
pandas/_libs/hashtable.c:63347:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
pandas/_libs/parsers.c:21453:18: warning: unused function '__pyx_f_6pandas_5_libs_7parsers__string_box_factorize' [-Wunused-function]
pandas/_libs/tslibs/conversion.c:33247:20: warning: unused function '__pyx_memview_get_object' [-Wunused-function]
pandas/_libs/tslibs/conversion.c:33252:12: warning: unused function '__pyx_memview_set_object' [-Wunused-function]
pandas/_libs/tslibs/nattype.c:3496:18: warning: unused function '__pyx_pw_6pandas_5_libs_6tslibs_7nattype_4_NaT_1__hash__' [-Wunused-function]
pandas/_libs/src/ujson/python/objToJSON.c:403:14: warning: unused function 'PyIntToINT64' [-Wunused-function]
```

I'm not sure what the `memview_set_object` and `memview_get_object` complaints are coming from just yet, but the rest are cleaned up in this PR

Labels:
name: Clean

Comments:
looks like a nice cleanup
thanks @WillAyd
DEPR: MultiIndex.to_hierarchical, labels
State: closed


Labels:
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
Looks like pyarrow is still using the deprecated usage.  We aren't pinning the version, and it looks like the py36_macos azure build is using 0.9.0 (which is also what we have listed in `compat._optional`).  Testing locally, it looks like 0.12.0 is the first minor release that doesn't usage the deprecated usage.

Anyone mind bumping the supported pyarrow min version to 0.12.0?  cc @jorisvandenbossche 
ok with bumping pyarrow
bumped pyarrow min version to 0.12.0 and updated usages
The azure mac-py36 build has numpy=1.13.3 pinned but pyarrow0.12.0 requires numpy>= 1.14.0

Should we bump the numpy pin in this build or remove pyarrow?
or we could just bump the numpy min version.  1.13.3 was released 2017-09-29
we should bump numpy anyhow to 1.14/1.15

let’
for now bumped the pin in this build to 1.14, can bump the numpy min ver in a dedicated PR
Bumping pyarrow to 0.12 is certainly OK I think!
thanks, 2 small followons.
TST: add test for ffill/bfill for non unique multilevel
State: closed

- [x] closes #19437
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Great thanks @ganevgv - loving all of the PRs
TST: add tests for period resample sum with min_count
State: closed

- [x] closes #19974
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Resample
description: resample method

Comments:
can you merge master and will look again
@jreback - rebased onto latest master.
@ganevgv - sorry can you merge master again - there are some conflicts
superseded by #31196