BrandonFors/pandas-github-issues huggingface dataset:
DOC: Fix heading capitalization in doc/source/whatsnew - part3 (#32550)
State: closed

- [ ] Modify files v0.8.0.rst, v0.8.1.rst, v0.9.0.rst, v0.9.1.rst, v010.0.rst, v0.10.1.rst, v0.11.0.rst, v0.12.0.rst, v0.13.0.rst, v0.13.1.rst

Labels:
name: Docs

Comments:
thanks @cleconte987
ENH: update feather IO for pyarrow 0.17 / Feather V2
State: closed

Upcoming pyarrow 0.17 release will include an upgraded feather format. 

This PRs updates pandas for that, more specifically ensures the new keywords can be passed through (the basics should keep working out of the box, since the public API did not change), and small update to the tests 

Labels:
name: IO Parquet
description: parquet, feather

Comments:
Hi,  is feather v2 now supported by pandas?  It seems tag 1.0.4 but I cannot find it in the release note,  thanks! 

the current to_feather() in pandas 1.0.5 seems does not support the compression that was introduced in feather V2 as well.

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_feather.html
this is in 1.1
@noklam Feather V2 is already supported by pandas 1.0.4, as long as you have pyarrow>=0.17 installed. 
But only the default behaviour, so no additional keywords (that is what this PR is enabling: correctly passing through keywords such as `compression`)
Thanks!  Got it.  Looking forward to release 1.1 😀
API: More permissive conversion to StringDtype
State: closed

- [x] closes #31204
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is a proposal to make using ``StringDtype`` more permissive and be usable inplace of ``dtype=str``.

ATM converting to StringDtype will only accept arrays that are ``str`` already, meaning you will often have to use ``astype(str).astype("string")`` to be sure not to get errors, which can be tedious. For example these fail in master and work in this PR:

```python
>>> pd.Series([1,2, np.nan], dtype="string")
0       1
1       2
2    <NA>
dtype: string
>>> pd.array([1,2, np.nan], dtype="string")
<StringArray>
['1', '2', <NA>]
Length: 3, dtype: string
>>> pd.Series([1,2, np.nan]).astype("string")
0     1.0
1     2.0
2    <NA>
dtype: string
>>> pd.Series([1,2, np.nan], dtype="Int64").astype("string")
0       1
1       2
2    <NA>
dtype: string
```

etc. now work. Previously the above all gave errors.

Obviously tests and doc updates are still missing, but I would appreciate feedback if this solution looks ok.

Labels:

Comments:
@topper-123 is there a reason you closed this?
I had only tested against a subset of the tests, and it didn't pass all tests when I pushed to upstream.

In general, I've found that type conversions in Pandas are quite complex, so I'm still working on getting something to work that is more than a temporary solution.
DOC: Fix EX01 in DataFrame.duplicated
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

```
################################################################################
################################## Validation ##################################
################################################################################

Labels:
name: Docs

Comments:
Thanks @farhanreynaldo - nice improvement
WIP: Timestamp/DTA match stdlib tzawareness-compat behavior
State: closed

- [x] closes #28507
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I'm not super-happy with the implementation adding `op` kwarg to check_compatible_with and assert_tzawareness_compat, looking for something cleaner.

Labels:
name: Timezones
description: Timezone data dtype

Comments:
Mothballing to clear the queue
BUG: DatetimeIndex.astype matches Series.dtype with datetime tz dtype
State: closed

- [x] closes #33401
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
As @jbrockmendel mentioned, this may not be the idea behavior in the first place. Going to close so we can discuss further in the issue
REF: call pandas_dtype up-front in Index.__new__
State: closed

This leaves us in position to drop in dtype-only is_foo_dtype checks

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Clean
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
seems fine, i imagine this has a tiny perf hit, which is likely acceptable, can you do a quick measurement.
> i imagine this has a tiny perf hit

yah, i think this only becomes a win once the dtype-only checks go in
BUG: to_period() freq was not infered
State: closed

- [x] closes #33358
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

---

When running ```pytest -q --cache-clear pandas/tests/ -k "period"``` some tests in ```pandas/tests/indexing/``` are failing, but when running ```pytest -q --cache-clear pandas/tests/indexing/``` __all__ the tests are passing (on my local machine), so I figured and debug from here, thoughts?

Labels:
name: Frequency
description: DateOffsets
name: Period
description: Period data type

Comments:
> thoughts?

we cant rely on grepping to get all the relevant tests.  i recommend `pytest pandas/tests --skip-slow --skip-db`, takes about 15 minutes for me.

Look at the failing tests, choose one and start executing it step-by-step in the REPL

```
off = "QS"
rng = pd.date_range("01-Jan-2012", periods=8, freq=off)

>>> prng = rng.to_period()   # <-- raises, so recurse into the `to_period` method
```

looks good, can you add a whatsnew note in 1.1 in the datetime section for bug fixes. ping on green.
ping @jreback 
thanks @MomIsBestFriend
BUG: Series.__getitem__ with MultiIndex and leading integer level
State: closed

- [x] closes #33355
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
looks good, does your patch also fix these cases by @jorisvandenbossche https://github.com/pandas-dev/pandas/issues/33355#issuecomment-611152308?
> does your patch also fix these cases

one of them, am about to push a commit with a test.  the list-like cases no
Follow-up coming up that fixes the listlike MultiIndex case as well as a related IntervalIndex bug
thanks
DOC: Fix heading capitalization in doc/source/whatsnew - part2 (#32550)
State: closed

- [ ] Modify files v0.6.1.rst, v0.7.0.rst, v0.7.1.rst, v0.7.2.rst, v0.7.3.rst

Labels:
name: Docs

Comments:
Hi @cleconte987, what docs are you going to be working on next?
Hi, Im going to work on the next files in doc/source/whatsnew. I think I can finish today


> Le 9 avr. 2020 à 12:34, themien <notifications@github.com> a écrit :
> 
> ﻿
> Hi @cleconte987, what docs are you going to be working on next?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

thanks @cleconte987
TST: Parameterize tests
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
a bunch of nitpicks, but this is substantively excellent.  we have tons of legacy tests that need to be split up like this
@MomIsBestFriend can you merge master?
@MomIsBestFriend this looks fine, can you merge master and a few comments
closing as stale
ENH: Support passing compression args to gzip and bz2
State: closed

This commit closes #33196 but takes a more generic approach than the suggested
solution. Instead of providing a 'fast' kwarg or global compression level
setting, this commit extends the ability to pass compression settings as a
dict to the gzip and bz2 compression methods. In this way, if the user
wants faster compression, they can pass
compression={'method': 'gzip', 'compresslevel'=1} rather than
just compression='gzip'.

Note: For the API to be consistent when passing  paths vs. filelikes, GZipFile and gzip2.open() must accept the same kwargs.

- [x] closes #33196
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
One question to consider: Instead of providing the ability to provide compression level args on a case-by case basis depending on protocol (originally just zip, now gzip and bz2 as well), would it be better to simply provide this ability to *all* zip protocols? I don't see anything in the code that prevents this--we would simply add `**compression_args` to each protocol's `open()` method or file constructor.
Hello @jessefarnham! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-10 20:39:31 UTC
@jessefarnham looks like there is a merge conflict - can you fix that up?
> @jessefarnham looks like there is a merge conflict - can you fix that up?

Fixed
@WillAyd Conflict resolved, but check seems to be failing because linting on master is currently broken
Gotcha; should be fixed in #33461 so can re-merge after that gets into master
> Gotcha; should be fixed in #33461 so can re-merge after that gets into master

All set, ready for merge to master.
thanks @jessefarnham very nice!
(WIP) CI: Check private function not used across modules
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/32942#issuecomment-609893282
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

cc @jbrockmendel 


Labels:
name: CI
description: Continuous Integration

Comments:
Nice.  I'm not aware of any alternative to sys._getframe, but the others we should be able to either track down alternatives or de-privatize
> Nice. I'm not aware of any alternative to sys._getframe, but the others we should be able to either track down alternatives or de-privatize

It will be very easy to white-list ```sys._getframe``` or for that matter, any other class/module/function...
just pushed with rebase
@MomIsBestFriend can we exclude the tests directory here
@MomIsBestFriend you ok?  havent heard from you in a while
> @MomIsBestFriend you ok? havent heard from you in a while

Yes I am perfectly fine, due to COVID-19 I have much more work IRL, and I am not able to contribute to pandas ATM :disappointed: 

When I have the time I'll be back to contribute once again.

Thank you for thinking about me @jbrockmendel <3
nice idea, please comment if you can come back to this / open a new PR
CI: Check private function not used across modules
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/32942#issuecomment-609893282
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @MomIsBestFriend! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`scripts/validate_unwanted_patterns.py`](https://github.com/pandas-dev/pandas/blob/286e49488bf748f0eb563c7e9d58e53e944c8c78/scripts/validate_unwanted_patterns.py):

> [Line 154:9](https://github.com/pandas-dev/pandas/blob/286e49488bf748f0eb563c7e9d58e53e944c8c78/scripts/validate_unwanted_patterns.py#L154): [E266](https://duckduckgo.com/?q=pep8%20E266) too many leading '#' for block comment


@MomIsBestFriend can i convince you to pick this and/or #33394 back up?  I found myself manually tracking down incorrectly-private functions today, would be nice to avoid that need.
BUG/PLT: Multiple plots with different cmap, colorbars legends use first cmap
State: closed

- [x] closes #33389 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Since this is a plot PR, just post what it looks like after the change

![Screen Shot 2020-04-08 at 3 43 54 PM](https://user-images.githubusercontent.com/9269816/78791204-d94a8a00-79af-11ea-9ebb-e216251c7908.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
thanks @charlesdong1991
FEATURE: pandas.Series.query() #22347
State: closed


Labels:

Comments:
Hello @yaolan4! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





* In the file [`pandas/tests/series/test_query_eval.py`](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py):

> [Line 124:13](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L124): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 132:13](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L132): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 309:80](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L309): [E251](https://duckduckgo.com/?q=pep8%20E251) unexpected spaces around keyword / parameter equals
> [Line 310:13](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L310): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent
> [Line 320:13](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L320): [E124](https://duckduckgo.com/?q=pep8%20E124) closing bracket does not match visual indentation
> [Line 328:13](https://github.com/pandas-dev/pandas/blob/4fc2b5c47e2cc49d739b775fa9bf165930d88104/pandas/tests/series/test_query_eval.py#L328): [E128](https://duckduckgo.com/?q=pep8%20E128) continuation line under-indented for visual indent



##### Comment last updated at 2020-04-10 01:50:37 UTC
Closing as this looks stale, but ping if you'd like to continue and can merge master. Thanks!
FEATURE: pandas.Series.query()(#22347)
State: closed


Labels:

Comments:
Hello @yaolan4! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/conftest.py`](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py):

> [Line 768:20](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py#L768): [W604](https://duckduckgo.com/?q=pep8%20W604) backticks are deprecated, use 'repr()'
> [Line 768:28](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py#L768): [E226](https://duckduckgo.com/?q=pep8%20E226) missing whitespace around arithmetic operator
> [Line 1201:1](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py#L1201): [E305](https://duckduckgo.com/?q=pep8%20E305) expected 2 blank lines after class or function definition, found 1
> [Line 1216:9](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py#L1216): [E113](https://duckduckgo.com/?q=pep8%20E113) unexpected indentation
> [Line 1216:53](https://github.com/pandas-dev/pandas/blob/5a263b459795d710dcb9b452ec7a1177ca10f37c/pandas/conftest.py#L1216): [E211](https://duckduckgo.com/?q=pep8%20E211) whitespace before '('
DOC: Fix capitalisation in doc/source/whatsnew - part1 (issue #32550)
State: closed

- [ ] modify files index.rst, v0.4.x.rst, v0.5.0.rst, v0.6.0.rst

- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Labels:
name: Docs
name: Clean

Comments:
Should version numbers begin with a capital 'v' or not?
> I think version numbers are ok in lowercase, but if that fails the script and complicates things too much, I guess in upper case would also be ok.

Ok so changing it to uppercase
> Ok so changing it to uppercase

Just realized that we'll be capitalizing the first one of `V0.4.1 through v0.4.3`, which I think it's not great. May be we can use `Versions v0.4.1 through v0.4.3` or something similar instead?
> > Ok so changing it to uppercase
> 
> Just realized that we'll be capitalizing the first one of `V0.4.1 through v0.4.3`, which I think it's not great. May be we can use `Versions v0.4.1 through v0.4.3` or something similar instead?

Ok changing it to "Versions ..."
Looks like in recent versions we use `v0.0.0` instead of `v.0.0.0`. But I think if we add the word version, we can actually get rid of the `v`, and just leave `Version 0.6.0`... What do you think?
> Looks like in recent versions we use `v0.0.0` instead of `v.0.0.0`. But I think if we add the word version, we can actually get rid of the `v`, and just leave `Version 0.6.0`... What do you think?

For me it would be ok. But Im not in the project enough to know if it would make sense
You're welcome!
can merge master and ping on green.
@jreback
Updated, merged master. 
Thanks @cleconte987 
> Thanks @cleconte987

You're welcome!
BUG: iloc setting columns not taking effect
State: closed

- [x] closes #33198
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
are you running the full test suite locally before pushing?
> are you running the full test suite locally before pushing?

Not fully. usually only the parts I suspect that are related to the changes I make, in this case I tried on ```pytest -q --cache-clear pandas/tests/frame/methods/``` before pushing.
Unless you need to save the wattage, its usually good to run the full thing.  e.g. i have in .bash_aliases `alias ptest="python3 -m pytest pandas/tests --skip-slow --skip-db"`

In this case we might expect tests.indexing, tests.frame.indexing, tests.series.indexing, and possibly tests.indexes to be potential trouble spots
> Unless you need to save the wattage, its usually good to run the full thing. e.g. i have in .bash_aliases `alias ptest="python3 -m pytest pandas/tests --skip-slow --skip-db"`

That's a good tip.

Looking at the test logs, there's still some more troubleshooting to be done.  What's the next step?
> Looking at the test logs, there's still some more troubleshooting to be done. What's the next step?

@jbrockmendel Looks like we need to check for the length of the index of each one, to decide if we are taking the split path.

what I tried was:

```python
                val = list(value.values()) if isinstance(value, dict) else value
                # https://github.com/pandas-dev/pandas/issues/33198
                if isinstance(indexer, tuple) and com.is_null_slice(indexer[0]):
                    take_spit_path = length_of_indexer(
                        self.obj.index
                    ) == length_of_indexer(indexer[0])
```

---

But it does not work.

Let's continue troubleshooting in #33198.  Closing to clear the queue, can re-open when its ready to go.
API/TST: Call __finalize__ in more places
State: closed

Progress towards https://github.com/pandas-dev/pandas/issues/28283. This
adds tests that ensures `NDFrame.__finalize__` is called in more places.
Thus far I've added tests for anything that meets the following rule:

> Pandas calls `NDFrame.__finalize__` on any NDFrame method that returns
> another NDFrame.

I think that given the generality of `__finalize__`, making any kind of
list of which methods should call it is going to be somewhat arbitrary.
That rule errs on the side of calling it too often, which I think is my
preference.

*For reviewers* I think the two most helpful pieces would be

1. Do you agree with the general rule of `NDFrame -> NDFrame` should call `__finalize__`?
2. A spot check on the tests I've added

If you don't agree with the general rule, then we'll need to go through method by method and determine whether that method should call `__finalize__`. 

Once we have agreement that the tests I've added accurately reflect the desired outcome, this can be merged and we can fix `xfails` in smaller batches.

Labels:
name: API Design
name: metadata
description: _metadata, .attrs

Comments:
CI has finished here, just hasn't updated on the UI.

On the call yesterday, people generally seemed OK with the rule that methods returning NDFrames should call `__finalize__`.

Does anyone have issues with the structure of the `finalize` test (parametrize over constructor, constructor args, and method)? Or are we OK to move ahead here?

If so, I'll start to work on the xfails (methods that don't yet call `__finalize__`)
nice work @TomAugspurger
DOC: Fix capitalization among headings in doc/source/whatsnew (#32550)
State: closed

Updated headers for the following files:

```
- [x] doc/source/whatsnew/v0.19.0.rst
- [x] doc/source/whatsnew/v0.20.0.rst
- [x] doc/source/whatsnew/v0.24.0.rst
- [x] doc/source/whatsnew/v0.25.0.rst
- [x] doc/source/whatsnew/v1.0.0.rst
```


Labels:
name: Docs

Comments:
thanks @themien
BUG: is_categorical shouldnt recognize Dtype objects
State: closed

is_categorical(obj) is used in a couple places where we then go on to access `obj.dtype`, so this should not return `True` if we have a `CategoricalDtype` object.

Labels:

Comments:
This will need a deprecation cycle right?
> This will need a deprecation cycle right?

id classify this as a bugfix, but OK either way
the other option is just to deprecate it, since we only use is_categorical in a few places and its more likely than not to cause confusion
Closing in favor of #33385.
DOC: Fix capitalization among headings in doc/source/whatsnew (#32550)
State: closed

Updated headers for the following files:

```
- [x] doc/source/whatsnew/index.rst
- [x] doc/source/whatsnew/v0.14.0.rst
- [x] doc/source/whatsnew/v0.15.0.rst
- [x] doc/source/whatsnew/v0.18.0.rst
- [x] doc/source/whatsnew/v0.18.1.rst
```

Labels:
name: Docs

Comments:
thanks @themien
DOC/TYP: Fixed typing and doc
State: closed

- [x] xref https://github.com/pandas-dev/pandas/pull/33263#pullrequestreview-389118991
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@MomIsBestFriend can you rebase (and address any actionable comments)
Thanks @MomIsBestFriend closing as stale. ping to reopen.
BUG: #31464 Fix error when parsing JSON list of bool into Series
State: closed

Add a missing exception type to the except clause, to cover
the TypeError that is thrown by Cythonized array_to_datetime
function when trying to convert bool to nonseconds.

- [x] closes #31464
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks @jessefarnham very nice.