BrandonFors/pandas-github-issues huggingface dataset:
DOC: Fix EX02 and SA01 in DataFrame.select_dtypes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

output of `python scripts/validate_docstrings.py pandas.DataFrame.select_dtypes`:
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
        No extended summary found
```

Notes:
It seems like passing `int` to the parameter do not include/exclude all integer dtypes. But, passing `integer` would do otherwise. I think it is related to this issue https://github.com/pandas-dev/pandas/issues/29394. It would be more convenient if passing `int` as an input would match the result with passing `integer`.



Labels:
name: Docs

Comments:
Great thanks @farhanreynaldo
TST: sparse and dense concat test GH30667
State: closed

- [x] closes #30668
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Adds testing to ensure concat between dense and sparse


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
Thanks @ricalanis
34297 sub slow in first call
State: closed

- [x] closes #34297
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

@jbrockmendel 

I added a lenght check as suggested by you. I could not find another method to check the indices (nlevels for example does not work). I'm open to recommendations about adding additional checks before running into the values call.

I have added a test, which measures the execution time of both calls to `sub`. Is there a better way to test, if they are equally fast?

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: MultiIndex
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
> 
> 
> can you also add a note in perf section for 1.1

Added the whats new entry.
thanks @phofl
Use repr of fill_value in SparseDtype repr
State: closed

This proposes changing the StringDtype repr to show the fill_value's repr rather than its string representation:

```python
>>> arr = pd.arrays.SparseArray([0, 1])
>>> arr.dtype
Sparse[int64, 0]  # master & this PR, unchanged
>>> arr.astype(str).dtype
Sparse[object, 0]  # master
Sparse[object, '0']  # this PR, notice quotes
```


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Sparse
description: Sparse Data Type

Comments:
Ok, I've moved the tests.
looks good can you add a release note (sparse bug fix section is ok). ping on green.
Thanks!
btw if would be helpful to survey other dtypes (mainly category) to see if we are consistent for various types
Non of the other dtypes have the concept of a "fill_value" (or print another value in its dtype repr), so I am not sure what should be checked?
of course; i am speaking generally

we expose other things in the repr eg categories 
The CategoricalDtype is fine:

```
In [9]: pd.Categorical([1, 2, '3']).dtype 
Out[9]: CategoricalDtype(categories=[1, 2, '3'], ordered=False)
```

Categorical itself is not:

```
In [11]: pd.Categorical([1, 2, '3']) 
Out[11]: 
[1, 2, 3]
Categories (3, object): [1, 2, 3]
```

but that is actually already being handled in https://github.com/pandas-dev/pandas/pull/34222
BUG: don't plot colorbar if c is column containing colors
State: closed

- [x] closes #34316
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
thanks @MarcoGorelli
Added missing import in boxplot_frame_groupby example
State: closed

- [X] closes #28903 
- [x] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Thanks @gsyqax - did you build this page of the documentation locally to check it works?
@MarcoGorelli yes, it is working.

I noticed the example also makes use of 'pd' and 'np' alias without importing and aliasing pandas and numpy. I don't think this can lead to confusion. Do you think I should also add those imports?
That's OK, it fits with the [docstring guide](https://pandas.pydata.org/docs/development/contributing_docstring.html)

> Import required libraries (except numpy and pandas)
> @MarcoGorelli yes, it is working.

I just tried building it with your changes and I don't see any plot though, do you?

See the guide I linked to above for how to display plots in documentation examples
I'm not seeing any plots either. I thought a plot was not needed since the current version is not showing any either.

I will add the plots and commit again
Hello @gsyqax! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-06 14:53:37 UTC
@MarcoGorelli I added some plots and made a few minor changes to make everything clearer, hope it's OK. Also removed whitespace as indicated by pep8speaks.
Also, I don't think a whatsnew note is necessary for documentation improvements, could you please revert that change?
Great! Many thanks @MarcoGorelli for all the help, really appreciate it!
cc @datapythonista @jorisvandenbossche @WillAyd 
try `boxplot_frame_groupby(grouped, rot=45, fontsize=12, figsize=(8, 10))` for instance? @gsyqax 

it will avoid having overlapped subplots and make doc look better 
Slight preference for what @jorisvandenbossche has suggested above but not a blocker so OK either way
I applied the change to the plot fig size and replaced boxplot_frame_groupby by df.groupby.boxplot. I'm keeping the intermediate var 'grouped' to make things easier to read
lgtm. @TomAugspurger 
Thanks @gsyqax!
DOC: Fix reference, old file was separated last year
State: closed

- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

The comment referenced a file deleted last year after separating the tests into a few files. I referenced the correct folder again

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
> 
> 
> Small nit-pick, otherwise good find / clean-up

Done :)

Thanks
thanks @phofl
BUG: fix concat of Sparse with non-sparse dtypes
State: closed

Closes #34336

This adds tests with the behaviour as it was on 0.25.3 / 1.0.3, and some changes to get back to that behaviour.  
(but whether this behaviour is fully "sane", I am not sure ..)

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
OK, will go forward with this PR as is (matching the previous behaviour) to unblock the other concat PRs, but will open a set of follow-up issues.
And opened https://github.com/pandas-dev/pandas/issues/34459 for the general "what should concat(sparse, categorical) do?" question
don't need len around shape
State: closed

#Fix #34274

- [x] closes #34274
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Clean

Comments:
Thanks @SilasK - could you add a test to show how this fixes the issue?
@SilasK this needs a test where this breaks / why you are changing
@SilasK can you add a test? https://github.com/pandas-dev/pandas/issues/34645 has an example.
@SilasK do you have time to add a test?
Sorry, not for the moment. And It would be a test that test that the error message is produced correctly, how do I do that?

like this: https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#testing-warnings
@SilasK Thanks for the PR. closing as stale. LMK if you want to continue and I'll reopen.
BUG: Fix render as column name in to_excel
State: closed

- [x] closes #34331
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Should I add a whats new enty? Under which section?

https://github.com/phofl/pandas/blob/34331_render_column_name_raises_error_in_to_excel/pandas/io/formats/excel.py#L389

The previous code checked if the `df` had a attribute `render`. This check was True, if there was a `DataFrame` given as input with a column `render`. I think that this check should only be True if a `Styler` object is given, so I added an `isinstace` check for `Styler`. 

Does the `Styler` object always an attribute `render`? If not, we would run into problems, if we get a `Styler` object with a column named `render` but without the `render` attribute. If a `Styler` object has always an attribute `render`, we could delete the `hasattr` check.

Edit: I changed the check to `not isinstance(df, DataFrame)` to avoid the jinja issues.

Labels:
name: IO Excel
description: read_excel, to_excel
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@phofl You can add a bug fix whatsnew note under the I/O section in pandas/doc/source/whatsnew/v1.1.0.rst
> 
> 
> @phofl You can add a bug fix whatsnew note under the I/O section in pandas/doc/source/whatsnew/v1.1.0.rst

Àdded the whats new entry
thanks @phofl very nice!
BUG: xticks unnecessarily rotated
State: closed

- [x] closes #29460
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

On `master`, xticklabels are rotated if
```python
            not self._use_dynamic_x()
            and data.index.is_all_dates
            and not self.subplots
            or (self.subplots and self.sharex)
```

There's a comment saying
```python
        if condition:
            # irregular TS rotated 30 deg. by default
            # probably a better place to check / set this.
            if not self._rot_set:
                self.rot = 30
```
, and it seems that the _intention_ was to rotate ticklabels if there's an irregular timeseries.

With the condition as it is written, they will also be rotated if the following is true:
```python
(self.subplots and self.sharex)
```

I don't see why that should warrant rotating the ticklabels, so I'd like to suggest that there was a bracketing error.

Also, the condition `data.index.is_all_dates` is (as far as I can tell) only relevant if `use_index=True`, so I've changed that.

Here's what the OP's plot looks like with these changes:

![image](https://user-images.githubusercontent.com/33491632/82729010-04810280-9cec-11ea-825c-5244a2a1f828.png)


Here's what it looks like on master:

![image](https://user-images.githubusercontent.com/33491632/82729022-1bbff000-9cec-11ea-9954-4e75488f91b8.png)

Irregular time series still have their ticklabels rotated:
![image](https://user-images.githubusercontent.com/33491632/84986706-e796ee00-b136-11ea-9df1-d067a0de134b.png)

Regular time series' ticklabels don't rotate:
![image](https://user-images.githubusercontent.com/33491632/84986756-fd0c1800-b136-11ea-857d-738f17bb74c1.png)

(unless you pass `xcompat=True`, but that's unaffected by this change)


Labels:
name: Visualization
description: plotting

Comments:
@jreback ping
@MarcoGorelli can you move release note to 1.2
@charlesdong1991 @jorisvandenbossche @TomAugspurger if comments.
Nothing here.

On Wed, Aug 12, 2020 at 10:55 AM Jeff Reback <notifications@github.com>
wrote:

> @charlesdong1991 <https://github.com/charlesdong1991> @jorisvandenbossche
> <https://github.com/jorisvandenbossche> @TomAugspurger
> <https://github.com/TomAugspurger> if comments.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/34334#issuecomment-672960097>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXRYE56GRJHYMFDT6LSAK3QZANCNFSM4NIMUDHA>
> .
>

@MarcoGorelli can you rebase.  looks like this was about ready
thanks for this @MarcoGorelli nice! (and good to add the testing like you did).
BUG: Fix using dtype with parse_dates in read_csv
State: closed

- [x] closes #34066
- [x] tests added/passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: IO CSV
description: read_csv, to_csv

Comments:
Comments from @jreback and @dsaxton notwithstanding, the changes look good !
@mproszewska Are you still interested in working on this? If so can you merge master?
Closing as stale, @mproszewska let us know if you'd like to reopen
I can go back to working on this. Could you reopen?
@mproszewska : Done
Hello @mproszewska! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-08 20:36:43 UTC
seems not unreasonable, but his PR is stale.
Fix MultiIndex melt when col_level is used
State: closed

- [x] closes #34129
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thanks @ashtou very nice. keep em coming!
CI: Numpy compat
State: closed

- [x] closes #34306

Likely related to: https://github.com/pandas-dev/pandas/pull/32510.

Seems doc builds have started using: 
numpy                     1.17.5           py38h95a1406_0    conda-forge
https://github.com/pandas-dev/pandas/pull/34317/checks?check_run_id=700337761

(Previous pipelines where using 1.18)

In numpy 1.17.5 BitGenerator path is:
`np.random.bit_generator.BitGenerator`

In numpy 1.18.0 path is
`np.random.BitGenerator`

Labels:
name: CI
description: Continuous Integration

Comments:
I don't really follow the issue / fix here: Did the changes go like this?

- numpy 1.17.4: `np.random.BitGenerator`
- numpy 1.17.5: `np.random.bit_generator.BitGenerator`
- numpy 1.18: `np.random.BitGenerator`

Or am I misunderstanding?
> * numpy 1.17.4: `np.random.BitGenerator`

For 1.17.4

I see `np.random.bit_generator.BitGenerator` ?

 -> hence I thought the easiest fix is just to require numpy 1.18 ?

Or am i missing something?

Ill also will need to update the test changes made in https://github.com/pandas-dev/pandas/pull/32510/files

@TomAugspurger - mind doing the azp run command -> I don't have permissions.
@alimcmaster1 restarted the failing job on travis (was a network error i think). but this looks good to go.
REGR: fix op(frame, series) with extension dtypes
State: closed

Closes #34311 

cc @jbrockmendel 

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
I'll port the test here to #34277
lgtm.. can you rebase as just merged a bunch of stuff.
merge this *then* rebase / merge #34277?
> merge this then rebase / merge #34277?

or just merge 34277
closing as merged #34277
Fix: StringArray non-extensible due to inconsisent assertion
State: closed

Fix for #34309

- [X] closes #34309
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Strings
description: String extension data type and string data
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
@dsaxton Thanks for your review. I've processed your proposed changes and added an additional string dtype test.
> Tiny nit, but otherwise LGTM

Done
cc @jorisvandenbossche @TomAugspurger 
I think the issue has an ongoing discussion that needs to be resolved. 
@sbrugman can you merge master just to keep this from drifting
Hello @sbrugman! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-23 21:22:44 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
CLN: consolidate arrow roundtrip tests for nullable dtypes in base masked tests
State: closed

We recently splitted the test_boolean and test_integer in multiple files in each a directory, but I think a next step can be to deduplicate some of the common tests for the different nullable/masked dtypes. 
Certainly given that we are going to add float tests as well (cfr https://github.com/pandas-dev/pandas/pull/34307)

Starting here with the arrow-compat/roundtrip tests for both integer/boolean.

I didn't yet write a full blown conftest.py and just defined a `data` fixture inline, since here it's just a single file. But if adding more later, we probably want to move this to a `conftest.py` (or a base test class), although this duplicates a bit the `tests/arrays/integer/conftest.py` and `tests/arrays/boolean/conftest.py`

cc @dsaxton @jbrockmendel 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
seems reasonable
if there are follow-ups you have in mind but are liable to not get to right away, can you either add TODO comments or open issue(s)
thanks @jorisvandenbossche 
> if there are follow-ups you have in mind but are liable to not get to right away, can you either add TODO comments or open issue(s)

None specific, but just in general any test in `integer/` and `boolean/` that is almost identical for both directories and could be shared, can be consolidated
TST: GH24798 df.replace() with duplicate columns
State: closed

- [x] xref #24798 
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Duplicate Report
description: Duplicate issue or pull request

Comments:
thanks @matteosantama 

I don't think this completely closes #24798 ?
fix to is_string
State: closed

- [X] closes #34295
- [x] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type

Comments:
Can you add a test? Also I think may have linked to the wrong issue?
ahh thank you @WillAyd, transcription error on my part, I've updated the associated issue and added a unit test.
@TomAugspurger 
Thanks!
CLN: always dispatch-to-series
State: closed

A couple more cleanups follow this.

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Clean

Comments:
ahh need to rebase
Looks like the combination of #34324+#34323 have a subtle merge conflict that broke master.  this should be fixed by any of my just-rebased PRs
kk ping when ready in one
pseudo-green. the remaining CI fail is anoteher ipython directive thing in the docbuild
thanks

if u wouldn’t mind pushing a change for the doc thing
BUG: flex op with DataFrame, Series and ea vs ndarray
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


I have a couple ideas in mind that might make the edit in blocks here unnecessary, but want to profile them first, so will do that cleanup in a refactor following the bugfix.
 Updated with a much cleaner implementation, it also de-special-cases one of our timedelta64 tests.

Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
This overlaps now with https://github.com/pandas-dev/pandas/pull/34312
I could also port your test to my PR, so can you try to explain why this fix would be better? (it might be doing more now, but considering only what is required for fixing the original issue, i.e. the interval flex op test you added)
> I could also port your test to my PR

please do!
Only if you agree it's a good fix (I don't fully understand the rationale of the change in this PR, so I can't really judge at this time which fix is "better")
> Only if you agree it's a good fix (I don't fully understand the rationale of the change in this PR, so I can't really judge at this time which fix is "better")

I agree that #34312 is a valid fix to a legitimate regression, and is well-written, minimally-invasive.  So I would have no problem with merging it on green (especially if the test is ported).  It is especially worthwhile if we don't expect this to get merged for 1.1

The advantage of the approach in this PR is that it continues operating blockwise in cases where #34312 does not.  It also gets a bunch of code out of _combine_series_frame, which will allow for streamlining in upcoming pass.
OK, I added the test in the other PR, then we can continue the discussion here regardless of the regression fix.
docbuild fail looks unrelated
> docbuild fail looks unrelated

It's failing everywhere -> https://github.com/pandas-dev/pandas/issues/34306
green
thanks @jbrockmendel
ENH: fix arrow roundtrip for ExtensionDtypes in absence of pandas metadata
State: closed

An oversight in our Arrow Extension Types for period and interval to also define the conversion to the equivalent pandas dtype. This ensures that the arrow->pandas conversion also works if there is no "pandas_metadata" in the schema's metadata, eg when this metadata was lost during some operations (up to now, the pandas ExtensionDtype was reconstructed based on the string name in the stored metadata).

See https://arrow.apache.org/docs/python/extending_types.html#conversion-to-pandas

Labels:
name: IO Parquet
description: parquet, feather

Comments:
lgtm. do we need a whatsnew note?
A whatsnew note for it might be a bit "obscure", but can certainly add one if preferred
TST: GH28813 test .diff() on Sparse dtype
State: closed

- [x] closes #28813 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add a quick test to confirm that .diff() function handles sparse dataframes the same way as regular dataframes. 

First time making a pandas PR so comments, advice, critiques are encouraged.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Gotcha ok. Updated the PR with your fixes
Thanks @matteosantama