BrandonFors/pandas-github-issues huggingface dataset:
ENH: try to preserve the dtype on combine_first for the case where the two DataFrame objects have the same columns
State: closed

…e two DataFrame objects have the same columns

- [ ] closes #7509 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @danielhrisca! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-15 14:44:28 UTC
@jreback I've changed the implementation and added tests
I think all request have been met. The failed test does not seem to be related to the changes done in this PR, most probably come after I re-synced with the master branch 
@jreback I've done the changes  but again there are failed builds for which I can see no connection with this PR
minor nitpick, otherwise looks good
thanks @danielhrisca very nice
BUG: pd.to_numeric does not copy _mask for ExtensionArrays 
State: closed

- [x] closes #38974 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry (not needed as bug on master only)

Fixing a bug I introduced in  #38746

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
thanks @arw2019
TST: extend check for leaked files
State: closed

Use `ResourceWarning`s as indicators of leaked files. I left the `psutil`-based check for leaked files.

I enabled the check for leaked files for all io tests.

boto3 (#17058) is probably causing all of the unclosed ssl connections. This PR ignores ResourceWarnings that contain the string `'ssl'`. The fixture applied to all io tests further ignores all connections found by `psutils` for the same reason. Connections are checked for code that was previously covered by `check_file_leaks`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-31 11:44:04 UTC
I added a test to make sure that `ResourceWarning` are being generated+caught: it works for me locally and on some of the CI test runs.

The test to check this leaks a file ending with "_resoruce_test" [1]. For the test runs that fail it seems that this exact file's `ResourceWarning` is associated with different tests!?! I don't understand how the warnings are associated with different tests!

[1] https://github.com/pandas-dev/pandas/pull/39047/files#diff-6e61080cbd905ab74df62dfef5763fe30dc51efd2db7a56a5a9d38281dd96e7bR450


edit: this phenomenon might probably also explain some ssl `ResourceWarning`s in test that definitely have no ssl/networking component. 
@twoertwein how's this coming?
> @twoertwein how's this coming?

I assume pytest-xdist might cause warnings to be associated with the other tests but I haven't yet tested disabling pytest-xdist on the CI.
@twoertwein do you have a plan in mind for how to get this finished?
> @twoertwein do you have a plan in mind for how to get this finished?

It still fails when running pytest with `-n 0 --dist=no`. I'm not sure of to further debug this: there must be some difference between me running the tests locally and the CI. I do not get the failing tests on my machine.
> I'm not sure of to further debug this: there must be some difference between me running the tests locally and the CI. I do not get the failing tests on my machine.

Sorry to hear that, and that i dont have any helpful suggestions.

If you're not planning on working on this in the near future, let's close this to clear the queue.
BUG: reindexing empty CategoricalIndex would fail if target had duplicates
State: closed

- [x] closes #38906
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
Hello @batterseapower! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-10 00:36:47 UTC
These test failures about ports being in use don't seem to be related to my change.
thanks @batterseapower
BUG/API: make setitem-inplace preserve dtype when possible with PandasArray, IntegerArray, FloatingArray
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

xref #38896 (doesn't close).  In general `df[:, "A"] = foo` tries to operate in-place before falling back to casting.  This makes sure we do that when `foo` is a `PandasArray`, `IntegerArray`, or `FloatingArray`

I guess could/should do the same for BooleanArray


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
rebased + green
> ok maybe if you can add a doc-string to the function will be more obvious that we are trying to match the len here

docstring updated + green
am ok with this, @jorisvandenbossche  if any addl comments.
There are some questions in my last review above for which I was still waiting on an answer
let's stick a pin in this until #39163 goes in; it will be easier to address outstanding questions/comments at that point
mothballing until after #39163
CI: use conda incubator in benchmarks
State: closed

The benchmark ci currently has the longest runtime. Using conda incubator would save about 6-8 min.


Labels:
name: CI
description: Continuous Integration

Comments:
@fangchenli this looks like a nice clean-up in any case (thanks!), but how would it save 6-8 minutes? (it currently does not seem to take less time as on master, unless the caching is not yet working here in this PR?)
> @fangchenli this looks like a nice clean-up in any case (thanks!), but how would it save 6-8 minutes? (it currently does not seem to take less time as on master, unless the caching is not yet working here in this PR?)

My mistake... It's more like 3 minutes. The caching is not working in this PR (maybe it doesn't work on PRs?). The two database builds are using conda incubator currently. It takes less time to setup the environment on those two builds. So I think caching is working there.
caching is only enabled in the master branch pushes 
> weirdly the doc build is failing

`doc/make.py --warnings-are-errors | tee sphinx.log ; exit ${PIPESTATUS[0]}`

`PIPESTATUS[0]` is 1 now. Not sure why. Looking at the output of `doc/make.py --warnings-are-errors`, nothing is failing.

## update

I missed this error.

```bash
>>>-------------------------------------------------------------------------
Exception in /home/runner/work/pandas/pandas/doc/source/whatsnew/v0.8.0.rst at block ending on line 201
Specify :okexcept: as an option in the ipython:: block to suppress this message
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-65b822729fc4> in <module>
----> 1 plt.figure()

NameError: name 'plt' is not defined
<<<-------------------------------------------------------------------------
```

maybe the doc build is not picking up the correct file / deps as it is in master?
> maybe the doc build is not picking up the correct file / deps as it is in master?

I fixed the error mentioned above. Right now the outputs of `doc/make.py --warnings-are-errors` of this PR is identical to master's. Only the exit codes are different.
I'll revert the doc build and open an issue for it.
thanks @fangchenli
REF: Prep for moving Series.apply into apply
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Will be moving Series.apply into apply next. The plan is to have a base class Apply which is parent to both FrameApply and SeriesApply.

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Failure is unrelated.

> FAILED pandas/tests/io/formats/test_to_excel.py::test_css_to_excel_good_colors[slategrey-708090]
= 1 failed, 148191 passed, 4557 skipped, 1087 xfailed, 3 xpassed, 5 warnings in 1261.08s (0:21:01) =

great
CLN, TYP use from __future__ import annotations
State: closed

Seeing as pandas is Python3.7+ we can clean up some annotations. Not yet figured out how to write a good automated check for this, but here's a start

Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
see also #36091
@MarcoGorelli I'll merge this now as without a check it's a waste of valuable reviewer time to check that all quotes are removed from a module when the import is added.

but there we a few here that are obvious without a through check.
a separate PR dealing with return types is welcome.
API: concatting DataFrames does not skip empty objects
State: closed

See discussion in https://github.com/pandas-dev/pandas/pull/38843

To repeat the main argument: when not skipping empty objects, the resulting dtype of a concat-operation only depends on the input *dtypes*, and not on the exact content (the exact values, how many values (shape)). In general we want to get rid of value-dependent behaviour. In the past we discussed this in the context of the certain values (eg presence of NaNs or not), but I think also the shape should not matter (eg when slicing dataframes before contatting, you can get empties or not depending on *values*).

This starts with reverting the change for DataFrame. If we opt for this behaviour, in addition we should also discuss how to change the Series behaviour.

cc @jbrockmendel

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
so if we revert this, then we actually need to change real logic here and use the combined dtypes of the empties, which is fine.
I'm generally OK with this.

>  If we opt for this behaviour, in addition we should also discuss how to change the Series behaviour.

If we do, we should do the same for Index setops, which has special-casing for empty indexes.
> I'm generally OK with this.
> 
> > If we opt for this behaviour, in addition we should also discuss how to change the Series behaviour.
> 
> If we do, we should do the same for Index setops, which has special-casing for empty indexes.

ok fair enought. let's revert then i guess try to unify and see what breaks.
Opened a follow-up issue for this at https://github.com/pandas-dev/pandas/issues/39122
Fix regression in setitem when expanding DataFrame with specific column name format
State: closed

- [x] closes #39010
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

We obviously want cast to object if value is string.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Alternatively, I suppose that `is_bool_dtype` should not raise an error on this:

```
In [97]: pd.api.types.is_bool_dtype('0 - Name')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-97-c990eff26eac> in <module>
----> 1 pd.api.types.is_bool_dtype('0 - Name')

~/scipy/pandas/pandas/core/dtypes/common.py in is_bool_dtype(arr_or_dtype)
   1395         return False
   1396     try:
-> 1397         dtype = get_dtype(arr_or_dtype)
   1398     except TypeError:
   1399         return False

~/scipy/pandas/pandas/core/dtypes/common.py in get_dtype(arr_or_dtype)
   1631         arr_or_dtype = arr_or_dtype.dtype
   1632 
-> 1633     return pandas_dtype(arr_or_dtype)
   1634 
   1635 

~/scipy/pandas/pandas/core/dtypes/common.py in pandas_dtype(dtype)
   1823     # raise a consistent TypeError if failed
   1824     try:
-> 1825         npdtype = np.dtype(dtype)
   1826     except SyntaxError as err:
   1827         # np.dtype uses `eval` which can raise SyntaxError

~/miniconda3/envs/dev/lib/python3.7/site-packages/numpy/core/_internal.py in _commastring(astr)
    177                     raise ValueError(
    178                         'format number %d of "%s" is not recognized' %
--> 179                         (len(result)+1, astr))
    180                 startindex = mo.end()
    181 

ValueError: format number 1 of "b'0 - Name'" is not recognized
```
You mean catching the numpy error?
I didn't check exactly what it would entail to fix it, but `is_bool_dtype` is a function that we control ourselves, so we can ensure it returns False for this case, instead of erroring. 
But yeah, since we are already catching errors when trying to create a numpy dtype, we might need to catch some more errors
Yeah, this works too. Thx
nice teamwork! @phofl can you add a test specific to is_bool_dtype
Done
thanks @phofl 
@meeseeksdev backport 1.2.x
Deprecate DataFrame indexer for iloc setitem and getitem
State: closed

- [x] closes #39004
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
> See my note on the PR: [#39004 (comment)](https://github.com/pandas-dev/pandas/issues/39004#issuecomment-756344692)
> 
> In addition, if we disallow this, I think it should be deprecated first for assignment (since that case was working right now)

this is reasonable to deprecate for setitem and simply raise for getitem. @phofl if you can amend.

though I am tempted to simply call this a bug fix if setitem actually gives wrong behavior? (or is it actually somewhat correct?)
Don't know if I would call this wrong. It is counterintuitive for sure. 

If you *know* that iloc does not align, you may expect the result, Additionally it only works if indexer has the same dimension as the object to index.

Will add a DeprecationWarning
> Will add a DeprecationWarning 

FutureWarning but thanks
Added the Warning
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-02 19:53:34 UTC
Added a deprecation whatsnew
@jreback green
Let's give this a bit more time, since the discussion in https://github.com/pandas-dev/pandas/issues/39004 is still ongoing
@phofl conflict and if you can rebase. i think this is fine to merge.
Done, failure unrelated
this looks fine to me. @jorisvandenbossche are you still objecting?
@phofl can you merge master and update
@jreback merged master, failure unrelated
everyone ok here ?

@jorisvandenbossche @phofl @jbrockmendel @gooney47
LGTM
Are we good here?
@jreback greenish
thanks @phofl
Backport PR #39019 on branch 1.2.x (DOC: np.bool -> np.bool_)
State: closed

Backport PR #39019: DOC: np.bool -> np.bool_

Labels:
name: Docs

Comments:
test failure seen before. maybe flakey test, restarting ci anyway.

```
=========================== short test summary info ============================
FAILED pandas/tests/io/test_gcs.py::test_to_csv_compression_encoding_gcs[zip-utf-8]
= 1 failed, 146094 passed, 3169 skipped, 1096 xfailed, 32 warnings in 2494.44s (0:41:34) =
```
DOC: np.bool -> np.bool_
State: closed

xref #34848, #34835

a couple missed that are causing doc build failures with numpy-1.20.0rc2, https://github.com/pandas-dev/pandas/pull/36092/checks?check_run_id=1662818738

probably want to backport this to prevent possible future ci failures

Labels:
name: Docs

Comments:
should we have a checker rule to prohibit?
> should we have a checker rule to prohibit?

possible, i used `np.bool[^_]` but that misses

```
pandas/tests/io/excel/test_writers.py:    @pytest.mark.parametrize("np_type", [np.bool8, np.bool_])
pandas/tests/io/excel/test_writers.py:        # Test np.bool values read come back as float.
```
could change the comment as well and use `np.bool[^_8]`
@simonjayhawkins you *may* have to remove the rule for backport but we shall see
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
CLN: add typing to dtype arg in core/common.py (GH38808)
State: closed

Follow the issue - https://github.com/pandas-dev/pandas/issues/38808

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @tushushu
DOC: Add whatsnew
State: closed

- [x] closes #38878
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @vangorade - can you rebase so as to drop the unrelated commits please?
did i just messed up something @MarcoGorelli?
possibly :) no worries though - at this point I think the easiest thing might be to just do
```
git fetch upstream --prune
git reset --hard upstream/master
<make the same changes you'd made previously>
git add -u
git commit -m <commit message>
git push origin HEAD -f
```
> possibly :) no worries though - at this point I think the easiest thing might be to just do
> 
> ```
> git fetch upstream --prune
> git reset --hard upstream/master
> <make the same changes you'd made previously>
> git add -u
> git commit -m <commit message>
> git push origin HEAD -f
> ```

Thanks :). I thought i broke something, after some google search i learned about this issue and how to time travel in git and thanks to your reply i finally resolve this issue :+1: !
thanks @vangorade
BUG: read_csv raising ValueError for tru_values/false_values and boolean dtype
State: closed

- [x] closes #34655
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I am not really experienced with cython, so I would appreciate feedback on the switiching function. This was not done previously in case of ea boolean dtype, hence why this was failing before.

Labels:
name: IO CSV
description: read_csv, to_csv
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hm this sounds good. Will try this tomorrow or Friday. Thx.
This is way better. To avoid special casing the boolean case, I added **kwargs to ``_from_sequence_of_strings``, Is there another way you would prefer?
Passing such additional keywords to `_from_sequence_of_strings` is not backwards compatible for externally defined EAs, if they do not accepts kwargs. 
So I would maybe keep a special case check for boolean dtype in _parsers.pyx, and only pass the keywords to `BooleanArray._from_sequence_of_strings`. 

(alternative could be to try/except, catch the error, and then try again without passing the additional keywords (and potentially raising a warning so that external EAs can update to accept `**kwargs`)
> Passing such additional keywords to `_from_sequence_of_strings` is not backwards compatible for externally defined EAs, if they do not accepts kwargs.
> So I would maybe keep a special case check for boolean dtype in _parsers.pyx, and only pass the keywords to `BooleanArray._from_sequence_of_strings`.
> 
> (alternative could be to try/except, catch the error, and then try again without passing the additional keywords (and potentially raising a warning so that external EAs can update to accept `**kwargs`)

we don't have these kinds of guarantees on EA. I think we should accept them.
`_from_sequence_of_strings` is a public method of the ExtensionArray Interface, documented through means of the base class implementation:

https://github.com/pandas-dev/pandas/blob/510bc200659668bf49effd41176e874b8b3e1e21/pandas/core/arrays/base.py#L213-L237

(and in the online API docs)

If we can easily avoid breaking third-party EA implementations, then we should do that. 

> `_from_sequence_of_strings` is a public method of the ExtensionArray Interface, documented through means of the base class implementation:
> 
> https://github.com/pandas-dev/pandas/blob/510bc200659668bf49effd41176e874b8b3e1e21/pandas/core/arrays/base.py#L213-L237
> 
> (and in the online API docs)
> 
> If we can easily avoid breaking third-party EA implementations, then we should do that.

I think we *need* to in this case (for BooleanArray) to make a proper api. yes I agree we can avoid breaking all others (see above my comment).
Yep, no problem for adding it to `BooleanArray`, that's our own EA implementation, there we can add keyword however we want.
thanks @phofl very nice!
CLN: inspect_excel_format
State: closed

- [x] closes #38823 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

cc @rhshadrach 

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @lithomas1!
DOC: Clarify index_col behavior for read_csv
State: closed

- [x] closes #38830
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
cc @gfyoung