BrandonFors/pandas-github-issues huggingface dataset:
Added test_set_index_dtypes_on_empty_DataFrames to check dtypes after…
State: closed

… using set_index on a DataFrame with empty Series that have dtype

- [x] closes #38419
- [x] tests added / passed test_set_index_dtypes_on_empty_DataFrames
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry. Added a test to check that dtypes don't change after using set_index on a DataFrame with empty Series


Labels:

Comments:
Hello @IngErnestoAlvarez! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-12 21:35:33 UTC
@IngErnestoAlvarez sorry to do this but closing as we already have a PR for this issue which was submitted first.

Thanks so much for opening the PR in the first place. We would of course love to have you contribute to resolve other tickets! [Issues filed under the Contributions Welcome](https://github.com/pandas-dev/pandas/issues?q=is%3Aissue+is%3Aopen+milestone%3A%22Contributions+Welcome%22+) are a good place to look, and let us know if you need help at any point!
BUG: read_csv raising TypeError for engine=c with names and parse_dates
State: closed

- [x] closes #33699
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry



Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
Fixed #38419 - BUG: set_index screws up the dtypes on empty DataFrames
State: closed


Fixes #38419 - BUG: set_index screws up the dtypes on empty DataFrames
- [ test_set_index_empty_dataframe] tests added / passed



Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Index
description: Related to the Index class or subclasses

Comments:
Hello @jordi-crespo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-13 12:05:16 UTC
thanks @jordi-crespo
REGR: Assigning label with registered EA dtype raises
State: closed

- [x] closes #38386
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Previous behavior in `io.parsers` depended on `is_bool_dtype("boolean")` erroneously raising `AttributeError`.

Test currently iterates over the names:

> ['category', 'interval', 'boolean', 'Float32', 'Float64', 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64', 'string']

It skips any EA where the name is a property; I don't see a good way to iterate over these in the test.

I only added a note about `is_bool_dtype` in the whatsnew as the `DataFrame.__setitem__` issue does not occur on 1.1.x. It didn't seem appropriate for any section besides other.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @rhshadrach 
@meeseeksdev backport 1.2.x
CLN: MultiIndex.union align with Index.union()
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Similar to MultiIndex.intersection()

Stumbled over a issue in the maybe_match_names function. Can not return plain None, because MultiIndex.rename can not handle this.

cc @jbrockmendel 

Labels:
name: MultiIndex

Comments:
Can Index.union be updated to use the same code pattern?
We could rewrite ``_get_reconciled_name_object`` to accept the result as additional input, this would simplify a few things for us and make both cases more  similar. 

Edit: That is probably not that good of an idea. We are missing something like ``_wrap_setop_result`` to unify both cases
this looks like a bug fix no? e.g. let's add a whatsnew and merge (1.3)
Only technically a bug, got in a few day ago only on master. technically because this function was only used for intersection before, which could not reach this case. So I think no need for whatsnew?
thanks
ENH/POC: EA.isin
State: closed

- [x] closes #38340
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Needs tests; the MaskedArray.isin method is not currently reached.

cc @jorisvandenbossche

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Closing in favor of #38379
@jbrockmendel thanks for starting this (and sorry for the slow response here), adding `isin` to the interface sounds like a good idea (eg also in the arrow-backed StringArray, we are going to want to override it).

Taking a look now at  #38379, but it might still be worth adding the general interface method in a separate PR (like this one), as it will need some more base extension tests + docs for that.
BUG: read_csv not recognizing numbers appropriately when decimal is set
State: closed

- [x] closes #31920
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The old regex was shorter but pretty buggy.

Let's assume ``decimal=","`` and ``thousands="."``

For example something like ``1a.2,3`` was interpreted as numeric by the regex and converted to ``1a2.3``, because the ``.`` was not escaped in the regex.
Also something like ``1,2,3`` was interpreted as numeric by the regex and converted to ``1.2.3``. 

This one is not quite finished. We have to define a few thousand separator related things:

How strict would we want to be? Should ``1.2,3`` be interpreted as numeric and be converted to ``12.3`` or is only something like ``1.234,5`` relevant as thousands separator? C Engine validation is not strict
Additionally should ``,2`` be the number ``0.2``? -> Currently it is, because the C engine behaves the same

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-01 22:36:56 UTC
great ping on green (suggestion to combine the python/c tests as a followup)
@jreback green
thanks @phofl if you can do a PR or issue for the followup would be great
Will open an issue. Hopefully I will be able to get back to this in the coming days
Opened #38926
ENH: Add method argument to rolling constructor to allow table-wise rolling
State: closed

- [x] closes #15095
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: API Design
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-27 19:53:30 UTC
Some performance timings:

```
df = pd.DataFrame(np.random.rand(10, 1000))

def f(x):
    return np.sum(x, axis=0) + 1

roll_single = df.rolling(2, method='single')

roll_single.apply(f, raw=True, engine='numba')  # cache

%timeit roll_single.apply(f, raw=True, engine='numba')
37.8 ms ± 136 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

roll_table = df.rolling(2, method='table')

roll_table.apply(f, raw=True, engine='numba')  # cache

%timeit roll_table.apply(f, raw=True, engine='numba')
201 µs ± 4.06 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
Looks like everything is passing but I think the tests I added are having trouble running on Travis (but they run fine on Azure).

Possibly want to wait until https://github.com/pandas-dev/pandas/pull/38344 is in
Looks like Travis is timing out

```
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
```

Locally, the tests I added to do take a long time to run (parameterized over a lot of values).

We have the same tests running & completing in Linux py37 min versions and Windows py38 


cc @jbrockmendel if you can have a look
hmm https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=50763&view=logs&j=ba13898e-1dfb-5ace-9966-8b7af3677790&t=b1c574fb-a97a-52bc-a09f-24b62226e66f some legit failing
Thanks @jreback. Merge conflict mess up on my part.  
thanks @mroeschke very nice here!
TYP: pandas/core/frame.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I plan to do a pass through the whole file - putting it up in case anybody has feedback on what I have

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
suggest that you do reasonably sized pieces so this doesn't sit in review for too long - better multiple but smaller prs as this is a huge file
cc @simonjayhawkins @WillAyd 
> would prefer you don't mix easy and hard ones. e.g. a pass on bool / strings ones. and Axis / Level. These are easy to review. by mixing lots of things its very hard to give good scrutiny here.
> 
> would prefer that.

For sure. I'll keep the hard ones that I've already attempted here so all the comments are in one place. I'll resubmit easy and follow-ons in separate PRs
needs merge master and will look
mypy green + all comments addressed (or moved out to orthogonal PRs)
Green + addressed comments
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@arw2019 if you want to merge master and update (or close and open a new one ok too)
@jreback @simonjayhawkins updated, CI green, comments addressed
> @jreback @simonjayhawkins updated, CI green, comments addressed

ok. maybe we should merge this. there has been plenty of discussion here.
@jreback @WillAyd i'm merging later today if no objections.

mypy is green and typing PRs easily become stale or get abandoned.

There are no ignores, casts or asserts added here and no existing annotations changed/deleted.

if any of the typing additions added here are problematic they will be picked up by mypy as more types are added.



sure let's make sure this is rebased first 
yep did that locally and ran mypy before I commented. will push and merge on green
Thanks @arw2019
TST: add messages to bare pytest raises
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This PR adds messages to the bare pytest raises in pandas/tests/generic/methods/test_pipe.py and pandas/tests/reshape/merge/test_merge_asof.py. This PR references #30999.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @marktgraham nice!
REF: move __array_ufunc__ to base NumericArray
State: closed

xref https://github.com/pandas-dev/pandas/issues/38110

The implementations for integer and float array is almost identical, so move to base class. In addition, this also enables that ufuncs on IntegerArray can return FloatingArray (instead of float numpy array)

I should probably consolidate the tests as well, and move those to `tests/arrays/masked/test_ufunc.py` or so.

In addition, `BooleanArray.__array_ufunc__` is actually also very similar, but didn't include that here, yet. I should probably do that, but wondering if I should move the common `__array_ufunc__` implementation to BaseMaskedArray, or rather make BooleanArray subclass NumericArray.

Labels:
name: Refactor
description: Internal refactoring of code
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
mypy complaint, otherwise nice code sharing!
@jorisvandenbossche can you merge master
can you merge master
Updated, and mypy is happy now
thanks @jorisvandenbossche
ENH: add end and end_day origin for resample
State: closed

- [x] closes #37804
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Resample
description: resample method

Comments:
I have revert the backward arg and only add the end and end_day.
Could u take a look at this? Thanks! @jreback 
Do u think it's worthwhile to explain more about these two backward resample method in ENH of whatsnew (maybe two examples added)?
Probably good to add to `timeseries.rst#use-origin-or-offset-to-adjust-the-start-of-the-bins` explaining the new options
ok
failures unrelated
Now I change it to the following format.
```
origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp
    or str, default 'start_day'
    The timestamp on which to adjust the grouping. The timezone of origin
    must match the timezone of the index.
```
Will this be wrongly rendered to this?

**origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp**
            or str, default 'start_day' The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index.

Instead of:

**origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp or str, default 'start_day'**
            The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index.

@jreback 
thanks @GYHHAHA
BUG: DataFrame.to_markdown with an empty frame
State: closed

- [x] closes #31771
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Picking up #36233. 

Bumping the minimum version of tabulate as per https://github.com/pandas-dev/pandas/issues/31771#issuecomment-685401053

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @arw2019
BUG: Series/DataFrame construction from scalars
State: closed

- [x] closes #38032
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Copied whatsnew from #38040
TST: Add test to ensure DF describe does not throw an error (#32409)
State: closed

- [x] closes #32409
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Notes:
- Tested using `pytest pandas/tests/frame/methods/test_describe.py::TestDataFrameDescribe::test_describe_does_not_raise_error` and tested error format by calling a raise error line in the try block `raise TypeError("Test message")`
- Re-created PR initial PR with rebase attempt in the following link: https://github.com/pandas-dev/pandas/pull/35270 -- but the diff didn't seem reasonable

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @luckyvs1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-20 14:40:02 UTC
Looks like the CI is failing with this new test.
@luckyvs1 can you merge master and ping on green
@jreback I have updated this branch and also is up to date with the latest master and build is now passing, please take another look when you have a chance -- thanks!
thanks @luckyvs1 
I confirmed that returning DataFrame from a function will still raise exception "unhashable type: 'list'" if the DataFrame has any columns with illegal characters. I raised this exception with column named '\xa0'.
ENH: make "closed" part of IntervalDtype
State: closed

- [x] closes #19371
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Most of the work here is in accomodating backwards-compat.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Deprecate
description: Functionality to remove in pandas
name: Interval
description: Interval data type

Comments:
added whatsnew for deprecation
cc @jschendel if you can have a look.
> Performing static analysis using mypy
pandas/core/dtypes/dtypes.py:1250: error: Item "ExtensionDtype" of "Union[Any, ExtensionDtype]" has no attribute "closed"  [union-attr]
pandas/core/dtypes/dtypes.py:1251: error: Item "ExtensionDtype" of "Union[Any, ExtensionDtype]" has no attribute "closed"  [union-attr]
There's an annoying new failure mode:

```
dtype = pd.IntervalDtype("interval[int64]")   # <-- now warns and sets closed="right" as default

idx = pd.IntervalIndex([], dtype=dtype, closed="left")   # <-- raises ValueError bc of mismatch
```
lgtm. @jorisvandenbossche @jschendel if you can take a look.
Why is it needed to deprecate the default value for `closed`?
> Why is it needed to deprecate the default value for closed?

ATM there is (presumably) code in the wild calling the IntervalDtype constructor without specifying any `closed`.  Eventually failure to specify closed will raise.  Needs a deprecation cycle.
Yes, but so the question is: why does it need to eventually raise? 
Other constructors like `Interval` and `interval_range` also have a default. 

Another option could also be to have it "not initialized" as default, to enable inferring it from the passed Interval objects
> Another option could also be to have it "not initialized" as default, to enable inferring it from the passed Interval objects

id be OK with this
> > Another option could also be to have it "not initialized" as default, to enable inferring it from the passed Interval objects
> 
> id be OK with this

then the default should be 'infer' (which we can do if we have an unitialized value)
ok ill give this a shot and update
ping when ready here
this was ok, can you merge master
Most recent discussion is that I need to do a partial re-write, which I haven't gotten around to yet.
> Most recent discussion is that I need to do a partial re-write, which I haven't gotten around to yet.

ok!
updated as discussed.  agreed this is much nicer
I must be missing it, where exactly is the FutureWarning in the Dtype constructor triggered?
this no longer deprecates?
> this no longer deprecates?

correct.  passing "closed" to IntervalDtype is optional now
> say that the repr for Interval is changing.

let's save that for the follow-up that changes that is aimed specifically at the repr
Solved desing problem with to_datetime when "ignore" error was being used.
State: closed

- [x] closes #28120
- [x] tests added / passed `test_to_datetime_some_invalid_numbers`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Now when u use some array of integers on to_datetime function passing as arguments: error="ignore" and format="%Y%m%d" it only skips none fitting values instead of all the values.


Labels:
name: Datetime
description: Datetime data dtype
name: Stale

Comments:
Hello @IngErnestoAlvarez! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-12-23 04:52:10 UTC
@arw2019  Hi! I leave the code here, with the changes you recommend me to do. I thought it was an open issue, not necessarily a bug, but a feature that can be good for it to be used. What I would understand at first, seeing this, `to_datetime(dt_col, format='%Y%m%d', errors='ignore')`, is that it will transform the array to datetime omitting all the errors and exceptions. It doesn't sound like "if I found an error I won't change anything". But I understand if you see it differently. Thank you for the feedback!
I'm not done yet, but It seems that in one Travis build conda had an error "solving the environment", it just gets stuck, it's something I should be worried about? Or first I finish what I'm doing and then take care of that?
ignore travis as it's being worked elsewhere (though certainly would appreciate help in that - we r trying to move off it)
What u r trying to do? Changing the CI environment? 
What things do u need help with? Maybe I can help
so this build: https://travis-ci.org/github/pandas-dev/pandas/jobs/751146161

something changed very recently like in last day or so - and now conda solve time blows up

so we want to fix the install env for just this build to make it work

independent we are working on moving these builds to azure / but that's slightly orthongoal 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. if you want to continue, pls ping and can re-open.
BUG: read_csv interpreting NA value as comment when NA contains comment string
State: closed

- [x] xref #34002
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I  looked into the ``c`` case, but this is not that trivial as here. Comments are removed way before checking for na values. Would have to move that part way down to be able to do this.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
looks fine here. can you merge master. 

cc @gfyoung if any comments.
can u merge master and ping on green
@jreback green 
thanks @phofl