BrandonFors/pandas-github-issues huggingface dataset:
REGR: Fixed slicing DatetimeIndex with date
State: closed

Closes https://github.com/pandas-dev/pandas/issues/31501

Labels:
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
We don’t combine for get_loc right?  Ok to fix regression, but we should deprecate for internal consistency
thanks @TomAugspurger 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 2495068adad8cd8b4c75fe1e6102a777f0c2e262
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #31521: REGR: Fixed slicing DatetimeIndex with date'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-31521-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #31521 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Not sure how this and get_loc overlap, but this seems like generally useful behavior. If there aren’t any ambiguities then I’d say it’s worth supporting. 

> On Jan 31, 2020, at 17:53, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> We don’t combine for get_loc right? Ok to fix regression, but we should deprecate for internal consistency
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@TomAugspurger this still needs a backport?
@MeeseeksDev backport to 1.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 2495068adad8cd8b4c75fe1e6102a777f0c2e262
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #31521: REGR: Fixed slicing DatetimeIndex with date'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-31521-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #31521 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REF: simplify DTI._parse_string_to_bounds
State: closed

Sits on top of #31475.  i.e. once that is merged, the only diff remaining here will be in indexes.datetimes.

After this, we are within striking distance of sharing the method between DTI/PI.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Period
description: Period data type

Comments:
nice, really simplfies
TYP: Arraylike
State: closed

@simonjayhawkins mypy doesn't like this

```
$ mypy pandas
pandas/core/base.py:679: error: "ArrayLike" has no attribute "nbytes"
pandas/core/indexes/base.py:246: error: Type variable "pandas._typing.ArrayLike" is unbound
pandas/core/indexes/base.py:246: note: (Hint: Use "Generic[ArrayLike]" or "Protocol[ArrayLike]" base class to bind "ArrayLike" inside a class)
pandas/core/indexes/base.py:246: note: (Hint: Use "ArrayLike" in function signature to bind "ArrayLike" inside a function)
pandas/core/indexes/base.py:1152: error: "ArrayLike" has no attribute "copy"
pandas/core/indexes/base.py:3881: error: ArrayLike? has no attribute "view"
pandas/core/series.py:252: error: "ArrayLike" has no attribute "copy"
pandas/io/pytables.py:2240: error: "ExtensionDtype" has no attribute "itemsize"
pandas/io/pytables.py:2245: error: "ExtensionArray" has no attribute "size"
pandas/io/pytables.py:2248: error: "ExtensionArray" has no attribute "codes"
pandas/io/pytables.py:4985: error: "ExtensionArray" has no attribute "codes"
```

Is this a product of ArrayLike being a TypeVar instead of a Union?  At least the last two look like they need to be Categorical instead of ExtensionArray, not sure about the others.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> pandas/core/indexes/base.py:246: error: Type variable "pandas._typing.ArrayLike" is unbound
> pandas/core/indexes/base.py:246: note: (Hint: Use "Generic[ArrayLike]" or "Protocol[ArrayLike]" base class to bind "ArrayLike" inside a class)
> pandas/core/indexes/base.py:246: note: (Hint: Use "ArrayLike" in function signature to bind "ArrayLike" inside a function)

@simonjayhawkins might know better but I don't think you can use a TypeVar for a class instance; the point of it is to make for generic functions and just assigned to a standalone variable that might not make sense

so should we implement something else in pandas._typing for `Union[np.ndarray, "ExtensionArray"]`?
closing to clear the queue
REGR: DataFrame.__setitem__(slice, val) is positional 
State: closed

- [x] closes #31469 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
The change that this reverts was unintentional (was a code-cleanup intended to remove accessing a private `loc` method).  That said, I do think we should deprecate it.
OK, will need a followup issue for that deprecation if you want to push for it.

Let's give @jreback a chance to look, but this should be good to go.
This still needs a backport?
@MeeseeksDev backport to 1.0.x
REGR: Fixed truncation with na_rep
State: closed

Closes https://github.com/pandas-dev/pandas/issues/31447

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @TomAugspurger
BUG: fix reindexing with a tz-aware index and method='nearest'
State: closed

- [x] closes #26683
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

First pandas PR, happy to make changes as needed.  Is a whatsnew entry needed?  And if so, I assume it should go in v1.0.1.rst? 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
Thanks @MarcoGorelli for the guidance!  Is there anything else I can improve? 
Docs build fail seems unrelated, transient network error?  https://github.com/pandas-dev/pandas/pull/31511/checks?check_run_id=419939179#step:6:37
@kanderso-nrel likely a network issue, you could add a whatsnew note in `1.1` and try it again
@jreback all green after #31594
thanks!
CLN: remove IndexEngine.set_value
State: closed

made possible bc Series._values now returns DTA/TDA for datetime64/timedelta64

Small perf improvement

```
In [3]: dti = pd.date_range('2016-01-01', freq='D', periods=10**4)              
In [4]: idx = list('abcdefghijklmnop')                                          
In [5]: arr = np.random.random(len(idx)*len(dti)).reshape(len(dti), -1)         
In [6]: df = pd.DataFrame(arr, index=dti, columns=idx)                          

In [7]: %timeit df2 = df._set_value("2043-05-14", "c", 4)                      
381 µs ± 14.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)   # <-- master
330 µs ± 13 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR

```

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
ok this is fine
BUG: Fixed IntervalArray[int].shift
State: closed

Closes https://github.com/pandas-dev/pandas/issues/31495

In ExtensionArray.shift, we have the note

```
        # Note: this implementation assumes that `self.dtype.na_value` can be
        # stored in an instance of your ExtensionArray with `self.dtype`.
```

I wonder, should we have a method / property like

```
@property
def _can_hold_na_value(self):
    return True
```

And for IntervalArray, we would do something like

```python
@property
def _can_hold_na_value(self):
    return is_float_dtype(self.dtype.subtype)
```


That would let us deduplicate things, since the call to `_from_sequence` would know to not pass `dtype=self.dtype` and trigger a re-inference.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Interval
description: Interval data type

Comments:
cc @jschendel for thoughts on that.
This is still failing for `interval[datetime64]`, since we try to concat a float (all NaN) and datetime. Will poke around a bit.
The datetime64 failure is present on master. I've opened https://github.com/pandas-dev/pandas/issues/31504 for it.
LGTM, agreed it would be nice to find a way to make this work in the general case.
All green now. Sorry about the string of CI failures earlier.
CLN: named parameters for GroupBy.(mean|median|var|std)
State: closed

Drops *args & **kwargs, replace with named parameters for groupby methods mean, median, var & std. Similar to #31473.

This PR has the side effect that the raised error when a parameter is not allowed, is now ``TypeError`` instead of ``UnsupportedFunctionCall``, so technically a API change...

Labels:
name: Groupby

Comments:
Maybe a decorator similar to ``deprecate_kwargs`` could be added. That way we'd get the correct signature, and ``UnsupportedFunctionCall`` would be raised in the decorator.
> looks good to me. I'd be ok with 1.1 but maybe worth a 2.0 target as well

we don’t have the bandwidth or infrastructure to handle this type of versioning
if it’s not targeted then would rather close

this is fine for 1.1
Ok, so I don't do a decorator solution?
Ok, I've added doc strings.

Also, I've found out that ``Groupby.median`` actually didn't accept ``min_count`` in master, but raised an ``AssertionError``. So I removed the parameter from that method signature.

thanks @topper-123 
> Ok, I've added doc strings.
> 
> Also, I've found out that `Groupby.median` actually didn't accept `min_count` in master, but raised an `AssertionError`. So I removed the parameter from that method signature.

happy to have an issue / PR about this.
I did fix that is this PR, so no issue necesary.

So in short, in v1.0.0/master we'd have that ``grp.mean(min_count=1)`` raised a ``AssertionError``, while now it raised a ``TypeError``, which is the same error code as we'd get for ``df.mean(min_count=1)``, so is the correct error type, IMO.
BUG: Array.__setitem__ failing with nullable boolean mask
State: closed

- [x] closes #31446 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@charlesdong1991 I haven't looked closely, but you may need to skip that setitem test for PandasArray. There are some issues with setting sized values in an ndarray.
ahh, thanks a lot for letting me know!! @TomAugspurger 

i am indeed struggling with one failed test which is caused by `PandasArray`
seeing the error message, looks like original array is `pd.array([0, 0, 0, (3,), (4,)])`, while somehow `expected` became: `pd.array([(0,), (0,), (0,), (3,), (4,)])`.

I will skip this for now, and pls let me know if you think a new issue needs to be created or not.

just FYI, the only failed test 
```

self = <pandas.tests.extension.test_numpy.TestSetitem object at 0x14d3eea20>
data = <PandasArray>
[    0,     0,     0,  (3,),  (4,),  (5,),  (6,),  (7,),  (8,),  (9,), (10,),
 (11,), (12,), (13,), (14,...(87,),
 (88,), (89,), (90,), (91,), (92,), (93,), (94,), (95,), (96,), (97,), (98,),
 (99,)]
Length: 100, dtype: object

    def test_setitem_nullable_mask(self, data):
        # GH 31446
        arr = data[:5]
        expected = data.take([0, 0, 0, 3, 4])
        mask = pd.array([True, True, True, False, False])
        arr[mask] = data[0]
>       self.assert_extension_array_equal(expected, arr)

pandas/tests/extension/base/setitem.py:205: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/_testing.py:1057: in assert_extension_array_equal
    obj="ExtensionArray",
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, msg, lobj, robj)
E   AssertionError: ExtensionArray are different
E   
E   ExtensionArray values are different (60.0 %)
E   [left]:  [(0,), (0,), (0,), (3,), (4,)]
E   [right]: [0, 0, 0, (3,), (4,)]
```
thanks @charlesdong1991
TST: preserve dtypes on assignment
State: closed

- [x] closes #31340 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Couldn't find any matching tests (some tests were close but mainly based on datetime).
Have added one. Didnt look deep into the issue since its already fixed
thanks @prakhar987
BUG: objToJson.c - fix return value
State: closed

- [x] closes #31463

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=27217&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=fe74a338-551b-5fbb-553d-25f48b1836e8&l=687

Seems like this warning is somehow causing an error in the users builds of pandas

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Seems to not work on mac:

![image](https://user-images.githubusercontent.com/16733618/73523731-677c2000-4403-11ea-9c43-e51267cb36e1.png)

Not at a computer to check but just might want to return NULL at the end of the function instead of changing it from a void pointer to void. I think original error is that the import_array() macro may only return that in some versions, so can force to work for all by returning NULL (though also a little strange)
Thanks Will - confirmed we no longer see the below warnings with this change.
`control reaches end of non-void function`

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=27360&view=logs&j=3a03f79d-0b41-5610-1aa4-b4a014d0bc70&t=fe74a338-551b-5fbb-553d-25f48b1836e8


thanks @alimcmaster1 ping on green (ex the 37 build which i just have to backport to CI).
@jreback - All green ex 37. I can merge https://github.com/pandas-dev/pandas/pull/31573 into this branch
thanks @alimcmaster1
REGR: Fix TypeError in groupby min / max of period column
State: closed

- [x] closes #31471
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Hello @dsaxton! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-04 21:30:21 UTC
@dsaxton can you fix the merge conflict in the release notes?
Thanks!
BUG: Period[us] start_time off by 1 nanosecond
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Period
description: Period data type

Comments:
Updated to catch overflow correctly.
ping on green
updated with a fix to missing import, which I expect will be causing most CI runs to fail until it gets patched
ping
gentle ping @WillAyd; this is a blocker
DOC: Fix examples in documentation
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @MomIsBestFriend! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-02-27 17:35:24 UTC
Flake8 errors in CI:

##[error]./pandas/core/generic.py:2217:89:E501:line too long (94 > 88 characters)
Restarting azure
A bit unsure about this, but I think it's reasonable. At least we test all what we can test.

You have conflicts to fix. Also, for inline comments (with `#`), please leave two spaces instead of one before the hash. That's PEP-8, not sure why the PEP-8 validation is not complaining, I guess we don't have it active in the CI because there are failing cases.

Also, why `describe` output is not deterministic? I think it should.
Thanks for the fixes @MomIsBestFriend. About the `describe`, why do we need to skip the tests? Isn't the output deterministic?

@jreback, if you want to have another look and see if your comments were addressed...
> Thanks for the fixes @MomIsBestFriend. About the `describe`, why do we need to skip the tests? Isn't the output deterministic?
> 

I haven't got it to work without the skip maybe one of the core developers knows something?
> I haven't got it to work without the skip maybe one of the core developers knows something?

Do you know what was the error?
Thanks for fixing those @MomIsBestFriend.

Do you mind opening an issue for the `describe`? I think the output should be deterministic and we shouldn't need the `SKIP` there. Would be good to have a look and know what's wrong.
REF: use inherit_names for DTI
State: closed

xref #31427, #31433.  With all three of these in, we can then remove indexes.datetimelike.DatetimelikeDelegateMixin

Using inherit_names, we also have the option of making tz, tzinfo, dtype cache_readonly if that turns out to make a difference.

Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype

Comments:
k great
Fix to_csv and to_excel links on read_csv, read_table and read_excel See Also docs
State: closed

- [x] closes #31448
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
Thanks @srvanrell !
TYP: Type annotaion for test decorators
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I am not 100% sure this is something we want, or at least with this implementation.

cc @simonjayhawkins 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @MomIsBestFriend! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-30 15:52:49 UTC
> I am not 100% sure this is something we want, or at least with this implementation.

eventually.

typing was original done like this, but changed in #28134, since `_pytest.mark.structures.MarkDecorator` currently resolves to `Any`.

for the current status of typing in pytest see https://github.com/pytest-dev/pytest/issues/3342
I see that is related to https://github.com/pandas-dev/pandas/issues/28133, I thought about helping there, but I don't know how to do it without stub files.

So I am closing this :)

sorry for your time @simonjayhawkins
DOC: Fixed example section in pandas/core/dtypes/*.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@datapythonista @jorisvandenbossche if any comments
> @MomIsBestFriend can you add those files to the "DOCTESTS" section in code_checks.sh? (or are not yet all docstrings of the files passing?)

@jorisvandenbossche Yes, all the doctests are now passing, I have added it in 2a330b4
Thanks!
BUG: Fix qcut for nullable integers
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Related to #31389 but doesn't close any issues that I'm aware of. Seems the problems were indexing into a numpy array with a BooleanArray and using searchsorted in the presence of pd.NA.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
pls merge master as well
great long on green
@jreback All green, thanks for the review
thanks @dsaxton
DOC: separate section with experimental features in 1.0.0 whatsnew
State: closed

Just a thought I had, to put the experimental features (pd.NA, string, boolean dtypes) in a separate section, better highlighting that those are experimental. 

Since @TomAugspurger basically wants to release now, this is certainly not a blocker :)

Labels:
name: Docs

Comments:
Would maybe put the "Experimental New Features" before the "Other Features", but +1 otherwise.
Doc build finished. Merging.
@Meeseeksdev backport to 1.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 eda68cfcda90efdc9c11c1a77094eb0f13a8b1b4
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #31429: DOC: separate section with experimental features in 1.0.0 whatsnew'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-31429-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #31429 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Handling the backport. Not sure why it wouldn't be clean.
BUG: regression when applying groupby aggregation on categorical colu…
State: closed

Backport of https://github.com/pandas-dev/pandas/pull/31359

Labels:
name: Groupby
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@simonjayhawkins btw, I don't think you need to do any effort in labeling backports, it's the original PR that is relevant 
(it rather adds noise to the label I would say)