BrandonFors/pandas-github-issues huggingface dataset:
Bump asv Python version
State: closed

Got this error trying to run the asvs locally and bumping the Python version in asv.conf.json fixed it:

```
   STDOUT -------->
   Processing /Users/danielsaxton/pandas/asv_bench/env/a36d556a9d1611aba4092dc48036d261/project
       Preparing wheel metadata: started
       Preparing wheel metadata: finished with status 'done'
   STDERR -------->
   ERROR: Package 'pandas' requires a different Python: 3.6.10 not in '>=3.7.1'
```

Let me know if this is needed / correct in general or if it was just a thing with my machine.

cc @TomAugspurger 

Closes https://github.com/pandas-dev/pandas/issues/34677

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
If we change this, let's directly bump to Python 3.8?

It has come up a few times before recently (updating the python version, eg https://github.com/pandas-dev/pandas/pull/34699), and the conclusion was to do this after the 1.1 release, so now is the time ;)
Yep, might as well do it now.

Though it would be good if anyone could scan through the recent values to
look for regressions, since that will become harder after the break.

On Wed, Aug 19, 2020 at 1:57 AM Joris Van den Bossche <
notifications@github.com> wrote:

> If we change this, let's directly bump to Python 3.8?
>
> It has come up a few times before recently (updating the python version,
> eg #34699 <https://github.com/pandas-dev/pandas/pull/34699>), and the
> conclusion was to do this after the 1.1 release, so now is the time ;)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/35798#issuecomment-675890034>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIRPXXY5QGEHUXEHW7LSBNZWJANCNFSM4QEOMLVA>
> .
>

Does this then close https://github.com/pandas-dev/pandas/issues/34677? There's a comment there about things failing on Python 3.6 which seems not to be relevant anymore.
Taking a quick look at the asv time series a few that jumped out to me:

https://pandas.pydata.org/speed/pandas/#timeseries.TimeDatetimeConverter.time_convert
https://pandas.pydata.org/speed/pandas/#reindex.LevelAlign.time_reindex_level
https://pandas.pydata.org/speed/pandas/#timedelta.TimedeltaIndexing.time_intersection
https://pandas.pydata.org/speed/pandas/#io.hdf.HDFStoreDataFrame.time_read_store_mixed

Looks like the first two are somewhat recent regressions that may or may not already have issues (I think you reported a bunch of these recently @TomAugspurger), the other two go a bit further back.

Edit: Seems Tom already has an issue open for reindexing, and the datetime converter one might be a necessary for matplotlib compatibility https://github.com/pandas-dev/pandas/pull/35393?
Let's give this a shot. I'll hopefully have time to debug stuff in the next couple days if the benchmark machine doesn't pick this up automatically.
BUG: issubclass check with dtype instead of type, closes GH#24883
State: closed

- [x] closes #24883
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Clean

Comments:
lgtm, can you add a whatsnew note.
updated+green
milestone 1.2 whatsnew 1.1.2 do you want this backported?
i think so yes
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 7e919373236a538b9157359af9c7b8ca3d32e920
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #35794: BUG: issubclass check with dtype instead of type, closes GH#24883'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-35794-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #35794 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CLN/BUG: Clean/Simplify _wrap_applied_output
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

First part of #35412. I could not find a relevant issue for the test change/whatsnew update.

Labels:
name: Bug
name: Groupby
name: Clean
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
cc @WillAyd @jbrockmendel @jreback 

mac timed out, but otherwise green.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Thanks @WillAyd for restarting the test. Passing now.
Hello @rhshadrach! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-26 02:38:15 UTC
@jreback - comment added; only failures are due to pyarrow.

Edit: Reran Travis, all green now.
Benchmarks ran, no errors, and no significant performance changes.

> asv continuous -f 1.1 upstream/master _wrap_apply_cleanup -b ^groupby
> ...
> BENCHMARKS NOT SIGNIFICANTLY CHANGED.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Thanks @jreback, merged.
@rhshadrach as a rule of thumb, we try not to merge our own PRs
@meeseeksdev backport 1.1.x

This will fail but give instructions
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 d90b73bdec5600d188d631150f83633f40c5b8a4
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #35792: CLN/BUG: Clean/Simplify _wrap_applied_output'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-35792-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #35792 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: clean v1.1.1 release notes
State: closed


Labels:
name: Docs

Comments:
@TomAugspurger I don't think we need the subheaders for bugfixes in patch releases
Looks good.
Thanks @jorisvandenbossche I've also updated `Styler` link.

That should just leave and `CategoricalIndex.format()` as dysfunctional (will do final check after merging this before adding release date) 

https://pandas.pydata.org/docs/dev/whatsnew/v1.1.1.html

Could link to Index.format() instead and re-write note for context.
DOC: Updated docstring for Series.str.cat (pandas-dev#35556)
State: closed

- [X] closes #35556
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I have updated the docstring for `Series.str.cat` to make the issue in #35556 more explicit to the user.

Labels:
name: Docs
name: Stale

Comments:
I tried to figure out the cause of the two Azure pipeline tests failing, but could not understand how that would be caused by a change in the docstring.
I'd really appreciate some help or hints regarding this, thanks.
Hi @souris-dev - this is off to a good start, if it's still active could you address Will's comments please?
My apologies for the tardiness (was a bit busy the past week), I'll make the changes and commit in a day. Thanks.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Sorry for the delay, I'm in the process of addressing the comments and will update soon.
Closing for now. @souris-dev let us know whenever you'd like to continue and we'll reopen!
CI: unpin numpy and matplotlib #35779
State: closed

- [x] closes #35779



Labels:
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
```
>>>-------------------------------------------------------------------------
Warning in /home/runner/work/pandas/pandas/doc/source/user_guide/visualization.rst at block ending on line 764
Specify :okwarning: as an option in the ipython:: block to suppress this message
----------------------------------------------------------------------------
/home/runner/work/pandas/pandas/pandas/plotting/_matplotlib/core.py:1547: MatplotlibDeprecationWarning: normalize=None does not normalize if the sum is less than 1 but this behavior is deprecated since 3.3 until two minor releases later. After the deprecation period the default value will be normalize=True. To prevent normalization pass normalize=False 
  results = ax.pie(y, labels=blabels, **kwds)
<<<-------------------------------------------------------------------------
```
thanks @VirosaLi
Added numba as an argument
State: closed

- [x] closes #35550 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
ping @jreback 
thanks @erfannariman
BUG: DataFrame.apply with result_type=reduce incorrect index
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This may xref #35683 pending discussion there.

Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
looks ok to me, could put on 1.1.2
@jreback is this ready? and does this close #35683

test failures are known pyarrow related
updated+green
lgtm can you rebase
rebased+green
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 82f9a0eaa6b7bdf841ace05f5a23f0e54b63dd32
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #35777: BUG: DataFrame.apply with result_type=reduce incorrect index'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-35777-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #35777 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Changed 'int' type to 'integer' in to_numeric docstring
State: closed

- [x] closes #35760 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @edwardkong. The docstring for SparseArray also seems to've been changed in error by the same PR, can you fix that here also?
i'll add this to the 1.1.1 milestone as effectively a regression in the docs. feel free to change if disagree
> can you also revert the changes to pandas/core/arrays/sparse/array.py made in #33093

Thanks @edwardkong Since I want to make sure this is fixed for 1.1.1 I pushed this change. 
@jreback ok to merge (and backport)
Thanks @edwardkong
TST: Verify whether read-only datetime64 array can be factorized (35650)
State: closed

- [x] closes #35650
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @avinashpancham
REGR: follow-up to return copy with df.interpolate on empty DataFrame
State: closed

follow-up to #35543

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
do we want to test this explicitly in this case
> do we want to test this explicitly in this case

sure could add a test
@jreback OK to merge (and backport)
thanks. I still don't tihink this is actually the correct place to do this (as it should be handled gracefully in the internal interpolate), but not a blocker.
@meeseeksdev backport 1.1.x
CI: close sockets in SQL tests
State: closed

closes #35660
closes #29514
 
Broken off from #35711

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Nice this one has been hanging round for awhile. Think it also closes: https://github.com/pandas-dev/pandas/issues/29514
thanks @jbrockmendel 
can the `@pytest.mark.xfail(reason="ResourceWarning #35660", strict=False)` added in #35672 be removed now?
> can the @pytest.mark.xfail(reason="ResourceWarning #35660", strict=False) added in #35672 be removed now?

I expect so, yes
BUG: Raise ValueError instead of bare Exception in sanitize_array
State: closed

- [x] closes #35744  
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Series
description: Series data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
@micahjsmith Can you add a note in /doc/source/whatsnew/v1.2.0.rst (you can reference v1.1.0.rst for notes on similar changes) and also a test for the error type / message?
@dsaxton sure, working on this now
@micahjsmith ping when this is updated, also see if you can track down an update any tests that *just* test for this raising Exception (and not ValueError).
this is hit in tests.series.test_constructors.TestSeriesConstructors.test_constructor
@micahjsmith can you update the appropriate test
@jreback @jbrockmendel updated -- sorry for the delay. 
@micahjsmith can you merge master and ping on green
thanks @micahjsmith
Backport PR #35543 on branch 1.1.x (REGR: Fix interpolation on empty dataframe)
State: closed

Backport PR #35543: REGR: Fix interpolation on empty dataframe

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
known flaky test
=========================== short test summary info ===========================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 72721 passed, 1743 skipped, 1054 xfailed, 58 warnings in 667.83s (0:11:07) =
MAINT: Initialize year to silence warning
State: closed

Initialize year to silence warning due to subtracting from
value that compiler cannot reason must be either initialized
or never reached

closes #35622

- [X] closes #35622
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
LGTM
thanks @bashtage
PERF: Allow jitting of groupby agg loop
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

New performance comparison for 10,000 groups

```
In [1]: In [1]: df_g = pd.DataFrame({'a': range(10**4), 'b': range(10**4), 'c': range(10**4)})

In [2]: In [2]: def f(x):
   ...:    ...:     return np.sum(x) + 1
   ...:

In [3]: df_g.groupby('a').agg(f)
Out[3]:
          b      c
a
0         1      1
1         2      2
2         3      3
3         4      4
4         5      5
...     ...    ...
9995   9996   9996
9996   9997   9997
9997   9998   9998
9998   9999   9999
9999  10000  10000

[10000 rows x 2 columns]

In [4]: %timeit df_g.groupby('a').agg(f)
1.2 s ± 70.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: def f(values, index):
   ...:     return np.sum(values) + 1
   ...:

In [6]: df_g.groupby('a').agg(f, engine='numba', engine_kwargs={'parallel': True})
Out[6]:
            b        c
a
0         1.0      1.0
1         2.0      2.0
2         3.0      3.0
3         4.0      4.0
4         5.0      5.0
...       ...      ...
9995   9996.0   9996.0
9996   9997.0   9997.0
9997   9998.0   9998.0
9998   9999.0   9999.0
9999  10000.0  10000.0

In [8]: %timeit df_g.groupby('a').agg(f, engine='numba', engine_kwargs={'parallel': True})
2.07 ms ± 64.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-21 23:15:46 UTC
@jreback all green
thanks @mroeschke 

as discussed if u can follow up to consolidate 2 clean
REGR: re-add encoding for read_excel
State: closed

- [ ] closes #35753
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Needs tests!

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
> Needs tests!

There were changes to `test_to_excel_output_encoding` in #34464, can that test be updated to cover this.



Does this do nothing or am I misreading it? Doesn't the excel format already define how to interpret its data?
> Does this do nothing or am I misreading it? Doesn't the excel format already define how to interpret its data?

I think it doesn't do what I expected it to do:

It will forward the `encoding` to https://github.com/pandas-dev/pandas/blob/13940c7f3c0371d6799bbd88b9c6546392b418a1/pandas/io/excel/_base.py#L494

but this is the wrong place (`encoding` finally ends up in `TextFileReader` but it seems to be never used there), it should (if we have an Excel file that needs a specific encoding) go to

https://github.com/pandas-dev/pandas/blob/13940c7f3c0371d6799bbd88b9c6546392b418a1/pandas/io/excel/_xlrd.py#L37

since this is the place (or one of the places) where the file is actually opened. After that, all functions seem to operate on strings so there shouldn't be a need for an encoding form there one.

I will put this PR in draft mode until we have an excel file that needs an encoding.

Edit: closed, a future PR might go the route of forwarding the encoding to each "Excel backend" (if there is a need for that)
CI: Unpin Pytest + Pytest Asyncio Min Version
State: closed

- [x] closes #35620

Pytest 6.0.0+ will require pytest-asyncio>=0.12.0 for these async tests to run.

Also worth noting `pytest-asyncio 0.12.0 requires pytest>=5.4.0`. (So maybe we should think about bumping min pytest version)

```
____________________________________________________________________________________ ERROR collecting pandas/tests/indexes/test_base.py _____________________________________________________________________________________
../../.conda/envs/pandas-dev/lib/python3.8/site-packages/pluggy/hooks.py:286: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
../../.conda/envs/pandas-dev/lib/python3.8/site-packages/pluggy/manager.py:93: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
../../.conda/envs/pandas-dev/lib/python3.8/site-packages/pluggy/manager.py:84: in <lambda>
    self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
../../.conda/envs/pandas-dev/lib/python3.8/site-packages/pytest_asyncio/plugin.py:39: in pytest_pycollect_makeitem
    item = pytest.Function(name, parent=collector)
../../.conda/envs/pandas-dev/lib/python3.8/site-packages/_pytest/nodes.py:95: in __call__
    warnings.warn(NODE_USE_FROM_PARENT.format(name=self.__name__), stacklevel=2)
E   pytest.PytestDeprecationWarning: Direct construction of Function has been deprecated, please use Function.from_parent.
E   See https://docs.pytest.org/en/stable/deprecations.html#node-construction-changed-to-node-from-parent for more details.
```

cc @simonjayhawkins 

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
> Also worth noting `pytest-asyncio 0.12.0 requires pytest>=5.4.0`. (So maybe we should think about bumping min pytest version)

pytest 5.4.0 Released: Mar 12, 2020

this is a bit recent to requires as minimum version.

but 

>Pytest 6.0.0+ will require pytest-asyncio>=0.12.0 for these async tests to run.

if we have a minimum version of pytest-asyncio>=0.12.0 then a minimum version of pytest>=5.0.1 this would be inconsistent

does conda not resolve these inconsistencies?
Yep looks like conda solver handles this, in 38 locale build logs below

So I think we can keep pytest min version as you suggested :)

```
pytest                    6.0.1            py38h32f6830_0    conda-forge
pytest-asyncio            0.12.0           py38h32f6830_1    conda-forge
pytest-azurepipelines     0.8.0                      py_0    conda-forge
pytest-forked             1.2.0              pyh9f0ad1d_0    conda-forge
pytest-xdist              2.0.0                      py_0    conda-forge
```
thanks @alimcmaster1 

@simonjayhawkins didn't backport to 1.1.1
ok won't do it now, but will backport after 1.1.1 so marking as still needs manual backport
thanks both - I can backport this post 1.1.1 

Ali
@meeseeksdev backport 1.1.x
CI: Min Pytest Cov Version/Restrict xdist version
State: closed

- [x] closes #35737

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Travis was successful, although I'm not sure why 2.10.1 isn't pulled in without this change. Any idea there?
cc @jreback issue created for unpin: https://github.com/pandas-dev/pandas/issues/35756
Thanks @alimcmaster1 
@meeseeksdev backport 1.1.x
Pass check_dtype to assert_extension_array_equal
State: closed

- [x] closes #35715
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @dsaxton! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-17 17:22:43 UTC
i've milestoned this 1.1.1. It does not appear to be a regression but no harm in backporting this. can put in bugfix section of whatsnew
thanks @dsaxton very nice
@meeseeksdev backport 1.1.x