BrandonFors/pandas-github-issues huggingface dataset:
PERF: MonthOffset.apply_index
State: closed

- [x] closes #35048
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

```
import pandas as pd
import numpy as np

N = 10000
rng = pd.date_range(start="1/1/2000", periods=N, freq="T")
offset = pd.offsets.MonthBegin()

In [6]: %timeit offset + rng
463 µs ± 19 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)   # <-- PR
508 µs ± 18.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- 1.0.4
736 µs ± 15 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- master
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Frequency
description: DateOffsets

Comments:
thanks
Create sayan2
State: closed

code to extract row from database and replace values within row

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Needs Discussion
description: Requires discussion from core team before further action
name: DataFrame
description: DataFrame data structure

Comments:
take
Is this in reference to a particular issue?
not sure what this is
Add date overflow message to tz_localize (#32967)
State: closed

- [x] closes #32967
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Added an error message to tz_localize when the timestamp overflows. I got a little confused by the history in #32979 as some changes were lost on the force pushes, but I tried to handle all review comments.

Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
> 
> 
> can u modify one of the tests that hits these to check that the error message matches (likely it just tests the type of the error message now)

Hmmm, the only test I found that was actually raising this purposefully was `test_tz_localize_pushes_out_of_bounds` (already covered in this PR). Did I miss anything?
thanks @gabrielNT
Performance regression in stat_ops.FrameMultiIndexOps.time_op
State: closed

- [ ] closes #35050
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

```
import pandas as pd
import numpy as np

levels = [np.arange(10), np.arange(100), np.arange(100)]
codes = [
    np.arange(10).repeat(10000),
    np.tile(np.arange(100).repeat(100), 10),
    np.tile(np.tile(np.arange(100), 100), 10),
]
index = pd.MultiIndex(levels=levels, codes=codes)
df = pd.DataFrame(np.random.randn(len(index), 4), index=index)
%timeit df.std(level=1)
# 9.09 ms ± 59.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) -> master
# 7.39 ms ± 71.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) -> 0.25.3
# 7.21 ms ± 113 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) -> PR
```

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@jreback if we're not branching, should I reopen this and merge to master? see https://github.com/pandas-dev/pandas/issues/34730#issuecomment-656182086
no reverting is not useful here
If we're not reverting, the only three options I can see are:

1. Let the regression stand.
2. Make _get_cythonized_result a 2d computation rather than operating on each column (which I first tried, and backed out once I realized how difficult this would be). I also expect that doing so would also involve a performance regression for single-column frames that go through this path.
3. Add pre/post-processing to _cython_agg_general.

I don't know how viable/difficult 3 would be, and I'd be happy to work on it, but I don't think we want it in 1.1.0 at this point even if it could get done.
CI: MacPython failing TestPandasContainer.test_to_json_large_numbers
State: closed

- [x] closes #35147
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
I am OK with this I think we can add these tests in when we actually make changes to the readers. @jreback ?
can you make this a separate test and just xfail them?
Hello @arw2019! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-15 01:42:21 UTC
Pushed a commit xfailing the test for 32-bit. Opened https://github.com/pandas-dev/pandas/issues/35279 to track it.
My commit broke the other case. Will get it fixed up here soon.
OK, just skipping that test on 32-bit systems, under the assumption that it's supposed to return an int64 dtype when `sys.maxsize - 1` fits in an int64. Leaving updating the test to https://github.com/pandas-dev/pandas/issues/35279.
Merging to get CI passing at MacPython. Thanks @arw2019.
ENH: Add compute.use_numba configuration for automatically using numba
State: closed

- [x] closes #33966
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: numba
description: numba-accelerated operations

Comments:
thanks @mroeschke
TST Verifiy that dropna returns none when called inplace (#35179)
State: closed

Closes https://github.com/pandas-dev/pandas/issues/35179


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @r-toroxel 

mind checking both drop_duplicates and duplicated for series in a similar way?
no. made a pr: https://github.com/pandas-dev/pandas/pull/35210
DOC: fix code snippets for generic indexing
State: closed

The code example for generic indexing is the same as positional/label indexing above. 

I think generic indexing here intended to be referred to something like `df[0:3]`  and `df['bar':'kar']`

Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @maximesong
DOC: Add pint pandas ecosystem docs
State: closed

- [ ] closes #xxxx (N/A)
- [ ] tests added / passed (N/A)
- [ ] passes `black pandas` (N/A)
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` (N/A)
- [ ] whatsnew entry


Labels:
name: Docs
name: Web
description: pandas website

Comments:
@datapythonista thanks for your help with this (and #22582). What did you mean when you said the docs had moved?
Didn't realize we still have the ecosystem page in the docs. We moved it to the [web](https://github.com/pandas-dev/pandas/blob/master/web/pandas/community/ecosystem.md), but I think we finally didn't have an agreement on which version to keep. Do you mind duplicating the changes in the web file here, so they keep synchronized.
Ok done (also added pandas path into the list of accessors for completeness)
Great thanks @znicholls
Fixed Series.apply performance regression
State: closed

Set the option once, rather than in the loop.

Closes https://github.com/pandas-dev/pandas/issues/35047


```python
import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(1000, 3), columns=list("ABC"))
%timeit df.apply(lambda x: x["A"] + x["B"], axis=1)
```

```
# 1.0.4
22.5 ms ± 4.73 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# PR
10.4 ms ± 714 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Nice!
thanks @TomAugspurger
Fixed apply_index
State: closed

Closes https://github.com/pandas-dev/pandas/issues/34580

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
cc @jbrockmendel 
CI is passing, travis just hasn't reported back.
cc @jreback 
All green again.
thanks @TomAugspurger
Infer compression if file extension is uppercase
State: closed

Inferring compression fails for files with uppercase extensions (e.g. `x.zip` works but `y.ZIP` does not)

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Do we do this elsewhere? Outside of Windows most file systems I think are case sensitive so while relatively harmless I also am not sure this is worth doing
I think the path is retained with the original case and couldn't see any uses beyond identifying the compression method. Ran into this issue on OS X, tested locally with `read_csv` and capitalised zip files
> Do we do this elsewhere?

Not that I'm aware of.

> Outside of Windows most file systems I think are case sensitive so while relatively harmless I also am not sure this is worth doing

I *think* it would be okay to add this, but only after checking if we are on Windows.

cc @jreback
Does pathlib offer any abstractions here? I don’t think it’s worthwhile for pandas to implement its own rules for file name equivalency. 

Sent from my iPhone

> On Jul 9, 2020, at 3:02 PM, Jeff Reback <notifications@github.com> wrote:
> 
> ﻿
> @jreback requested changes on this pull request.
> 
> i think this is ok to do everywhere.
> 
> In doc/source/whatsnew/v1.1.0.rst:
> 
> > @@ -1050,6 +1050,7 @@ I/O
>  - Bug in :meth:`~HDFStore.create_table` now raises an error when `column` argument was not specified in `data_columns` on input (:issue:`28156`)
>  - :meth:`read_json` now could read line-delimited json file from a file url while `lines` and `chunksize` are set.
>  - Bug in :meth:`DataFrame.to_sql` when reading DataFrames with ``-np.inf`` entries with MySQL now has a more explicit ``ValueError`` (:issue:`34431`)
> +- Bug in :meth:`io.common.infer_compression` where capitalised files extensions were not decompressed by read_* functions.
> can you add this PR number here (use the issue format). also add 'on windows platforms'
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.

this is fine; we already have our own rules for when to decompress - this is just expanding to capitalized which is not a big deal
thanks @willbowditch
CI: update to isort 5 (#35134)
State: closed

- [x] closes #35134
- [x] 0 tests added / 0 passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
can you rebase
IIUC isort 5 is sorting imports that we not sorted before (in .pyx and imports not at the top)

maybe possible to break some changes here off into smaller PRs before updating isort?
> IIUC isort 5 is sorting imports that we not sorted before (in .pyx and imports not at the top)
> 
> maybe possible to break some changes here off into smaller PRs before updating isort?

I posted a tracking list on the original issue. And I'll break this PR into samller PRs.
> I posted a tracking list on the original issue. And I'll break this PR into samller PRs.

Thanks. I think this makes sense. when we introduced isort, we did it on small batches of files. These can be quickly reviewed and merged.
BUG: transform with nunique should have dtype int64
State: closed

- [x] closes #35109
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

cc @WillAyd 

Removes casting on transformations which go through `_transform_fast`. The result is a reduction that is broadcast to the original index, so casting isn't necessary once special care is taken for categoricals when `observed=False`

If this PR is accepted, I'll close #35130 which is for the same issue.

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
lgtm ping on green.
@jreback green
thanks @rhshadrach
CI: move py36 slow to azure (#34776)
State: closed

- [x] closes #34776
- [x] 0 tests added / 0 passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Thanks for the PR. We would need to configure MySQL and PostGres on Azure Pipelines for this to work.
> Thanks for the PR. We would need to configure MySQL and PostGres on Azure Pipelines for this to work.

Thanks, I'll look into this.
I haven't figured out how to correctlly setup sql on Azure Pipelines. The current commit has sql removed for py36 slow.
thanks @VirosaLi 

I created a followup issue #35160 to add back the sql tests :->
Isn’t this the only place where the slow SQL tests where running? Think we should add back
yes we should 
but ok as a follow up 
> but ok as a follow up

Gotcha missed your comment above
BUG: pd.crosstab fails when passed multiple columns, margins True and normalize True
State: closed

- [x] closes #35144
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you merge master and will look again
> can you merge master and will look again

Done. @jreback I'll go on vacation on Friday would be great if we can get this done before that. Otherwise I'll finish this at the beginning of September.
thanks @CloseChoice lgtm.
Move mark registration
State: closed

Should avoid warnings like

```
  D:\a\1\s\test_venv\lib\site-packages\pandas\tests\plotting\test_series.py:655: PytestUnknownMarkWarning: Unknown pytest.mark.slow - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.slow

```

in https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=38672&view=logs&j=c0130b29-789d-5a3c-6978-10796a508a7f&t=e120bc6c-1f5e-5a41-8f0a-1d992cd2fbfb

Labels:
name: CI
description: Continuous Integration

Comments:
lgtm
thanks @TomAugspurger
Fix regression on datetime in MultiIndex
State: closed

- [x] closes #35015
- [x] tests added / passed
  - `tests.indexing.multiindex.test_datetime:test_multiindex_datetime_columns`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
   - Not needed - this was a regression


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jbrockmendel this is what you had in mind right?
Yes
> can you add a whatsnew, bug fixes 1.1 in conversion section, patch looks good, some comments

@jreback I'm not going to add a whatsnew, as this was NOT a bug in 1.0.5, but was introduced during the development of 1.1 .  I will do the other things.
> > can you add a whatsnew, bug fixes 1.1 in conversion section, patch looks good, some comments
> 
> @jreback I'm not going to add a whatsnew, as this was NOT a bug in 1.0.5, but was introduced during the development of 1.1 . I will do the other things.

thanks
CI: pin isort version
State: closed

- [X] xref #35134
- [X] 0 tests added / 0 passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry. __this is a CI PR. I don't think we touch whatsnew for those.__

### Scope

Remove the `--recursive` keyword from `code_checks.sh`, so that the newest version of isort can run without errors (it does recursive sorting by default).

Pin the version instead, because version 5 has changed the way isort sorts imports. Needs more work to switch safely.


Labels:

Comments:
Perhaps some other changes in 5.0 that will take more time. I was just about to push a commit pinning to <5 for now. Thanks @AlexKirko. 
Thanks @TomAugspurger for the quick fix!
Docs are being fixed in https://github.com/pandas-dev/pandas/pull/35139.
BUG: transform with nunique should have dtype int64
State: closed

- [x] closes #35109
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry



Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
superseded by #35152
TST: Add test for category equalness on applies (#21239)
State: closed

- [x] closes #21239
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The test checks if equalness comparisons through `df.apply()` works correctly for the categorical `dtype`, especially with `np.NaN` and `None` values.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @gabrielNT
CLN: remove kwargs in Index.format
State: closed

Removes kwargs in ``Index.format`` and subclasses. Also changes ``Index._format_with_header`` to have parameter ``na_rep`` not have a default value (in order to not have default value set in two locations).

Related to #35118.

Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @topper-123! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-14 20:50:39 UTC
Updated. @WillAyd?
Ping.
I missed that, I've updated again.
thanks @topper-123
TST add test case for drop_duplicates
State: closed

Add a test case to drop_duplicates for inplace=True.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Hello @r-toroxel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-07 17:55:33 UTC
No.
Its just expanding on test coverage, after looking at pytest-cov and existing tests.
thanks @r-toroxel 

happy to take PRs to verify other inplace ops return None
thanks. yes. created an issue. https://github.com/pandas-dev/pandas/issues/35179
CLN: Index._format_with_header (remove kwargs etc.)
State: closed

Refactor of ``Index._format_with_header`` and subclass methods:
* remove ``kwargs`` for cleaner signature.
* Add return type
* Move functionality specific to ``CategoricalIndex`` to that class.


Labels:
name: Clean
name: Index
description: Related to the Index class or subclasses

Comments:
LGTM
thanks @topper-123
BUG: get_loc with time object matching NaT micros
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@simonjayhawkins any idea what to do about this:

```
Performing static analysis using mypy
pandas/core/indexes/datetimes.py:781: error: "DatetimeIndex" has no attribute "tz_convert"
Found 1 error in 1 file (checked 1024 source files)
```
> @simonjayhawkins any idea what to do about this:

see #34630
cool
BUG: Mixed DataFrame with Extension Array incorrect aggregation
State: closed

- [ ] closes #34520 by reverting #32950


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
cc @jbrockmendel 
ping @jbrockmendel 
I think there are better fixes available.  The underlying problem is that we're calling `nanops.nansum` on an IntegerArray and that raises `ValueError`.  So we should either make `nanops.nansum` handle `IntegerArray` or change the definition of `f` to dispatch to `EA._reduce` cc @jorisvandenbossche has been working on the latter approach
closing in favour of #32867
BUG: reset_index is passing a bad dtype to NumPy
State: closed

- [ ] closes #35095
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @simonjayhawkins
DOC: Fix code formatting and typos in Series.tz_localize
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The missing line feed prevented the example from being formatted as code sample.

Labels:
name: Docs

Comments:
Thanks @StefRe
Extension base test
State: closed

- [x] closes #28488 
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The style is not consistent with guidelines etc, and not at all ready to be merged. 


Labels:

Comments:
Closing until ready for CI testing.
PERF: Fix quantile perf regression
State: closed

Closes https://github.com/pandas-dev/pandas/issues/35049

```pyhon
591 µs ± 29.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

440 µs ± 22.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Should we take a more general look at where we can/should do string formatting more lazily?
Interesting - so the evaluation of q / 100 makes things slower right? I would have thought generally f-strings would be more performant given things like this:

https://stackoverflow.com/questions/43123408/f-strings-in-python-3-6

But I suppose that assumes that the expression needs to be evaluated, which isn't the case here unless an exception occurs
Thanks @TomAugspurger