BrandonFors/pandas-github-issues huggingface dataset:
DOC clean up positional-args deprecation warnings in whatsnew
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @MarcoGorelli i'll merge this as already an improvement. can address @jbrockmendel comment in a follow-up
@MarcoGorelli this hasn't rendered correctly as a list https://pandas.pydata.org/pandas-docs/dev/whatsnew/v1.3.0.html#deprecations
BUG: pytz.AmbiguousTimeError not caught in hypothesis test test_on_offset_implementations (GH41906)
State: closed

- [x] closes #41906
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry: let me know if needed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
is it possible to add the offending test case?
> is it possible to add the offending test case?

Sure, any preference between 1) a separate test with `assertRaises` and 2) just an `@example(dt, offset)` to make sure the case is always tested? 
prefer 1
@jreback @mroeschke 

I took at a quick look at the failing `Africa/Kinshasa` test case (also the original offending test case) under `actions-37-minimum_versions.yaml`. As expected, `pytz=2017.3` does not yet contain the DST information for `Africa/Kinshasa` that triggers `AmbiguoutTimeError` - namely `datetime.datetime(1905, 6, 30, 23, 46, 25),`. 

`python=3.8 pytz=2020.1`:
```
In[6]: import pytz
In[7]: pytz.__version__
Out[7]: '2021.1'
In[8]: pytz.timezone('Africa/Kinshasa')._utc_transition_times
Out[8]: 
[datetime.datetime(1, 1, 1, 0, 0),
 datetime.datetime(1905, 6, 30, 23, 46, 25),      # ----> triggering transition time
 datetime.datetime(1908, 7, 1, 0, 0),
 datetime.datetime(1913, 12, 31, 23, 46, 25),
 datetime.datetime(1919, 8, 31, 23, 30)]
```
`python=3.7 pytz=2017.3`:
```
>>> import pytz
>>> pytz.__version__
'2017.3'
>>> pytz.timezone('Africa/Kinshasa')._utc_transition_times
[datetime.datetime(1, 1, 1, 0, 0), datetime.datetime(1919, 8, 31, 23, 46, 24)]
```
Are you okay with removing this original offending test case in `test_nontick_offset_with_ambiguous_time_error`? 


@jackzyliu yeah go ahead and skip for py37 (going to merge this to 1.4 which is going to have 3.7 removed anyhow), so just xfail for rn.
@jreback Got it! Just updated it accordingly and rebased after master. 
thanks @jackzyliu
Fixed file format based on coding standards
State: closed

Moved late imports, fixed whitespace issues, cleaned up comments.

Labels:

Comments:
Hello @ranOutaNames! Thanks for opening this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/arrays/integer.py`](https://github.com/pandas-dev/pandas/blob/2d0fcc21cc5e5b8a8f1848800eef47aa5f047f80/pandas/core/arrays/integer.py):

> [Line 468:13](https://github.com/pandas-dev/pandas/blob/2d0fcc21cc5e5b8a8f1848800eef47aa5f047f80/pandas/core/arrays/integer.py#L468): [E125](https://duckduckgo.com/?q=pep8%20E125) continuation line with same indent as next logical line


> Moved late imports, fixed whitespace issues, cleaned up comments.

Did you run the tests locally after making these changes? Usually imports done not at the top of the file are because of circular import issues.

Also, the whitespace is a purposeful style choice to separate sections following isort (https://github.com/PyCQA/isort)
Thanks for the PR @ranOutaNames. 

As mentioned, these runtime imports are meant to avoid circular imports so they cannot be changed (easily).

Going to close this PR, but happy to have your help on other issues in the pandas tracker, especially ones labeled "good first issue"
TYP: to_csv accepts IO[bytes] and fix FilePathOrBuffer
State: closed

Two small typing fixes:

- `to_csv` accepts `IO[bytes]` (replace `FilePathOrBuffer[str]` with `FilePathOrBuffer[AnyStr]`)
- pyright was not able to parse `FilePathOrBuffer` because the unconstrained TypeVar `pandas._typing.T` was used when the constrained TyepVar `typing.AnyStr` should have been used: Could not specialize type "Buffer[AnyStr@Buffer]"   Type "T@FileOrBuffer" is incompatible with constrained type variable "AnyStr"

Labels:
name: IO CSV
description: read_csv, to_csv
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
the macOS run failed because it timed out
![image](https://user-images.githubusercontent.com/6618166/121413816-10aa2300-c934-11eb-849d-01e73498152e.png)

thanks @twoertwein 
It _seems_ that the mac build started failing with this PR (it's failing now on master consistently since this PR, and I didn't notice any change in versions of dependencies in the build environment that might otherwise explain it).

Above it was mentioned that the Mac build timed out, but the last commit actually has the failing build without time out. 

It's a bit strange (and didn't look further in detail, just reporting here), the actual tests seems to pass (pytest prints the end result), but then there is:

```
ci/run_tests.sh: line 31:  2533 Segmentation fault: 11  sh -c "$PYTEST_CMD"
##[error]Bash exited with code '139'.
```

Although nothing in the diff of this PR would explain a new failure, though
There is also a

```
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "pyarrow/types.pxi", line 2638, in pyarrow.lib._unregister_py_extension_types
SystemError: unregister_extension_type() method: bad call flags
```

in the output
Noticed there is already an issue about it -> https://github.com/pandas-dev/pandas/issues/41937
DOC: update the msgpack IO section to not refer to pyarrow.serialize
State: closed

Closes #39956

Stop referring to pyarrow.serialize/deserialize, since those are deprecated (see https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization)

Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
sure
BUG: to_hdf append string column to incompatible column
State: closed

- [ ] closes #xxxx
- [X] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Hello @lrepiton! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-14 08:17:47 UTC
thanks @lrepiton 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
ENH: `Styler.apply_index` and `Styler.applymap_index` for conditional formatting of column/index headers
State: closed

Checks an item on #41648 and #41693

Currently Styler has the `apply` and `applymap` methods to **perform conditional styling on the data elements**.

Recent additions to `Styler.set_table_styles` allows styling of index and column headers if the key-labels are directly specified. But this solution cannot work with LaTeX, nor excel output, and is obviously not conditionally applied.

This PR adds the mechanics to **perform conditional styling on per level index labels**

Follow-ons are:

- [x] updating LaTeX templates to incorporate the styles. (#41993)
- [x] updating Excel output to incorporate the styles. (#41995)
- [x] update Styler user guide

![Screen Shot 2021-06-13 at 10 32 40](https://user-images.githubusercontent.com/24256554/121800538-d3df7400-cc32-11eb-9aaf-8124ba221171.png)


Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
> wouldn't apply_axis be more appropriate? e.g. don't see _header anywhere else in the codebase
> 
> do we need both (i get that we have symmetry, but is this overkill)?

i came up with 'header' becuase in the HTML that styler renders the index and column header cells are `<th>` (table header) elements, and these are the ones targeted by this method.

I'm not tied to the name but `apply_axis` feels a bit odd, since `axis` is already used by pandas for a common meaning and not really to do with impacting the values of the index or column headers. 

`apply_index(axis="columns")` might be an alternative to make it clear that the function is being applied to either the index values or the column header values??   
thanks @attack68 very nice!
TYP: use from __future__ import annotations more - batch 1
State: closed

I'm fairly sure that we agreed we would do this but leave it to just before the 1.3 release to reduce the chance of auto backports failing


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
after you have changed everything should create a rule to avoid Optional if this is possible.
> after you have changed everything should create a rule to avoid Optional if this is possible.

it is still needed when creating aliases although I could move all aliases definitions into if TYPE_CHECKING blocks or into pandas._typing where that makes sense.

A couple of other points that have arisen from this exercise..
we are using Optional/Union alot in docstrings
some imports from pandas._typing instead of typing... could maybe limit what's reexported from pandas._typing.
I think is we have a rule that if we import from typing we also use `from __future__ import annotations` and pyupgrade and flake8 should make sure that the typing imports of Union, Optional etc are only used in casts and aliases.
> I think is we have a rule that if we import from typing we also use `from __future__ import annotations` and pyupgrade and flake8 should make sure that the typing imports of Union, Optional etc are only used in casts and aliases.

Might be easier to require that every file (except for __init__.py , api.py, tests, setup.py, and probably just a handful of others) contains this import. This could be done with `pygrep` and `--negate`, I'll open an issue for this
sounds good
can you rebase thisone
> after you have changed everything should create a rule to avoid Optional if this is possible.

Sorry for barging in but can someone link me to context for this? My understanding was that the recommendation is to prefer `Optional` over a union with `None`  (but maybe that changed with support for the `|` operator in type hints?)
@mwaskom yeah it's just to use the latest `| None` syntax, rather than anything against `Optional` specifically
BENCH: Remove unnecessary random seeds
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

We are implementing a common seed explained in https://asv.readthedocs.io/en/latest/writing_benchmarks.html.

Added a code check as well


Labels:
name: CI
description: Continuous Integration
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
thanks @mroeschke
ENH: Add online operations for EWM.mean
State: closed

- [x] xref #41673
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
After a discussion with @jorisvandenbossche have added a new tag so that the 1.3 milestone can be added to PRs that are for 1.3 but not needed for the rc.

Have added a blocker tag here since I'm not sure whether this was to be included in the rc. feel free to remove
this is not a blocker for the rc (though would like to get it in)
OK barring anything else then, assume we are releasing tomorrow if this is not a blocker. If you really want to include it. add back the blocker tag and I won't release.
likley to merge if ci checks run ok this time
this lgtm.

any comments @pandas-dev/pandas-core 
Thanks @mroeschke
PERF: nancorr_spearman fastpath
State: closed

xref #40956

The diff looks much more complicated than it is because of indentation changes - the change is basically allowing a fast path for computation without nans by pulling up a slightly modified computation of `sumx, sumxx, sumyy` before missing value handling.

Benchmarks:
```
before           after         ratio
[b73c38e2]       [efeac916]
<perf/grp_cumsum_int~7>       <nancorr_spearman_perf2>
            70.3M              71M     1.01  stat_ops.Correlation.peakmem_corr_wide('spearman')
-     1.39±0.06ms      1.01±0.02ms     0.73  stat_ops.Correlation.time_corr('spearman')
         669±20μs         641±40μs     0.96  stat_ops.Correlation.time_corr_series('spearman')
-      26.3±0.6ms       8.36±0.3ms     0.32  stat_ops.Correlation.time_corr_wide('spearman')
          491±3ms          473±9ms     0.96  stat_ops.Correlation.time_corr_wide_nans('spearman')
       8.04±0.5ms       7.97±0.6ms     0.99  stat_ops.Correlation.time_corrwith_cols('spearman')
          215±1ms          213±2ms     0.99  stat_ops.Correlation.time_corrwith_rows('spearman')
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
can you  add this PR number to the existing perf note on corr, ping on green.
Added a small fix and test in latest commit for edge case of min periods > frame length
REF: simplify indexes.base._maybe_cast_data_without_dtype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
> any perf change?

i did some timeits and it was a wash
Replaced value assignment by location accessor.
State: closed

Changed self.obj[key] = value to self.obj.loc[:, key] = value to disable warning.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Needs Info
description: Clarification about behavior needed to assess issue
name: Stale

Comments:
Thanks for the pr @kosmitive.  Is there an issue associated with this? What is the behavior change here (and if there is, it should be tested). Also seems like a change that might be performance sensitive

I get a warning otherwise that the old command operates on a copy, so I thought changing it to in-place assignment would make sense and be a really quick fix. But I didn't do further investigation on that line of code. 
@mzeitlin11 @jreback  Here is the warning I get with python 3.9

.../.venv/lib/python3.9/site-packages/pandas/core/indexing.py:1597: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
@kosmitive pls file an issue with a reproducible test. The code you are touching is not trivially changed and has quite some implications.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but agreed it would be better if an issue could be raised first so the entirety of the issue could be understood before diving into a fix. Closing.
REF: avoid mutating Series._values directly in setitem but defer to Manager method
State: closed

There are currently two cases where the Series code directly modifies the underlying values itself (`self._values[..] = ..`), while most setitem code paths end up going through a specific indexing method that all end up in the Manager doing the actual modification of the values.

Apart from being a bit inconsistent (it's good to have all eventual modifications consistently happening inside the manager, I think), I also need to move this to the manager for being able to check for copy-on-write in https://github.com/pandas-dev/pandas/pull/41878. 
Since I think this is a generally useful change, moving it out of the CoW PR.

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
I lean towards using a new method instead of overloading `setitem`, which in general can cast.
> let's try not to overload setitem (e.g. its always inplace), so not in place is odd.

@jreback to be clear, it's the existing manager `setitem` method that is _not_ inplace (the odd thing), and I am adding a variant that _is_ inplace .

> I lean towards using a new method instead of overloading setitem, which in general can cast.

I can certainly add a new method (eg `setitem_inplace(..)`) instead of adding a keyword (`setitem(.., inplace=True)`), no problem. But just note that this is really doing the same operation. It's just that in some cases, we know that the setitem will in practice be inplace, and we can be stricter in that case by not going through the general Block method that indeed can cast (that optimization was now done inside Series, and this moves it to the manager). Given that it is about the same operation but with different semantics, I think a keyword actually makes sense.
The last commits adds the version with a separate method instead of keyword. Either way is fine for me (it's also easy to revert)
BTW im picky on the implementation, but in general i think this is a very good idea
Any further response here?
i will look at some point
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This is not stale, but waiting on further comments.
i think the failures are related here, e.g. the stacklevel is changed. so prob need to use `find_stack_level` instead of hard-coding for a lot of the indexing tests.
> i think the failures are related here, e.g. the stacklevel is changed. so prob need to use `find_stack_level` instead of hard-coding for a lot of the indexing tests.

actually these are likley https://github.com/pandas-dev/pandas/pull/42857

but this PR may affect as well (so let's fix that one first)
BUG: unstack with object dtype of tzaware timestamps
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
this is ok for 1.3, can you rebase
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
It seems that it's not fixed for the case where mask_all is true. I'll send another PR
https://github.com/pandas-dev/pandas/blob/42082a80d0c3b3483f08b887271b1b08c66551e1/pandas/core/reshape/reshape.py#L294
CLN/PERF: no need for kahan for int group_cumsum
State: closed

Surprised at lack of impact here - doesn't noticeably affect benchmarks.

Targeting the cython algo specifically shows an improvement (but smaller than I'd expect given the removed operations):
```
import numpy as np
import pandas._libs.groupby as libgroupby

N = 4_000_000
vals = np.random.randint(0, 10, (N, 5), dtype=np.int64)
result = np.empty_like(vals)

%timeit libgroupby.group_cumsum(result, vals, np.ones(N, dtype="int"), 1, False)
# 28.9 ms ± 805 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # this pr
# 37.3 ms ± 206 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   # master
```

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Clean

Comments:
hmm seemingly unrelated failures. maybe on master? cc @jbrockmendel 
thanks @mzeitlin11
Tst interval index NaN uniqueness
State: closed

add test cases for interval-indexer with unique NaN value and non-unique NaN values.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Interval
description: Interval data type

Comments:
Thanks @r-toroxel is this not currently tested or is it one of the open issues that are fixed on master

is this related to #41831 and the associated open PR #41863
@simonjayhawkins yes, these were not tested before and seem to be related to #41831.
thanks @r-toroxel 
@jbrockmendel does this close #41831 ?
@jreback I'm afraid it does not fix the bug and hence close the issue. just a test added.
PERF: clipping with scalar
State: closed

Removes conversion of boolean series to array-like in `_clip_with_scalar`.

- [x] closes #41434
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

I may also add type annotations to `_clip_with_scalar`, let me know if that's welcome.


Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> I may also add type annotations to `_clip_with_scalar`, let me know if that's welcome.

always welcome. separate PR OK
can you show the asv's on this.
Here it is:
```
[ 50.00%] · For pandas commit ef773a6d <time-clip-perf> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· series_methods.Clip.time_clip                                   ok
[ 75.00%] ··· ======== =============
                 n                  
              -------- -------------
                 50       492±40μs  
                1000      502±30μs  
               100000   1.46±0.05ms 
              ======== =============

[ 75.00%] · For pandas commit 3f67dc33 <master> (round 2/2):
[ 75.00%] ·· Building for conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[100.00%] ··· series_methods.Clip.time_clip                                   ok
[100.00%] ··· ======== =============
                 n                  
              -------- -------------
                 50       540±30μs  
                1000      611±40μs  
               100000   1.79±0.09ms 
              ======== =============
```
that we can also compare with before the regression:
```
[ 50.00%] · For pandas commit 4f3acf10 <stash~155> (round 2/2):
[ 50.00%] ·· Benchmarking conda-py3.8-Cython0.29.21-jinja2-matplotlib-numba-numexpr-numpy-odfpy-openpyxl-pytables-pytest-scipy-sqlalchemy-xlrd-xlsxwriter-xlwt
[ 75.00%] ··· series_methods.Clip.time_clip                                   ok
[ 75.00%] ··· ======== =============
                 n                  
              -------- -------------
                 50       411±40μs  
                1000      449±20μs  
               100000   1.56±0.06ms 
              ======== =============
```
thanks @TLouf