BrandonFors/pandas-github-issues huggingface dataset:
Backport PR #37613 on branch 1.1.x: BUG: read-only values in cython funcs
State: closed

Backport PR #37613

will update milestones on PR and associated issues once this is green-ish. Will also need to update release notes on master. (move from 1.2 to 1.1.5)

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@simonjayhawkins whats your preferred way to move forward here?
i would prob just close this backport (and fix the release note), but either way ok
any resolution on this one (happy to just not backport as well).
leave it open, I keep putting this off!, but the open issues on 1.1.5 milestone is getting less, so can't put it off much longer!
closing for now. nothing needs to change on master as milestone and release note are 1.2 for #37613
Added dataclass into dataframe API reference
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
Added dataclass constructor for dataframe into the Pandas in pandas/init.py line 363 and 426.

Labels:
name: Docs

Comments:
thanks @taytzehao
ENH: named agg with multiple columns
State: closed

- [ ] closes #29268
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
pls add tests always

in fact tests should be done before any code changes to make sure they behave properly 
Thanks for the heads up @jreback , actually I was hoping to get some feedback which was the reason I pushed this as a draft while developing to get some comments. Some parts are challenging since I am not sure if I make the right decisions.
Hello @erfannariman! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-09-12 11:44:58 UTC
@jreback @ivanovmg I made some progress, but I am at the point that I need to pass the DataFrame as a obj to the aggregator instead of a Series so we can do the aggregation over the columns.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@erfannariman status here?
> @erfannariman status here?

This went stale, but is definitely one I would like very much to finish. I just think it gets quite complex on some points where I could use some guidance. Will take asap to clean up, merge master etc so it's updated.
@erfannariman Can you merge master/update? Thanks.
Thanks @erfannariman for the PR. closing as stale. ping if you want to continue.
> Thanks @erfannariman for the PR. closing as stale. ping if you want to continue.

Started to work on this PR again, can you re-open? Thanks @simonjayhawkins 
this is quite old, happen to reopen if actively worked on.
ENH: memory_map for compressed files
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Apply memory mapping first and then compression. The existing order didn't work.

This adds support for memory mapping in `read_csv` when compression is specified (python and c engine).

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-04 20:01:25 UTC
@jreback green'ish :)
BUG: read-only values in cython funcs
State: closed

- [x] closes #34857
- [x] closes #37264
- [x] closes #37312
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@simonjayhawkins this might be worth backporting
> @simonjayhawkins this might be worth backporting

sure. will wait for #37628 to be merged to avoid merge conflicts in release notes.
`to_datetime` not works when `unit` set,
```python
import pandas as pd
import numpy as np

a = np.array([1,2,3])
a.setflags(write=False)
pd.to_datetime(a, utc=True, unit='s')
```
This would call `tslib.array_with_unit_to_datetime` eventually.
Add tests for categorical with null ea as input
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

cc @jreback Is this what you had in mind for ExtensionArrays and Categoricals?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Categorical
description: Categorical Data Type
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@phofl can you merge master and ping on green.
BUG: nunique not ignoring both None and np.nan
State: closed

- [x] closes #37566
- [x] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Hello @GYHHAHA! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-10 01:20:07 UTC
looks good, just @phofl comment i think is left, ping on green.
cc @jreback 
thanks @GYHHAHA very nice
BUG: set index of DataFrame.apply(f) when f returns dict (#37544)
State: closed

- [x] closes #37544
- [x] tests (added) updated / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks, sure.
thanks @ma3da
DEPR: DataFrame/Series.slice_shift
State: closed

- [x] ref #18262
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hello @erfannariman! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-03 14:41:23 UTC
thanks @erfannariman very nice
BUG: Index.where casting ints to str
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Two things going on here (can separate if asked, but they got mixed together in the process).

1) don't re-raise in `DatetimeLikeIndexMixin.where`, so we end up giving the more standard exception message.
2) Use validate_fill_value in Index.where, and have that disallow strs for numeric.  This fixes the following in master:

```
>>> idx = pd.Index(range(3))
>>> mask = np.array([True, False, True])
>>> 
>>> idx.where(mask, "2")
Index(['0', '2', '2'], dtype='object')
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@pganssle I'm seeing this in the logs for many of the windows py38 builds:

```
Exception ignored in: 'pandas._libs.tslibs.tzconversion._tzlocal_get_offset_components'
Traceback (most recent call last):
  File "C:\Miniconda\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 144, in fromutc
    return f(self, dt)
  File "C:\Miniconda\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 261, in fromutc
    _fold = self._fold_status(dt, dt_wall)
  File "C:\Miniconda\envs\pandas-dev\lib\site-packages\dateutil\tz\_common.py", line 196, in _fold_status
    if self.is_ambiguous(dt_wall):
  File "C:\Miniconda\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 254, in is_ambiguous
    naive_dst = self._naive_is_dst(dt)
  File "C:\Miniconda\envs\pandas-dev\lib\site-packages\dateutil\tz\tz.py", line 260, in _naive_is_dst
    return time.localtime(timestamp + time.timezone).tm_isdst
OSError: [Errno 22] Invalid argument
```

Any idea why _naive_is_dst would throw an OSError?
@jbrockmendel Yes, that will happen any time the timestamp is negative or close to negative (i.e. pre-1970), because at the OS level Windows doesn't support negative timestamps. It's documented in [`fromtimestamp`](https://docs.python.org/3/library/datetime.html#datetime.date.fromtimestamp).

There's a [bug here](https://bugs.python.org/issue36439), and we're hoping in CPython that eventually we can write our own timestamp conversion logic that has consistent behavior cross-platform, but it's particularly tough when it involves interacting with locales and whatnot.
this maybe an outstanding issue (of course finding it might be non-trivial)
CLN refactor rest of core
State: closed

Some refactorings found by Sourcery https://sourcery.ai/

I've removed the ones of the kind
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master ping on green
I still need to address https://github.com/pandas-dev/pandas/pull/37586#discussion_r519926657 , will ping when it's ready
moving off 1.2 until fixed up

Sure, no problem, though I believe I've responded to the review comments, I think it should be ready now
@MarcoGorelli can you rebase and see if you can get passing
ok this looks fine, but a ci /checks is failing. ping on greenish
thanks @MarcoGorelli this is a nice cleanup. in the future pls do these types of changes in smaller batches.
CLN refactor core dtypes
State: closed

Some refactorings found by Sourcery https://sourcery.ai/

I've removed the ones of the kind
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
look ok. these may have broken things. 
Indeed - the refactoring
```diff
-    if len(types) == 0:
+    if not types:
```
was wrong, as `types` could be a `Series` here (as in the test `pandas/tests/arrays/sparse/test_accessor.py::TestFrameAccessor::test_to_coo`), in which was `not types` raises
```
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

Have fixed up and changed the annotation
can you merge master ping on green
> can you merge master ping on green

Green, barring (probably unrelated?) travis error:
```
____________________________ test_deprecated_kwargs ____________________________

[gw0] linux -- Python 3.7.8 /home/travis/miniconda3/envs/pandas-dev/bin/python

    def test_deprecated_kwargs():

        df = pd.DataFrame({"A": [2, 4, 6], "B": [3, 6, 9]}, index=[0, 1, 2])

        buf = df.to_json(orient="split")

        with tm.assert_produces_warning(FutureWarning):

>           tm.assert_frame_equal(df, read_json(buf, "split"))

pandas/tests/io/json/test_deprecated_kwargs.py:15: 

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <contextlib._GeneratorContextManager object at 0x7fc4d8ce5110>

type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):

        if type is None:

            try:

>               next(self.gen)

E               AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed <ssl.SSLSocket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.20.0.66', 55192), raddr=('52.216.144.227', 443)>"), '/home/travis/build/pandas-dev/pandas/pandas/core/dtypes/base.py', 435), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedWriter name='/tmp/tmpnm7c5epi.xls'>"), '/home/travis/build/pandas-dev/pandas/pandas/core/dtypes/base.py', 435), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedWriter name='/tmp/tmplgig23ur.xlsx'>"), '/home/travis/build/pandas-dev/pandas/pandas/core/dtypes/base.py', 435)]

../../../miniconda3/envs/pandas-dev/lib/python3.7/contextlib.py:119: AssertionError
```
moving off 1.2 until ready
Sure, no problem (though as far as I can tell, this is ready - anything else need changing?)
> Sure, no problem (though as far as I can tell, this is ready - anything else need changing?)

I think it was fine, just want to make sure after merge
sure, have merged
thanks @MarcoGorelli
CLN refactor core indexes
State: closed

Some refactorings found by Sourcery https://sourcery.ai/

I've removed the ones of the kind
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Index
description: Related to the Index class or subclasses

Comments:
can you rebase this again to make sure failures are not systemtic
The problem is that
```
pytest pandas/tests/reductions/test_reductions.py::TestIndexReductions::test_minmax_period
```
gets to
```python
        if self.hasnans:
            if skipna:
                min_stamp = self[~self._isnan].asi8.min()
            else:
                return self._na_value
        else:
            min_stamp = i8.min()
        try:
            return self._data._box_func(min_stamp)
        except ValueError:
            return self._na_value
```
and throws a ValueError on `min_stamp = self[~self._isnan].asi8.min()`

----

latest commit addresses that
can you merge master ping on green
@MarcoGorelli if you rebase and fix this up we can get into 1.2
> @MarcoGorelli if you rebase and fix this up we can get into 1.2

sure, have fixed conflicts and responded to review comments
can you rebase 
thanks @MarcoGorelli
CLN refactor core/arrays
State: closed

Some refactorings found by Sourcery https://sourcery.ai/

I've removed the ones of the kind
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
this is failing checks
> this is failing checks

Yes, the change
```diff
-        for i in range(len(self)):
-            yield self[i]
+        yield from self
```
was throwing a RecursionError
thanks @MarcoGorelli
CLN refactor non-core
State: closed

Some refactorings found by Sourcery https://sourcery.ai/

I've removed the ones of the kind
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MarcoGorelli
sourcery refactoring
State: closed

I tried applying https://sourcery.ai/github/ , seems like there's some nice suggestions here. I removed the ones where they change things like
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```

Labels:

Comments:
+1 in principal. opinionated is good imo on a project with a large number of contributors.
@simonjayhawkins do you think we should keep the
```diff
- if param:
-     var = a
- else:
-     var = b
+ var = a if param else b
```
-style changes then?
those changes can sometimes be beneficial for mypy, sometimes the opposite. It depends on the inference. So i'm ok with allowing those changes and seeing if we run into issues.
DOC: Add windows.rst
State: closed

Adding a dedicated `user_guide/windows.rst` for `rolling/expanding/ewm` operations.

Mainly ported over from `computation.rst`. 


Labels:
name: Docs
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke very nice.

can you followup with a pointer from the computation section to the new windowing section (or maybe a note at the top of that file), e.g. a way for folks to see that things moved.
BUG: to_dict should return a native datetime object for NumPy backed dataframes
State: closed

- [x] closes #21256
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Added a fix for datetime. Added tests for datetime and for bool

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Ready for re-review (CI  green + addressed comments)
Ready for re-review. Addressed comments + CI green
thanks @arw2019 very nice
PERF: faster numeric indexes comparisons when self is identical to other 
State: closed

Further performance improvement over #37130, this time improving all numeric indexes. The improvement is larger for `Int64Index`, because that doesn't need to check for `na` values.

Examples:

```python
>>> n = 100_000
>>> idx1 = pd.Int64Index(range(n))
>>> idx2 = idx1.view()
>>> %timeit idx1 == idx2
145 µs ± 2.43 µs per loop  # master
11.4 µs ± 290 ns per loop  # this PR
>>> idx3 = pd.Float64Index(range(n))
>>> idx4 = idx3.view()
>>> %timeit idx3 == idx4
104 µs ± 2.12 µs per loop  # master
49.8 µs ± 886 ns per loop  # this PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @topper-123
BUG: Fix bug in combine_first with string dtype and only NA
State: closed

- [x] closes #37519
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This caused bugs in ``Index.union()`` and ``Index.join()``  with ``how=outer`` too, but they are called from combine_first. Could add test, if this would be preferable. Hope Adding ExtensionArray to the if condition does not crash anything else. Run most categorical tests already locally.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
this is fine, thanks. likely we would want some more direct testing of categoricals here (e.g. pass a dtype='string' EA array in (or actually *any* EA array) and it should hit this path). if you can followon with some additional tests
CLN: de-duplicate asof_locs
State: closed

We might want to deprecate allowing DatetimeIndex `where` in `PeriodIndex.asof_locs`.  We only have one test for it, and allowing it is inconsistent with how we treat other comparisons (unless we want to start allowing dt64 in PeriodArray.searchsorted...)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
> We might want to deprecate allowing DatetimeIndex `where` in `PeriodIndex.asof_locs`. We only have one test for it, and allowing it is inconsistent with how we treat other comparisons (unless we want to start allowing dt64 in PeriodArray.searchsorted...)

can you point to where this is (confusing because we use the variable name `where`)
> can you point to where this is (confusing because we use the variable name where)

tests.series.methods.test_asof test_periodindex
> > can you point to where this is (confusing because we use the variable name where)
> 
> tests.series.methods.test_asof test_periodindex

ok why don't you create an issue with a pointer to the code.
ENH: add DataFrame.is_unique method
State: closed

Currently, to see columns with unique values, we have to do `df.apply(lambda x: x.is_unique)`, which is a bit awkward.

IMO checking for unique columns in a DataFrame is a common enough need to justify a method for that specific purpose.

Xref: #11948

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
For a DataFrame, `is_unique` could also be interpreted as whether the DataFrame has all unique rows?
@jorisvandenbossche that's true, and could be fixes by having an `axis` parameter. I didn't ínclude it, because `duplicated` doesn't have one and worried that doing `self.T.is_unique()` would change the types for the values and could give wrong results. Not sure about that one, actually.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@topper-123 this was close, can you merge master and move the note to 1.3
Ok, done.
Looks pretty good overall. Should there be a corresponding `Series.is_unique` method as well? There is a `Series.duplicated` method (as your referenced `duplicated` earlier)
Thanks for the PR, but it appears that it's gone stale. Please let us know if you'd like to continue by merging master and targeting for 1.4.
REGR: pd.to_hdf(..., dropna=True) not dropping missing rows
State: closed

- [x] closes #35719
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Ready for re-review (CI green + addressed comments)
thanks @arw2019 very nice
REF: move IntervalIndex.equals up to ExtensionIndex.equals
State: closed


Labels:
name: Interval
description: Interval data type

Comments:
any more equals you can remove because of this? (followon)
> any more equals you can remove because of this? (followon)

Not without API changes, but it may be worthwhile to get these more strict/consistent
>> any more equals you can remove because of this? (followon)

> Not without API changes, but it may be worthwhile to get these more strict/consistent

So the least invasive would be CategoricalIndex.  Changing it to match this strict version (actually just removing it) breaks 5 tests.  Four of these are asserting `index.equals(index.astype(object))`, which we test for all Index subclasses except IntervalIndex (which suggests IntervalIndex possibly should support this)

The fifth is an indexing test that _may_ indicate a bug elsewhere:

```
        df = DataFrame(np.random.randn(3, 3), index=list("ABA"), columns=list("XYX"))
        cdf = df.copy()
        cdf.index = CategoricalIndex(df.index)
        cdf.columns = CategoricalIndex(df.columns)
    
        exp_index = CategoricalIndex(list("AA"), categories=["A", "B"])
>       expect = DataFrame(df.loc["A", :], columns=cdf.columns, index=exp_index)
[...]
E           ValueError: cannot reindex from a duplicate axis
```
Add ss02
State: closed

- [x] closes #25113, closes https://github.com/pandas-dev/pandas/issues/36745
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs

Comments:
@jreback I resolved your comments, I think this is ready for review. Test that is failing in CI is pandas/tests/test_sorting.py which passes on my machine and is unrelated to this PR.
thanks @abhishekmangla
Fixed the documentation for pandas.DataFrame.set_index inplace parameter
State: closed

Fixed the documentation for `pandas.DataFrame.set_index` 's `inplace` parameter.
Added proper explanation for True case.


Labels:

Comments:
Hello @PurushothamanSrikanth! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-13 16:08:20 UTC
@arw2019 @pep8speaks Is it fine now?
thanks @PurushothamanSrikanth