BrandonFors/pandas-github-issues huggingface dataset:
BUG: Setting DTI/TDI freq affecting other indexes viewing the same data
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The issue this addresses is related to #31218.

Labels:
name: Bug

Comments:
This is going to take more work, mothballing.
this is good thanks. pls push a whatsnew note at some point if you would as this is a slight behavior change.
[WIP] Add remote file io using fsspec.
State: closed

- [x] closes #33452
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
@jrderuiter , is there any way in which I can help here?
ping ?
Closing in favor of https://github.com/pandas-dev/pandas/pull/34266.
Preserving boolean dtype in Series.any/all function with level keyword #33449
State: closed


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Stale
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Hello @KenilMehta! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-12 14:15:01 UTC
@jbrockmendel  @jorisvandenbossche 
As you can see in the following image, one of the CI checks is failing.
![ss](https://user-images.githubusercontent.com/25628466/79246614-0b1f8d00-7e97-11ea-8c4b-69ec770e416b.PNG)


How do I solve this?
For the linting, running `black pandas` should fix it
@jreback 
I have made the changes you requested. Can you please review it once again?
@jreback @jbrockmendel 
Are there any more changes that need to be done?
@KenilMehta if you want to merge master and respond to comments will look
@KenilMehta can you merge upstream/master to resolve conflict.
@KenilMehta is this still active?
Closing for now. @KenilMehta ping us whenever you'd like to continue and we'll reopen!
@arw2019 
Sorry
I couldn't complete the merge. Can you reopen this issue?
I would like to continue this issue.
Note that the issue is also going to be solved indirectly by https://github.com/pandas-dev/pandas/pull/40819, which fixes the actual groupby algorithm to work better for boolean dtypes. 
Thanks for the PR @KenilMehta, but does appear that https://github.com/pandas-dev/pandas/pull/40819 has solved the original issue. Going to close this PR, but we're always happy to have other PRs tackling other issues.
BUG: Series[listlike_of_ints] incorrect on MultiIndex
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is one of the cases that @jorisvandenbossche identified in #33355.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
whatsnew added + green
rebased+green
ENH: Implement IntegerArray.sum
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I think this is mostly interesting in that it allows normalize=True for value_counts on an IntegerArray backed Series, which currently doesn't work:

```python
[ins] In [1]: s = pd.Series([1, 2, 3], dtype="Int64")

[ins] In [2]: s.value_counts(normalize=True)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-2bf1a78353e5> in <module>
----> 1 s.value_counts(normalize=True)

~/pandas/pandas/core/base.py in value_counts(self, normalize, sort, ascending, bins, dropna)
   1252             normalize=normalize,
   1253             bins=bins,
-> 1254             dropna=dropna,
   1255         )
   1256         return result

~/pandas/pandas/core/algorithms.py in value_counts(values, sort, ascending, normalize, bins, dropna)
    725
    726     if normalize:
--> 727         result = result / float(counts.sum())
    728
    729     return result

AttributeError: 'IntegerArray' object has no attribute 'sum'
```

Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
furthermore the numpy_ EA do this in an opposite way, meaning that the ops themselves are defined, and _reduce dispatches TO them.

So we should decide on a single way forward here. I think the way integer array does this is better, e.g. in _reduce.
As mentioned in https://github.com/pandas-dev/pandas/pull/33351#issuecomment-610868353, we should probably have a general discussion about how to deal with those reduction methods in our EAs, otherwise those comments like Jeff's concern above (https://github.com/pandas-dev/pandas/pull/33538) are going to keep coming up in each PR that adds something like this.
thanks @dsaxton !
@dsaxton would you like to open a general issue about this? (https://github.com/pandas-dev/pandas/pull/33538#issuecomment-618949767)
REF: use cached inferred_type when calling lib.infer_dtype(index)
State: closed

Among other things, this will let us avoid a couple of ugly calls in Series

```
        if isinstance(key, Index):
            key_type = key.inferred_type
        else:
            key_type = lib.infer_dtype(key, skipna=False)
```

Cleanup of nearby EA-handling code

Labels:
name: Refactor
description: Internal refactoring of code
name: Clean

Comments:
lgtm. can you merge master and ping on green.
ping
ENH: Clear index cache after reindex
State: closed

Add flag to clear index cache after reindex.

By default, reindex causes index to cache values which potentially increases the memory consumption significantly in the case of multiindexes. 

```
In [2]: idx = pd.MultiIndex.from_product([pd.date_range('2010-01-01', '2015-01-01'), range(1000)], names=['date', 'id'])
In [3]: idx2 = pd.MultiIndex.from_product([pd.date_range('2010-01-01', '2015-01-01'), range(500)], names=['date', 'id'])
In [4]: df =  pd.DataFrame({'a': 1}, index=idx)
In [5]: df.memory_usage(deep=True, index=True) # Original Memory Usage
Out[5]:
Index     7453600
a        14616000
dtype: int64
In [6]: df.reindex(idx2)  # df is still the same as original. 
In [7]: df.memory_usage(deep=True, index=True)  # Memory usage after reindex
Out[7]:
Index    91339680
a        14616000
dtype: int64
```

With  clear_cache=True

```
In [20]: idx = pd.MultiIndex.from_product([pd.date_range('2010-01-01', '2015-01-01'), range(1000)], names=['date', 'id'])
In [21]: idx2 = pd.MultiIndex.from_product([pd.date_range('2010-01-01', '2015-01-01'), range(500)], names=['date', 'id'])
In [22]: df =  pd.DataFrame({'a': 1}, index=idx)
In [23]: df.memory_usage(deep=True, index=True) # Original Memory Usage
Out[23]:
Index     7453600
a        14616000
dtype: int64
In [24]: df.reindex(idx2)  # df is still the same as original.
In [25]: df.memory_usage(deep=True, index=True)  # Memory usage after reindex
Out[25]:
Index     7453600
a        14616000
dtype: int64

```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
Hello @BaiBaiHi! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-14 04:22:14 UTC
@jreback I was trying to be cautious and preserve the default behavior, but after digging into it a bit more, it doesn't seem like keeping the cache there actually provides any benefits. 
I've updated the PR so it automatically clears the mapping. 
@BaiBaiHi closing this PR as stale. maybe raise an issue for discussion.
CLN: Change isocalendar to be a method
State: closed

For consistency with `Timestamp.isocalendar`, `Series.dt.isocalendar` and
`DatetimeIndex.isocalendar` should rather be methods and not attributes.

Followup of #33220, see the discussions following the merge of that PR

Labels:
name: Datetime
description: Datetime data dtype

Comments:
@mroeschke dev build was fixed in #33534 by pinning CPython.  PR rebased, all green.
Thanks @mgmarino for the timely and clean PRs!
CLN: General cleanup in `_libs/lib.pyx`
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
Um... why?  I try to be encouraging, but i dont see the point of any of this.
> Um... why? I try to be encouraging,

You are doing a wonderful job :)

> but i dont see the point of any of this.

I'll remove the noise so you can see the actual changes.

Much better, thanks.  I think parsers.pyx has an unnecessary import of `icom` too.

The lowest-hanging fruit for CLN PRs at this point is going through the tests and identifying giant tests that can be split into targeted tests.
Looks good whenever green
Thanks @MomIsBestFriend
BUG: Grouped-by column loses name when empty list of aggregations is specified
State: closed

- [ ] closes #32580
- [ ] 0 tests added / 0 passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:

Comments:
Hello @echozzy629! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/base.py`](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py):

> [Line 441:17](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L441): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 442:89](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L442): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (106 > 88 characters)
> [Line 443:17](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L443): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 443:89](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L443): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (137 > 88 characters)
> [Line 444:17](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L444): [E265](https://duckduckgo.com/?q=pep8%20E265) block comment should start with '# '
> [Line 445:28](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L445): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 447:34](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L447): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 448:32](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L448): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 450:32](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L450): [E225](https://duckduckgo.com/?q=pep8%20E225) missing whitespace around operator
> [Line 451:89](https://github.com/pandas-dev/pandas/blob/4b7f225558c95ad474535760bc0645eaa58de03e/pandas/core/base.py#L451): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 88 characters)



##### Comment last updated at 2020-04-13 16:55:13 UTC
needs a test for the resolved issue
@echozzy629 closing as stale. ping if you want to continue.
BUG: Fixed concat with reindex and extension types
State: closed

Closes https://github.com/pandas-dev/pandas/issues/27692
Closes https://github.com/pandas-dev/pandas/issues/33027

I have a larger cleanup planned, but this fixes the linked issues for now.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> I have a larger cleanup planned

Can you give a brief overview?  ive got a couple of branches going, may need to mothball them until you're done
> Can you give a brief overview?

Nothing actually written down yet, and no timeline. Just a general sense of "this is probably more complex than it needs to be". So I wouldn't recommend planning around me.
Might conflict/overlap with #33486
needs rebase
just pushed a rebase
Just (force) pushed a rebase.

I think this will fail till https://github.com/pandas-dev/pandas/pull/34338 is in.
@TomAugspurger this will need an update now https://github.com/pandas-dev/pandas/pull/34339 is merged
Merged. No need for `ignore_2d_ea` now that #34339 is in.
This should be good to go now.
thanks @TomAugspurger
TYP: use overload to refine return type of reset_index
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> would rather try to integrate a Literal type
 
can't use Literal yet.

> or actually deprecate inplace.

can't do this till 2.0

closing to clear queue as requested changes not actionable. can potentially reopen depending on outcome of #33274 
> closing to clear queue as requested changes not actionable. can potentially reopen depending on outcome of #33274

reopening as #33274 is closed.

> would rather try to integrate a Literal type

Literal types are not an option for this PR at this stage.
@simonjayhawkins @WillAyd is there a consensus on how to move forward here?
So I wouldn't object to vendoring typing_extensions and doing this with Literal values - @simonjayhawkins do you have any particular thoughts on that?

As is though I think this creates as many nuances as it solves, so not a huge fan of going forward in current state
DOC: Add missing descriptions in See Also section
State: closed

Fixed SR04 for `pandas/core/window/rolling.py`
- [X] xref #28792
- [ ] 0 tests added / 0 passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs

Comments:
Please ref the issue in the description. Also if you merge master the test failures will be resolved
Is this PR about the tests or the docstrings?

@datapythonista are the docstring edits here desired?  I dont find the extra verbosity helpful, but not a strong opinion
> @datapythonista are the docstring edits here desired? I dont find the extra verbosity helpful, but not a strong opinion

Adding descriptions to the see also items is desired. But I agree with your point. I think adding `DataFrame.aggregate: Alias of DataFrame.agg.` is useful for users. The description added here is probably not very useful, and even confusing, since users will wonder which one they should use.

@devjeetr can you have another look, remove the unrelated changes as requested, and provide a more useful description.
@devjeetr closing as stale. ping if you want to continue.
BUG: Fix Categorical.min / max bug
State: closed

- [ ] closes #33450
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
pls do a PR where you test on Series as well
joris pls don’t merge so fast
ENH: clearer error message if read_hdf loads upsupported HDF file (GH9539)
State: closed

- [x] closes #9539
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The error message changed since the original bug report but still could be made more clear.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Thanks @a-y-khan
Added two tests for issue #29697
State: closed

I added two basic tests for `merge` as requested in #29697, one for `how="outer"` and one for  `how="right"`. 

I'm not sure if the way I'm creating duplicate columns for the `expected` dataframe is the recommended way but I could not find anything on this. Please let me know if there's a better way to do this.

- [x] closes #29697
- [x] 2 tests added / 2 passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`





Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Think the best place in the file for this test method would be directly after `def test_merge_suffix`
> Think the best place in the file for this test method would be directly after `def test_merge_suffix`

I gave it a look and I agree. Moved it in the latest commit.
Hello @devjeetr! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-23 18:29:46 UTC
LGTM please fix up the pep8 issues for CI to pass. @devjeetr 
force-pushed a lint fixup
thanks @devjeetr
Removed xfail for issue #31883
State: closed

I removed the xfail in lines 412-414 as discussed in https://github.com/pandas-dev/pandas/issues/31883.

- [x] closes #31883
- [x] 0 tests added / 0 passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM pending green
Great thanks @devjeetr
ENH: Make Series.update() use objects coercible to Series
State: closed

- [x] closes #33215 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Allows `Series.update()` to accept objects that can be coerced into a `Series`, like a `dict`. This mirrors the existing behavior of `DataFrame.update()`.

Labels:
name: Enhancement

Comments:
Hello @dericke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-26 01:39:36 UTC
thanks @dericke
BUG: Fix a bug in 'timedelta_range' that produced an extra point on a edge case (fix #30353)
State: closed

The issue from #30353 came actually from `timedelta_range`.

```python
import pandas as pd

def mock_timedelta_range(start=None, end=None, periods=None, freq=None, name=None, closed=None):
    epoch = pd.Timestamp(0)
    if start is not None:
        start = epoch + pd.Timedelta(start)
    if end is not None:
        end = epoch + pd.Timedelta(end)
    res = pd.date_range(start=start, end=end, periods=periods, freq=freq, name=name, closed=closed)
    res -= epoch
    res.freq = freq
    return res

print(mock_timedelta_range("1day", "10day", freq="2D"))
print(pd.timedelta_range("1day", "10day", freq="2D"))
```

The outputs from `mock_timedelta_range` and from `pd.timedelta_range` are supposed to equivalent, but are not on pandas v1.0.0:
Outputs without this PR:
```
TimedeltaIndex(['1 days', '3 days', '5 days', '7 days', '9 days'], dtype='timedelta64[ns]', freq='2D')
TimedeltaIndex(['1 days', '3 days', '5 days', '7 days', '9 days', '11 days'], dtype='timedelta64[ns]', freq='2D')
```

Outputs with this PR:
```
TimedeltaIndex(['1 days', '3 days', '5 days', '7 days', '9 days'], dtype='timedelta64[ns]', freq='2D')
TimedeltaIndex(['1 days', '3 days', '5 days', '7 days', '9 days'], dtype='timedelta64[ns]', freq='2D')
```

----
It also solve an issue (that fail on master) related to this comment: https://github.com/pandas-dev/pandas/issues/13022#issuecomment-238382933

```python
import pandas as pd
rng = pd.timedelta_range(start='1d', periods=10, freq='d')
df = pd.Series(range(10), index=rng)
df.resample('2D').count()
```
Outputs with this PR:
```
1 days    2
3 days    2
5 days    2
7 days    2
9 days    2
Freq: 2D, dtype: int64
```

----

On the firsts commits I just fixed the issue in `_generate_regular_range`, then I decided to do a refactor and to use the same code that generate ranges in `core/arrays/_ranges.py` for `date_range` and `timedelta_range`.

Linked with #10887.

----

- [X] closes #30353
       closes #13022
       closes #7687 (for loffset when resampling Timedelta)
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
Hello @hasB4K! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-09 16:24:27 UTC
A few comments on the refactor, but i really like the idea of re-using generate_regular_range
LGTM
Just rebased with master
@WillAyd over to you
> lgtm a couple of questions.

Thank you! :slightly_smiling_face:
I have just answered on those and fixed the issue on the test you mentioned earlier.

Let me know if you need anything else!
thanks @hasB4K very nice!
BUG: Allow apply of ufunc over Series of list-like
State: closed

- [ ] closes #33492
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
@jreback I wasn't sure myself but figured I'd put this up. I'll go ahead and close since it's failing and may not be worth supporting anyways
REF: de-duplicate code in libperiod
State: closed

discussed a few months ago, this gets rid of `date_info_from_days_and_time` and instead re-uses `pandas_datetime_to_datetimestruct`

Labels:
name: Refactor
description: Internal refactoring of code
name: Clean

Comments:
cc @WillAyd trying to figure out which of the current CI failures are false-positives, is this one from Travis a real problem?

```
pandas/_libs/tslibs/period.c:9227:11: error: ‘__pyx_v_factor’ may be used uninitialized in this function [-Werror=maybe-uninitialized]

   __pyx_r = (__pyx_v_sub * __pyx_v_factor);

           ^

pandas/_libs/tslibs/period.c:8995:26: note: ‘__pyx_v_factor’ was declared here

   __pyx_t_5numpy_int64_t __pyx_v_factor;

                          ^

cc1: all warnings being treated as errors

building 'pandas._libs.tslibs.resolution' extension
```
I'm assuming your latest commit fixes the issue, but yea I think just telling you that `factor` could potentially be uninitialized if none of the conditions branch, so either need to guarantee it branches or initialize to a value