BrandonFors/pandas-github-issues huggingface dataset:
REF: separate out indexer/mask preprocessing code in algorithms.take_nd
State: closed

Precursor for https://github.com/pandas-dev/pandas/pull/39692 (just moving code into a separate helper function)

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
FWIW im thinking a lot of `take` code could go into an array_algos.take
thanks. yeah `array_alogs.take` sgtm.
SAS validate null dates
State: closed

- [x] closes #39725 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: IO SAS
description: SAS: read_sas

Comments:
cc @bashtage 
It looks good aside from my personal preference over multi-line lambda functions which are really hard to read.
REF: move reshaping of array for setitem from DataFrame into BlockManager internals
State: closed

I think ideally the DataFrame does not need to be aware of how the underlying manager stores the data (as 2D, transposed or not), so moving the logic of ensuring 2D/transposed values from the DataFrame set_item-related method into `BlockManager.iset`.

This will help for the ArrayManager, so we don't have to re-reshape there.

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
sorry for so quick @jbrockmendel 

will wait in future
Yeah, I explicitly marked Brock for review ;)
[ArrayManager] Implement .equals method
State: closed

xref #39146

Moved the common part (checking of dimension and indices) to the base class `equals` method so this can be shared, and the actual BlockManager/AraryManager then only need to check the column values.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
cool this is a nice pattern (e.g. have the top-level api in base & implementation methods separate as needed).
[ArrayManager] Test DataFrame reductions + implement ignore_failures
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

This adds support for ignoring failures in `ArrayManager.reduce`.



Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
lgtm. if you can merge master to make sure up to date and merge on greenish
thanks @jorisvandenbossche
CLN: strip whitespaces on `set_td_classes`
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

A use case with string DataFrames is to construct multiple and add them, needing space separators:

```
cls1 = pd.DataFrame([['', 'cls1 ']])
cls2 = pd.DataFrame([['cls2 ', '']])
cls3 = pd.DataFrame([['cls3 ', 'cls3 ']])
Styler.set_td_classes(cls1 + cls2 + cls3)
```

Stripping cleans up the output for HTML, and might be helpful in other ways..


Labels:
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
why is this needed? IOW what does this change from a user POV
@jreback In hindsight I agree. leave the formatting to the user.
DOC: Group optional dependencies by category
State: closed

- [x] closes #39594 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

This PR splits the Optional Dependencies table into multiple tables one for each category. It should help people compare different options when looking for a specific capability, e.g. "I need to read an Excel file. What are my options?".

The specific grouping felt pretty subjective. If you think of a better way to group them, we can do that instead.

Labels:
name: Docs

Comments:
looks fine. can you post a rendered screenshot?
@jreback 

Here's a [pastebin](https://pastebin.com/EJtracLm) of the full html page. And here's a couple screenshots.

![1](https://user-images.githubusercontent.com/35386821/107526903-8d3f1780-6b75-11eb-98bb-7cb1b883928b.png)
![2](https://user-images.githubusercontent.com/35386821/107526907-8e704480-6b75-11eb-8f48-e80010942f2f.png)
![3](https://user-images.githubusercontent.com/35386821/107526912-8f08db00-6b75-11eb-928b-4f1aab2e6cef.png)

@nmay231 thanks, yeah looks good. ping after @jorisvandenbossche suggestions are pushed & green.
@jorisvandenbossche
can you merge master
@jreback Merged with master
@jreback Updated
@jreback Sorry for the misunderstanding. It's fixed now.
thanks @nmay231 very nice.

https://pandas.pydata.org/pandas-docs/dev/ will have the built docs (once CI is done), check that everything renders properly.

thanks!
REF: implement ensure_can_hold_na
State: closed

Much simpler than maybe_promote, which for the most part id like to replace with something like

```
def maybe_promote(dtype, fill_value):
    inferred, fv = infer_dtype_from(fill_value)
    new_dtype = find_common_type([dtype, inferred])
    return inferred, fv
```

Labels:
name: Refactor
description: Internal refactoring of code
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
sure can you rebase
CI: pin numpydoc
State: closed

xref #39688 

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
gets the CI to green
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
DOCS: Added links to official docs in cheat sheets
State: closed

- [x] closes #39706
- [x] tests added / passed (not necessary, only edited ppt and pdf)
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them (not necessary, only edited ppt and pdf)
- [x] whatsnew entry (not necesarry)


Labels:
name: Docs

Comments:
I am not sure how to resolve the check-errors. I changed nothing in the codebase.
Can you merge master and add screenshots of your changes(before/after)?
[ArrayManager] TST: get tests running for /tests/frame
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

This is on top of https://github.com/pandas-dev/pandas/pull/39612, so only the last commit is relevant for this PR: https://github.com/pandas-dev/pandas/pull/39700/commits/ec83091284ac9a388ef147e006e2d4d43d946ea7

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Rebased this now https://github.com/pandas-dev/pandas/pull/39841 is merged
couple comments, otherwise LGTM
thanks, hopefully nothing breaks from prior merges :-<
[ArrayManager] Enable downcast in fillna / enable where indexing tests
State: closed

Instead of https://github.com/pandas-dev/pandas/pull/39539, for now keeping the downcasting behaviour of `fillna`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Failure is an unrelated resource warning failure
can you resolve conflicts and ping on green. lgtm otherwise.
STYLE use ast.parse instead of regex to fixup inconsistencies in namespace
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

The regular expression wasn't working too well, this solution should be better. Also, like this, you can pass `--replace` and have the inconsistencies fixed up for you (though I haven't passed this by default as it may be confusing)

Just running on `pandas/tests/frame/` here, can expand to the rest in separate PRs if this is alright

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@momisbestfriend can you take a look?  IIRC you did a lot of the ast-based checks
> @momisbestfriend can you take a look? IIRC you did a lot of the ast-based checks

@ShaharNaveh 
> lgtm ex @jorisvandenbossche comments

Thanks - I believe these were addressed in d7825aa, but Joris do let me know if you had something different in mind
i think i got into the habit of importing array as pd_array bc there are a number of places where we use "array" as a variable name (or in principle could be using the stdlib array module) and name overlap of that form bothers me more than most
> @momisbestfriend can you take a look? IIRC you did a lot of the ast-based checks

Yes correct I did a lot of AST checks:)

This script looks amazing, LGTM:)
> This script looks amazing

Thanks - I learned loads by studying the [pyupgrade](https://github.com/asottile/pyupgrade) source code
@MarcoGorelli can you merge master and ping on green. i think this is fine (and followup PRs to add coverage are good).
Sure, have merged - @jreback 

EDIT I'll try re-writing this slightly so it's just standard library, then the same local virtualenv as is used for lots of other checks can be reused


UPDATE
------

Now, `tokenize-rt` is only required if using the `--replace` flag. This way, the pre-commit hook can re-use the same virtualenv as other local Python hooks
Hello @MarcoGorelli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-18 12:12:43 UTC
thanks @MarcoGorelli pls open an issue to fix the non-checked parts of the codebase.
> thanks @MarcoGorelli pls open an issue to fix the non-checked parts of the codebase.

Thanks - I'm running a little event with PyLadies London on the 6th of March, so I'll get participants to work on this then
REF: remove unnecessary PeriodIndex._get_indexer
State: closed


Labels:
name: Period
description: Period data type
name: Clean
name: Index
description: Related to the Index class or subclasses

Comments:
hmm lots of things are failing here. maybe an ensure_indexer issue?
worse: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=53846&view=logs&j=acc1347f-8235-55b0-95c7-0e2189e61659&t=486f34f1-eda6-516a-fc5d-d8195128dacd

```
>   value = parse_timedelta_string(value)
E   TypeError: Expected unicode, got numpy.str_
```
> hmm lots of things are failing here. maybe an ensure_indexer issue?

i think i switched the order of a np.asarray/to_timedelta call, just tried re-ordering them
TST: split chaining tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
lgtm. small comment, ok for followon or here.
PERF: special case numpy.dtype in is_extension_array_dtype
State: closed

Master:

```
dtype = np.dtype(float)
dtype2 = pd.Int32Dtype()

In [2]: %timeit pd.api.types.is_extension_array_dtype(dtype)
645 ns ± 59.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [3]: %timeit pd.api.types.is_extension_array_dtype(dtype2)
251 ns ± 9.68 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

with this branch, it keeps the speed for ExtensionDtype the same, and improves for numpy.dtype (avoiding a extension dtype registry lookup):

```
In [2]: %timeit pd.api.types.is_extension_array_dtype(dtype)
328 ns ± 24.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [2]:  %timeit pd.api.types.is_extension_array_dtype(dtype2)
240 ns ± 19.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
sure (assume you found this from existing asvs), if not can you add the one from above.
also a whatsnew mention is prob worth it.
> assume you found this from existing asvs, if not can you add the one from above.

Yes, it came from profiling the concat benchmarks (https://github.com/pandas-dev/pandas/pull/39612), gives a small improvement. Now, apparently we don't have specific benchmarks for our dtype checks, so added one for `is_extension_array_dtype` (that could be supplemented with benchmarks for some of the other `is_` checkers, but going to leave that for another issue/PR).

> also a whatsnew mention is prob worth it.

Not sure it is worth it, this only gives a tiny improvement anywhere it is used (it was just lowing hanging fruit to shave off a percent, and with ArrayManager it gets used more, so getting more worth fixing it)



thanks @jorisvandenbossche
CI: run database tests only #39550
State: closed

- [x] closes #39550
- [x] closes #39758

Labels:
name: CI
description: Continuous Integration

Comments:
hmm maybe its the db - cov build that is a problem. i guess any way to split up these cov builds? (this might be harder, as now we would have multiple uploads to code-cov, but also maybe unavoidable)
> hmm maybe its the db - cov build that is a problem. i guess any way to split up these cov builds? (this might be harder, as now we would have multiple uploads to code-cov, but also maybe unavoidable)

https://docs.codecov.io/docs/merging-reports Codecov automatically merges multiple reports. So we don't even need a separate build for coverage. We could select a set of builds that include all tests, run coverage, and upload reports for all of them.
> > hmm maybe its the db - cov build that is a problem. i guess any way to split up these cov builds? (this might be harder, as now we would have multiple uploads to code-cov, but also maybe unavoidable)
> 
> https://docs.codecov.io/docs/merging-reports Codecov automatically merges multiple reports. So we don't even need a separate build for coverage. We could select a set of builds that include all tests, run coverage, and upload reports for all of them.

that would be great! (but should tag the builds as -cov just to indicate that they are part of the coverage build(s))
I probably won't have time to work on the coverage stuff in the next couple of days. I opened #39758. So people can take from there.
thanks @fangchenli very nice
PERF: EWMGroupby
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


```
(pandas-dev) asv_bench % asv continuous -f 1.1 upstream/master HEAD -b rolling.GroupbyEWM
       before           after         ratio
     [508b9f98]       [c4340436]
     <master>         <ref/ewm_groupby>
-        22.1±4ms       2.57±0.3ms     0.12  rolling.GroupbyEWM.time_groupby_mean('cython')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
love it. we have sufficient asv's for these cases? (e.g. ewmcov, not that its used much). ?
Here are the new benchmarks I added for other methods

```
       before           after         ratio
     [e889b941]       [1f1d2ef9]
     <ref/ewm_groupby~1^2>       <ref/ewm_groupby>
-      22.6±0.7ms       2.29±0.2ms     0.10  rolling.GroupbyEWM.time_groupby_method('var')
-        22.2±2ms       2.21±0.3ms     0.10  rolling.GroupbyEWMEngine.time_groupby_mean('cython')
-      30.9±0.1ms       2.42±0.3ms     0.08  rolling.GroupbyEWM.time_groupby_method('std')
-       234±0.9ms       4.42±0.2ms     0.02  rolling.GroupbyEWM.time_groupby_method('cov')
-        251±20ms       4.54±0.7ms     0.02  rolling.GroupbyEWM.time_groupby_method('corr')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Looks like the CI / Checks is failing for an unrelated reason

```
Run ci/code_checks.sh docstrings
Validate docstrings (GL03, GL04, GL05, GL06, GL07, GL09, GL10, SS01, SS02, SS04, SS05, PR03, PR04, PR05, PR10, EX04, RT01, RT04, RT05, SA02, SA03)
Traceback (most recent call last):
  File "ci/../scripts/validate_docstrings.py", line 50, in <module>
    from numpydoc.validate import validate, Docstring  # isort:skip
ImportError: cannot import name 'Docstring' from 'numpydoc.validate' (/usr/share/miniconda/envs/pandas-dev/lib/python3.8/site-packages/numpydoc/validate.py)
```
very nice @mroeschke 

yeah the doc-strings are failing unrelated
Enhancement dataframe to csv use zip method
State: closed

Enhancement dataframe to csv use zip method,now df.to_csv('myfile.csv.zip') works like df.to_csv('myfile.csv.gz'), generate a zip file with a myfile.csv as content

- [ ] closes #39465 
- [ ] tests added / passed,simply test
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

## test code
>> import pandas as pd
a=pd.DataFrame({'a':[1,2,3,4,5],'b':[4,5,6,7,8]})
a.to_csv(r'z:/myfile.csv.zip')


Labels:

Comments:
Hello @HQDragon! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



* In the file [`pandas/tests/frame/methods/test_to_csv.py`](https://github.com/pandas-dev/pandas/blob/f21de23589aafe0af00fbe3a47968704bba10dab/pandas/tests/frame/methods/test_to_csv.py):

> [Line 1336:1](https://github.com/pandas-dev/pandas/blob/f21de23589aafe0af00fbe3a47968704bba10dab/pandas/tests/frame/methods/test_to_csv.py#L1336): [W391](https://duckduckgo.com/?q=pep8%20W391) blank line at end of file

* In the file [`pandas/tests/io/formats/test_to_csv.py`](https://github.com/pandas-dev/pandas/blob/f21de23589aafe0af00fbe3a47968704bba10dab/pandas/tests/io/formats/test_to_csv.py):

> [Line 651:21](https://github.com/pandas-dev/pandas/blob/f21de23589aafe0af00fbe3a47968704bba10dab/pandas/tests/io/formats/test_to_csv.py#L651): [E126](https://duckduckgo.com/?q=pep8%20E126) continuation line over-indented for hanging indent



##### Comment last updated at 2021-03-11 07:51:12 UTC
DOC: pin sphinx theme to avoid mobile dropdown bug
State: closed

In the latest master version of the pydata-sphinx-theme, there is a bug in the menu dropdown on mobile mode (https://github.com/pydata/pydata-sphinx-theme/issues/304), and since this will probably not be fixed by tomorrow when releasing 1.2.2, I am pinning it here to a few commits earlier to avoid this specific bug (but still using the latest version to have the same looks as the current docs, since we are using a few features that are not yet in the latest stable release).

Labels:
name: Docs

Comments:
@meeseeksdev backport 1.2.x
thanks @jorisvandenbossche 
Something went wrong ... Please have  a look at my logs.
BUG: Series[dt64tz].__setitem__(mask, different_tz) raises
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

sits on top of #39650

after this we'll be ready to cleanup/de-duplicate some of the putmask/where methods

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
@jreback gentle ping, we're getting close to getting all the setitem/putmask/where casting behavior consistent
great!
BUG: equals/assert_numpy_array_equals with non-singleton NAs
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> Eg currently we have see None or np.nan as equivalent:

good catch, this changes that.  will update to keep current behavior (though id be OK with deprecating it separately)
whatsnew added + green
DOC: fix an incomplete sentence
State: closed

"We will address array-based indexing like s[[4, 3, 1]] in section." -> "We will address array-based indexing like s[[4, 3, 1]] in section on indexing."

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @partev
add default  zip file name in dataframe to_csv when use 'infer' + 'zip' method
State: closed

fix #39465 

copy some code from gzip ,now df.to_csv('myfile.csv.zip') works like df.to_csv('myfile.csv.gz')


- [ ] closes #39465 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Hello @HQDragon! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

























































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-08 01:17:17 UTC
sorry ,i messed up some git routine, i need to close this and open new one
pls review the new PR #39662 
@twoertwein
DOC: typo in 1.2.2 whatsnew
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Labels:
name: Docs

Comments:
Thanks!