BrandonFors/pandas-github-issues huggingface dataset:
BUG: close file handles in mmap
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Broken off from #35711

Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
thanks
REF: insert self.on column _after_ concat
State: closed

The idea here is to push towards #34714 by making _wrap_results do things in the same order as other other similar methods do.  cc @mroeschke LMK if there is a simpler way to accomplish this.

Orthogonal to #35470, #35730, #35696.

Labels:
name: Refactor
description: Internal refactoring of code
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @jbrockmendel
TST: encoding for URLs in read_csv
State: closed

- [x] closes #10424
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Reading CSVs from URLs with a non UTF-8 encoding should already work.

Labels:
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @twoertwein
DEPR: Deprecate pandas/io/date_converters.py
State: closed

- [x] closes #24518
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jreback addressed all your comments. Could you have a second look?
this looks good can you merge master and ping on green.
@jreback merged master; all checks in CI pass except two, due to a flaky test (see below)

`hypothesis.errors.Flaky: Hypothesis test_shift_across_dst(offset=<31 * MonthEnds>) produces unreliable results: Falsified on the first call but did not on a subsequent one`
Reran CI, now everything passes
@jreback updated doc, merged master and CI on green
thanks @avinashpancham very nice!
BUG/ENH: to_pickle/read_pickle support compression for file ojects
State: closed

- [x] closes #26237, closes #29054, and closes #29570
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This was basically already supported but `to/read_pickle` set `compression` to `None` for file objects.

Some functions called `get_filepath_or_buffer` (might convert a string to a file object) before calling `infer_compression` (doesn't work with file objects). I moved `infer_compression` and `get_compression_method` inside `get_filepath_or_buffer`.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Pickle
description: read_pickle, to_pickle

Comments:
@jreback ping
very nice @twoertwein keep em coming!
BLD: update min versions #35732
State: closed

- [x] closes #35732


Labels:
name: Build
description: Library building on various platforms
name: Dependencies
description: Required and optional dependencies

Comments:
Think this travis build will need a restart: https://travis-ci.org/github/pandas-dev/pandas/jobs/718069611

(I no longer seem to have permissions to do so)

```
INTERNALERROR>     self.cov_controller.configure_node(node)
INTERNALERROR>   File "/home/travis/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_cov/engine.py", line 274, in configure_node
INTERNALERROR>     workerinput(node).update({
INTERNALERROR>   File "/home/travis/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/pytest_cov/compat.py", line 42, in fn
INTERNALERROR>     return getattr(obj, attr, *args)
INTERNALERROR> AttributeError: 'WorkerController' object has no attribute 'slaveinput'
```
Ref: https://github.com/pandas-dev/pandas/issues/35737 for CI failures
thanks @VirosaLi
DOC: Fix broken link in cookbook.rst
State: closed

The original link [Aggregation and plotting time series](http://nipunbatra.github.io/2015/06/timeseries/) found in the [Pandas Cookbook](https://pandas.pydata.org/pandas-docs/stable/user_guide/cookbook.html?highlight=get_group#timeseries) is broken.

This appears to have been moved to the authors [Blog](https://nipunbatra.github.io/blog/visualisation/2013/05/01/aggregation-timeseries.html)

While the date does not match ( 2013/05 vs 2015/06 ) the contents appear identical. I was able to determine this after viewing the archive at [WayBack Machine](https://web.archive.org/web/20161202094122/http://nipunbatra.github.io/2015/06/timeseries/)


- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry




Labels:
name: Docs

Comments:
thanks @estasney
BLD: bump xlrd min version to 1.2.0
State: closed

Warnings about time.clock are filling up the py37 min_version build.  xlrd 1.2.0 was released Dec 15, 2018, so I think we're safe to bump it.

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
not sure if there is an open issue about this?
> not sure if there is an open issue about this?

doesn't look like it. `ignore_xlrd_time_clock_warning` added in #23727 see also https://github.com/pandas-dev/pandas/pull/23727#discussion_r235957934
agg with list of non-aggregating functions
State: closed

- [x] closes #35490
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Reverts to 1.0.5 behavior while maintaining the bugfix that introduced the "regression". Ideally `agg` would instead raise in these scenarios ([ref](https://github.com/pandas-dev/pandas/issues/35490#issuecomment-672981745)), but that would be an API change. Much of this PR should be reverted once this is done, I've marked such places with TODOs.


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach can you open an issue to 'fix' this properly in 1.2
The issue is here: #35725
@meeseeksdev backport 1.1.x
update io documentation
State: closed

The HDFS dropna=True parameter does not drop the NaN values. Perhaps this is a bug to the core Pandas, or really a typo in the documentation

- [X] closes #35719
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:

Comments:
The way you're changing the documentation, setting `dropna=True` keeps NA values. That isn't correct, I believe this is a bug, not an issue with the docs.
Ah, glad this bug is found soon, 'cause I myself kinda doubt that the errata comes from the docs, but apparently it's not the case. Thanks to @XiaozhanYang observations.
@simonjayhawkins Thanks for the quick updates with @rhshadrach. I am actually new to contributing to open source. I myself am quite familiar with Python. Perhaps with regards to this bug, you have any suggestion or tips that would allow helping contribute? thanks in advanced, really love pandas hahaha.
@grrlic in the first instance, if you are unfamiliar with the codebase, I would probably bisect the regression, see #35685 to get a idea of where a fix should be applied.

although for this case, it might be quite straightforward to follow the code from `to_hdf` (I notice from git blame that there we changes to the signature in #29957) It appears that dropna is not passed onto `store.put` in the if append else block
CLN: remove extant uses of built-in filter function
State: closed

No internet for a good part of yesterday so some time to experiment with code cleanups.

Labels:
name: Clean

Comments:
thanks @simonjayhawkins if all are removed then pls add a code_check to ban these (may not be easy though); its possible have to use a static analyzer
would be a good idea

the regex that I used to find these contained false positives (such as simplefilter) will need to look again I just used `[^\.]filter\(` to get filter calls that weren't methods

will need to put on todo list.

  
I think `(?<!def)[\(\s]filter\(` could work
CI: Check for file leaks in _all_ tests
State: closed

Sits on top of #35693 

Still have two tests failing locally, but I'm not seeing any socket leaks that I occasionally see in the CI logs.


Labels:

Comments:
Excellent, Travis is showing us where the socket leaks are coming from.

@simonjayhawkins thoughts on how to handle the mypy complaint?
> @simonjayhawkins thoughts on how to handle the mypy complaint?

not sure which is best until look at bigger cleanup. (can replace `handles` (or whatever used) in `get_handle` calls with `_` in some cases, since it appears that not used in some cases)

IIUC we should only need to close the files in handles and not access them for any other purpose, so could use Protocol, say  Closable (used in https://docs.python.org/3/library/typing.html#typing.runtime_checkable) or SupportsClose

from https://github.com/python/typeshed/blob/master/CONTRIBUTING.md#conventions

>A few guidelines for protocol names below. In cases that don't fall into any of those categories, use your best judgement.

>Use plain names for protocols that represent a clear concept (e.g. Iterator, Container).
>Use SupportsX for protocols that provide callable methods (e.g. SupportsInt, SupportsRead, SupportsReadSeek).
>Use HasX for protocols that have readable and/or writable attributes or getter/setter methods (e.g. HasItems, HasFileno).

or maybe could make `_MMapWrapper` a subclass (or virtual) of IO

or simpliest, to get mypy to green, is to change type declaration of `handles` to `handles: List[Union[IO, "_MMapWrapper"]] = list()`. The return of `get_handle` is not typed, so this simple solution shouldn't create any consistency issues for now.
The isort complaint looks like it is disagrees with black on how to format a runtime import.
Remaining test failures look like they're coming from moto/boto3, no luck tracking down where the left-open file is.  @TomAugspurger do we have anyone knowledgeable about these packages?
Dunno, but @martindurant has a PR open changing moto to run as a server (in a separate process?), so it's possible those failures will go away.
Correct, moto will run as a separate process - but the cleanup was apparently leaving *something* behind, so the latest iteration is to run with broader scope for the pytest fixture, so they should go away (worked in https://github.com/dask/dask/pull/6528 ). I'm not sure those would show up in what you are testing here.

Maybe some "open files" will tend to show up in tests, if file closing is left to the garbage collector, which is fine in general use.
Following #35836 i think this has been pushed as far as it can ATM, closing.
add web/ directory to isort checks
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
pls rebase as this *should* fail (as it doesn''t include the check) and then ping on green
@jreback all clear!
thanks @el-iot
Reorganize imports to be compliant with isort (and conventional)
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
can you add this file to the files checked by isort in ci/code_checks.sh
No problem, check out #35709 
> No problem, check out #35709

ok idealy it would have been in this PR but ok
REGR: fix DataFrame.diff with read-only data
State: closed

Closes #35559

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
@jreback that's what I already did (` # TODO(cython 3) update to "const diff_t[:, :] arr"`)
> @jreback that's what I already did (` # TODO(cython 3) update to "const diff_t[:, :] arr"`)

ahh right, ok then.
Thanks @jorisvandenbossche 
@meeseeksdev backport 1.1.x
ENH add na_action to DataFrame.applymap
State: closed

For symmetry with Series.map

- [x] closes #23803
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Apologies for my omissions! Thanks for the review.
Failure appears unrelated
looks good, can you merge master and ping on green.
@jreback it's green :)
thanks @jnothman very nice
Backport PR #35654 on branch 1.1.x (BUG: GH-35558 merge_asof tolerance error)
State: closed

Backport PR #35654: BUG: GH-35558 merge_asof tolerance error

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
isort 5.3.2 -> 5.4.0 errors unrelated to changes in this PR
Backport PR #35647 on branch 1.1.x (BUG: Support custom BaseIndexers in groupby.rolling)
State: closed

Backport PR #35647: BUG: Support custom BaseIndexers in groupby.rolling

Labels:
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
can ignore doctest failure for now (need to manually backport changes to pandas/io/pytables.py from #35214)
```
##[error]/home/runner/work/pandas/pandas/pandas/io/pytables.py:275:PR03:pandas.read_hdf:Wrong parameters order. Actual: ('path_or_buf', 'key', 'mode', 'errors', 'where', 'start', 'stop', 'columns', 'iterator', 'chunksize', '**kwargs'). Documented: ('path_or_buf', 'key', 'mode', 'where', 'start', 'stop', 'columns', 'iterator', 'chunksize', 'errors', '**kwargs')
```
REGR: Don't ignore compiled patterns in replace
State: closed

- [x] closes #35680
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Strings
description: String extension data type and string data
name: replace
description: replace method

Comments:
Currently failing mypy and not quite sure of the fix. https://github.com/pandas-dev/pandas/pull/35697/commits/5006980843a382cd27291e75a8173ec70c00fda6 did not work.

```
mypy --version
mypy 0.730
Performing static analysis using mypy
pandas/core/internals/managers.py:1952: error: Module has no attribute "Pattern"  [attr-defined]
Found 1 error in 1 file (checked 1030 source files)
Performing static analysis using mypy DONE
##[error]Process completed with exit code 1.
```
> @dsaxton looked into that CI error you're getting.
> 
> xref [python/typing#23](https://github.com/python/typing/issues/23)
> 
> On my machine adding
> 
> ```
> from typing import Pattern
> ```
> 
> at the top of `pandas/core/internals/managers.py` and then using plain `Pattern` in the line you modify fixed the `mypy` complaint.
> 
> Hope this helps!

Awesome thanks @arw2019 
> Currently failing mypy and not quite sure of the fix. [5006980](https://github.com/pandas-dev/pandas/commit/5006980843a382cd27291e75a8173ec70c00fda6) did not work.
> 
> ```
> mypy --version
> mypy 0.730
> Performing static analysis using mypy
> pandas/core/internals/managers.py:1952: error: Module has no attribute "Pattern"  [attr-defined]
> Found 1 error in 1 file (checked 1030 source files)
> Performing static analysis using mypy DONE
> ##[error]Process completed with exit code 1.
> ```

I think this is a mypy issue that is fixed in later mypy, but can't see a relevant issue on mypy issue tracker (probably a later typeshed)
not a regression, but I guess `regex=True` shouldn't be necessary when compiled regex is passed and if `regex=False` is explicitly passed should error.

Thanks @dsaxton merging and backporting this as addresses reported regression. followups can be left for 1.2.
@meeseeksdev backport 1.1.x
REF: implement reset_dropped_locs
State: closed

We get to get rid of the comment `# really should be done in internals :<`

Making the window.rolling usage use this helper method is the next step.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
sure, rebase just in case and ping on green.
ping
ENH: GH-35611 Tests for top-level Pandas functions serializable
State: closed

- [x] closes #35611
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Pickle
description: read_pickle, to_pickle

Comments:
thanks @ikedaosushi
BUG: to_pickle/read_pickle do not close user-provided file objects
State: closed

- [x] closes #35679
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Do not close user-provided file-objects in `to_pickle` and `read_pickle`.

Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
thanks @twoertwein very nice!
BUG/ENH: compression for google cloud storage in to_csv
State: closed

- [x] closes #35677, closes #26124, and closes #32392
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

By inferring the compression before converting the path to a file object `df.to_csv("gs://mybucket/test2.csv.gz", compression="infer", mode="wb")` works. By wrapping fsspec file-objects in a `TextIOWrapper` `df.to_csv("gs://mybucket/test2.csv", mode="wb")` works as well. Path-like objects that are internally converted to file-like objects (in `get_filepath_or_buffer`) are now always opened in binary mode (unless text mode is explicitly requested) and the potentially changed mode is returned (no need to specify `mode="wb"` for google cloud files). As long as the google file is opened in binary mode (which is now always the case), we also honor the requested `encoding`.

This PR also fixes Zip compression for file objects not having a name.

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
it seems that only the windows py37 machine on azure actually tests the google cloud interface?!

EDIT: and one linux machine on travis
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-31 23:19:00 UTC
thanks for your review @jreback! I was lazy and used tuple unpacking to keep the diff small. I will use the named tuple explicitly in the next commits.

The return types of `getfilepath_or_buffer` were previously not checked (return types were only mentioned in the doc string - and were partly inconsistent with the input arguments). I will double check the types and add asserts instead of ignore comments.
thanks @twoertwein yeah i really like the named tuple for its readability

(you could actually use a DataClass) but not sure if that adds anything
Typing of `filepath_or_buffer` is a mess. I also added `IOBase` to `FilePathOrBuffer` (`io.BytesIO` and `io.StringIO` do not seem to be covered by `isinstance(..., typing.IO)`).

I ended up using a dataclass for IOargs, `NamedTuple` does not support `TypeVar`s in it. I use `TypeVar`s for encoding (stays  None/str) and mode (stays None/str or it changes from None to str).

If IOargs was retrieved inside an if-branch, I 'unpack'  it (using the tuppe names)  into the previous variables, otherwise I use `ioargs.` instead of the previous variable names.
@WillAyd @TomAugspurger @simonjayhawkins if any comments; also using Generic protocols now so if you guys can review types (not to get bogged down of course.....)
PERF: make RangeIndex iterate over ._range
State: closed

Minor performance issue.

By adding a custom ``__iter__`` method to ``RangeIndex``, we partly avoid needing to create/cache the expensive ``_data`` attribute and partly it's just faster to iterate over a ``range`` than a ``ndarray``:

```python
>>> idx = pd.RangeIndex(100_000)
>>> %%timeit
... for _ in idx:
...     pass
10.9 ms ± 74.7 µs per loop  # master
6.11 ms ± 48.8 µs per loop  # this PR
>>> "_data" in idx._cache
True  # master
False  # this PR
```

xref #35432, #26565.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
cool, can you add an asv which hit this (or do we have enough coverage)?
Ok, i’ve added ASVs.
thanks @topper-123
Fix cython3
State: closed

I'm not sure if this actually works, but it gets cython to compile with py310.

- [x] closes #34014
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Should this take over from #34997?
Ah, I missed #34997.  I probably should pick off the second commit into its own PR and then sort out if my attempt here is the right path or the one in the other PR.

That said, my guess is that this code is performance critical and was written the way it was as an optimization and I really don't have more than a passing understanding of the block manager so I am not the right person (and don't really have the bandwidth right now to become the right person) to take this on...
Sorry for both the low-quality PR and dropping the ball on moving this forward.  I had this sitting on my machine for a while,  thought it was better to get it out in the public at all and that I would have more bandwidth to follow up :(.
All good @tacaswell ! I think this is getting covered in some other PRs so closing this one to clear the queue, but effort much appreciated in any case
Avoid redirect
State: closed

A minor change to avoid 301 redirect on link.

Labels:
name: Docs

Comments:
Looks good - mind merging master for green CI?
@alimcmaster1 done, not sure why it's failing...
Ref: #35737 for CI failures
If you merge master the CI issues should be fixed - thanks
If you merge master again CI failures should be resolved - thanks for the PR!
Thanks @chankeypathak
REGR: Dataframe.reset_index() on empty DataFrame with MI and datatime level
State: closed

- [ ] closes #35606
- [ ] closes #35657
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
thanks @simonjayhawkins very nice
@meeseeksdev backport 1.1.x