BrandonFors/pandas-github-issues huggingface dataset:
use a try block in _expand_user instead of a typing condition
State: closed

Noticed that `_expand_user`'s docstrings stated that expansion was performed "if possible", while the actual implementation was only applying expansion to str objects.
With a try block it should work with str, os.Pathlike and bytes objects (and everything else supported by `os.path.expanduser` in the future).
I except this change may be undesirable since it will force the output type to `str`. In this case I can replace this with a docstring update.
I'll complete the checklist with tests and whatsnew entry if the change is deemed desirable.

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@arw2019 thanks, will do!
I gave up on this when I couldn't figure out how to make mypy happy with the change, any piece of advice here ?

> @arw2019 thanks, will do!
> I gave up on this when I couldn't figure out how to make mypy happy with the change, any piece of advice here ?

Hmmm it seems like there are a ton of CI failures rn. Can you rebase against master and see which problems persist
> Can you rebase against master and see which problems persist here ?

Done !
I note the logic block I'm touching has been updated on master since I opened this, but I still think a try-except pattern is probably more robust than what's already there. I'll be back to add tests to this :)
Think I fixed the branch now !
> Think I fixed the branch now !

Ok, looping in some of the core team to see if there's appetite for this change

@simonjayhawkins @jreback @mroeschke 

Hello @neutrinoceros! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-18 12:27:33 UTC
I've now added tests to check for the new behaviour. The new tests fail on master and pass on this branch.
@neutrinoceros Is this still active? Can you merge master to fix conflicts if so?
Hi ! Yes I am still willing to go through with this. I just rebased in a pretty agressive way, maybe I'll need to do some back pedalling. I see the parts I changed in this PR have seen considerable churn since I originally read the code so maybe the added value is gone now. 
Ok I must admit I'm having a hard time trying to put the pieces back together and I am not sure this is worth it.
I want thanks @jreback for taking the time to review this, and everyone involved in the discussion, but I now think the easiest way to fix this would be to just start fresh. Maybe I'll reissue this later, but I don't wish to clutter the issue tracker.
DOC: Fixed formatting and errors in whatsnew v1.1.0
State: closed

cc @TomAugspurger 

Minor fixes for formatting, spelling, and grammar. Please let me know if any of this is too nit-picky, I can revert to just the definite spelling/syntax errors.

Labels:
name: Docs

Comments:
Failure is unrelated.
Reported upstream in https://github.com/numpy/numpy/issues/16939.

Thanks @rhshadrach.
CLN: Remove leftover partial-result code from apply
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Clean

Comments:
hmm yeah i don't really recall while this was needed

any perf differences?
@jreback I added the partial results in order to solve https://github.com/pandas-dev/pandas/issues/33879 ( PR https://github.com/pandas-dev/pandas/pull/34183)
But it seems that the whole use of `libreduction` was removed, and there is no need.

All the code i removed isn't actually used, so there shouldn't be any perf difference

Do you happen to know why `libreduction` isn't used anymore? is there a slowdown after its removal?
libreduction just added complexity 


ok.
Anything left for this to be merged?
yeah this is nice @alonme thanks
PKG: Set max pin for Cython
State: closed

We know that pandas doesn't work with Cython 3.0
(https://github.com/pandas-dev/pandas/issues/34213,
https://github.com/pandas-dev/pandas/issues/34014)

This sets the maximum supported version of Cython in our pyproject.toml
to ensure that pandas 1.1.0 can continue to be built from source without
Cython pre-installed after Cython 3.0 is released.

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
cool
Matplotlib 3.3 compatibility fixups
State: closed

Closes #34850 
Closes https://github.com/pandas-dev/pandas/issues/35350

Haven't tested with matplotlib's older than 3.2 yet. We'll see what CI says.

Labels:
name: Visualization
description: plotting
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @jklymak. The only change I'm uncertain about being a bug vs. deliberate is https://github.com/pandas-dev/pandas/pull/35393#discussion_r459519522.
> do we need to bump mpl min? (I am not sure what the oldest we test)

Matplotlib has supported datetime64 since v2.2.0, though there have been some bug fixed along the way....
2.2.0 is our minimum right now, so I think we're good.
thanks @TomAugspurger
CLN: resolve isort mypy import confilict test_transform
State: closed

Part of #35134
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Clean

Comments:
@WillAyd I merged #35339 since you approved that. Are you OK with this and #35380
Sure looks good

Sent from my iPhone

> On Jul 29, 2020, at 5:17 AM, Simon Hawkins <notifications@github.com> wrote:
> 
> ﻿
> @WillAyd I merged #35339 since you approved that. Are you OK with this and #35380
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Thanks @VirosaLi
BUG: ValueError: Cannot convert non-finite values (NA or inf) to integer only when DF exceed certain size
State: closed

- [ ] closes #35227
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Stale

Comments:
Can you add a test for the behavior you are trying to fix? As reviewers that's typically the first thing we look at
@ben-arnao can you update here in response to comments?
@TomAugspurger I wasn't sure if this was something people wanted to continue with. I can make the changes @simonjayhawkins suggested if that's what's needed.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Closing for now. @ben-arnao ping us whenever you'd like to continue and we'll reopen!
Storage options
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: API Design

Comments:
Refs: #28377 , #30359 , #33639 , #16692 (and probably more)
This PR is near complete.
There may be more places where storage_options can be added, but the most important cases are covered and tested.

Question: the docstring for the kwarg should be added, but it would be repeated verbatim in every place; is this the right thing to do?

(cc @TomAugspurger )
> Question: the docstring for the kwarg should be added, but it would be repeated verbatim in every place; is this the right thing to do?

Yeah I think it needs to be duplicated. We do share some sharing between some readers, but not all. It'll be easier to just add the same lines to each, and add an example in one place (perhaps read_csv).
(fails against numpy dev with "ValueError: invalid __array_struct__" while reading stata)
Failing test is https://github.com/pandas-dev/pandas/issues/35426 (i.e., unrelated)
Added text to io.rst
I disagree with "file-like objects must be iterable" in `is_file_like` - most uses (in pandas) require only `read`, and certainly for stata.


IIRC we've gone back and forth on exactly what is required. I don't especially care what is required, but I'd prefer to be consistent in what we accept across readers unless there's a good reason not to be. Hopefully @gfyoung can comment here.
isort is complaining, but passes for me locally. Can someone tell me what's wrong?
```text
Check import format using isort
##[error]ERROR: /home/runner/work/pandas/pandas/pandas/conftest.py Imports are incorrectly sorted and/or formatted.
Check import format using isort DONE
##[error]Process completed with exit code 1.
```
code:
```python
@pytest.fixture()
def fsspectest():
    pytest.importorskip("fsspec")
    from fsspec.implementations.memory import MemoryFileSystem
    from fsspec import register_implementation
    from fsspec.registry import _registry as registry
```
Perhaps check that the version of isort matches?

Also some warnings in io.rst: https://github.com/pandas-dev/pandas/pull/35381/checks?check_run_id=954318397#step:6:2333.
@TomAugspurger you were right, newer isort solved it; also fixed the RST link.
ping
My only outstanding question is https://github.com/pandas-dev/pandas/pull/35381#discussion_r465250632.
@gfyoung has not yet responded, and I would rather not require iterable files, nor change the utils code that other things might be depending on. I suppose I'm stuck for now! This particular case is *only* for stata, and the previous code didn't use `is_file_like` either.

(note that IOBase does guarantee iterability, but I'm not certain that all file-like implementations do - not that I can imagine why they wouldn't)
> ##[error]doc/source/user_guide/io.rst:1706:83:E501:line too long (86 > 79 characters)

lint issue
> lint issue

@jreback : fixed I think. Even linting the docs...
Thanks!
DOC: Expanded Using a Docker Container section
State: closed

- [x] closes #35345
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Added Docker installation link, Docker commands, and PyCharm sections.

Labels:
name: Docs

Comments:
Thanks @alexhlim for the PR. test failure in Windows py37_np18 is unrelated.

```
INTERNALERROR> assert not 'pandas/tests/base/test_unique.py::test_unique_bad_unicode[idx_or_series_w_bad_unicode0]'
```
Thanks for the reply, @simonjayhawkins! Let me know if there are any revisions I need to do on my end to get these checks passing.
restarted failed job
No problem, @MarcoGorelli. Glad I could help!
Thanks @alexhlim
Change defaults for rolling/expanding.apply engine kwargs to None
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This change was supposed to be apart of https://github.com/pandas-dev/pandas/pull/35182 but was overlooked. Since 1.1 isn't out, this doesn't need to be backported correct?


Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke
ENH: Add validation checks for non-unique merge keys
State: closed

- [x] closes #27430
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

After implementing the suggestion in #27430 I realized it should also apply to `1:m` and `m:1` merges. When any type of `many` merge is specified, an error will be raised if the `many` side is actually unique.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Stale

Comments:
Thanks @LewisDavies for the PR. The current behaviour is documented so this is an breaking API change. Would first need a deprecation.

from https://pandas.pydata.org/pandas-docs/version/1.1.0/reference/api/pandas.merge.html

> validate : str, optional
> If specified, checks if merge is of specified type.
>
> “one_to_one” or “1:1”: check if merge keys are unique in both left and right datasets.
>
> “one_to_many” or “1:m”: check if merge keys are unique in left dataset.
>
> “many_to_one” or “m:1”: check if merge keys are unique in right dataset.
>
>“many_to_many” or “m:m”: allowed, but does not result in checks.

 FWIW i'm -1 on changing this.
Happy to contribute! I didn't realize the current behaviour was intentional so I understand not wanting to implement this. That said, it feels odd that if a user specifies `m:m` they won't get an error if both sides have unique keys.

With the benefit of hindsight the names could be clearer, especially for one-to-many merges. Adding `1:` & `:1` options could make this explicit but could just add confusion so I'm not sure what's best. What do you think @simonjayhawkins?
> Happy to contribute! I didn't realize the current behaviour was intentional so I understand not wanting to implement this.

Thanks @LewisDavies. That's just my opinion. others may disagree so we'll leave this open for a while.

> With the benefit of hindsight the names could be clearer, especially for one-to-many merges. Adding `1:` & `:1` options could make this explicit but could just add confusion so I'm not sure what's best. What do you think @simonjayhawkins?

another alternative could be to have, say, a `check_unique` keyword parameter that accepts, say, `left`, `right` and `both`. I think something that that may also make the intention of the functionality clearer.

There is functionality currently being added to pandas, to disallow allow duplicates and propagate that property in further operations, see #28394. It could be that in the future the `validate` parameter of merge becomes redundant.
@LewisDavies are you able to address @jreback comment https://github.com/pandas-dev/pandas/pull/35373#pullrequestreview-462947864
Closing as stale, @LewisDavies let us know if you'd still like to work on this
DOC: Fix heading capitalization in doc/source/whatsnew - part6 (#32550) (open PR again due to branch removed)
State: closed

- [ ] Modify files v0.22.0.rst, v0.23.1.rst, v0.19.0.rst, v0.24.0.rst, v0.24.2.rst
-[ ] Add exceptions in 'validate_rst_title_capitalization.py'

Labels:
name: Docs

Comments:
@simonjayhawkins @datapythonista is this our standard for capitalization?
Thanks @cleconte987
FIX: remove dependence on matplotlib.dates.epoch2num
State: closed

- [ ] closes #34850
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

`converter._dt_to_float_ordinal(dt)` special cased `datetime64` objects, and used `matplotlib.dates.epoch2num`.  However, matplotlib has supported `datetime64` natively since Matplotlib 2.2 (https://github.com/matplotlib/matplotlib/pull/9779), so this special casing shouldn't be necessary (unless there is a subtlety I'm missing).  Matplotlib would also prefer to deprecate `epoch2num`, so this would help us streamline.  

I didn't check your support policy - if you need to support pre MPL 2.2, then I guess some try/except logic will need to go in here.  

ping @TomAugspurger @dstansby 

Labels:

Comments:
Thanks. I'm looking into this a bit too. Will take a closer look later.
So, pandas has a quasi-public API at https://github.com/pandas-dev/pandas/blob/3e88e170a436b5e3cfdfc29f8a7416220658c616/pandas/plotting/_matplotlib/converter.py#L136 that gets tested in `test_converter.py` and assumes a matplotlib epoch of 0000-12-31.  Most plotting seems OK with the change in this PR, but the hard-coded tests that assume the epoch is the same as `datetime.toordinal()` naturally fail.  Not sure how you want to handle this dance - I think it would be easiest from the users point of view to just use the Matplotlib epoch and adjust the test (and the quasi-public results of calling `convert`).  But I don't claim expertise in how pandas users are used to handling datetime objects.
Thanks for starting on this @jklymak. I've incorporated it into https://github.com/pandas-dev/pandas/pull/35393.
fix bug when combining groupby with resample and interpolate with dat…
State: closed

…etime-index (GH 35325)

- [x] closes #35325
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method

Comments:
Thanks @CloseChoice for the PR. The test failures should be fixed on master. can you merge upstream/master.
> Thanks @CloseChoice for the PR. The test failures should be fixed on master. can you merge upstream/master.

Done, though I am not quite sure if my whatsnew entry is correct. There seemed to be some problems with building the documentation. I also had a similar issue on another PR.
thanks @CloseChoice very nice
REF: implement _apply_blockwise
State: closed

Preliminary pushing towards #34714

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
> I think ultimately you want to put this _apply_blockwise in a BlockwiseMixin, similar to SelectionMixin, so you can add it to groupby and so on, but suppose could be done later

I'm planning a BlockManager method that subsumes all of the apply-blockwise usages.

For that to work, we need to get all of the relevant looping-over-blocks functions into a form that can be passed to such a method.  Thats the intermediate goal of this and #35356.
@simonjayhawkins is this new/expected?

```
pandas/core/window/rolling.py:500: error: Type variable "pandas._typing.ArrayLike" is unbound
pandas/core/window/rolling.py:500: note: (Hint: Use "Generic[ArrayLike]" or "Protocol[ArrayLike]" base class to bind "ArrayLike" inside a class)
pandas/core/window/rolling.py:500: note: (Hint: Use "ArrayLike" in function signature to bind "ArrayLike" inside a function)
```
that worked, thanks @simonjayhawkins 
thanks
DOC: Fix small spelling mistake in style docs
State: closed

This pull request fixes a small spelling mistake in the pandas style user guide.

I believe this change does not justify a whatsnew entry, since it is extremely small. It does not refer to a particular issue on GitHub. I have not run `black` or `flake8`, since I believe the change is out of scope for these tools.

- [ ] ~~closes #xxxx~~
- [ ] ~~tests added / passed~~
- [ ] ~~passes `black pandas`~~
- [ ] ~~passes `git diff upstream/master -u -- "*.py" | flake8 --diff`~~
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
Thanks @flrs
REGR: MultiIndex Indexing
State: closed

- [ ] closes #35351
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
I'll add tests if nothing breaks. I'm not keen on special casing the base Index method, but overriding _reindex_non_unique maybe overkill. maybe better to revert part of #33691
test failures unrelated, see #34850 so marking as ready for review. whatsnew needed for regression in rc?
Thanks, LGTM.

> whatsnew needed for regression in rc?

I don't think so.
mpl issue breaking the docs?
rebased and green. since this is for 1.1, I suppose this could be merged even without a firm decision on the branch strategy.
Yep, thanks.
fix generate_range function in datetimes.py file
State: closed

change some issues be metioned at #35342

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Stale

Comments:
![testresult_20200720172043.png](https://i.loli.net/2020/07/20/52RGWSEsKng9uav.png)
I test all the cases on my machine but there still have some errors like `pytz.exceptions.NonExistentTimeError: 1907-01-01 00:00:00`
I want to know whether should I change NonExistentTimeError detect ways.
> This would need a test.
> 
> Also I don't think this is an appropriate place for the fix. I imagine the bug is in `generate_range`
> 
> Could you give this PR a more informative title as well? Thanks.

I have make some tests and they all passed,I will let PR have  a more informative title。
![image](https://user-images.githubusercontent.com/9131265/88256438-e9cc0980-cced-11ea-8c70-31860229dbd7.png)


This PR still needs a test.
@zky001 can you address https://github.com/pandas-dev/pandas/pull/35348#issuecomment-663341641 and https://github.com/pandas-dev/pandas/pull/35348#pullrequestreview-454628733
See my comment in #35342 on the appropriate fix
Closing as stale, @zky001 let us know if you'd still like to work on this
TYP: remove # type: ignore for unpacking compression_args
State: closed

eg. compresslevel for GzipFile is an int and compression in get_handle is Optional[Union[str, Mapping[str, Any]]] = None

once we adopt TypedDict, then maybe we could be more explicit here, but for now Any is appropriate.

the ignored message for GZipFile is `error: Argument 2 to "GzipFile" has incompatible type **Dict[str, str]"; expected "int"`

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> lgtm; these are just kwargs that get passed to zip file .write correct?

you mean .open?
from https://docs.python.org/3/library/gzip.html?highlight=gzipfile#module-gzip

> Changed in version 3.3: Added support for filename being a file object, support for text mode, and the encoding, errors and newline arguments.

so maybe (in a followon) remove the `if is_path` for `compression == "gzip"`
ENH: Select numeric ExtensionDtypes with DataFrame.select_dtypes
State: closed

- [x] closes #35340
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Hello @andrewgsavage! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-19 13:14:33 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@andrewgsavage can you merge master to resolve conflicts. Also needs a release note. (put in doc\source\whatsnew\v1.2.0.rst for now, but my need to moved as 1.2rc is scheduled before the end of the month)
Closing in favor of #38246