BrandonFors/pandas-github-issues huggingface dataset:
Infer filesystem from path when writing a partitioned DataFrame to remote file systems using pyarrow
State: closed

Infer the file system to be passed to pyarrow based on the path provided.

- [x] closes #34841
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Parquet
description: parquet, feather

Comments:
Hello @kylase! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-18 09:04:33 UTC
https://github.com/pandas-dev/pandas/pull/34266 has been merged in the meantime. @kylase if you have time to revisit this?
> #34266 has been merged in the meantime. @kylase if you have time to revisit this?

Will take a look at it over the weekend. With a quick glance at the change, I think it should be fixed but we should add the test in to make sure the file system is inferred with just the path given.
> > #34266 has been merged in the meantime. @kylase if you have time to revisit this?
> 
> Will take a look at it over the weekend. With a quick glance at the change, I think it should be fixed but we should add the test in to make sure the file system is inferred with just the path given.

@kylase are you still interesting in working on this? whatsnew will be 1.2 now

Thanks
I have

> 
> 
> > > #34266 has been merged in the meantime. @kylase if you have time to revisit this?
> > 
> > 
> > Will take a look at it over the weekend. With a quick glance at the change, I think it should be fixed but we should add the test in to make sure the file system is inferred with just the path given.
> 
> @kylase are you still interesting in working on this? whatsnew will be 1.2 now
> 
> Thanks

I have looked at the tests in 1.1 and there are tests for [explicit filesystem](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/test_parquet.py#L540) and [inferred](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/test_parquet.py#L558). Therefore I feel that the issue has been resolved.

Will proceed to close this PR and the issue.
ENH: Add support for calculating EWMA with a time component
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


This enhancement allows EWMA to be calculated relative to the timestamp in which an observation occurs instead of assuming each observation is equally spaced.

Labels:
name: Enhancement
name: Window
description: rolling, ewma, expanding

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-06 21:33:14 UTC
cc @pandas-dev/pandas-core if anyone has comments.
thanks @mroeschke
REF: implement _shared_docs to de-circularize dependencies
State: closed

A side effect of this is that a bunch of pd.concat imports can be done at import-time instead of at runtime, will do this in a separate pass.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
nice idea - can you merge master?
master merged; are the mypy complaints here real or ignorable?
Not sure on first glance; haven’t seen that before. @simonjayhawkins any idea?

Sent from my iPad

> On Jun 18, 2020, at 6:02 PM, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> master merged; are the mypy complaints here real or ignorable?
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.

This is weird...

with the change

```
        if isinstance(objs, (ABCSeries, ABCDataFrame, str)):
            raise TypeError(
                "first argument must be an iterable of pandas "
                f'objects, you passed an object of type "{type(objs).__name__}"'
            )
```
it appears mypy assumes \_\_init__ always raises and does not process the rest of \_\_init__ and the attributes are not defined.

This doesn't seem right. maybe mypy bug with combination of objs, ABCSeries and ABCDataFrame being Any.

adding type annotation for obj seems to fix.


can you merge master again. I think this module as `shared_docs.py` is fine as we are going to move core to _core anyhow.
> I think this module as shared_docs.py is fine as we are going to move core to _core anyhow.

updated+green
ex merge conflict this lgtm
great. you might want to create an issue and mark as good first so folks can move shared docs to the new location.
CLN: GH29547 change string formatting with f-strings (6 files changed)
State: closed

Replace old string formatting syntax with f-strings #29547

- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Changed 6 files:
- pandas/tests/io/parser/test_header.py
-  pandas/tests/io/test_sql.py
- pandas/tests/io/test_html.py
- pandas/tests/reductions/test_reductions.py
- pandas/tests/reshape/test_melt.py
- pandas/tests/scalar/timedelta/test_timedelta.py



Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MatteoFelici
REF: move registry, Registry to dtypes.base
State: closed

These are more closely related to ExtensionDtype than to our internal EADtypes.  More concretely, they are low-dependency, while dtypes.dtypes has some unfortunate circular dependencies.

I'd also like to move pandas_dtype and is_dtype_equal from dtypes.common so that those can be stricty "above" dtypes.dtypes in the dependency structure.  Saving that for another pass.

Labels:
name: Refactor
description: Internal refactoring of code
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
no objections from me - @TomAugspurger thoughts?
Hmm doctests failure may be related though.
I see how the doctest is failing, but its not at all clear why it _isnt_ failing in master
I couldn't seem to reproduce that error locally. Looks like the assertion was added back in #30619 to close a mypy issue - @simonjayhawkins any idea?
> yeah i agree, let's create registry.py

hmm not sure about this, since the other two funcs im interested in (pandas_dtype and is_dtype_equal) depend on registry, so not sure where those would go.

but its a moot point until i figure out the doctest thing, which im getting nowhere on
> I couldn't seem to reproduce that error locally. Looks like the assertion was added back in #30619 to close a mypy issue - @simonjayhawkins any idea?

I think the comments in #30619 explain the reasoning. Could either sort out the class hierarchy so that Liskov is obeyed (maybe add something like a _basename class attribute to access from the construct_from_string classmethod) or simply change the assert to a cast or #type: ignore for now.
lgtm ping on green
ping
Add support for gitpod
State: closed

This PR proposes to add support for Gitpod, which can provide (new) contributors with one-click environment to use for working on the pandas code base. 

The PR includes two main additions:
- .gitpod.yml - Config file for gitpod that specifies which image/Dockerfile to use for gitpod and which init command(s) to run when booting the pod.
- .gitpod.Dockerfile - Dockerfile used to build a base image for the pod. Partially adapted from the existing Dockerfile in the repo.

Some things maybe still to consider:
- Gitpod has some options for providing pre-built workspaces (https://www.gitpod.io/docs/prebuilds/). This may be interesting to enable.
- There is some duplication between the two dockerfiles, but unfortunately the VS Code Dockerfile is not entirely compatible with the gitpod one. As such, the duplication may not be a bad thing. 

- [x] closes #xxxx (N/A)
- [x] tests added / passed (N/A)
- [x] passes `black pandas` (N/A)
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` (N/A)
- [x] whatsnew entry (N/A)


Labels:
name: Enhancement
name: Build
description: Library building on various platforms

Comments:
Didn't realize this was copied from an existing Dockerfile. With that being the case, any reason to not just leverage the existing file? Looks like you can do that in a config

https://www.gitpod.io/docs/config-docker/
Thanks @jrderuiter - I wasn't aware of gitpod, but this looks really cool!

However, when I clicked on "gitpod - ready to code" from your branch, I get an environment that doesn't look quite set up correctly:
```python-traceback
gitpod /workspace/pandas $ python -c "import pandas"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/workspace/pandas/pandas/__init__.py", line 16, in <module>
    raise ImportError(
ImportError: Unable to import required dependencies:
numpy: No module named 'numpy'
pytz: No module named 'pytz'
```

Is there any extra configuration required? And as Will suggested, does it work if you use the existing Dockerfile in
```yaml
image:
  file: .gitpod.Dockerfile
```
?
closing as stale, but please ping if you'd like to keep working on this
BUG fix _Unstacker int32 limit in dataframe sizes (pandas-dev#26314)
State: closed

- [x] closes #26314 
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry



I have tested it with :
`df = pd.DataFrame(np.random.randint(low=0, high=1500000, size=(90000, 2)), columns=['a', 'b'])`
`df.set_index(['a', 'b']).unstack()`

I am not sure if I should add it as a test, it requires quite some memory.  I am also hesitant about where the test should be located in case it's added.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> I am not sure if I should add it as a test, it requires quite some memory. I am also hesitant about where the test should be located in case it's added.

how long does this take / how much memory.?

you can mark tests as well.
can u merge master and see if can get passing
@KaonToPion closing as stale. please LMK if you want to continue and the PR will be reopened.
@simonjayhawkins could you re-open it? I can take it again
I am having trouble with the fix. This is the code
```

# Bug fix GH 20601
# If the data frame is too big, the number of unique index combination
# will cause int32 overflow on windows environments.
# We want to check and raise an error before this happens

num_rows = np.max([index_level.size for index_level in self.new_index_levels])
num_columns = self.removed_level.size

# GH20601: This forces an overflow if the number of cells is too high.

num_cells = np.multiply(num_rows, num_columns, dtype=np.int32)
if num_rows > 0 and num_columns > 0 and num_cells <= 0:
    raise ValueError("Unstacked DataFrame is too big, causing int32 overflow")
```
The issue right now is that "raise ValueError" generates troubles in all the other systems so there are two ways out of this:

1) Checking with int64 
num_cells = np.multiply(num_rows, num_columns, dtype=np.int64)
This solves the issue, but I am not sure how to test this, because It doesn't matter the size of my Ram (I have even tried with a 500 Gb RAM machine) , that I have a memoryError way before I am able to raise the ValueError exception using num_cells

2) Just deleting the num_cells, num_rows, and num_columns and the raise_exception so go forward with the MemoryErrors that a lot of people are going to have when doing the df.unstack() because their data will unwrapp.

My first commit tried the first option but trying to test it I find that the second option is probably the best option.

Any opinion on this @jreback @simonjayhawkins ?
DOC: move 'Other API changes' under correct section
State: closed

The "Other API changes" bullet points were split in two (one part under "Other API changes", and one part directly under "Backwards incompatible API changes"), and in previous whatsnew files, the "Other API changes" is also a subsection of "Backwards incompatible API changes".

So moved a few things around to make this consistent.

Labels:
name: Docs

Comments:
This was inspired by looking at https://github.com/pandas-dev/pandas/issues/34801, but this PR does not yet "fix" anything, just cleans up the organization.
BUG: Respect center=True in rolling.apply when numba engine is used
State: closed

- [x] closes #34784
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding
name: numba
description: numba-accelerated operations

Comments:
thanks!
REF: move Resolution to tslibs.dtypes
State: closed

This is pretty much a clean move, in preparation for making FreqGroup an enum and trying to de-duplicate our 3+ enum-like classes

Labels:
name: Frequency
description: DateOffsets

Comments:
At some point in one of the previous PRs, you said that deduplication might not be possible after all (https://github.com/pandas-dev/pandas/pull/34587#issuecomment-639530365). Can you give a bit of context on what your idea / goal is on that now?
> Can you give a bit of context on what your idea / goal is on that now?

The mapping between FreqGroup and Resolution is not going to be a bijection, but im now thinking that both FreqGroup.FR_DAY and FreqGroup.FR_BUS map to Resolution.RESO_DAY.  This will allow us to clean up semi-kludge [here](https://github.com/pandas-dev/pandas/blob/master/pandas/core/indexes/period.py#L520).
BUG: apply() fails on some value types
State: closed

- [x] closes #34529
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Thanks @Veronur - can you add a test to show how this fixes the linked issue?
Sure it will be done but i think ill need to look into what made it fail the pandas-dev.pandas and the pandas-dev.pandas (Windows py37_np18)  first

I am working on the test right now, i linked this first pull request just to start the conversaion on on whether looking at this table the change makes sense.
![photh](https://user-images.githubusercontent.com/26484334/84786729-7edb3480-afc3-11ea-8273-8485aea18f3d.png)

Hello @Veronur! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-06-18 00:05:00 UTC
Alright so i made the test for the issue and i fail before the change and stops failing after. If there is anything else to be done, please let me know. If not let me know and i will make my last pull request with the test and the change
Rebased the all commits into one so it becomes easier to look into them
Merging master should fix CI. Can you add a release note to whatsnew/v1.1.10.rst?
is there anything else that needs to be done here?
thanks @Veronur
BUG: reading line-format JSON from file url #27135
State: closed

- [x] closes #27135
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew note entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
The Travis CI check for python3.9-dev failed. Not sure what's going on.
I think looks good. Can you add a whatsnew note for v1.1?
Thanks @VirosaLi
API: Allow non-tuples in pandas.merge
State: closed

Closes https://github.com/pandas-dev/pandas/issues/34741, while
retaining the spirit of the spirit of
https://github.com/pandas-dev/pandas/pull/34208.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thanks @TomAugspurger
Regression in to_timedelta with errors="coerce" and unit
State: closed

Introduced in https://github.com/pandas-dev/pandas/commit/d3f686bb50c14594087171aa0493cb07eb5a874c

In pandas 1.0.3

```python
In [2]: pd.to_timedelta([1, 2, 'error'], errors="coerce", unit="ns")
Out[2]: TimedeltaIndex(['00:00:00.000000', '00:00:00.000000', NaT], dtype='timedelta64[ns]', freq=None)
```

In master, we raise.

```pytb
In [2]: pd.to_timedelta([1, 2, 'error'], errors="coerce", unit="ns")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-a3691c044041> in <module>
----> 1 pd.to_timedelta([1, 2, 'error'], errors="coerce", unit="ns")

~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/tools/timedeltas.py in to_timedelta(arg, unit, errors)
    101         arg = arg.item()
    102     elif is_list_like(arg) and getattr(arg, "ndim", 1) == 1:
--> 103         return _convert_listlike(arg, unit=unit, errors=errors)
    104     elif getattr(arg, "ndim", 1) > 1:
    105         raise TypeError(

~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/tools/timedeltas.py in _convert_listlike(arg, unit, errors, name)
    140
    141     try:
--> 142         value = sequence_to_td64ns(arg, unit=unit, errors=errors, copy=False)[0]
    143     except ValueError:
    144         if errors == "ignore":

~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/arrays/timedeltas.py in sequence_to_td64ns(data, copy, unit, errors)
    927     if is_object_dtype(data.dtype) or is_string_dtype(data.dtype):
    928         # no need to make a copy, need to convert if string-dtyped
--> 929         data = objects_to_td64ns(data, unit=unit, errors=errors)
    930         copy = False
    931

~/Envs/dask-dev/lib/python3.7/site-packages/pandas/core/arrays/timedeltas.py in objects_to_td64ns(data, unit, errors)
   1037     values = np.array(data, dtype=np.object_, copy=False)
   1038
-> 1039     result = array_to_timedelta64(values, unit=unit, errors=errors)
   1040     return result.view("timedelta64[ns]")
   1041

pandas/_libs/tslibs/timedeltas.pyx in pandas._libs.tslibs.timedeltas.array_to_timedelta64()

ValueError: unit must not be specified if the input contains a str
```

This restores the 1.0.3 behavior, and adds an additional test for `errors="ignore"`, and cleans up the  `to_timedelta` docstring.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timedelta
description: Timedelta data type

Comments:
thanks @TomAugspurger
TST: ensure read_parquet filter argument is correctly passed though (pyarrow engine)
State: closed

xref https://github.com/pandas-dev/pandas/issues/26551#issuecomment-643882543

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Parquet
description: parquet, feather

Comments:
@MeeseeksDev backport to 1.0.x
i'll make a start on this shortly
Thanks for the backport!
Backport PR #34718 on branch 1.0.x (Removed __div__ impls)
State: closed

https://github.com/pandas-dev/pandas/pull/34711#issuecomment-644148383


@WillAyd on 1.0.x. we have  cython>=0.29.13 and on master cython>=0.29.16. is this an issue?


Labels:

Comments:
> @WillAyd on 1.0.x. we have cython>=0.29.13 and on master cython>=0.29.16. is this an issue?

this was done in #33334, don't think it's sensible to bump cython on a patch release
Yea seems like the PR you linked would only be needed if we were also back porting #33220 but don't think that's the case. So should be OK to not touch Cython version
CI: Checks-and-Web-and-Docs-on-1.0.x
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
DO NOT MERGE - probably don't want the web and docs to run on push
> DO NOT MERGE - probably don't want the web and docs to run on push

although that'll be less to do in the release process if we have the backport branch docs published somewhere. although if they build successfully on the PR then we should be OK?
closing for now. will need to make sure we have github actions running on 1.1.x from the start after the 1.1.0rc release
Backport PR #34711 on branch 1.0.x (BLD: Pin cython for 37-locale build)
State: closed

xref #34711

Labels:
name: Build
description: Library building on various platforms

Comments:
see https://github.com/pandas-dev/pandas/pull/34711#issuecomment-644148383, abort this backport?
closing for now. I probably should have pinned at 0.29.13 anyway.
cc @jorisvandenbossche just to be aware i've closed this.
OK!
CI testing for s3 reads from public buckets on 1.0.x - DO NOT MERGE
State: closed

xref #34626

Labels:

Comments:
@jorisvandenbossche this works for me locally. If you can have a look at the failures in Travis and let me know if we're confident that #34626 is fixed on 1.0.x
Does the test run on one of the azure builds as well?
cc @TomAugspurger do you have any experience with this? On travis, the S3 public bucket reading is failing with boto / permission errors.

I checked out the 1.0.x branch locally, and can confirm it is working locally.
Not sure. I notice that the two jobs with failing moto / boto issues have moto==1.3.7. https://travis-ci.org/github/pandas-dev/pandas/jobs/698478225#L1128 has 1.3.14. Perhaps we failed to backport a dependency update?
But moto shouldn't be used for that test? (unless that is done automatically?)
Ah I missed that this was for the public bucket. You're correct, I think moto shouldn't be involed here..
When you're testing locally, do you run the whole file or just the single test? s3fs does cache things, including the S3FSFileSystem constructor, which might cause this.
> Does the test run on one of the azure builds as well?

i'll create a failing test to make sure
> When you're testing locally, do you run the whole file or just the single test? s3fs does cache things, including the S3FSFileSystem constructor, which might cause this.

I just ran ``pd.read_csv("s3://nyc-tlc/misc/taxi _zone_lookup.csv")`` in a console
But running it with `pytest pandas/tests/io/test_s3.py` also works
if we get the tests passing, does pandas have an amazon account where we can place a test file in a public bucket?
hmm travis didn't run. 

> Does the test run on one of the azure builds as well?

on 3 azure builds, will revert the fail shortly
OK, if it thus was passing on azure, I think we can be confident enough for the 1.0.5 release that this is  properly working. And can then figure out the travis failure later when we fix this in master
> OK, if it thus was passing on azure, I think we can be confident enough for the 1.0.5 release that this is properly working. And can then figure out the travis failure later when we fix this in master

OK closing
We could also already merge it (but then in the master branch), with an xfail, if you want
i'm not sure about the ethics of using someones public bucket for testing
Ref my comment here: https://github.com/pandas-dev/pandas/issues/34626#issuecomment-640286333

I couldn’t get a test for public buckets working using moto.

I propose we use an Amazon Open Data public s3 bucket to test https://aws.amazon.com/opendata/public-datasets/

We can use the tm.network decorator? Would be helpful to catch future regressions - especially since we are moving to fsspec. Thoughts @jorisvandenbossche @simonjayhawkins 

I can open a PR. 
Using an existing public s3 bucket seems fine for this (we should just find a small dataset). 
And indeed, we have network tests, so we can just test this for real, without moto, with the network decorator.  Thanks @alimcmaster1 !