BrandonFors/pandas-github-issues huggingface dataset:
TST: 2d index when constructing dataframe (#25416).
State: closed

- [X] closes #25416
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This change adds a regression test to check for the handling of 2d index when creating a dataframe (#25416).

Since there is no visible change no whatsnew entry has been added.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @qudade
`values=['c']` returns `aggfunc` on all columns, instead of just the passed column
State: closed

- [ ] closes #29080



Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @quanghm! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-12 04:58:12 UTC
@quanghm can you add tests for the bug this fixes
> 
> 
> @quanghm can you add tests for the bug this fixes

@jbrockmendel I'm not sure how to add the tests. Can you point to some links? Thanks.
> I'm not sure how to add the tests. Can you point to some links? 

Take a look in pandas/tests/groupby/ for examples
@quanghm is this still active? Can you add tests?
> 
> 
> @quanghm is this still active? Can you add tests?

Sorry, has been caught at work lately. I'll write the test over the weekend.
@quanghm looks like CI is failing - can you take a look?
closing as stale. if you'd like to continue, pls ping. this needs tests.
TST: Add regression test for Series dropping uint datatype (#18311)
State: closed

- [x] closes #18311
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @Reksbril
DOC: updated categorical docstring
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed *(tests failed even on master)*
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I think `api.types.CategoricalDtype` in *see also* section of [pandas.Categorical](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Categorical.html) documentation which does not have any link, is better to point to [pandas.CategoricalDtype](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.CategoricalDtype.html?highlight=categoricaldtype) for better understanding of this type. So I made this change and created a pr.

<img width="960" alt="chrome_2019-10-18_14-22-01" src="https://user-images.githubusercontent.com/7765309/67089231-0746d280-f1b4-11e9-9bba-c88ccfd7fa93.png">


Labels:
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
hmm the link doesn't work in *either* location. ``pd.api.type.CategoricalDtype`` might be needed here
That's strange. When I build documentation with `python make.py html` command, it has the link. but when I build them with `python make.py --single pandas.Categorical` command, it does not have a link. There is a bug in `--single` switch?
if you only build a single page links are not built by definition as you don’t have anything to link

all build should show the link

Oh, I see. So the link works with the change I made earlier.
![image](https://user-images.githubusercontent.com/7765309/67134074-f1b9c300-f21c-11e9-9841-bd8dfd734374.png)

great thanks @javad94 
@jreback Your welcome. Thanks to you too.
TST: add regression test for all-none-groupby
State: closed

- [X] closes #21624 
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Is the whatsnew entry required if the fix was somewhat silent and only a regression test was added?


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
thanks @crepererum
TST: regression test for groupby with datetime and timedelta (#15562)
State: closed

- [X] closes #15562
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This change doesn't add a new feature but adds an regression test for #15562 which has been fixed with pandas 0.20. Hence there is no whatsnew entry.

The bug was reported for pandas 0.18.1 and persisted until 0.19.2 but was fixed in 0.20.0(-rc1). I managed to reproduce the error (with `numpy==1.12.1` on python 3.6, otherwise some errors unrelated to this bug appear).

I added an regression test to `pandas/tests/groupby/test_apply.py` and verified that the code will fail with 0.19.2 and succeed with 0.20.0 and master.

I wasn't able to further reduce the test provided by @field-cady.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
@mroeschke Change has been pushed. Some azure pipeline timed out. Is there a way to restart tests? Or should I re-push my branch?

(update: Apparently the command/comment [`/AzurePipelines run`](https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#comment-triggers) would work but I don't have the privileges. I replaced that comment after nothing happened apparently)
<samp>
Commenter does not have sufficient privileges for PR 29063 in repo pandas-dev/pandas<br>

</samp>
If you merge master it will run the CI (master was having problems yesterday)
ok, ping on green.
@jreback Tests are green
thanks @qudade
API: Restore getting name from MultiIndex level
State: closed

xref https://issues.apache.org/jira/browse/ARROW-6922 /
https://github.com/pandas-dev/pandas/pull/27242#issuecomment-543302582
/ https://github.com/pandas-dev/pandas/issues/29032

No docs yet, since it isn't clear how this will eventually sort out. But
we at least want to preserve this behavior for 1.0

cc @jorisvandenbossche @topper-123 

Labels:
name: MultiIndex

Comments:
what is exactly the error from pyarrow?
I'm working on the implementation discussed in #27242, including new to be deprecated levels with names. Won't that fix this issue?
> I'm working on the implementation discussed in #27242, including new to be deprecated levels with names. Won't that fix this issue?

As I mentioned in the other issue, IMO we should first properly discuss if we actually want to deprecate `.levels` 
And in general, I would prefer to not have master in a "broken" state (meaning that we removed something that will only be temporary). For projects trying to be good citizens and testing against pandas master that is a bit annoying.


> what is exactly the error from pyarrow?

pyarrow is using the name of a level, and this is now no longer preserved.
Let's merge this, and discuss future plans in https://github.com/pandas-dev/pandas/issues/29032? This at least gets CI passing for downstream projects.
sgtm.
do we have sufficient tests for downstream though (can be added later in test_downstream.py)
> do we have sufficient tests for downstream though (can be added later in test_downstream.py)

I am not sure additional tests are needed. We have tests for it in out own tests. We just broke purposefully our API (and change our own tests to reflect that), so it is to be expected that behaviour that relies on accessing the level names like this will be broken.
@TomAugspurger thanks for the quick PR! 

@topper-123 As mentioned above, IMO we should still first have a discussion about a deprecation of `.levels`. Do you want to open a dedicated issue for that? (or is there already an issue to discuss that?)
The discussion about the future of .levels can happen in https://github.com/pandas-dev/pandas/issues/29032.
Stop catching TypeError in groupby methods
State: closed

cc @WillAyd @jreback 

Note this doesn't touch cython_agg_blocks, I've run out of things to meddle with there.

Open to suggestions for the DecimalArray case.

Labels:
name: Groupby
name: Clean

Comments:
> don't we need to have tests for the complex path ?

we have a couple of tests that get here with an ndarray[object] containing complex objects
thanks @jbrockmendel
Backport PR #29057 on branch 0.25.x (CI: xfail on numpy 1.18)
State: closed

Backport PR #29057: CI: xfail on numpy 1.18

Labels:

Comments:
restarted the stalled 3.8 build.
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=19134 timed out getting miniconda. I think we can ignore it.
woops, restarted azure just as you were commenting
No worries. I think we're good to go here.
CI: xfail on numpy 1.18
State: closed

xref #29053 

i think this addresses the failing azure builds, not sure about Travis.

Will open an issue for a longer-term fix.

Labels:
name: CI
description: Continuous Integration

Comments:
cool, merge on green.

do you need to port this to 0.25.2 as well?
Probably need to backport.
Restarted the stalled 3.8 build.
updated the 3.8 build to use the numpy wheel
Support customised S3 servers endpoint URL
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add support for customised S3 servers via checking the environment variable `S3_ENDPOINT`.

If set, the value of `S3_ENDPOINT` will be passed to the `endpoint_url` parameter for `boto3.Session`. It should be the complete URL to S3 service following the format: http(s)://{host}:{port}.

This feature is useful for companies who use their own S3 servers like MinIO, Ceph etc, as mentioned in issue  #26195

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Is S3_ENDPOINT a common variable in the S3 ecosystem? Should s3fs just be using it as the default value for `endpoint_url` if it's not otherwise specified?

Ideally, pandas has as little s3-specific code as possible. We'd like to delegate it to other libraries.
Hi @TomAugspurger.  In this commit, if `S3_ENDPOINT` is not specified, s3fs will be using the default value for `endpoint_url`, i.e., `None`, as documented in [Boto3 client API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client). In this case, s3fs will connect to AWS S3 by default. Therefore, people need minimal effort to use s3 with pandas: for those who are using AWS S3, simply use it as it was before; while for people use their own s3 servers, define `S3_ENDPOINT` environment variable. 

`S3_ENDPOINT` is a common variable to be set if people use their own S3 compatible object storage servers instead of AWS S3. This scenario is becoming more and more common as solutions like MinIO, Ceph gains more popularity. One example is that [tensorflow supports S3 via `S3_ENDPOINT` environment variable](https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/s3.md).   Pandas used to support this option by specifying `AWS_S3_HOST` before switching from boto to s3fs in this [commit](https://github.com/pandas-dev/pandas/commit/dc4b0708f36b971f71890bfdf830d9a5dc019c7b#diff-a37b395bed03f0404dec864a4529c97dL94) as discussed in #26195. 
Agreed with Jeff. I would prefer to see this in s3fs.
closing as out of scope for pandas, encourage to submit to s3fs.
Would appreciate is people on this thread would comment in https://github.com/dask/s3fs/issues/273 as to how they would expect users to codify such arguments. I suppose environment variables is certainly an option, but passing arguments would be preferable. The linked issue talks about embedding these arguments in the URL itself.
CLN: derivation of nogil param
State: closed

This moves the `nogil` parameter of in the template from being hard-coded to being derived from the other parameters.

This changes the parameter value on line 18 (int8/int8) which currently has `nogil=False` which I think is incorrect.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
any perf changes? (would expect +)?
> any perf changes? (would expect +)?

I'll do some timeit checks, but I dont expect much.  Only int8/int8 cases should be affected
Still poking a timeit, but so far looks indistinguishable
cool thanks
Fix mypy in pandas/tests/arithmetic/test_datetime64.py
State: closed

GH28926, mypy did not like DatetimeIndex's __new__,
and couldn't find a method

- [x] addresses part of #28926
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@saskakarsi can you rebase
@saskakarsi Mind merging master - this looks like its nearly there
closing as stale, if you want to continue, pls ping to reopen.
REF: remove groupby_helper
State: closed

no longer necessary now that the funcs there use fused types.

Labels:
name: Groupby
name: Clean

Comments:
Can you run benchmarks to make sure no difference?
> Can you run benchmarks to make sure no difference?

Sure, running now.  But this is _really_ benign, moving code that is already `#include`d directly into the file where it already lives post-include.
For sure. Those benchmarks also take a while...but for moving a large chunk like this better safe than sorry
Noise

```
       before           after         ratio
     [da3d0d92]       [c601d34f]
     <master>         <gbh>     
+      11.1±0.5ms       15.1±0.3ms     1.36  groupby.Nth.time_series_nth('datetime')
+      8.46±0.3ms       9.94±0.3ms     1.18  groupby.AggFunctions.time_different_numpy_functions
-        137±10μs        122±0.5μs     0.89  groupby.GroupByMethods.time_dtype_as_group('int', 'shift', 'direct')
-      2.27±0.2ms      2.01±0.01ms     0.89  groupby.GroupManyLabels.time_sum(1)
-        306±20μs          268±3μs     0.88  groupby.GroupByMethods.time_dtype_as_field('float', 'std', 'transformation')
-      1.10±0.1ms          958±5μs     0.87  groupby.GroupByMethods.time_dtype_as_field('float', 'value_counts', 'direct')
-        293±30μs          230±3μs     0.78  groupby.GroupByMethods.time_dtype_as_group('int', 'min', 'transformation')
-      1.70±0.3ms      1.33±0.02ms     0.78  groupby.SumBools.time_groupby_sum_booleans
-        531±80μs         408±10μs     0.77  groupby.GroupByMethods.time_dtype_as_group('float', 'sum', 'transformation')
-        435±30μs          332±9μs     0.76  groupby.GroupByMethods.time_dtype_as_field('float', 'head', 'direct')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.

```
thanks!
BUG: Fix TypeError in _cython_agg_blocks
State: closed

and change one of the high-level `except Exception`s in groupby.groupby to `except (TypeError, NotImplementedError)`.  The remaining `TypeError` will be addressed in a separate PR.  Haven't decided about the remaining NotImplementedError.

cc @jreback @WillAyd.

Labels:
name: Groupby

Comments:
> As mentioned in other PR I am hoping to do away with this block management entirely so not sure how much you want to invest in this...

I still very much want to see that happen.  FWIW this should be the last PR in this sequence directly touching cython_agg_blocks.  After this there is one more PR that implements group_add for complex dtypes at which point we can stop catching the `(TypeError, NotImplementedError)` in groupby.py:L1360.
thanks @jbrockmendel
TST: Test key dtype cast after merge
State: closed

- [X] closes #17044
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Hello @albertvillanova! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-21 19:37:44 UTC
Travis fail is likely unrelated, but there are some linting issues you need to address
Could you merge master? Master was broken recently, but it should be stable now.
Just need to run `black` otherwise LGTM.
Thanks @albertvillanova
fixed issue of mypy for test_ujson
State: closed

- [ ] xref #28926
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @iamrajhans! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-18 06:38:56 UTC
Thanks @iamrajhans
Fix mypy error in pandas\tests\indexes\interval\test_interval_tree.py
State: closed

- [x] relates to #28926
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Hello @pqzx! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-11-06 23:07:39 UTC
Hmm just out of curiosity does this work if you did `(list(x) for x in permutations(range(3)))`. Assuming equivalent I think would be easier to track than the type ignore
> 
> 
> Hmm just out of curiosity does this work if you did `(list(x) for x in permutations(range(3)))`. Assuming equivalent I think would be easier to track than the type ignore

Yes, this works. Updated to use this instead.
@pqzx can you fix lint error and merge master
@simonjayhawkins sorry for the noob question, what's the lint error in this case? Is it the line length?
Looks like the checks are fine. I think if you merge master and try again should work (have had intermittent failures recently)
@WillAyd OK to merge?
Thanks @pqzx !
CLN: Use ABC classes where possible
State: closed

- [x] closes #27353
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
Looks like the linter doesn't like the import order.  can you run `isort pandas/core/reshape/concat.py`
This will need to be profiled carefully given @jorisvandenbossche's comments in https://github.com/pandas-dev/pandas/issues/27353#issuecomment-541301711.
lgtm can you merge master and try again? I think should resolve CI
Rebased, we'll see how this goes.
@jreback When comparing 2ce2f4ff and 9f038370 there is no significant difference on the reshaping benchmark.

[reshaping-results.zip](https://github.com/pandas-dev/pandas/files/3748813/reshaping-results.zip)
@Kazz47 do you have bash output available you can post instead? Would be easier to review
@Kazz47 can you merge master and fix conflict? Also for asv run output just post the output from your terminal
Nice PR but needs rebase and I think has gone stale. Ping if you'd like to pick this back up
CLN: fix mypy error pandas/tests/plotting/test_backend.py
State: closed

GH28926, mypy didn't like setting non-existant attribute on module.
Dummy function, circumvented with `__setattr__`

- [x] addresses part of #28926
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@TomAugspurger mind taking a look at this one?
Thanks!
Backport PR #29010 on branch 0.25.x (Setuptools CI fixup)
State: closed

Backport PR #29010: Setuptools CI fixup

Labels:
name: CI
description: Continuous Integration

Comments:
Looks like the 3.8 build is failing when trying to build numpy
@jbrockmendel merge?
Yeah, I think so... The 3.8 build isn't using conda and so is unaffected by this change.

And hopefully NumPy wheels are out soon and we can just install that for the 3.8 build.
CLN: Fix typing in pandas\tests\arrays\test_datetimelike.py (#28926)
State: closed

- [x] partially adresses #28926
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Merging master should fix the CI failures.
Thanks @jjlkant
Remove to original callable
State: closed

- [x] tests passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

@datapythonista This is the pull request for replacing to_original_callable

Labels:
name: Docs

Comments:
Attached are the two json files with the errors. It looks like it's introducing more errors, but I think that it's just uncovering some errors that were previously covered. We can hold off on this PR though, if you think it's introducing too much. Some of the new errors should be sorted out as I go through the codebase for decorators without the `__wrapped__` atribute.

[change_formatted.txt](https://github.com/pandas-dev/pandas/files/3732146/change_formatted.txt)
[master_formatted.txt](https://github.com/pandas-dev/pandas/files/3732147/master_formatted.txt)

Let me know what you think when you can. There's a lot of data in there.

Thanks for that @Nabel0721. It's not a problem at all to have more errors, if they are genuine. What I don't is to add incorrect errors, because we're validating the wrong docstring.

Do you have the list of new errors? Would be great to have a look at some of them and try to understand why they're appearing after this change.
I’ll look over them more this week. I don’t have as much time as I used to,
so it’ll take a bit. But I should be able to figure out if they’re
legitimate errors or not.

On Tue, Oct 15, 2019 at 10:41 PM Marc Garcia <notifications@github.com>
wrote:

> Thanks for that @Nabel0721 <https://github.com/Nabel0721>. It's not a
> problem at all to have more errors, if they are genuine. What I don't is to
> add incorrect errors, because we're validating the wrong docstring.
>
> Do you have the list of new errors? Would be great to have a look at some
> of them and try to understand why they're appearing after this change.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/29012?email_source=notifications&email_token=ACT76URGIAO6CXLL55PMAQDQOZ5MLA5CNFSM4JBC2M52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBK2TAI#issuecomment-542484865>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACT76URD7SQBFOLUNB2BJF3QOZ5MLANCNFSM4JBC2M5Q>
> .
>
-- 
Sincerely,

*Nathan Abel*
Kettering University - Computer Science
989-780-3050

I think this is already done now that the script lives in numpydoc
CI: troubleshoot RemoveError: 'setuptools' is a dependency of conda
State: closed

- [ ] closes #29008
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@simonjayhawkins I may have an alternative. Will push in a few minutes.
> @simonjayhawkins I may have an alternative. Will push in a few minutes.

good. since I'm using windows locally and that works, so this trial and error could take forever :)