BrandonFors/pandas-github-issues huggingface dataset:
TYP: Fix chainmap typing for mypy 0.740+
State: closed

This will otherwise raise a failure during a `mypy` update. Also increase type precision of this file to 100%.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @simonjayhawkins 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @simonjayhawkins 
note we are currently pinned to 0.73 for a number of issues.
> note we are currently pinned to 0.73 for a number of issues.

Yes, I'm aware of them and this a tiny step to get rid of them.
> > note we are currently pinned to 0.73 for a number of issues.
> 
> Yes, I'm aware of them and this a tiny step to get rid of them.

great that's what I figured, thanks!
PR looks good to me.

There has been a build-in collections.ChainMap since python 3.3. Is this version different or could we switch over to the builtin version?
> PR looks good to me.
> 
> There has been a build-in collections.ChainMap since python 3.3. Is this version different or could we switch over to the builtin version?

The `typing` one used here is simply a typed alias for it that exists since Python 3.6.
> note we are currently pinned to 0.73 for a number of issues.

This will help for 0.73 too...

#30539 pandas\compat\chainmap.py:4: error: Implicit generic "Any". Use "typing.ChainMap" and specify generic parameters

thanks @xhochy
BUG: groupby apply raises ValueError when groupby axis has duplicates and applied identity function
State: closed

This is a more of a patch than a complete solution to this groupby apply paradigm. 
When there are duplicates in the groupby axis, we restore the axis to its original order, but not guaranteeing that the order of the data with the same axis values is restored.

- [x] closes #30667
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
Restarted the failing travis build 
Hello @fujiaxiang! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-20 13:53:25 UTC
ping
All tests passed on my local machine which has the most recent master merged. Don't know why this fails. I see many other PRs also failing. Is this a CI issue?
The CI process keeps failing. Can anyone tell me how I can retrigger CI without pushing new commits?
thanks @fujiaxiang very nice.

so a more longer term solution would be for ``Index.reindex(...)`` to handle duplicates (rather than raising), via a new keyword (e.g. errors='raise' or errors='coerce'`), though maybe have to think about the api a bit. this might be a fairly big change though.
BUG: bug in date_range with custom business hours and given periods
State: closed

- [x] closes #30593
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
thanks @fujiaxiang
TST: Add more tests for fixed issues
State: closed

- [x] closes #11244
- [x] closes #11020
- [x] closes #12754
- [x] closes #12261
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @mroeschke nice!
CLN: Simplify rolling.py helper functions
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Cleans helper functions in `rolling.py` and remove dead code paths.

Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
Nice cleanup
thanks @mroeschke 

doc-strings and types always welcome
MAINT: Change all pandas links to use HTTPS
State: closed

Also update the link to pandas' `whatsnew` page

Labels:
name: Docs
name: Clean

Comments:
> any way to check these all work?

This is a simple change from HTTP to HTTPS for all of the `pandas.pydata.org` links. `pandas.pydata.org` supports HTTPS across the board.

The question of whether these links work is actually not relevant here.  If they don't work, they wouldn't have worked on HTTP.
ENH: Implement PeriodIndex.intersection without object-dtype cast
State: closed

PeriodIndex._simple_new is not as simple as it should be.  In order to get it to have the appropriate signature we need to implement some of the set methods correctly.  This is the first of those.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type
name: Index
description: Related to the Index class or subclasses

Comments:
> How does the ASV for period.Indexing.time_intersection look?

For the length=1000 index in the benchmark we take a hit of about 37%.  When I change the benchmark to be length=10**5 it is no change.
> > How does the ASV for period.Indexing.time_intersection look?
> 
> For the length=1000 index in the benchmark we take a hit of about 37%. When I change the benchmark to be length=10**5 it is no change.

interesting, i though this would be faster. ok
CI: unpin IPython
State: closed

Closes https://github.com/pandas-dev/pandas/issues/30537

Labels:
name: CI
description: Continuous Integration

Comments:
LGTM
Does it matter this version was released two days ago? I guess IPython is technically a dev dependency so maybe not, but I would think its common enough outside of just pandas development that requiring such a new release could cause inconsistencies with other packages a user may have
This is only for the doc build / dev environment. We haven't changed the requirement for users.
thanks @TomAugspurger 

the less we pin in ci the better generally
TYP: check_untyped_defs pandas/io/sql.py
State: closed

pandas\io\sql.py:785: error: "insert" of "list" does not return a value
pandas\io\sql.py:1450: error: "insert" of "list" does not return a value


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Thanks @simonjayhawkins
Fix flakey base setitem test
State: closed

This test makes a potentially incorrect assertion about the data provided to the test. We already require that `data[0] != data[1]`, so it can be used instead.

This failed in https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=24731&view=logs&j=a67b4c4c-cd2e-5e3c-a361-de73ac9c05f9&t=33d2fdd0-c376-5f94-e6d3-957bdd23a3b8.

Labels:
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Thanks @TomAugspurger
TYP: check_untyped_defs io.sas.sasreader
State: closed


pandas\io\sas\sasreader.py:75: error: Incompatible types in assignment (expression has type "SAS7BDATReader", variable has type "XportReader")

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
rebase as well
need to sort out some import issues. closing to reduce queue for now


@simonjayhawkins if you can merge master and merge on green.
TYP: --disallow-any-generics pandas\core\reshape\concat.py
State: closed

xref #30539

pandas\core\reshape\concat.py:477: error: Missing type parameters for generic type "List"
pandas\core\reshape\concat.py:504: error: Missing type parameters for generic type "List"

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
Thanks @simonjayhawkins
REF: change TDI.delete behavior to match DTI.delete
State: closed

With this change, the two methods behave the same and can be shared.

Labels:
name: Timedelta
description: Timedelta data type
name: Index
description: Related to the Index class or subclasses

Comments:
Whats the behavior change between "infer" and None?
> Whats the behavior change between "infer" and None?

```
tdi = pd.TimedeltaIndex(["1 Day", "2 Days", None, "3 Days", "4 Days"])

>>> tdi.delete(2).freq
None   <-- PR
<Day>  <-- master
```

ping
TST: Test for merge_asof groupby=multiple with categorical column
State: closed

- [x] closes #16454
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @dwhu! Congrats on your first commit to pandas!
ENH: .equals for Extension Arrays
State: closed

- [x] closes #27081
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

--- Jan 3 10:00 PST

Hey team,

Looking for a bit of feedback on this API design and appropriate testing for this PR before I get too deep into it. Could you provide some guidance regarding how to appropriately handle updating this test? https://github.com/pandas-dev/pandas/blob/6f03e76f9d47ecfcfdd44641de6df1fc7dd57a01/pandas/tests/extension/base/ops.py#L162 It appears to contradict the issue #27081.

Appreciate the help,
Dave


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks for your contribution! @dwhu 

in general here, if it is `NotImplemted`, and you implement some new functionalities, you could remove this one, and replace it with the corresponding tests to test your new implementations!

Good luck! and hope this helps
> in general here, if it is `NotImplemted`, and you implement some new functionalities, you could remove this one, and replace it with the corresponding tests to test your new implementations!

@charlesdong1991 - Sounds good. Thanks for the feedback. Wanted to make sure I wasn't missing something or breaking a behavior contract by adding this behavior.

@dwhu looks like CI is red - can you take a look and fix up the merge conflict?
Closing as I think this has gone stale but ping if you'd like to pick back up. Thanks!
@dwhu I am going to look into further updating this, if that's OK with you
OK, I updated this with the following things:

- Made `__eq__` into an abstract method so it is required to implement (another option could be to `return NotImplemented` as default, but that's actually wrong, as it results in not doing element-wise comparison for `EA == EA`, but just returning False)
- Fixed the `equals` imlementation: it became a bit longer to ensure we check for BooleanArray (if the `==` comparison returns a nullable boolean array, we need to fill NAs with False)
- Added `ExtensionBlock.equals` (overriding the base Block method) to ensure we dispatch to the new EA method
- A bit of clean-up of the tests

Should be ready for a new review now @TomAugspurger @jbrockmendel 
@simonjayhawkins how do I solve this typing error?

```
pandas/core/arrays/base.py:338: error: Return type "Union[Any, ExtensionArray]" of "__eq__" incompatible with return type "bool" in supertype "object"
```

So our annotation of `__eq__` is incompatible with the base "object" class, but we do this on purpose. So is there another way to solve this than to suppress the error in some way?
Any more comments on this PR?
will look soon
@jreback you wanted to take a look here?
Travis failures are unrelated (network issues, S3)
REF: use _data.take for CI/DTI/TDI/PI.take
State: closed

cc @jschendel @jreback any idea why IntervalIndex doesn't use _assert_take_fillable like the others?  If it can/should, then we can share this method between all of our EA-backed indexes.  (Actually also need to make sure the slice behavior in the DTI/TDI/PI is OK to do for all of them)

Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
thanks
REF: restructure api import
State: closed

Previously, `pandas.api` was imported indirectly via `pandas.io.json._table_schema`. This makes things a bit more direct.

I think we'll want a code check for importing from `pandas.api` within `pandas.core` but I wasn't able to easily write one (things like docstrings may want to have `import pandas.api` for example).

Labels:
name: Clean

Comments:
LGTM
TST: Adding test to concat Sparse arrays
State: closed

- [ ] closes #23557 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
thanks @SdgJlbl
DOC: Change refs in docs from pandas.SparseArray to pandas.arrays.SparseArray
State: closed

- [x ] closes #30642
- [ ] tests added / passed
  - N/A
- [ ] passes `black pandas`
  - N/A
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
  - N/A
- [x] whatsnew entry

Change all references in docs to use `arrays.SparseArray` rather than just `SparseArray`


Labels:
name: Sparse
description: Sparse Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
In #30642 I think Jeff was proposing that accessing `pandas.SparseArray` from the top-level namespace emits a warning.
> In #30642 I think Jeff was proposing that accessing `pandas.SparseArray` from the top-level namespace emits a warning.

Ah, that's a bigger change.  I will change all references to `pandas.SparseArray` in the code to `pandas.arrays.SparseArray` and add the deprecation warning to `pandas.__init__.py`

It's probably worth discussing whether we actually want to do that for 1.0. I'm not sure if it's worth introducing that right now.
@TomAugspurger Well, I have all the changes done and am about to commit.  I'll separate it to a different PR, so you can merge this one as is (with the doc changes), and then we can use the new PR to discuss whether we do all those changes.
> It's probably worth discussing whether we actually want to do that for 1.0. I'm not sure if it's worth introducing that right now.

@TomAugspurger not sure what the issue is here. This makes things very consistent with other EA (I agree Categorical is still there, but this makes a good step).
I like #30566 better than this.
> I like #30566 better than this.

#30566 has everything this one does, so if we go forward with that, we can close this one.
closing in favor of #30566
BUG: Fix IntervalArray equality comparisions
State: closed

- [X] closes #24112
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

cc @TomAugspurger 

Labels:
name: Bug
name: Interval
description: Interval data type

Comments:
@jschendel ok, hopefully some of this can be pushed down to base classes; @jbrockmendel merge if your comments have been addressed.
> hopefully some of this can be pushed down to base classes

Some of this probably could be pushed down to base classes but need to be careful because it could cause a perf hit.  In the `IntervalArray` implementation I took a bit of care to not actually create any `Interval` objects and instead use comparisons against the `left`/`right` arrays.  I think using the base class implementation would end up creating `Interval` objects.  Did some quick ad hoc timings and looks like the perf hit can be ~3x in worst case.
ok fIr @jschendel yep we often balance perf of specialized code paths vs more generic code
k thanks @jschendel
BLD: address build warnings
State: closed

xref #30609

Labels:
name: Clean

Comments:
Nice! Does this address all of them?
Not sure if you've seen it, but besides the failure in the docs, this error is happening lots of times in the tests:
```
Exception ignored in: 'pandas._libs.window.aggregations.is_monotonic_start_end_bounds'

TypeError: Argument 'arr' has incorrect type (expected numpy.ndarray, got pandas._libs.window.aggregations._memoryviewslice)
```
> Does this address all of them?

No.  Locally (only checked OSX) it addresses the buffer unpacking in aggregations.pyx and the assignment warnings for algos.c.  The __pyx_memview_get_object ones I think are coming from an unused import; they dont correspond directly to any of the .pyx code
> > Does this address all of them?
> 
> No. Locally (only checked OSX) it addresses the buffer unpacking in aggregations.pyx and the assignment warnings for algos.c. The __pyx_memview_get_object ones I think are coming from an unused import; they dont correspond directly to any of the .pyx code

Nice! What's the best way of figuring out what unused import is causing? - For my own knowledge!
`pandas/_libs/hashing.c:25344:20: warning: ‘__pyx_memview_get_object’`

Also I randomly get this warning: (which doesn't seem to be as expected)
```
pandas/_libs/src/ujson/python/objToJSON.c: In function ‘NpyArr_encodeLabels’:
pandas/_libs/src/ujson/python/objToJSON.c:1616:18: warning: ‘len’ may be used uninitialized in this function [-Wmaybe-uninitialized]
```




> What's the best way of figuring out what unused import is causing? - For my own knowledge!

No idea
The one is caused by Cython so can't speak to that but for the JSON one if you look at the line:

https://github.com/pandas-dev/pandas/blob/4206fd42cc5cd20204c0c5f192f7e59f204ad48d/pandas/_libs/src/ujson/python/objToJSON.c#L1616

The warning is saying that there are some code paths where `len` is declared but never assigned, which would give it an undefined value and either give weird results or segfault if that line gets hit without `len` ever having been assigned.

Not clear what path that is though - you are only getting the warning intermittently?
Thanks @jbrockmendel
ENH: Create DockerFile and devcontainer.json files to work with Docker and VS Code in Containers
State: closed

- [x] closes #30614 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The instructions aren't that well written, and should probably be improved, but here's a first shot.

Labels:
name: Enhancement
name: Build
description: Library building on various platforms

Comments:
@WillAyd : Could you have a look?
I’ll look this weekend 

Sent from my iPhone

> On Jan 3, 2020, at 6:13 PM, gfyoung <notifications@github.com> wrote:
> 
> ﻿
> @WillAyd : Can you have a look?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Two questions, neither blockers.   Is the root directory the ideal/only place for this?  Do we need to worry about this getting out of date?
@jbrockmendel per the VS Code docs `devcontainer.json` needs to be either in the root directory or in `root/devcontainer`. And regarding becoming outdated, the `DockerFile` is really very simple, just updating the Linux base image, cloning the pandas repo, and building it. Not sure what will be out of date.
Hi all, just bumping this in case it gets stale @pandas-dev/pandas-core can we merge?
REF/BUG: DTA/TDA/PA comparison ops inconsistencies
State: closed

It will take a couple of steps to get these three methods to all behave the same.  Following that we can move to share code between them.

This came up when trying to make the indexes dispatch `searchsorted` and it turns out that the DTI/TDI/PI searchsorted methods are not consistent with their DTA/TDA/PA counterparts.  The fix is to have shared validation/casting methods, which should in turn be consistent with the comparison ops.

Labels:
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type
name: Period
description: Period data type

Comments:
Thanks @jbrockmendel
DEPR: Deprecate numpy argument in read_json
State: closed

Co-authored-by: Luca Ionescu <lucaionescu@users.noreply.github.com>

- Continuing https://github.com/pandas-dev/pandas/pull/28562
- [x] closes #28512

@lucaionescu - i've merged master and pushed to here (I don't have the permissions to push to your branch), I will aim to fix up the tests. Or feel free to take it from here if you have time?

Opening up WIP PR to see what test failures need addressing. 

I've also co-authored the commit.

Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Deprecate
description: Functionality to remove in pandas

Comments:
Can you merge master?
> Can you merge master?

done!
friendly ping @jreback 
thanks @alimcmaster1 very nice!
CI: Remove pin google-cloud-bigquery
State: closed

xref: https://github.com/pydata/pandas-gbq/issues/271 is now resolved.

This can be removed since `pandas-gbq` depends on `google-cloud-bigquery` ref: https://github.com/pydata/pandas-gbq/blob/master/setup.py

Labels:
name: CI
description: Continuous Integration

Comments:
lgtm
great.
Thanks @alimcmaster1
TPY: Add Types to gbq.py
State: closed

Add Types to args for `to_gbq` and `read_gbq`.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @alimcmaster1 

you might want to push these annotations to pandas-gbq as well
> thanks @alimcmaster1
> 
> you might want to push these annotations to pandas-gbq as well

Thanks will do!
CLN: Update old string formatting to f-string
State: closed

- [x] contributes to #29547 
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Updates:
 pandas/core/ops/array_ops.py
 pandas/core/ops/dispatch.py
 pandas/core/ops/docstrings.py
 pandas/core/ops/invalid.py
 pandas/core/ops/methods.py
 pandas/core/ops/roperator.py


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@thepaullee can you have a look at the CI please? Seems like some things are broken.
Thanks @thepaullee!

Do the maintainers have a preferred style for the changes introducing temporary variables just for f-strings? For example, https://github.com/pandas-dev/pandas/pull/30631/files#diff-f91291af4cee29af665968c2ffd05013R105-R106 isn't an obvious improvement to me.
CI: Fix Flakey GBQ Tests
State: closed

-ref https://github.com/pandas-dev/pandas/pull/30478#issuecomment-570068174
We see the below in the logs:
```
google.api_core.exceptions.Conflict: 409 POST https://bigquery.googleapis.com/bigquery/v2/projects/pandas-travis/datasets: Already Exists: Dataset pandas-travis:pydata_pandas_bq_testing_py31
```
https://travis-ci.org/pandas-dev/pandas/jobs/631599036

Despite attempting to delete the dataset in the previous line.
`self.client.delete_dataset(self.dataset, delete_contents=True)`

Since we run with `dist=loadfile` these tests are run sequentially by pytest. But they could potentially clash across builds?

We now create a unique dataset name per test function and teardown when complete

GBQ Tests will run against my fork will post results on here.

cc. @jreback, @tswast

Labels:
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
Hmm wouldn't this keep the dataset lingering around? Perhaps an alternate is to just split this into two fixtures and scope the dataset one to say the module level or even higher
> Hmm wouldn't this keep the dataset lingering around? Perhaps an alternate is to just split this into two fixtures and scope the dataset one to say the module level or even higher

Sure if we don't want to keep the dataset - then an easy way is to give the dataset a random name and create/delete in this fixture. I've pushed the update.

lgtm. very minor comment. ping on green.
> nitpick can you make this a module level function

Sure done.

[Link to my travis fork](https://travis-ci.org/alimcmaster1/pandas/builds/632086058?utm_medium=notification&utm_source=email) where these tests have passed: (since they only run on our master branch) 
great thanks @alimcmaster1 
Let me know if you see any more issues - thanks!
thanks @alimcmaster1