BrandonFors/pandas-github-issues huggingface dataset:
REF: share hashtable_func_helper code
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
all good here?  becoming blockery
Thanks @jbrockmendel (was waiting for Jeff but think he's still on vacation)
BUG/REF: Use lru_cache instead of NUMBA_FUNC_CACHE
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.



Labels:
name: Bug
name: numba
description: numba-accelerated operations

Comments:
> assume this has neutral perf impact

Yup, additionally this actually reduces some unnecessary jitting when e.g. going from `rolling.mean` to `expanding.mean` so that might be a slight performance bump in those odd cases
Typeinterval part1
State: closed

I would like to add `pandas/_libs/interval.pyi`,  and I have a tentative file for that, but when I started to do that, it uncovered issues in our existing typing, because currently, `mypy` sees `Interval` as `Unknown`, which allows a bunch of things to pass.

So this PR addresses that issue.  The way to see the problems that were fixed by this PR is to change in `_typing.pyi`:
```python
PandasScalar = Union["Period", "Timestamp", "Timedelta", "Interval"]
```
TO
```python
PandasScalar = Union["Period", "Timestamp", "Timedelta"]  # , "Interval"]
```

A followup PR will add `pandas/_libs/interval.pyi` and make changes to handle that.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Shameless plug :) PR based on the Microsoft-stubs: #44922
> Shameless plug :) PR based on the Microsoft-stubs: #44922

Ugh.  I wrote that stub in the MS stubs, which is why I tried to then convert it over to use in pandas.   Took me a long time to get this far!

I guess I should close this?  @twoertwein  ?


Sorry @Dr-Irv getting interval.py right is definitely a mess.

Let's keep this PR open in case my PR is not the preferred solution.
> Sorry @Dr-Irv getting interval.py right is definitely a mess.
> 
> Let's keep this PR open in case my PR is not the preferred solution.

If we go this direction, then I still have to create a PR for the `interval.pyi` part.  This "part 1" PR is just to fix up things that are a problem because `Interval` is undefined.
> This "part 1" PR is just to fix up things that are a problem because `Interval` is undefined.

IMO baby steps is definitely OK with typing.
> > Shameless plug :) PR based on the Microsoft-stubs: #44922
> 
> Ugh. I wrote that stub in the MS stubs, which is why I tried to then convert it over to use in pandas. Took me a long time to get this far!
> 
> I guess I should close this? @twoertwein ?

@jbrockmendel created nearly all the stubs for the cython code last year and also created a stub for libinterval #41059

> it uncovered issues in our existing typing, because currently, `mypy` sees `Interval` as `Unknown`, which allows a bunch of things to pass.

yep. from https://github.com/pandas-dev/pandas/pull/41059#issue-863084468

> This one is pretty ugly.

thanks @Dr-Irv
Backport PR #46065 on branch 1.4.x (BUG: groupby().rolling(freq) with monotonic dates within groups)
State: closed

Backport PR #46065: BUG: groupby().rolling(freq) with monotonic dates within groups

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Window
description: rolling, ewma, expanding

Comments:
Still open discussion on the fix. Closing
ENH: Updated read_html to add option
State: closed

- [x] closes #24766
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Stale

Comments:
@RomainL972 if you can merge master and get this passing will take a look
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge the main branch, address the review comments, and we can reopen.
BUG: .transform(...) with "first" and "last" fail when axis=1
State: closed

Part of #45986 (first/last)

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This removes warnings of dropping nuisance columns when using e.g. `.transform("mean")`. This makes it consistent with `.agg`, and in #46072 both `transform` and `agg` will warn about the default switching from `numeric_only=False` to `numeric_only=True`. Doing this first will make #46072 slightly easier.

Not falling back to `_transform_item_by_item` is also better for performance. For the benchmark, I disabled the warning being emitted in main to make sure that wasn't messing up the results.

```
size = 1_000_000
df = pd.DataFrame(
    {
        "A": size * ["foo", "bar"],
        "B": "one",
        "C": np.random.randn(2*size),
        "D": np.random.randn(2*size),
    }
)
%timeit df.groupby("A").transform("mean")

# This PR
108 ms ± 966 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
# main, but with size set to 100_000 (1mm was taking very very long)
920 ms ± 7.95 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Bug
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Clean
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
thanks @rhshadrach
Construction of series with na and datetime dtype
State: closed

- [ ] closes #41805 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Hello @weikhor! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-03-09 16:19:51 UTC
I close this since I stuck this for long time. I get simpler task. Thank for review.
DOC: added examples for index assignment using Series
State: closed

- [x] closes #39845
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
can you merge master
@marco-georgaklis do you mind using a mask when columns are being set in this docstring so we don't show code that we wouldn't use in practice? Would be good if you can do it for the whole dosctring, not only your changes. Thanks!
I'm not quite sure I understand. Should I be using a mask instead of .loc when the docstring is under the .loc property?
> I'm not quite sure I understand. Should I be using a mask instead of .loc when the docstring is under the .loc property?

Sorry I wasn't clear. The problem with your changes, is that in pandas we don't use:
```python
>>> df.loc[:, 'shield'] = pd.Series({7: 8, 8: 10, 9: 13})
```

Instead, we would use:
```python
>>> df['shield'] = pd.Series({7: 8, 8: 10, 9: 13})
```

The only case where we'd use `loc` for setting a column, is if there is a mask (a condition on which rows should be updated. Something like:
```python
>>> outstanding_employees = df['performance'] > 80
>>> df.loc[outstanding_employees, 'salary'] = ...
```

So, if you can have a look at the docstring you're updating, check if it makes sense to have an example with a mask, and update it, so the `.loc[:, 'shield']` idiom is not being used, so we don't show code in the docs we wouldn't use, that would be great.
Thanks for the pull request, but it appears to be stale. If interested in continuing, please merge in the main branch, address the review, and we can reopen.
BUG: BooleanArray match non-masked behavior div/pow/mod
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
yep ping with whatsnew
> yep ping with whatsnew

ping
DOC: Add details to DataFrame.combine_first docstring
State: closed

- [x] closes #41704
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @Moisan
CI: Test various Pyarrow version on Linux
State: closed

- [x] closes #45705 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature

As discovered in https://github.com/pandas-dev/pandas/pull/46014, pyarrow v2 seems to cause timeouts in read_csv (mamba resolves to v2 but conda resolves to v6 when pyarrow is unpinned, interestingly) so skipping that version.

Labels:
name: CI
description: Continuous Integration
name: Arrow
description: pyarrow functionality

Comments:
Locally, I am also getting the timeouts with v3, but have not investigated this
Thanks. Should hopefully get us to a more consistent green state.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 ea0184e339b7dbf27eec53e6c01c6514bda1fc83
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #46059: CI: Test various Pyarrow version on Linux'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-46059-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #46059 on branch 1.4.x (CI: Test various Pyarrow version on Linux)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Fix aligning a DataFrame with a Series with MultiIndex #46001
State: closed

When aligning a DataFrame to a Series we are using Series.reindex() to
broadcast the Series data to the new index.  That introduces NaNs when the new
index rows are not identical to the existing ones, which is not the case when
we introduce a new MultiIndex level.

In this patch we use the same approach as for aligning a Series to another
Series. That means that we have to replicate a part of Series._reindex_indexer
as DataFrame does not have it.

- [x] closes #46001 
  and #43321
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst`.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
thanks @johannes-mueller 
@johannes-mueller sorry would u mind doing a follow up address comments? thanks
DOC: Add "build C extensions" note
State: closed

- [x] closes #45914

This PR adds the missing information that the C extensions must be built when a local pandas repo was bind for the first time.

I also adjusted the docker commands to:
- show how you can use your forked repo (`--build-arg gh_username=yourname` was missing)
- specify the workdir `-w /home/pandas`, to directly start in the right directory

Labels:
name: Docs

Comments:
looks ok @rhshadrach merge when ready
Thanks @rendner!
BUG: read_csv not respecting converter in all cases for index col
State: closed

- [x] closes #40589 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
very nice @phofl
ENH: Add defaultdict support for dtype in read_csv
State: closed

- [x] closes #41574 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
cool can you rebase
this looked fine, can you merge master @phofl 
Done
thanks @phofl very nice as always!
BUG: error in read_excel with some ods files #45598
State: closed

- [x] closes #45598 
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
cc @phofl @rhshadrach 
Thx @dimitra-karadima
BUG: Multiindex.equals not commutative for ea dtype
State: closed

- [x] closes #46026 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: MultiIndex
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hey @phofl - did you mean to include the changes to multi.py? Looks the that is undoing #45931
Nope, this was an accident, thx
Should work now
FIX: clarify warning in read_sql_query()
State: closed

- [x] closes #xxxx (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

When using `pandas.read_sql_query()` with a non-sqlalchemy connection, the following warning is raised.

> UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy

For example, I noticed this warning today using `psycopg2` to connect to Amazon Redshift, like this:

```python
import os
import pandas
import psycopg2

query = "SELECT * from some_table LIMIT 10"

with psycopg2.connect(...) as conn:
    df = pd.read_sql_query(sql=query, con=conn)
```

This PR proposes small grammatical changes to the warning:

* adds space to prevent unintended concatenation (e.g. `"or database"` instead of `"ordatabase"`)
* adds `.` to separate individual statements

Thanks very much for your time and consideration!

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks @jameslamb
DOC: added note on str cons for read_sql
State: closed

- [x] closes #46023  
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but more information needs to be provide in the original issue before proceeding with this change.

Closing this for now, but feel free to address any other issue with `good first issue`!
PERF: MultiIndex slicing
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Similar to #45931, but for MultiIndex slicing.


```
import numpy as np
import pandas as pd

lev0 = pd.date_range("2000-01-01", "2020-12-31", freq="D")
lev1 = np.arange(10000)
mi = pd.MultiIndex.from_product([lev0, lev1])

df = pd.DataFrame({"A": 1.0}, index=mi)

%timeit df.loc["2010-12-31": "2015-12-31"]

132 ms ± 3.43 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)     <- main
571 µs ± 4.25 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)  <- PR
```


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
I just expanded the asvs for MultiIndex indexing as part of this. There are a lot of code paths for multi indexing and the existing asvs only covered a few of them. A few of the added asvs are currently a bit above the targeted 80ms, but I have a few follow on PRs that will hopefully put them all well below 80ms.  
very nice @lukemanley keem em coming!
ENH: Added support for the pd.read_sql to return a dictionary of all tables
State: closed

- [x] closes #45979 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but we require discussion for enhancement requests before moving forward with pull requests The referenced issue has not been triaged by the core team yet. (As a reference I would be -1 on this enhancement as our other `read_*` methods are not "multi-file/table" aware and trivial to perform in a loop).

Closing for now, but if you'd like to stimulate the discussion in the original issue or work on an enhancement that has been triaged and supported by the core team that would be much appreciated
Thanks for the update, I would look at other issues that have been triaged. 

How do I know when an issue has been triaged? Is there a specific label?

Lastly, what does -1 mean on enhancements? I noticed some other figures like -0.5 in some PRs too
-1 means almost no chance of the PR being accepted 
> How do I know when an issue has been triaged? Is there a specific label?

Any issue with `Needs ...` generally means it has not been fully vetted yet.
BUG: fix parquet roundtrip for Interval dtype with datetime64[ns] subtype
State: closed

Closes #45881

Labels:
name: Interval
description: Interval data type
name: IO Parquet
description: parquet, feather

Comments:
thanks @jorisvandenbossche 
Hi, is there a reason why this patch isn't included in pandas 1.4.x? I'm facing the same issue this PR was trying to address but I don't see the changes in the release. @jreback 

edit: I see, this is supposed to be included in the 1.5.0 release?
we don't backport except regressions
DOC: Minor addition to str.split
State: closed

- [x] Closes #44867  
- [ ] Added an entry in whatsnew


Labels:
name: Docs

Comments:
> Could you maybe add an example?

Nice idea.
I tried to add an integer and a boolean to the Series used for the already existing examples.

I'm thinking that it might be preferable to incorporate the method's behaviour in the already existing examples rather than adding new examples to show it off. Or am I wrong in this?
Thanks @ChrisAlbertsen