BrandonFors/pandas-github-issues huggingface dataset:
ASV: remove duplicated SelectDtypes benchmark
State: closed

https://github.com/pandas-dev/pandas/pull/36839 added an extensive benchmark class for `select_dtypes` (`dtypes.py::SelectDtypes`), but we actually already had a benchmark in `frame_methods.py`. Now, the new ones are more extensive, testing multiple dtypes and both include/exclude, instead of only including int, so I think the one in frame_methods can be safely removed to avoid duplication of benchmarks:

https://github.com/pandas-dev/pandas/blob/7b78b730aee122bb7d6e90723c4208da234de19e/asv_bench/benchmarks/dtypes.py#L51-L112

Labels:
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
thanks @jorisvandenbossche
PERF: passthrough mask to take_1d if already known
State: closed

In `take_1d` (if `allow_fill=True`), the mask gets calculated based one `indexer == -1`. But in the cases where this function is used, we either actually already calculated this mask in advance, or can calculate it once instead of multiple times again in `take_1d` (repeated calls with the same mask), which improves the performance in both cases.

Showing with two benchmarks where ArrayManager was quite a bit slower because of this:

`Reindex.time_reindex_upcast` benchmark:

```python
import numpy as np
import pandas as pd

N = 10 ** 3
df2 = pd.DataFrame(
    {
        c: {
            0: np.random.randint(0, 2, N).astype(np.bool_),
            1: np.random.randint(0, N, N).astype(np.int16),
            2: np.random.randint(0, N, N).astype(np.int32),
            3: np.random.randint(0, N, N).astype(np.int64),
        }[np.random.randint(0, 4)]
        for c in range(N)
    }
)
idx2 = np.random.permutation(range(1200))

df2_am = df2._as_manager("array").copy()


In [3]: %timeit df2_am.reindex(idx2)
16.7 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
9.73 ms ± 1.65 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR
```


`Unstack.time_without_last_row` benchmark:

```python
import numpy as np
import pandas as pd

dtype = 'int'
m = 100
n = 1000

levels = np.arange(m)
index = pd.MultiIndex.from_product([levels] * 2)
columns = np.arange(n)
values = np.arange(m * m * n).reshape(m * m, n)

df = pd.DataFrame(values, index, columns)
df2 = df.iloc[:-1]
df2_am = df2._as_manager("array").copy()

In [3]: %timeit df2_am.unstack()
594 ms ± 8.33 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- master
183 ms ± 21.2 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <-- PR
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
lgtm cc @jbrockmendel 

maybe more cases that could use this (for followon for sure)
nice perf bump.  any interest in applying it to BM while you're at it
>  any interest in applying it to BM while you're at it

I don't think the same optimization is directly applicable to BlockManager: 1) the one place where the Block uses `take_nd` directly in `reindex_indexer`, the mask is not precomputed (and it also wouldn't help to pre-compute, since computing it inside take_nd is as efficient, since this is done on the full 2D array), and 2) `unstack` doesn't use `take` (and in general wouldn't do repeated `take` calls with the same indexer, I think, since it is block-based).
TYP: changed variable cat_array to cat_array_list in dtypes.py and arr_list in categorical.py
State: closed

xref #37715

My apologies, I pushed hastily on my previous branch assign_err and did not check the diff of the files i was pushing. I accidentally pulled when i should have merged and I pushed a bunch of other peoples commits.  These are my previous changes.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
umm what are these in reference to?
@jreback #44373
thanks @nickleus27
TST: de-duplicate test_eval
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
green (several follow-ups in the pipeline to fixturize)
ENH: pd.Series.shift and .diff to accept a collection of numbers
State: closed

- [x] closes #44424 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale

Comments:
@phofl @jreback I noticed that I am failing a check in CI/Checks. It is due to Series.shift expecting a series as output, but the issue requested that in the case of an itterable being passed for the period parameter, that Series.shift returns a DataFrame. Is there some way to get around this?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
closing as stale if you want continue, pls ping
@jreback @skwirskj I am considering taking over this PR, would that be ok with you?
sure
Stop raising in read_csv when header row contains only empty cells
State: closed

- [x] closes #13054
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

If ok with the change in general, I would suggest deprecating and changing in 2.0

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
why deprecating and not implementing the unamed case?
Was not sure if people relied on this behavior. If you think is ok to immediately stop raising, will implement that
@jreback this immediately changes from raising to returning only Unnamed columns
thanks @phofl
BUG: fix .loc.__setitem__ not raising when using too many indexers
State: closed

- [x] closes #13831
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@phofl @jbrockmendel good here? (ex the question)
cc @phofl @jbrockmendel if any comments
@phofl @jbrockmendel good here?
@phofl @jbrockmendel ok here?
@DriesSchaumont can you merge master
Should I move this to 1.5?
@DriesSchaumont if you can merge master and address comments
@jreback @jbrockmendel merged master, please let me know which/if questions still need answering.
thanks @DriesSchaumont very nice!
REF: nanpercentile -> array_algos.quantile
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
> What's the rationale for this? Currently all nan* functions live in nanops.py, which is a nice pattern?

these two functions are used exclusively in this file, and don't have any intra-nanops dependencies (e.g. _get_values, _wrap_results).  There's nothing _wrong_ with them being in nanops, but array_algos.quantile is better choice, and remains tightly-scoped.
this is fine as it decouples the dependency structure a big. can you rebase
rebased + greenish
BUG: read_csv raising error for python engine when names longer than header but equal to data
State: closed

- [x] closes #38453
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
BUG: using .replace() on a Series containing np.nan raising ValueError when using regex=True.
State: closed

- [x] closes #43344
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
nice, thanks @DriesSchaumont
DOC: Clarify how to keep lines with too many fields
State: closed

- [x] closes #17319
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
very  nice @phofl keep em coming!
WARN: Clarify future warnings for read_csv
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

In other places we emit a specific warning message what to use in the future. I find this quite helpful for the user. 



Labels:
name: IO CSV
description: read_csv, to_csv
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
Thanks @phofl
TST: stop skipping 1986 tests in test_eval
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Get 8 xfails instead.  They may turn out to be easy, leaving that for another day.

This constitutes almost 40% of currently-skipped tests, at least with my setup.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> pandas/tests/computation/test_eval.py:834: error: Need type annotation for "exclude_bool" (hint: "exclude_bool: List[<type>] = ...")  [var-annotated]
pandas/tests/computation/test_eval.py:835: error: Need type annotation for "exclude_cmp" (hint: "exclude_cmp: List[<type>] = ...")  [var-annotated]
mypy issues fixed + green
CI/TST: Add network tests to CI
State: closed

- [x] closes #43559
- [x] tests added / passed

No network tests were running on the CI, so adding these tests to PY38 and PY39 Linux, MacOS and Window builds


Labels:
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
We might want to run these on the database builds as well and maybe a slow build. LGTM otherwise.
> We might want to run these on the database builds as well and maybe a slow build

Hmm wouldn't that just be essentially duplicating the same tests in multiple PY38 or PY39 environments (AFAICT these tests don't depend on optional dependencies since it used urllib to make the network call)?
> > We might want to run these on the database builds as well and maybe a slow build
> 
> Hmm wouldn't that just be essentially duplicating the same tests in multiple PY38 or PY39 environments (AFAICT these tests don't depend on optional dependencies since it used urllib to make the network call)?

yeah just a fast & slow build are fine
can you only enable this on a few builds? the linux ones are prob enough (e.g. 1 fast and 1 slow)
> can you only enable this on a few builds? the linux ones are prob enough (e.g. 1 fast and 1 slow)

These are enabled on:

Linux PY38
Linux PY39
MacOS PY38
Windows PY38
Windows PY39

which is good coverage IMO. Running on a slow build I don't think is worth it because it's equivalent to running on Linux PY38 twice.
BUG: read_csv not recognizing bad lines with names given
State: closed

- [x] closes #22144
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
wow killing the old issues, excellent!
Thx. Want to clean that up before starting the refactoring
BUG: mangle_dup_cols in read_csv replacing existing cols when conflict with target col
State: closed

- [x] closes #14704
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


This ensures that already existing column names are kept. The index of the duplicated col is one higher in this case

Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
@gfyoung if any comments
Thx
I think its worth making this a sub-section in the whatsnew docs to highlite it.
Added the sub section
@jreback greenish
thanks @phofl 
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=69777&view=logs&j=cb04ab33-336b-54ee-68fe-dcd3abde835b&t=0558d00d-ff4a-5538-0de9-a0e7f449700f

hmm this actually be related 
Are you sure? previous commit was green, only merged master and modified whatsnew afterwards.
not sure at all - if it's not a problem on master then ignore
Have seen similar things in the past like https://github.com/pandas-dev/pandas/runs/4331169003, hence assumed this would be unrelated. Lets see if it passes on master
BUG: read_csv not applying dtype to index col
State: closed

- [x] closes #9435
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

``_agg_index`` and ``_convert_to_ndarryays`` need a refactor in the future, they both contain more or less the same casting logic now


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
> _agg_index and _convert_to_ndarryays need a refactor in the future, they both contain more or less the same casting logic now

sgtm
Unfortunately, it does not work for cases where dtype is specified globally:

```py
>>> import pandas as pd
>>> from io import StringIO
>>> data = "1,a\n2,b"
>>> df = pd.read_csv(StringIO(data), index_col=0, dtype=str, header=None)
>>> df.index
Int64Index([1, 2], dtype='int64', name=0)
```

whereas this works

```py
>>> df = pd.read_csv(StringIO(data), index_col=0, dtype={0:str}, header=None)
>>> df.index
Index(['1', '2'], dtype='object', name=0)
```

However i need to set all columns str globally. So I expect the first chunk of code works and turns index into str.
Hm I think I prefer the current behavior. But you could open an issue to gather feedback from others.
One additional note: If we should decide to change that, we need a deprecation cycle.
WIP/CI: Debug segfault in ci/code_checks.sh docstrings
State: closed

xref https://github.com/pandas-dev/pandas/issues/44590

Labels:

Comments:
@phofl, @mroeschke: This may be nothing to do with it, but the last log output before the #44590 segfault was from ` errors/__init__.py`.

I ran (on Windows):
>  `python scripts/validate_docstrings.py pandas.errors.DtypeWarning`

repeatedly in multiple processes, and it quickly crashed with contention over test.csv.
> repeatedly in multiple processes, and it quickly crashed with contention over test.csv.

It might be good just to use `io.StringIO` here. I don't think it's great in general that the doctests are writing to a file.
Not yielding much as of late. Closing
DEP: Deprecate non boolean sort for concat
State: closed

- [x] closes #41518
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Deprecate
description: Functionality to remove in pandas

Comments:
``sort=sort`` in this test caused sort=None, fixed that
PERF/TYP: typing cast in __getitem__ gives quite some overhead
State: closed

While doing some unrelated profiling, I noticed typing overhead in `DatetimeArray.__getitem__`. Isolating that here, there is a typing `cast(..)` call in that method, and removing that as done in the PR, I see the following impact:

```python
arr = pd.date_range("2012", periods=3).array

In [4]: %timeit -r 20 arr[0]
4.76 µs ± 36 ns per loop (mean ± std. dev. of 20 runs, 100000 loops each)  # <-- master
3.44 µs ± 34.4 ns per loop (mean ± std. dev. of 20 runs, 100000 loops each)  # <-- PR
```

(I did this timing several times back and forth on a computer with no other activity at that moment, and the difference was stable)

Personally, I wouldn't have expected such a relative big impact of a typing cast. Because if that's the case, we should be very careful with using that in "hot paths". 

cc @jbrockmendel @simonjayhawkins @Dr-Irv 


Labels:
name: Performance
description: Memory or execution speed performance
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
wow.  ive been assuming that the casts are negligible.  if that wrong, then there are probably a lot of small improvements to be made by avoiding them across the codebase.

On the other hand, most of this is in the Union:

```
In [1]: from pandas._typing import *

In [2]: from pandas.core.arrays.datetimelike import *

In [3]: %timeit Union[DatetimeLikeArrayT, DTScalarOrNaT]
879 ns ± 19.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [4]:  %timeit cast(Union[DatetimeLikeArrayT, DTScalarOrNaT], NaT)
920 ns ± 8.15 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```

So at least in this case we can get most of the benefit by just doing the Union once at the module level
> So at least in this case we can get most of the benefit by just doing the Union once at the module level

Another (maybe easier) solution is to just put the type in double quotes:
```python
        result = cast(
            "Union[DatetimeLikeArrayT, DTScalarOrNaT]", super().__getitem__(key)
        )
```

I did a grep exercise, and that is what is done elsewhere in the code.  The only other place I could find (via grep) that we need to fix this is in `pandas/core/arrays/categorical.py`,  line 517

@jorisvandenbossche want to change the annotation to a string and call this a win?
@jorisvandenbossche can you also make a similar change in `pandas/core/arrays/categorical.py:astype()` so we are consistent in terms of doing casts with `Union`
Yeah, I had actually searched for other casts in the /arrays submodule, and this categorical one is the only other cast with a composite type (or that is not predefined). Since that's not typically a hot path, I didn't change it, but of course also can't harm, so pushed that change now anyway.
test added for dtype comparison
State: closed

- [43038 ] closes #xxxx
- [ 1] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Please finsih the other pr
BUG: read_fwf not handling skiprows correctly with iterator
State: closed

- [x] closes #10261
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry
This was relatively straightforward after the previous pr


Labels:
name: IO Fixed Width
description: read_fwf

Comments:
thanks @phofl
BUG: read_csv not replacing nans before date conversion
State: closed

- [x] closes #11019 
- [x] closes #26203
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Not a clue how this could influence mypy in read_xml
Ah it is the init file...
Fix issue #36271 - pd.read_json() fails for strings that look similar to fsspec_url
State: closed

- [ ] closes #36271
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@mntss can you address comments and merge master
thanks @mntss
DOC Make explicit in to_csv docs that compression kwarg is ignored if…
State: closed

I got confused because I tried `df.to_csv(compression='gzip')` and the result was not gzip compressed.
This PR makes it more clear in the documentation that this argument is silently ignored if there's no file object to save into.

Note that if you do `df.to_csv(io.StringIO(), compression='gzip')` that also ignores the `compression argument, but it prints a warning. If there is no file argument, there is no warning. Hence why this documentation is important.

Honestly I didn't bother doing most of the following because this is a two-line doc change.

- [ ] closes #xxxx 
- [ ] tests added / passed 
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
thanks @mdavis-xyz
make explicit in to_csv docs that compress kwarg is ignored if no arg
State: closed

I got confused because I tried `df.to_csv(compression='gzip')` and the result was not gzip compressed.
This PR makes it more clear in the documentation that this argument is silently ignored if there's no file object to save into.

Note that if you do `df.to_csv(io.StringIO(), compression='gzip')` that also ignores the `compression argument, but it prints a warning. If there is no file argument, there is no warning. Hence why this documentation is important.

Honestly I didn't bother doing most of the following because this is a two-line doc change.

- [ ] closes #xxxx 
- [ ] tests added / passed 
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Something strange is going on where my modifications to that branch aren't being reflected in this PR. I'm going to close this and create another.