BrandonFors/pandas-github-issues huggingface dataset:
BUG: read_csv and read_fwf not skipping all defined rows when nrows is given
State: closed

- [x] closes #44021
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
Fix FloatingArray.equals on older numpy
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
this fixes the fails?
> this fixes the fails?

thats the idea, yep
BUG: to_datetime/to_timedelta with memoryview
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
no objections but. an u run the performance on is_list_like
```
In [3]: arr = np.arange(5)
In [4]: %timeit is_list_like(arr)
87.1 ns ± 1.15 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)  # <- PR
84.6 ns ± 0.407 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each))  # <- master
```
> pandas/tests/dtypes/test_inference.py:121: error: Argument 1 to "memoryview" has incompatible type "ndarray[Any, Any]"; expected "Union[bytes, bytearray, memoryview, array[Any], mmap]"  [arg-type]
pandas/tests/dtypes/test_inference.py:122: error: Argument 1 to "memoryview" has incompatible type "ndarray[Any, Any]"; expected "Union[bytes, bytearray, memoryview, array[Any], mmap]"  [arg-type]


also prob need a note on this
can you merge master, haven't seen the 3.10 failing before
WIP/TST: troubleshoot xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
pls rebase
will keep rebasing, but i don't expect this to be mergeable any time soon
one more rebase pls
> one more rebase pls

will keep rebasing, but pls dont merge; even once green i want to re-run a handful of times to try to catch flaky failures
got it can u change the title to WIP or make a draft
prob should rebase this
status here?
> status here?

The OK-looking changes i've been breaking off into the TODO/FIXME PRs.  Mostly hoping to keep running until the pytest-timeout catches something useful.
[DOC] Transferring data from the 6th column to the first.  #44379
State: closed

Transferring data from the 6th column to the first.
This should resolve issue #44379 


Labels:
name: Docs

Comments:
Thanks @Mikhaylov-yv - could you post a screenshot of the docs please?
@MarcoGorelli

Oh sure. I just don't understand why the links in the first column do not work for me:

-- :ref:`categorical`

![skrin](https://user-images.githubusercontent.com/44315225/141656623-97a79ee7-e158-45b1-8f62-7e8444e66eb7.png)

Thanks - it may be that you need to build the whole documentation, rather than just the page, in order for the link to work?
Yes, you are right.

![Снимок экрана от 2021-11-14 01-07-40](https://user-images.githubusercontent.com/44315225/141660493-dd7f4cd9-1c4e-4993-a673-803e2daa13e8.png)

Thanks @Mikhaylov-yv - is it possible to have it such that the text redirects to the link? E.g. clicking on "Intervals" redirects to "IntervalIndex".

If not, an alternative would be to write it as "Intervals - see IntervalIndex", it just looks a bit funny like this
Thanks @MarcoGorelli
Now I figured out how to do it. It turned out grea
![Снимок экрана от 2021-11-14 23-03-25](https://user-images.githubusercontent.com/44315225/141696823-c89dfb12-93da-4372-8693-f9e17fd26bf8.png)
t.
@MarcoGorelli  Yes fine. No problem.

I am just checking for other updates.
thanks @Mikhaylov-yv 

let's check the built docs to make sure this looks good
BUG: Using boolean keys to select a column (GH44322)
State: closed

- [x] closes #44322
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
thanks @DriesSchaumont
TST: collect/share Index tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
can you rebase
REF: EA quantile logic to EA._quantile
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> (the dual use case as base class for internal usage vs external usage is sometimes a bit conflicting ..)

i sympathize with this

> Is it really annoying to have the code in array_algos/quantile.py ?

A bit, yes.  The idea of array_algos is to not depend on EA implementation details (the docstring in `array_algos.__init__` hints at this).  More importantly, the special-treatment of pandas-internal (i.e. MaskedArray and a bit more subtly Sparse) is a pattern we've been trying to avoid.  I don't see any other way around this, am open to ideas.
> though anyway to combine some of the added code that is in base & masked?

there might be, but i haven't found it.  trouble is we call the constructors differently in the masked code
WARN: Add FutureWarning for `DataFrame.to_latex`
State: closed

Instead of #41648, which performs a refactor and adds the warning in the same PR, I am proposing adding the warning only for 1.4.0 and then in 2.0 there will be scope to make breaking changes and refactor the arguments signature.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Deprecate
description: Functionality to remove in pandas
name: IO LaTeX
description: to_latex

Comments:
looks fine can you add a whatsnew note in the deprecation section to message this. ping on green.
I didn't stumble upon this until today; this warning is emitted every time a user uses this method even though this method is not deprecated. The message itself says the arguments _may_ change, and doesn't give any indication how. I think the bar for disrupting users (especially those that have a CI that fails on warnings) needs to be set higher.
@attack68 do you have bandwidth/appetite to align DataFrame.to_latex and Styler.to_latex since 2.0 will be the next version? If not, I suppose we should remove this warning for 2.0
see https://github.com/pandas-dev/pandas/pull/47970#issuecomment-1320579957
BUG: read_csv raising if parse_dates is used with MultiIndex columns
State: closed

- [x] closes #8991
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
BUG: .get_indexer_non_unique() must return an array of ints (#44084)
State: closed

GH#44084 boils down to the following.

According to the docs `.get_indexer_non_unique()` is supposed to return
"integers from 0 to n - 1 indicating that the index at these positions matches
the corresponding target values".  However, for an index that is non unique and
non monotonic it returns a boolean mask.  That is because it uses `.get_loc()`
which for non unique, non monotonic indexes returns a boolean mask.

This patch catches that case and converts the boolean mask from `.get_loc()`
into the corresponding array of integers if the index is not unique and not
monotonic.

- [x] closes #44084
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex
name: Interval
description: Interval data type

Comments:
thanks @johannes-mueller 

do we have sufficient testing on this for *other* index types aside from those explicityly tested here? if not would take a PR for that
Backport PR #44356 on branch 1.3.x (Fixed regression in Series.duplicated for categorical dtype with bool categories)
State: closed

Backport PR #44356

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version
name: Categorical
description: Categorical Data Type

Comments:
Thanks @phofl
BUG: DataFrame.stack with EA columns
State: closed

- [x] closes #43561
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
nice
DEPR: PeriodIndex.astype(dt64)
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Match the Series/EA behavior

Labels:
name: Period
description: Period data type
name: Deprecate
description: Functionality to remove in pandas

Comments:
web and docs failing

```
2021-11-11T21:46:19.7568073Z WARNING: 
2021-11-11T21:46:19.7569269Z >>>-------------------------------------------------------------------------
2021-11-11T21:46:19.7570278Z Warning in /home/runner/work/pandas/pandas/doc/source/user_guide/timeseries.rst at block ending on line 2098
2021-11-11T21:46:19.7571558Z Specify :okwarning: as an option in the ipython:: block to suppress this message
2021-11-11T21:46:19.7573968Z ----------------------------------------------------------------------------
2021-11-11T21:46:19.7576034Z <ipython-input-382-a92cd46ab454>:1: FutureWarning: Converting PeriodIndex to DatetimeIndex with 'astype' is deprecated and will raise in a future version. Use `obj.to_timestamp(how).tz_localize(dtype.tz)` instead.
2021-11-11T21:46:19.7577328Z   pi.astype("datetime64[ns]")
2021-11-11T21:46:19.7578226Z <<<-------------------------------------------------------------------------
```
Yah, this was premature.  I didn't fix the docbuild bc I was having second thoughts.  Trouble is that we _do_ allow dt64.astype(period), and we should either have both or neither.  The part I have a strong opinion about is that IF we allow period.astype(dt64), it should be in PeriodArray.astype, not in PeriodIndex.astype
ok can revert for now if like
ENH: Support timespec argument in Timestamp.isoformat()
State: closed

- [x] closes #26131 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This is an update of PR #38550.  I added support for "nanoseconds" as an argument, expanded the test cases, and addressed most of the comments in the original PR.

Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
thanks @swt2c lgtm.

cc @mroeschke if any comments
Great, thanks @swt2c
[ArrayManager] Array version of putmask logic
State: closed

xref #39146

Trying to remove the usage of `apply_with_block` for putmask. 
(I also need to this for the Copy-on-Write branch to be able to perform the copy on write in putmask)

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Stale

Comments:
can the fallback logic stay in internals?  maybe internals.methods?  there are a few funcs in internals.blocks that might go there too.
> can the fallback logic stay in internals? maybe internals.methods?

Yes, either way is fine for me. Happy to move it into internals.
> (I also need to this for the Copy-on-Write branch to be able to perform the copy on write in putmask)

Can you elaborate on this a bit?  If we're going to implement CoW for BM anyway, some of the duplication might be avoidable.
>> (I also need to this for the Copy-on-Write branch to be able to perform the copy on write in putmask)
> Can you elaborate on this a bit?

See the 3 lines at https://github.com/pandas-dev/pandas/pull/41878/files#diff-b8c89656832340e9986eeee40df66eb4282555045a257e7a489d98b21b0adfbeR390-R393 that I currently added in that PR. Since putmask is used in indexing, I need to check for a copy-on-write (and this is not needed for `apply` in general).

> If we're going to implement CoW for BM anyway, some of the duplication might be avoidable.

I think that in general the CoW detection on BlockManager will be a separate implementation (but of course difficult to say before actually trying). For example the simplest way is to do this on a block-by-block basis (which would already give different code as the 3 lines I referenced). 

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I increasingly think that this type of change should be left until right before BlockManager is ripped out. Until then, this just duplicates code and makes it easy for behavior to accidentally drift apart.
can you merge main
closing as stale
fix documentation on options in read_csv
State: closed

Noticed this due to pylance (typing) showing header=None was wrong. Don't know how it will propagate to the typings.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @LunarLanding
BUG: handle NaNs in FloatingArray.equals
State: closed

- [x] closes #44382
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
do we have reference docs for these anywhere? (i don't remember), if so can you add otherwise lgtm
actually would be great to tests Series equals with FloatDtype (does it dispatch to this?)
> do we have reference docs for these anywhere? (i don't remember), if so can you add otherwise lgtm

Yes. The relevant one for users is https://pandas.pydata.org/docs/dev/reference/api/pandas.Series.equals.html#, and the ExtensionArray version is also listed: https://pandas.pydata.org/docs/dev/reference/api/pandas.api.extensions.ExtensionArray.equals.html

> actually would be great to tests Series equals with FloatDtype (does it dispatch to this?)

Yes, it does dispatch, and that's already tested.
where is the series test explicitly added here? 
It is not added here because we already have a general extension test (`test_equals`) that ensures both DataFrame and Series dispatch to the EA method.
sure but do we have an example that was failing with Float dtypes that check nans - we couldn't have as it would have failed -  so need this case
We have a lot of specific tests in the `/tests/arrays` that are not all repeated each time for Series or DataFrame. And I think that is fine.
i think this wasnt ready, had failures on older numpys
It seems the `equal_nan` keyword is new in numpy 1.19 (https://numpy.org/doc/stable/reference/generated/numpy.array_equal.html)
shimmed a fix for this into #44428
Can you quickly put it in its own PR? That seems easier to merge it quickly, while the other PR still needs to be reviewed?
CI: Use conda-forge to create Python 3.10 env
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Anaconda messed up their recipe :(. Fixes the sdist job.

Labels:
name: CI
description: Continuous Integration
name: Python 3.10

Comments:
LGTM

should we expect an upstream fix at some point?  do we revert this at that point?
thanks @lithomas1 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
> LGTM
> 
> should we expect an upstream fix at some point? do we revert this at that point?

Probably. No need to revert this one, though, since we really should be using conda-forge's stuff for everything.
TST/COMPAT: update csv test to infer time with pyarrow>=6.0
State: closed

Closes #44231

Labels:
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
Thanks @jorisvandenbossche. This has now starting failing on master and so I guess probably 1.3.x too. Will milestone 1.3.5 for backport.
> Thanks @jorisvandenbossche. This has now starting failing on master and so I guess probably 1.3.x too. Will milestone 1.3.5 for backport.

The new pyarrow engine should only be present on master, so I guess backport is not needed.

@jorisvandenbossche Do you know if pyarrow will add an option to configure this auto-inference?
Good point.

I am not aware of any discussion about an option to turn it off (looking at https://issues.apache.org/jira/browse/ARROW-11247 now)
The sdist 3.10 build was failing (unrelated I suppose), is that a know failure ? (I don't directly see an issue about it)
REF/TST: collect index tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
can you rebase
Rebased + green
TST: Make tests for groupby median/mean more strict on dtype
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Followup to #41139.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @rhshadrach
TYP:  changed variable cat_array to cat_array_list in dtypes.py
State: closed

xref #37715

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel any idea why I am failing these tests?  Thanks for your time.
Best guess is the CI started using a newer version of pyarrow and ran into #44231, so its unrelated to this PR and you can ignore those particular failures for now
can you merge master
@jreback Sorry I pulled from master then pushed. I guess that was the wrong thing to do. I should  have git merge.  Not sure how to undo this. So I just closed the pull request.
TST: test that Int64 series can be created from pyarrow array
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Closes #44368 

Labels:

Comments:
(same comment as on https://github.com/pandas-dev/pandas/pull/44369, you might also want to combine it in a single test)
Closing in favor of combined test in #44369
TST: check compatibility with pyarrow types_mapper parameter
State: closed

- [x] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

Closes #44368

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Possible this was actually just a misunderstanding on my part of the `__from_arrow__` method. Creating an instance of the dtype resolves my issue.

https://pandas.pydata.org/pandas-docs/stable/development/extending.html#compatibility-with-apache-arrow

I still think it'd be worth having these tests, so I can update my PR to do that.
can you merge master & @jorisvandenbossche comment (agree)
thanks @tswast
ENH: add suffixes argument to compare
State: closed

- [x] closes #44354 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the pull request, but this pull request appears to have gone stale. If interested in continuing, please merge in main and address the comments and this PR can be reopened. Closing in the meantime
CI: xfail tests failing on numpy dev
State: closed

- [x] ref #44343



Labels:
name: CI
description: Continuous Integration

Comments:
thanks @alimcmaster1 merge on green!
@meeseeksdev backport 1.3.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.3.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 d779366ce9157eb55eef9f162cdded0dca73b9df
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #44362: CI: xfail tests failing on numpy dev'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-44362-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #44362 on branch 1.3.x (CI: xfail tests failing on numpy dev)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
CLN: Removed _SAFE_NAMES_WARNING in io.sql
State: closed

- [x] closes #44295
- [x] tests passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Please let me know if I should improve/change something. Thank you very much!

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Clean

Comments:
thanks @EdAbati.