BrandonFors/pandas-github-issues huggingface dataset:
TST: read_csv with custom date parser and na_filter=True results in ValueError
State: closed

previous PR #39079

- [ ] closes #36111
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
Hi @ftrihardjo - could you put a descriptive title please (the one from the issue is fine)?
@ftrihardjo Why is a new PR opened? It is preferable to continue in #39079 to keep the history more readily visible.
thanks @ftrihardjo
CI: update for numpy 1.20
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback gentle ping; this should get the CI back to green
thanks
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 435d1374ed1eb1f3bd68e39d259ab8190c281856
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39494: CI: update for numpy 1.20'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39494-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39494 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
ASV: add benchmarks for concatenating and appending of CategoricalIndex (GH38149)
State: closed


- [x] closes #38149
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
thanks @avinashpancham
DOC: Document how encoding errors are handled
State: closed

xref #39450

This should probably go in 1.2.2.

<1.2.0: `engine=c` and `engine=python` handled encoding errors differently when `encoding=None` (python: "replace", c: "strict"). The `engine=c` further supported to ignore lines (`skiprows`) with encoding errors iff `encoding=None`.

1.2.0 (shared file opening code between `engine=c` and `engine=python`): both engine mistakenly used "strict" when `encoding=None`. `engine=c`+`encoding=None` can no longer skip encoding errors.

1.2.1: both engine use "replace" when `encoding=None`. Both engines can ignore encoding errors in skipped rows iff `encoding=None`.

Option for 1.3: default to "strict" but expose `errors` in `read_csv` (and other read calls that might need it)?


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @twoertwein 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 2bcd31343a072a03913fea151a33857e420cd49b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39492: DOC: Document how encoding errors are handled'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39492-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39492 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 2bcd31343a072a03913fea151a33857e420cd49b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39492: DOC: Document how encoding errors are handled'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39492-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39492 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: Verify plot order of a Series (GH38865)
State: closed

- [x] closes #38865
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
Verified that the test passes on master, but fails on version 1.2.0 as expected
CI is greenish. Failure due to another issue not related to this PR.
thanks @avinashpancham
BUG: setting td64 value into numeric Series incorrectly casting to int
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

We fixed this a few days ago for some of the setting-methods, but there were a few we missed.  Re-using the TestSetitemEquivalents pattern is much more thorough.

We'll be able to de-duplicate the test class following #39477

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
> We'll be able to de-duplicate the test class following #39477

this is deduplicated?
> this is deduplicated?

haven't updated to de-dup yet, turns out to be a little non-trivial bc this makes some view-or-not assertions the other doesnt
prob should add a whatsnew for this change
CI: Pin pyarrow to 0.15.1 in 37 macos and linux
State: closed

Pyarrow 2.0.0 instead of 0.15.1 is fetched on these builds since today, which causes test failures

Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
BUG: Regression in astype not casting to bytes
State: closed

- [x] closes #39474
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Not sure if this is the right place

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Strings
description: String extension data type and string data

Comments:
@phofl if you'd merge master
merged master
Opened #39566
thanks @phofl 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 300d1fcafb02e822893d335bde890011a415ddf7
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39484: BUG: Regression in astype not casting to bytes'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39484-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39484 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: cumulative functions with ea dtype not handling NA correctly and casting to object
State: closed

- [x] closes #39479
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
More general solution is being worked on in https://github.com/pandas-dev/pandas/pull/28509 (which doesn't mean a short term solution can't be merged, I didn't closely look at either PR, just noting)
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> More general solution is being worked on in #28509 (which doesn't mean a short term solution can't be merged, I didn't closely look at either PR, just noting)

@phofl can you resolve conflicts
@phofl closing as stale. https://github.com/pandas-dev/pandas/pull/39483#discussion_r593823587 needs addressing. reopen when ready.
ERR: Unify error message for bad excel sheetnames
State: closed

- [x] closes #39250
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

IMO the core issue in #39250 is upstream in openpyxl, providing a better error message is really tangential to it. Since the error message that openpyxl raises is not a regression, it seemed best to me to (a) target this for 1.3 and (b) unify all error messages across engines.

If, on the other hand, we really do want to improve the error message in 1.2.2 because of the switch from xlrd to openpyxl, then I will open a new PR for 1.2.2 that only impacts openpyxl.

Will do a followup to remove the duplicate tests.

cc @simonjayhawkins @jorisvandenbossche 

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO Excel
description: read_excel, to_excel

Comments:
Thanks @jreback, note moved to 1.2.2. bugfixes
thanks @rhshadrach 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 9c33e5e4f4c5dd6941da866234e506dae7d9f1e0
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39482: ERR: Unify error message for bad excel sheetnames'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39482-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39482 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@rhshadrach can you follow the instructions and backport this
BUG/TST: Address GH28283 calling __finalize__ in pivot_table, groupby.median and groupby.mean
State: closed

Progress towards #28283 

1. refactor `pivot_table` function to avoid overhead of `_finalize__` calls when a list of aggfuncs passed
2. call `__finalize__` before returning `groupby.mean` and `groupby.median`
3. add tests for 1 and 2
4. add ignored test cases for `groupby.std/var/sem/size/ohlc/describe` as a reminder to fix(they are failing currently)
 
- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> add ignored test cases for groupby.std/var/sem/size/ohlc/describe as a reminder to fix(they are failing currently)

where is this?
@liaoaoyuan97 if you can fix up would be great. (and merge master)
@jreback I think all your comments are addressed. Would you ming taking a took?
@rhshadrach if you'd have a look
Thanks @liaoaoyuan97!
TYP, CLN annotate some variables, remove some possibly unused ones
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

The [mypy daemon](https://mypy.readthedocs.io/en/latest/mypy_daemon.html) throws some extra `var-annotated` errors compared with `mypy pandas`, so I've tried fixing some. Some of the variables it complains about seem to be unused (ExcelWriter.curr_sheet and NDFrame._ix) - are these safe to remove?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
looks fine. cc @simonjayhawkins 
can you merge master
thanks @MarcoGorelli
BUG: Fix sort_values bug that creates unprintable object
State: closed

- [x] closes #39426 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The problem occurs when sorting on columns and the `ignore_index` parameter is set to `True`. `sort_values` tries to refit the index after the columns have been sorted resulting in a corrupted dataframe.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @zitorelova! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-03 14:54:59 UTC
needs a test
precommit is failing
precommit is passing now
Seems to be a `ResourceWarning` causing one of the checks to fail. Is this related?
thanks @zitorelova
BUG: accepting ndarray[object] of dt64(nat) in TimedeltaIndex
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Timedelta
description: Timedelta data type

Comments:
cool, guess could use a whatsnew note
whatsnew added + green
BUG: assert_attr_equal with numpy nat or pd.NA
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This started off with editing na_value_for_dtype so that it could be re-used in #39446, ended up snowballing into assert_attr_equal.  If requested, I'll split off the bugfix into its own PR.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
looks fine. can you add a whatsnew note for the user.
REF: Move Resampler/Window.agg into core.apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Last move from aggregation.aggregate to apply. Will follow up with moving helper methods from aggregation into apply.

Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
cool
CLN: add typing to dtype arg in core/frame.py (GH38808)
State: closed

- [x] closes #38808
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @avinashpancham
CLN: clean error message
State: closed

Fix typo

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Stale

Comments:
Thanks for the PR! Looks like the message in the tests needs updated too.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@gkutiel Are you still interested in working on this? If so, reply here and we can reopen.
REF: simplify _get_empty_dtype_and_na
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
can you rebase as i think we merged some of the prior ones
> can you rebase as i think we merged some of the prior ones

git is telling me this is up to date (though github is showing me a weird diff)
> > can you rebase as i think we merged some of the prior ones
> 
> git is telling me this is up to date (though github is showing me a weird diff)

hmm interesting, i see pieces of several merged PRs AFAICT
ill force push to be on the safe side
rebased + green
i think merge master as other things have been merged which touch this
> i think merge master as other things have been merged which touch this

rebased 2 hours ago, following #39454, so we should be good here
BUG: raise when sort_index with ascending=None
State: closed

- [ ] closes #39434
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Added validation of ``ascending`` for ``sort_index`` methods.
Proposed a solution to raise ``ValueError`` when either of the elements of ``ascending`` turned out to be ``None``.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
can you merge master and fixup to comments.
ok looks fine. can you add a whtasnew note for 1.2.3 in regression section. merge master and ping on green.
thanks @ivanovmg 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 78371ccbd8744326e8535ac8bb9a661b073dfd88
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39451: BUG: raise when sort_index with ascending=None'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39451-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39451 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REF: dont pass fill_value to algos.take_nd from internals.concat
State: closed

The dtype-finding code in internals.concat is something else.  Tried untangling it, but now I think we can make it unnecessary altogether.  This is the first step in that process.

Labels:
name: Refactor
description: Internal refactoring of code
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks
TST: strictify xfails in sparse tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sparse
description: Sparse Data Type

Comments:
sure
REF: shallow_copy->simple_new/rename
State: closed


Labels:
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
> can you completely remove _shallow_copy?

Short answer: i think so.

Use cases that are just `self._shallow_copy()` could be replaced with `self.view()` (or implement a `self._view` fastpath) and `self._shallow_copy(name=foo)` that could use `self.rename` (or implement a `self._rename` fastpath).

There's one usage that can't be removed until #38140 (or we implement an equivalent shim in `Categorical._get_unique_index`)

That leaves 7 usages in core.indexes which i think can use simple_new directly (though MultiIndex may throw a wrench in that)
Want to move forward with the rest of the shallow_copy usages?
yep i think that would be good
BUG: DataFrame constructor reordering elements with ndarray from datetime dtype not datetime64[ns]
State: closed

- [x] closes #39422
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The ravel("K") was a problem when passing in a view on the underlying object, which was done here.

Labels:
name: DataFrame
description: DataFrame data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @phofl 
This was a regression, so I think worth backporting to 1.2.2. If we do that, we only need a small PR moving the whatsnew note to v1.2.2.rst (apart from the actual backport)
sure we could backport

@phofl if u wouldn't mind putting up a backport PR (need to edit the note)

and then separate PR to move the note in master
Done #39475 and #39476
Thanks @phofl !