BrandonFors/pandas-github-issues huggingface dataset:
Series append raises TypeError
State: closed

- [ ] closes #31413
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Series.append now raises TypeError when a DataFrame or a sequence containing DataFrame is passed.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Hello @hvardhan20! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-03 06:14:24 UTC
thanks @hvardhan20
REF: remove fast_apply
State: closed

xref #32083, one more coming that i think finishes off libreduction

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Clean

Comments:
Any perf differences here?
Progress on https://github.com/pandas-dev/pandas/issues/31845 hits this code path you are removing. Happy to remove - would just like to see this PR move forward sooner rather than later. 
> Any perf differences here?

Ran full asv run overnight and am shocked to see no differences.  Do we just not have asvs that hit this?
it’s possible we only have test that hit this - maybe have to construct one?
Tentatively looks like I ran the asvs on the wrong pair of commits, re-running now.
OK, a run with just `-b groupby` gives:

```
       before           after         ratio
     <master>         <no-fast-apply>
+     5.73±0.05ms          277±4ms    48.30  groupby.Apply.time_scalar_function_single_col
+      19.0±0.1ms          690±5ms    36.37  groupby.Apply.time_scalar_function_multi_col
-      13.2±0.1ms       11.7±0.4ms     0.89  groupby.Nth.time_series_nth('float64')
-         754±2μs          667±3μs     0.88  groupby.GroupByMethods.time_dtype_as_group('object', 'unique', 'direct')
-         755±3μs          666±1μs     0.88  groupby.GroupByMethods.time_dtype_as_group('object', 'unique', 'transformation')
-         813±5ms         702±20ms     0.86  groupby.Apply.time_copy_overhead_single_col
-      2.05±0.06s       1.75±0.02s     0.85  groupby.Apply.time_copy_function_multi_col
```

so that's pretty rough
Ouch. Guessing ripping it out is not justifiable given the performance hit.
> Ouch. Guessing ripping it out is not justifiable given the performance hit.

Yah.  On the plus side, profiling it has turned up some optimization opportunities, locally ive got the worst one down from 48x to 10x.
closing to clear the queue, will revisit after 1.0.2
POC/REF: remove SeriesBinGrouper, SeriesGrouper
State: closed

Maintenance of libreduction is a PITA, so I want to see how big a performance hit we take if we rip it out.

I get a test failure locally because apparently something in resample behaves differently if we go through the python path, which itself should be considered a bug.  cc @mroeschke any ideas how to fix the failing test/

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
asv run with `-b groupby`:

```
       before           after         ratio
     <master>         <no-_aggregate_series_fast>
+         135±3ms       1.75±0.03s    12.94  groupby.MultiColumn.time_lambda_sum
+        71.6±1ms         881±10ms    12.31  groupby.MultiColumn.time_col_select_lambda_sum
+      14.0±0.3ms       15.7±0.3ms     1.12  groupby.Nth.time_frame_nth('float64')

```
Closing to clear the queue, will revisit.  Still hoping to @mroeschke can take a look at the failing test case, since that suggests an extant bug hidden somewhere.
REGR: Series.str.encode("base64")
State: closed

- [ ] closes #32048
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Strings
description: String extension data type and string data

Comments:
> not sure if u saw @TomAugspurger comments
> 
> you can’t encode bytes only decode them

see https://github.com/pandas-dev/pandas/issues/32048#issuecomment-587613824
> > not sure if u saw @TomAugspurger comments
> > you can’t encode bytes only decode them
> 
> see [#32048 (comment)](https://github.com/pandas-dev/pandas/issues/32048#issuecomment-587613824)

maybe i don’t understand you comment
you can’t encode this in the standard library
why should we allow it
we don't use the standard library. we use codecs so maybe we need a deprecation warning.
> we don't use the standard library. we use codecs so maybe we need a deprecation warning.

no the point is we shouldn’t do this at all

if you have bytes you can’t encode 
i am not sure why that would be an option that is exposed
IIUC it's a regression. and if we don't want to allow it we should have a deprecation cycle.
@jreback is it worth fixing up the tests and adding a special case for `base64` encoding or is this a non-starter.
this was done on purpose. i see no reason to change anything
REF: de-nest Series.__setitem__
State: closed

Splitting this into a couple of steps for readability.  The two main things I have in mind here are a) de-nesting, b) try to parallel the structure of `__getitem__`

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
ping
lgtm. can you rebase
rebased+green
Thanks @jbrockmendel
CI: Remove docs build from pipelines
State: closed

- [X] closes #29472

At the moment we're building the web and the docs from both Actions and Pipelines. While is still under discussion where things should be hosted, it's probably worth to just build on Actions for now, and publish from there to both the OVH cloud, and to GitHub pages.

**This needs the GitHub pages ssh key to be added to the secrets of this repo with name "github_pages_ssh_key".*** Can someone with permissions please add it? Or if we already got it with a different name, please let me know the name.


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
> Can someone with permissions please add it?

cc @TomAugspurger 
I'm not sure what the value of the key is. Perhaps @jorisvandenbossche has it?
Sent them via email.
Thanks @datapythonista. Added to the pandas-dev/pandas github repo's secrets with the name `github_pages_ssh_key`.
Cool, this should be ready then. As a follow up, I'll synchronize the website and the dev docs in the rackspace server in the GitHub Actions build.
I just noticed https://github.com/ammaraskar/sphinx-action (a github action to build sphinx docs and puts errors inline), might be interesting (but didn't really look at it)
All green after rebasing, merging.
CLN: Remove unused script find_commits_touching_func.py
State: closed

- [X] xref #31039

I've been trying the script `find_commits_touching_func.py` and doesn't seem to be working. Tried for many existing functions, and never get any result. I guess it doesn't work with newer git versions, or with the latest dependencies. And if nobody realized of this before, I assume nobody is using this script (and in any case, a separate project would be more appropriate for it, since I don't think it's pandas specific).

The script was being refactored in #32044, that's why I was having a look.


Labels:
name: Clean

Comments:
LGTM
I vaguely recall some contributor using this recently (@3vts or @MomIsBestFriend - either of you?)

No objections to getting rid of it, though cc'ing them in case they have a counter point
I did not work on this file. Just tested and seems it does not work as stated earlier
@WillAyd I cannot recall when I used this script.

I think I was the one who were one on this as part of the ```f-strings``` formatting change.

I have no objections in removing this script as I never used it, and for the fact that I haven't seen anyone else use it.
OK thanks all for comments and @datapythonista for PR
TST: corrwith and tshift in groupby/groupby.transform
State: closed

This PR tests `corrwith` and `tshift`, so not sure if it fully closes the original issue or not, but the other functions`backfill`, `pad` and `cumcount` don't seem to be supported in `groupby.transform` as reported in issues  #27472 and #31269.

- [x] closes #27905 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@ryankarlos is there more to add. if not I don't think this closes the issue?
@simonjayhawkins  yes Im thinking of  adding for `tshift` and `corrwith` - anymore ?
> @simonjayhawkins yes Im thinking of adding for `tshift` and `corrwith` 

those are explicitly mentioned in the issue, so could add here or seperate PRs and change the OP of this PR to xref instead of closes.

> - anymore ?

not sure. @charlesdong1991 ?


So there is already a fixture called `transformation_func` which has all of these; I think should be using that in tests where there is a gap today instead of adding this
Thanlks @WillAyd , Ive now added new tests with the `transformation_func` fixture.
@jreback  @WillAyd  thanks for the comments - addressed them now. 

> @simonjayhawkins yes Im thinking of adding for `tshift` and `corrwith` - anymore ?

looking at 

https://github.com/pandas-dev/pandas/blob/53ece7009c02eefc0befccf911d72369b0ed37f8/pandas/tests/groupby/test_transform.py#L330-L332

it appears that pad and cumcount need to be tested also.
> > @simonjayhawkins yes Im thinking of adding for `tshift` and `corrwith` - anymore ?
> 
> looking at
> 
> https://github.com/pandas-dev/pandas/blob/53ece7009c02eefc0befccf911d72369b0ed37f8/pandas/tests/groupby/test_transform.py#L330-L332
> 
> it appears that pad and cumcount need to be tested also.

So these other functions were throwing `AttributeError` when used with transform which are also mentioned in  issues #31269 and #27472  
@simonjayhawkins  to be honest, think this original issue may be out of date and superseded by #32293 ,  #31269 , #27472 and #32344 reporting the associated bugs which need to be fixed first for `cumcount`, `pad`, `backfill`, `tshift`, `corrwith`, as seems rest of transform funcs are already tested.

If you want I could just leave the tests for `groupby` and put them in `test_groupby.py` - otherwise feel free to close this PR if not relevant anymore ?
> otherwise feel free to close this PR if not relevant anymore ?

Thanks @ryankarlos for the work here, much appreciated, but I'll close this PR and close the issue too.
DOC: Add example for multiindex series and dataframe merge
State: closed

- [x] closes #12550



Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@simonjayhawkins  all green now 
Pushed some changes and simplified the example  - seems CI failure is unrelated
Rebased, all green now
@WillAyd  All green 
Thanks @ryankarlos
@datapythonista can we see these merged changes or is #32279 preventing the publishing of the dev docs 
> @datapythonista can we see these merged changes or is #32279 preventing the publishing of the dev docs

Development (master) website and docs should be available at pandas.io (dev.pandas.io should have the same, but it's the one being affected by that problem).
cool. should the link to the contributing guide in README.md point to https://pandas.io/docs/development/contributing.html instead of https://dev.pandas.io/docs/contributing.html
@ryankarlos looking at the rendered output at  https://pandas.io/docs/user_guide/merging.html, the result is not displayed. see https://github.com/pandas-dev/pandas/pull/32068#discussion_r383482432
@datapythonista from the pandas.io link, is there a link to the latest released docs (i.e. 1.0.1). the documentation link under Latest version: 1.0.1 links to 1.1.0.dev0+641.g7d9c20073 docs whereas the docs links to previous versions link to the correct docs.
@simonjayhawkins I sent an email to pandas-dev some days ago to try decide on the domains. Originally everybody looked happy with just using directories under pandas.io. But looks like finally the preference is to drop pandas.io (including dev.pandas.io), and just use pandas.pydata.org (the dev docs should be a directory there). The first PR to move into that direction is giving some problems with the ssh key, and I don't have permissions to this repo settings and not yet to the server. So things will move slowly until we clean up the mess with the domains and the structure. 
Thanks @datapythonista So it maybe worth updating the README.md, even as a temporary measure, as I think it is important this link should point to the latest docs. I'll open a PR for this.
DOC: Use container directives instead of raw html for the doc "cards"
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The PR is a follow up of #31156 and #31148 translating the raw html to docutils/sphinx directives, making it more _readable_. 

Furthermore, it fixes the redundant css in `pandas.css` versus `getting_started.css` by removing the `install-card` class and merging it with the existing `intro-card` class. 

__Note:__ The setup required an [adjustment to the theme](https://github.com/pandas-dev/pydata-bootstrap-sphinx-theme/pull/92) to overcome the usage of the `container` class by both docutils and bootstrap. 

Labels:
name: Docs

Comments:
@stijnvanhoey can you merge master to get the ci green.
I think looks good but @jorisvandenbossche 

Is this directive provided by sphinx or someone else?
Yes, it's even from base `docutils` / restructuredtext. 

But, this still depends on a change in the sphinx theme to support the way it is styled (a conflict in a css class needs to be solved); so this needs to wait on that (https://github.com/pandas-dev/pydata-bootstrap-sphinx-theme/pull/92)
Looks like both this PR and the one is depending on have conflicts. Are we still moving forward with this?
Yes, I think we still want to forward with this. The linked PR mainly conflicts due to the rename, will try to move forward that PR.
Hey guys, just an FYI, that I took your good idea here and turned it into an extension: https://github.com/executablebooks/sphinx-panels

(in this extension I also provide a "fix" for pandas-dev/pydata-sphinx-theme#92)
First try to convert some of the cards to use sphinx-panels -> https://github.com/pandas-dev/pandas/pull/34065 (still needs to be further updated, though). So let's close this PR in favor of that one.
CLN: GH29547 replace old string formatting
State: closed

Hi there! I've added some missed f-strings to:
* tests/tslibs
* tests/tseries

- [x] tests passed
- [x] passes `black pandas`

Thank you.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
```
 Extension error:
Could not import extension contributors (exception: No module named 'gitdb.utils.compat')
##[error]Process completed with exit code 2.
```

We did not encounter this error during the setup, https://github.com/pandas-dev/pandas/pull/31156/checks?check_run_id=411825801. I do not have a specific idea to which this refers. @jorisvandenbossche, do you have an idea?
See https://github.com/pandas-dev/pandas/issues/32060
@pcandoalmeida can you merge master and re-push? Then CI should be passing.
Hi @TomAugspurger I've merged master and pushed now. Thanks!
Thanks @pcandoalmeida
Backport PR #31734 on branch 1.0.x (BUG: list-like to_replace on Categorical.replace is ignored or crash)
State: closed

Backport PR #31734: BUG: list-like to_replace on Categorical.replace is ignored or crash

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
can these backports be merged, or do we need to get to green here first?
Definitely want to be green for back ports - have you seen that error message yet in another PR? Couldn't find other backports to address but maybe overlooked
The numpy dev failures were handled in another PR -> https://github.com/pandas-dev/pandas/pull/32057
REGR: read_pickle fallback to encoding=latin_1 upon a UnicodeDecodeError
State: closed

When a reading a pickle with MultiIndex columns generated in py27
`pickle_compat.load()` with `enconding=None` would throw an UnicodeDecodeError
when reading a pickle created in py27. Now, `read_pickle` catches that exception and
fallback to use `latin-1` explicitly.

- [x] closes #31988
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Pickle
description: read_pickle, to_pickle

Comments:
LGTM
Thanks @pedroreys !
use numexpr for Series comparisons
State: closed

subsumes #31984.

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Does this give a better performance?
> Does this give a better performance?

After running some timeings, I found a surprising and unacceptable 2x slowdown in homogeneous dataframe ops, so I reverted the edit in dispatch_to_series (which I thought, apparently incorrectly, was not doing anything useful).

These timings are after that reversion (I'll push shortly).  We pick up some good ground for Series ops, its a wash for homogeneous DataFrame ops, and we lose a small (and confusing) amount of ground for heterogeneous DataFrame ops.

```
In [5]: ser = pd.Series(range(10**7))                                                                                                                           
In [6]: %timeit ser == ser                                                                                                                                      
6.86 ms ± 295 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   # <-- master
4.68 ms ± 129 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   # <-- PR

In [7]: df = pd.DataFrame({n: range(10**6) for n in range(10)})                                                                                                 
In [8]: %timeit df == df                                                                                                                                        
6.53 ms ± 61.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
6.47 ms ± 124 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR

In [8]: df[1] = df[1].astype(float)                                                                                                                             
In [9]: %timeit df == df                                                                                                                                        
12 ms ± 33.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
12.8 ms ± 338 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR
```

so the reason we orignally wrote eval was to provide a quite significant speedup using *multiple* chained ops / comparisions, e.g.

https://pandas.pydata.org/pandas-docs/stable/user_guide/enhancingperf.html#expression-evaluation-via-eval

(this is slightly orthogonal to this issue)

however,

https://pandas.pydata.org/pandas-docs/stable/getting_started/basics.html#accelerated-operations are very relevant. a) we should have asv's for these, and be sure we are actually using the accelerated options (and likley update this link)
yah, I think it was @TomAugspurger who recently discussed being more careful about what we pass to numexpr.  I think in particular the tests can be improved to be not just "was numexpr called" but "was it called without falling back to python"

This is a blocker for the branch that implements frame-with-frame ops blockwise, so id like to minimize side-tracking.
thanks.
Use fixtures in pandas/tests/base
State: closed

part of #23877
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

**Note:** The diff is a bit inflated. Most changes are just indentation because for loops are replaced by a parametrized fixture.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I think I addressed all your comments here @WillAyd 
@SaturnFromTitan if its easy for you, can you point out sections of the diff that are effectively just-de-indentation, should make it easier to focus review
I addressed all your comments @jbrockmendel 
Also, I added comments for all real changes in the tests to aid the review
@jreback CI is green and I addressed all your comments
thanks. ideally very targeted PRs are best as we can merge quickly. 
That's the plan @jreback I wanted to focus on fixturizing the tests before refactoring.

Seeing how this PR went, it apparently makes more sense to refactor right away and go test by test instead ✌️
CLN: 29547 replace .format() with f-strings
State: closed

This PR replaces .format() with f-strings as requested in https://github.com/pandas-dev/pandas/issues/29547.

It did this replacement in scripts/find_commits_touching_func.py

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
I was checking, and seems like this script is not working, and I don't think anyone is using it. I'm removing it in #32071, so better to close this PR. Thanks for the work anyway, and sorry for not detecting that earlier.
DOC: Mention black and PEP8 in pandas style guide
State: closed

- [ ] closes #31828
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks a lot for your help @datapythonista. Seems that currently it fails tests because of issue #32060. Should work after it resolved.
I think we've got a temporary fix, can you merge master into your branch? I think CI should pass after that.
Something went wrong, the diff is huge now, can you have a look please?
Yes, sorry, I did something wrong. Will try to fix it now.
If all else fail, run the commands

```
git fetch upstream
git rebase upstream/master
git push origin issue_31838 --force
```

(__USE ONLY AS LAST RESORT__)

---

Can anyone show me a better way, I am sure that there is one, I just don't know how to do it.
This is the only thing that actually works for me when this kind of thing happened.
In the first section of this post is explained what I do: https://datapythonista.me/blog/useful-git-commands.html
Thank you for your help! Very much appreciated.
> In the first section of this post is explained what I do: https://datapythonista.me/blog/useful-git-commands.html

Bookmarked, thank you @datapythonista 
Thanks @RaisaDZ