BrandonFors/pandas-github-issues huggingface dataset:
REGR: object column repr not respecting float format
State: closed

- [x] closes #40024
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This is a regression caused by #38759

Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @mzeitlin11 
@meeseeksdev backport 1.2.x
REF: share ArrayManager/BlockManager methods
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
@jbrockmendel I don't think moving those to the base class really helps (also design-wise, the base class shouldn't know about blocks). All those case of "apply_with_block" in ArrayManager are actually cases that still need to be fixed to not use the "fallback" of blocks. So I would rather leave them there as is.
@jorisvandenbossche my thought here is that the apply_with_block is temporary and these will gradually be changed into just using `self.apply` (like astype and to_native_types), and that in the interim we get the normal benefits of code de-duplication.

If you feel strongly about it, im not inclined to press the point.
> my thought here is that the apply_with_block is temporary and these will gradually be changed into just using self.apply (like astype and to_native_types), 

But it's not a given that at that point it will be possible to share the method. For example astype and to_native_types are not actually shared (eg `self.apply("astype", ..)` vs `self.apply(astype_array_safe, ...)`). 

So I would rather decide about sharing or not *when* doing the change to remove `apply_with_block`.
PERF: put BlockManager constructor in cython
State: closed

No idea why mypy started complaining about the io.parsers stuff

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
can you merge master
any perf on this vs master (now that the ndarray stuff is merged)? (ideally put the relevant timeits in the cython code, that is a great idea generally)
> any perf on this vs master (now that the ndarray stuff is merged)? (ideally put the relevant timeits in the cython code, that is a great idea generally)

On the [benchmark](https://github.com/pandas-dev/pandas/pull/40171#issuecomment-790219422) I've been using for these im seeing about a 4% gain (though the worst performance had a 0.2% slowdown).  The big win I'm expecting comes in the next step, which puts BlockManager.get_slice in cython, which allows us to chain together several no-longer-python calls
rebased + green
Just measured the branch after this (that implements BlockManager.get_slice in cython) and that gets a 15% improvement.
> On the benchmark I've been using for these im seeing about a 4% gain

As I have mentioned before (eg on the mailing list, in https://github.com/pandas-dev/pandas/issues/40263), this benchmark is basically useless / unrealistic. 
I think you should benchmark with an actual applied function to check wether it is worth moving things to cython for this.
DOC: Fix wrong URL in web/README.md
State: closed


- [x] closes #40828 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
sure
thanks @hasan-yaman
TST: make flaky zip check less strict
State: closed

Saw this failure the other day:
```
pandas/tests/io/test_gcs.py:137: 
71
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
72

73
result = b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xe0\xa1\x87R3ET~\x18\x06\x00\x00\x8c\n\x00\x00\x03\x00\x00\x00zip-\x96]WT;\x0c\x8...x00\x00\x00\x00\x80\x01\x00\x00\x00\x00zipPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x001\x00\x00\x009\x06\x00\x00\x00\x00'
74
expected = b'PK\x03\x04\x14\x00\x00\x00\x08\x00\xdd\xa1\x87R3ET~\x18\x06\x00\x00\x8c\n\x00\x00\x03\x00\x00\x00zip-\x96]WT;\x0c\x8...x00\x00\x00\x00\x80\x01\x00\x00\x00\x00zipPK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x001\x00\x00\x009\x06\x00\x00\x00\x00'
```

The failing byte was the 10th, based on the linked wikipedia page in `assert_equal_zip_safe`, looks like 10th and 11th are last modified timestamp related, so may as well not check the 10th byte as well.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
thanks @mzeitlin11
Dark background
State: closed


- [*] tests added / passed
- [*] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [*] Color in boxplot.py should be a hex value. Letter based color will be at matplotlib level. Using same color as that of boxes for the caps as well. The color will be fetched based on the selected theme. We are over riding color with "k" , rather need to use the one set at matplotlib level. 

Before Fix : 
> Caps for the box plot is not visible : 
![image](https://user-images.githubusercontent.com/9417467/114264636-ab0bea00-9a09-11eb-943a-5e7cda090ed9.png)

After Fix : 
> Caps for the box plow will be visible in dark background mode : 
![image](https://user-images.githubusercontent.com/9417467/114264600-7009b680-9a09-11eb-870c-5633bd336d1e.png)

Whats New : Box Plot's Caps will have same color as boxes unless color is explicitly specified by user arguments. Hence theme changes will not adversly effect caps color. 



Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Hello @yeshsurya! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-06 07:17:23 UTC
> if your code changes the attribute `_caps_c` to `colors[0]` why is the test still testing the value that the color for `caps` is `k`?

yes , it shouldn't be. I was invoing df.boxplot() in test but I should be testing df.plot.box(). I've changed this. 
looks fine if you can add a whatsnew note as @charlesdong1991 suggests and ping on green (bug fix section of 1.3 in visualization)
> looks fine if you can add a whatsnew note as @charlesdong1991 suggests and ping on green (bug fix section of 1.3 in visualization)

Added whatsnew note 
> > looks fine if you can add a whatsnew note as @charlesdong1991 suggests and ping on green (bug fix section of 1.3 in visualization)
> 
> Added whatsnew note

did you push ? it doesn't show up
> > > looks fine if you can add a whatsnew note as @charlesdong1991 suggests and ping on green (bug fix section of 1.3 in visualization)
> > 
> > 
> > Added whatsnew note
> 
> did you push ? it doesn't show up

Oops ! Missed it. Pushed it now .
emm, could you please do it again, seems still some docs in whatsnew are from other PRs. @yeshsurya 
> emm, could you please do it again, seems still some docs in whatsnew are from other PRs. @yeshsurya

Done !
thanks @yeshsurya it looks like there is a pre-commit error about trailing space in your whatsnew, could you fix it and rebase master? ping if all green ^^
Another cleaner pull request raised for this : https://github.com/pandas-dev/pandas/pull/41349
emm, okay, @yeshsurya , then I will close this PR to make way for the other one
BUG: Incorrect index shape when using a user-defined function for aggregating a grouped series with object-typed index.
State: closed

- [x] closes #40014
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Index
description: Related to the Index class or subclasses

Comments:
> Thanks for the pr! Some quick comments, plus can you please add a whatsnew?

Done. Added whatsnew  targetting 1.2.4, but let me know if this is not the case.
I am opening this PR for review again. I have added another test, not sure on which one to keep. Open to any suggestions.
> 
> 
> This looks great! Can you just check on my example in the issue and report back? Do you understand why the "> 0" was causing it to fail? It's still not clear to me from the diff in this PR.

I addressed your question here: https://github.com/pandas-dev/pandas/issues/40014#issuecomment-819489220
lgtm. cc @jbrockmendel if any comments
@rhshadrach over to you
Thanks @DriesSchaumont, very nice!
CLN: tighten cython declarations
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@jbrockmendel just fyi in case it's what you're troubleshooting, have run into the annoying `ModuleNotFoundError: No module named 'pandas._libs.interval'` error before - think it's usually caused by a failure in the build step, not an actual import problem
> think it's usually caused by a failure in the build step, not an actual import problem

you're right.  unfortunately the message we re-raise in `__init__` obscures any actually-helpful information in the original exception message, hence removing it for the moment
TST: add test on checking objects to concatenate
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

When looking through the enhancement xref #40820,
I noticed that there was no test checking that the ValueError is raised when concatenating an empty sequence.
Added this test.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
lgtm
@jreback I moved the test just above ``test_concat_exclude_none``.
thanks @ivanovmg
DEPR: `Styler.where` possible deprecation
State: closed

closes #40124 

Proposal to deprecate the method: `Styler.where`, since `Styler.applymap` and `Styler.where` are so similar in function, and return signatures, that it is almost confusing to have 2 methods for the same thing.

Note that `Styler.where` offers no advantage over `applymap` since internally it just restructures the callable and uses `applymap` itself:

```
def where(..):
    return self.applymap(lambda val: value if cond(val, **kwargs) else other, subset=subset)
```

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Code Style
description: Code style, linting, code_checks

Comments:
i would actually deprecate .applymap, as its a non-vectorized version of .where (not that this matters for Styler generally).

ok can you separate out the bug fix and do the deprecation in a different PR? (where can discuss)
> i would actually deprecate .applymap, as its a non-vectorized version of .where (not that this matters for Styler generally).
> 
> ok can you separate out the bug fix and do the deprecation in a different PR? (where can discuss)

my other PR #40845 fixes the bug. This one I left as the potential deprecation. `Styler.where` is not vectorized, it's effectively a one-liner invoking `applymap`.
cc @pandas-dev/pandas-core any thoughts here
To use `where` in order to accomplish `applymap` seems unnatural:

    df.where(cond=lambda _: True, some_style)

To use `applymap` to accomplish `where` seems natural:

    df.applymap(lambda x: some_style if cond(x) else other_style)
> To use `where` in order to accomplish `applymap` seems unnatural:
> 
> ```
> df.where(cond=lambda _: True, some_style)
> ```
> 
> To use `applymap` to accomplish `where` seems natural:
> 
> ```
> df.applymap(lambda x: some_style if cond(x) else other_style)
> ```

I think I agree with this. As a user I always find myself having to stop and think what `where` actually does, on the other hand methods like `map`, `apply`, and `applymap` are more obvious.
ok thanks, yeah no harm in deprecating
REV: Revert #40510
State: closed

@jorisvandenbossche reports that #40510 was performance-hurting, so this rolls it back until we can sort out why that is.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Thanks @jbrockmendel !

I checked some of the benchmarks that were affected, and they improved again, see eg https://pandas.pydata.org/speed/pandas/#groupby.GroupByCythonAgg.time_frame_agg?python=3.8&Cython=0.29.21&jinja2=&p-method='mean' 
(this one might also be a good case to check what was actually the cause, if you're interested)
BUG-7023 allow error bars to use style
State: closed

- [x] closes #7023
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Visualization
description: plotting
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Closing as stale. @GLeurquin if you want to add tests and address the comment, just ping me, and more than happy to reopen.
Fix line_terminator escaping in python doc
State: closed

The line terminators are not escaped so we don't see the backslash here:

![image](https://user-images.githubusercontent.com/9317502/113784381-9ee80a00-9735-11eb-8e5e-6d4b84da7527.png)

Source : https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html

Labels:
name: Docs

Comments:
thanks @Drarig29
BUG: Support numpy.random.Generator as random_state input
State: closed

- [x] workaround for #38100
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
I don't think this is actually a good idea. I think it's better to change random_state to produce Generator directly, rather than RandomState. Then this will be fixed directly.
Also I don't understand why the MacOS tests failed (timeout)
Subprocess to validate flake8
State: closed

- [x] closes #40784 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MarcoGorelli for your comments!

I updated the PR. Now it passes 

`pytest scripts/tests/test_validate_docstrings.py 
`

Hi @MarcoGorelli 
I run the failed test locally, it passes without problem. (Actually, the changes made here, shouldn't effect the failed test case, i think)

Do you have any idea? How do we proceed from here?

thanks @hasan-yaman 
the failure is unrelated
Backport PR #40768 on branch 1.2.x (DOC/CI: temp pin of decorator (IPython dependency))
State: closed

Backport PR #40768: DOC/CI: temp pin of decorator (IPython dependency)

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
this doesn't need to be backported, as it's already reverted on master
see https://github.com/pandas-dev/pandas/pull/40801#issuecomment-814224625 although not sure if this will fix. currently recreating this env locally
That are different warnings than addressed in this PR. Comparing the two envs, another difference is the docutils version. They had a new release a few days ago, and since that broke a bunch of stuff in sphinx, I pinned the docutils version in the pydata-sphinx-theme. But the 1.2.x branch is still using an older version of the theme (not having this pin), so it might be https://github.com/pandas-dev/pandas/pull/40407 we need to backport.
Thanks @jorisvandenbossche . yes this doesn't fix. will try  #40407 instead.
STY: remove --keep-runtime-typing from pyupgrade Final_Part-5
State: closed

To fix issue #40759 :
Changes here are from PEP585 (e.g. List[str] -> list[str]) and from PEP604 (Optional[str] -> str | None), both of which are only applied in the presence of from future import annotations. Some imports became unused after this so those have been removed.
As there were changes in many files (97) this has been split up into 5 smaller PRs.
This is the final PR 5 of 5.
Checks in this one will pass only when other parts are merged as --keep-runtime-typing is removed the pyupgrade precommit hook.

Refer the discussion here : #40773

- [x] closes #40759 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@github-actions pre-commit

---

the above hasn't worked - @fathomer could you run pre-commit again please?
Thanks @fathomer !
> Thanks @fathomer !

Thanks @MarcoGorelli for all the help and clear details of the task.
Glad to make my first ever open source contribution here! :)
STY: remove --keep-runtime-typing from pyupgrade Part-3
State: closed

xref #40759 :
Changes here are from PEP585 (e.g. List[str] -> list[str]) and from PEP604 (Optional[str] -> str | None), both of which are only applied in the presence of from future import annotations. Some imports became unused after this so those have been removed.
As there were changes in many files (97) this has been split up into 5 smaller PRs. 
This is PR 3 of 5.

Refer the discussion here : #40773 

- 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
looks like this is breaking some CI checks, even though it was all green here - will take a look now

EDIT
----

looks like it's unrelated, something to do with conda-incubator
Backport PR #38574 on branch 1.2.x (TST: fix some mpl warnings)
State: closed

Backport PR #38574: TST: fix some mpl warnings

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
picking up decorator 5.0.5 on 1.2.x whereas on master 4.4.2. will merge this and backport #40768 for now
TYP fixup overload in reset_index
State: closed

If we have `t.py`:
```python
import pandas as pd

inplace: bool

reveal_type(pd.DataFrame([1,2,3]).reset_index(0, inplace=True, col_fill='a'))

```

then on `master`, the last line would be revelead as `Revealed type is 'Union[pandas.core.frame.DataFrame, None]'`. With the current changes, it will be revealed as `Revealed type is 'None'`

This function takes arguments `level, drop, inplace` - if `drop` isn't passed, then `inplace` needs to be passed by keyword, but `level` can be passed positionally

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks Simon - merging following your approval then (also, as this is really small, and serves as an example to others doing overload issues)
TST: groupby of idxmax/min with mixed types
State: closed

- [x] closes #40346
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry - didn't think this was necessary, just added a test case to a solved problem.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
thanks @bsun94
ENH: Upgrade sql module for SQLAlchemy 2.0 #40686
State: closed

- [ ] closes #40686



Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Dependencies
description: Required and optional dependencies

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
status here?
> status here?

This might require API change. I made a comment in https://github.com/pandas-dev/pandas/issues/40686#issuecomment-906880967. If sqlalchemy won't release 2.0 soon, I think we could wait until we release 1.4 at the end of this year.
@fangchenli status of this?
> @fangchenli status of this?

I'll close this for now.
BUG: groupby.agg/transform casts UDF results
State: closed

- [x] closes #21240
- [x] closes #17035
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This PR stops casting when the func is a callable since we can't tell when casting will be more harmful than helpful. I think this results in a more expected/consistent behavior for users. 

Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback -  change made and green. TODOs are added in regards to mean/median/var, opened #41137 and will be taken care of by #41139

> may also need to update groupby.rst, we might have this documented

I looked through, there is no mention of return dtype for agg/transform/apply.
right i think we should try to start documenting return types for the udfs in the main docs with some examples (and ideally add to the doc strings too)

@jreback - docs added and green.
@jreback - changes made and green. I added versionchanged to the Notes section as well as the Examples sections.
thanks @rhshadrach
BUG: RollingGroupby ignored as_index=False
State: closed

- [x] closes #39433
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
Does this fix #31007? Haven't looked closely but seen this in the past
@phofl unfortunately doesnt look like it

```
In [1]: import pandas as pd
   ...:
   ...: data = {
   ...:     'groupby_col': ['A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', ],
   ...:     'agg_col': [1, 1, 0, 1, 0, 0, 0, 0, 1, 0],
   ...: }
   ...: df = pd.DataFrame(data)
   ...: df.groupby(['groupby_col'], as_index=False).rolling(4).agg({'agg_col': 'mean'})
Out[1]:
               agg_col
groupby_col
A           0      NaN
            1      NaN
            2      NaN
            3     0.75
            4     0.50
B           5      NaN
            6      NaN
            7      NaN
            8     0.25
            9     0.25
```

`agg` still goes partially through groupby code I think to calculate the index result. 
Pitty, thanks for checking
thanks @mroeschke
REF: share methods between ExtensionBlock, NDArrayBackedExtensionBlock
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jreback adds test discussed in #40763