BrandonFors/pandas-github-issues huggingface dataset:
BUG: styler multiindex hiding indexes and columns alignment
State: closed

#### Follows #43686

- [x] closes #43637 
- [x] closes #43644
- [x] closes #43703
- [x] tests added / passed
- [x] whatsnew entry (note part of the bug seems to have been created by hiding `levels` which is a new feature in 1.4.0).

This PR has two parts:

1) Addressing the bug fixes as per above.

The key change made here was to add a single line: `if r not in self.hidden_rows:` in the loop, and in `_translate_latex` update for the new object structure that is input to it.

2) Refactoring the code so that the main part of the code is now easier to grok:

```
def _translate(...):
    # calls _translate_header()
    # callls _translate_body()

def _translate_header(...):
   # calls _generate_col_header_row()
   # calls _generate_index_names_row()

def _translate_body(...):
   # calls _generate_trimmed_row()
   # calls _generate_body_row()
```







And tests were update and comprehensive tests for the issues added.

Labels:
name: Bug
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
because of moving the hiding functions to outer loop:

```
before           after         ratio
     [cd13e3a4]       [75de033c]
     <master>         <bug_styler_multiindex_hiding>
-      11.2±0.1ms       9.44±0.3ms     0.85  io.style.Render.time_apply_format_hide_render(24, 12)
-     6.57±0.09ms      5.39±0.06ms     0.82  io.style.Render.time_apply_format_hide_render(12, 12)
-        19.2±7ms       12.2±0.2ms     0.64  io.style.Render.time_apply_format_hide_render(36, 12)
-      45.1±0.6ms       20.7±0.5ms     0.46  io.style.Render.time_apply_format_hide_render(24, 120)
-        26.6±3ms       11.9±0.5ms     0.45  io.style.Render.time_apply_format_hide_render(12, 120)
-      68.6±0.8ms       28.8±0.2ms     0.42  io.style.Render.time_apply_format_hide_render(36, 120)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

```
thanks!
BUG: REGR: read_csv with memory_map=True raises UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc4 in position 262143: unexpected end of data 
State: closed

- [x ] closes #43540 
- [x ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x ] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
I updated `whatsnew` docstring according to @jreback request
thanks @michal-gh very nice. always happy to take further updates to master.
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
Thank you @twoertwein , @jreback for help with this PR
CI: azure timeouts
State: closed

Based on the logging in #43611, in both timeout cases, the last test `gw0` ran was before the hypothesis test in `test_parse_dates` (this is a giant test - 56 parameterizations, hypothesis does 100 examples by default).

Not sure why this would be the cause, couldn't find any issues about it that might explain a deadlock or something like that, but maybe this will help? Regardless of fixing timeouts, it makes sense for these to be slow


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Maybe something similar to https://github.com/HypothesisWorks/hypothesis/issues/2340?
Hmm this didn't work - new guess - the logging seems to stop around parser tests - maybe the fact that we use the same parser object across tests can cause deadlock (or issue is some mutation is happening?)
We're at ~5 in a row on azure not timing out. Will keep running azure, but this is ready for review from my side. Summary of changes:
1. Mark hypothesis tests as slow
2. For parser fixtures, ensure we generate new objects instead of sharing them
3. skip pyarrow tests which can deadlock, xref #43650
(EDIT: just noticed more timeout cases on deadlock in #43611. Another option might be to just replace all `pyarrow` xfails with skips)
> Mark hypothesis tests as slow

is there some option that can set on hypthosesis instead? the point of these tests is to find holes in our tests, which are almost all fast, so this just remove this entirely (which maybe ok). but then we should just do that.
> is there some option that can set on hypthosesis instead? the point of these tests is to find holes in our tests, which are almost all fast, so this just remove this entirely (which maybe ok). but then we should just do that.

We could set fewer examples to run (but at some point that defeats the purpose of `hypothesis`). I thought on running on `slow` was a good compromise since it will still run on some builds, just fewer. 

Regardless, the important change here is skipping `pyarrow` for the `hypothesis` parse_dates test - will just remove the slow markers for now since it turned out `pyarrow` was the cause of the timeout, not the `hypothesis` tests sometimes running extremely slowly
kk lgtm. ping when ready to merge (or just go ahead)
note that I think we are seeing something similar on 1.3.4, but we didn't merge the pyarrow csv reader so prob something else.
> note that I think we are seeing something similar on 1.3.4, but we didn't merge the pyarrow csv reader so prob something else.

Good to know - certainly possible there are other potential timeout-causing issues unrelated to pyarrow. Will keep running azure pipelines on #43611 to see if anything else comes up. This should at least make timeouts less frequently hopefully
thanks @mzeitlin11 nice improvement here. Yeah let's keep an eye on 1.3.4
CLN: Python Parser code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
can you merge master
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
superseded.
backport 43638: BUG: styler to_latex doesnt respect hide_index
State: closed


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
> the backport (alternative fix) of #43638 should be done only after #43638 is merged to master?

yes, should really have done this as wip, just did both prs now to save time later/
closed by #43649 ?
indeed.
wip: BUG: styler hide index fails for latex translate
State: closed

- [x] closes #43637
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Is this a regression or bug fix? we are quite late in the release cycle of 1.3.x now. Probably only one release after the next one. so opportunities to resolve any issues are limited.

probably only regression fixes backported now going forward.
this is a bug fix. pretty simple one so I bothered to backport it. if too late, no worries, close the backport and i put it in 1.4.0
we don't have any hard and fast rules. see what others think.
just converted this to wip as #43644 was detected whilst solving this and a fix for that might be a universal fix
@attack68 replaced by #43649 ?
> @attack68 replaced by #43649 ?

yes that one has become more developed and is better than this.
WIP: DEPR: remove methods deprecated in 1.3.0 `Styler`
State: closed

removing methods from `Styler` for 1.4.0 that were deprecated as of 1.3.0.

Labels:
name: Stale

Comments:
these need to stay 
depreciating shouldn't be removed so soon
sure, can we just leave the pr in place on the back burner - wip maybe.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing this for now. we cannot remove these till 2.0
DOC: Update Matplotlib Colormaps documentation link in chart visualization #43624
State: closed

- [x] closes #43624 

Fixes: Fixed a chart visualization coiormap section link that linked to an outdated version of matplotlib. Now links to a [stable version](https://matplotlib.org/stable/gallery/color/colormap_reference.html#sphx-glr-gallery-color-colormap-reference-py).


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
Rather than hardcoding the url, why not use an intersphinx reference here?
> 
> 
> Rather than hardcoding the url, why not use an intersphinx reference here?

I don't know what intersphinx is, could you elaborate a little bit more?
https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html

you can reference the label they use for that section so it will still resolve properly if they reorganize their site in the future
I've done some research and in order to use intersphinx, both docs must use intersphinx as extension to their documentation and I don't know if either pandas or matplotlib use it.
https://github.com/pandas-dev/pandas/blob/master/doc/source/conf.py#L429-L439
Where can I find the label they use for that section? And wouldn't it be more useful to create a new issue to change all the documentation to use intersphinx, rather than a single link/element? 
> Where can I find the label they use for that section?

If you click "show page source" on the matplotlib docs (and most sphinx sites) you can see the raw rst. Right above the H1 element there's a reference target:

```
.. _sphx_glr_gallery_color_colormap_reference.py:
```

> And wouldn't it be more useful to create a new issue to change all the documentation to use intersphinx, rather than a single link/element?

(I am not a pandas maintainer and have no opinion on the best path forward, just dropping by with a suggestion).
I'd be happy merging this as is since a definite improvement, but will wait a bit to see if others weigh in.

Thanks for the idea @mwaskom - if you wouldn't mind opening an issue with the suggestion of using more intersphinx (and the related advantages), that would be awesome
> Where can I find the label they use for that section? And wouldn't it be more useful to create a new issue to change all the documentation to use intersphinx, rather than a single link/element?

agreed, I would just merge as is and open an issue for intersphinx, Perhaps it could be part of a beginners sprint or soemthing.
thanks @ricardobohadana 
Thank you!
REF: implement mask_setitem_value
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

cc @phofl I think this is what is needed to get #40380 working (https://github.com/pandas-dev/pandas/pull/40380#discussion_r680289744), could use your input when you have a chance.

This only does the masking for the non-tuple case.  Need to figure out if we can construct test cases that require the masking with tuples.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Closing to clear the queue; has merged into #40380.  @phofl id still appreciate any comments when you get a chance.
Backport PR #43467 on branch 1.3.x (Make group_mean compatible with NaT)
State: closed

Backport PR #43467: Make group_mean compatible with NaT

Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
arm failure (circle CI)

```
=========================== short test summary info ============================
FAILED pandas/tests/frame/methods/test_rename.py::TestRename::test_rename_inplace
= 1 failed, 158075 passed, 16037 skipped, 860 xfailed, 1 xpassed, 2 warnings in 1740.54s (0:29:00) =
```

This is a known flaky test.
DOC: add json_normalize in See Also for read_json
State: closed

I believe this should link https://pandas.pydata.org/pandas-docs/version/1.2.0/reference/api/pandas.json_normalize.html in https://pandas.pydata.org/docs/reference/api/pandas.io.json.read_json.html#pandas.io.json.read_json

`json_normalize` is great and it may help people working with JSON data who are likely to have an entry point of the `read_json` page

Labels:
name: Docs
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks @raybellwaves
BUG: Groupby-Apply with modifying dataframe and sorting overwrites group values 
State: closed

xref #36602 

- [x] closes #36602
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Labels:
name: Docs
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Great! Thanks @kasmith11
BUG: dropna changes index even when nothing dropped
State: closed

- [x] closes #41965
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Stale

Comments:
need to merge master and address comments
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale if you want to continue, pls ping to reopen.
CLN: let `codecs` validate the possible values of encoding and encoding errors
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

This makes the errors checking in `get_handle` simpler and consistent with the XML code.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Clean

Comments:
thanks @twoertwein
Backport PR #43608: REGR: spearman corr raising on 32-bit
State: closed

Backport PR #43608

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
cc @lithomas1 does the circleci pr need to be backported?
On it, thanks for the reminder. This was not backported initially because we wanted to see if it worked first. This might still fail if 1.3.x doesn't work well with ARM64 but we'll see.
@mzeitlin11 if you'd merge 1.3.x master and push again (to pick up the circle config)
thanks @mzeitlin11 
@mzeitlin11 when doing the manual backport , we need to make sure that the merge conflicts in the release notes are resolved in the same order as master otherwise future autobackports could fail.

I'll do a PR syncing the notes on 1.3.x with master shortly.

---

There is also a missing double quote in the release note. if this is fixed on master, the autobackport will fail until the notes are synced. I'll do that after since there is also an error in the release note for #43132 which can be done at the same time.


> I'll do a PR syncing the notes on 1.3.x with master shortly.

will merge #43630 first, but restated ci on that one to get to green.
> @mzeitlin11 when doing the manual backport , we need to make sure that the merge conflicts in the release notes are resolved in the same order as master otherwise future autobackports could fail.
> 
Sorry about messing that up, will keep in mind going forward.  Please let me know if there's anything I can do to help


> Sorry about messing that up

no problem. easy to do and easy to fix #43636
CI: Add back CircleCI for ARM64
State: closed

- [ ] closes #41829
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Successful run https://app.circleci.com/pipelines/github/lithomas1/pandas/3/workflows/ebe9e4aa-f86b-4bb2-9324-2b7a5eb9c243/jobs/3 (my fork)

I think someone with admin permissions in the github organization needs to turn on CircleCi on the pandas repo again. It probably would be better if this were merged first.

I also figured out how the credits work. Basically there's a setting for it in project settings. I think it should already be on.
![image](https://user-images.githubusercontent.com/47963215/133693306-9398585d-c131-40ad-9fd8-2d56682580f0.png)
(my fork settings)

Labels:
name: CI
description: Continuous Integration
name: ARM
description: aarch64 architecture

Comments:
wow @lithomas1 do I need to enable anything on the pandas repo for this to work?
Yeah. I think you need to go here https://app.circleci.com/projects/project-dashboard/github/pandas-dev/ and click the setup button for the pandas repo. It'll probably better if this is merged first so that CircleCI can detect the config file. 

Then, I think in Project Settings > Advanced we should probably Build Forked Pull Requests, and Auto Cancel Redundant Builds, but the only thing that really needs to be on is Free and Open Source(it should be on by default, but can't hurt to double-check).
ok done. Build Forked PRs was already checked as was auto cancel.
ok let's see how this goes
Nice. Just a heads up that a lot of PRs will be red unless they merge master, since Circle needs the config to run.
that's fine
@meeseeksdev backport 1.3.x
PERF: Improve performance in rolling.mean(engine="numba")
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

This also starts to add a shared aggregation function (mean) that can shared between rolling/groupby/DataFrame when using the numba engine.

```
df = pd.DataFrame(np.ones((10000, 1000)))
roll = df.rolling(10)
roll.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})
%timeit roll.mean(engine="numba", engine_kwargs={"nopython": True, "nogil": True, "parallel": True})

260 ms ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- PR
431 ms ± 9.21 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) <- master
```


Labels:
name: Performance
description: Memory or execution speed performance
name: numba
description: numba-accelerated operations

Comments:
how does this compare to the cython mean?
> how does this compare to the cython mean?

```
In [3]: %timeit roll.mean()  # cython
371 ms ± 15 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
is there an issue somewhere for discussing making numba required and just using this instead of the cython versions?
> is there an issue somewhere for discussing making numba required and just using this instead of the cython versions?

Not an ongoing issue, but the last time it was discussed was in https://github.com/pandas-dev/pandas/issues/28987. Looks like a lot of the discussion back then revolved around stability & maturity but those issues may not be as bad anymore.
> is there an issue somewhere for discussing making numba required and just using this instead of the cython versions?

we ought to have this discussion as that would greatly simplify code generally. this is a good start though.
thanks @mroeschke
CI: debug azure timeouts
State: closed

More attempts to see what's causing the timeouts.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
arm failure is ok if you merge master
> arm failure is ok if you merge master

Sounds good. Sorry, should've marked this as draft - want to let CI run a few times because I added `pytest-timeout` and want to see if we can catch whatever the test(s) which get stuck and cause timeouts or actions jobs that take hours
Thanks for taking the lead on this.  Looks like its one of the mac builds that timed out this last time.
> Thanks for taking the lead on this. Looks like its one of the mac builds that timed out this last time.

But no timeout was detected :(
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Logs from a timeout: https://dev.azure.com/pandas-dev/e01c9671-576d-40f3-a1a1-7bef78604087/_apis/build/builds/66368/logs/77

```
...
2021-09-18T04:51:29.0847391Z pandas/tests/io/parser/test_parse_dates.py::test_invalid_parse_delimited_date[pyarrow-32/32/2019] 
2021-09-18T04:51:29.0898854Z [gw0] XFAIL pandas/tests/io/parser/test_parse_dates.py::test_invalid_parse_delimited_date[pyarrow-32/32/2019] 
2021-09-18T05:22:53.6837334Z ##[error]The operation was canceled.
2021-09-18T05:22:53.6851486Z ##[section]Finishing: Test
```

Just goes quiet for 30 min, can't think of anything around the tests running there that would cause this. Wonder if timeout shows up same place every time
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
Hmm latest timeout looks to have `gw0` stuck in same place - right before hypothesis tests...
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
PERF: Block.apply
State: closed

```
from asv_bench.benchmarks.groupby import *
self = GroupByMethods()
self.setup('object', 'ffill', 'direct', 1)

%timeit self.time_dtype_as_group('object', 'ffill', 'direct', 1)
67.6 µs ± 708 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- PR
82 µs ± 2.24 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <- master
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
greenish ex circleci