BrandonFors/pandas-github-issues huggingface dataset:
Backport PR #42323 on branch 1.3.x (BUG: `Styler.to_latex` now doesn't manipulate the `Styler` object.)
State: closed

Backport PR #42323: BUG: `Styler.to_latex` now doesn't manipulate the `Styler` object.

Labels:
name: Bug
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@attack68 this backport is failing. depends on another PR not backported?
> @attack68 this backport is failing. depends on another PR not backported?

just looked, and I strongly believe that since it uses `obj = self._copy(deepcopy=True)` it will depend on (#42187), which was a cleanup assigned to 1.4.0, although no reason it can't also be backported?
thanks @attack68 

yep this is fine.
WEB: Added theme dependent favicon 
State: closed

- [x] closes #42314
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Added a light mode version of the favicon based upon the .svgs in the static folder as well as added a small js scipt to the template page to detect the theme change.

![mgCmqcreiy](https://user-images.githubusercontent.com/53790060/123965934-fe1c8a00-d9c5-11eb-8838-ce72ee988a3c.gif)


Labels:
name: Web
description: pandas website

Comments:
cc @datapythonista if any comment
@datapythonista I probably have done it this way in the first place, but I've made the changes you've requested, now updates the favicon without requiring refresh through use of a event listener. And now just updates the href of the current favicon. Definitely a lot simpler. 
![363GRl3Xq6](https://user-images.githubusercontent.com/53790060/124248105-91270280-db33-11eb-804c-2aa41866ed93.gif)

Are there any more changes that need to be made? Or can it be safely merged without conflicts?
Sorry for the delay @DeviousLab. We usually require two core devs to review a PR before merging, let's wait a bit more and see if someone else can have a look. Thanks for your patience.
> Sorry for the delay @DeviousLab. We usually require two core devs to review a PR before merging, let's wait a bit more and see if someone else can have a look. Thanks for your patience.

No worries! just thought I might have been missing something
@jreback Does this look good to you?
thanks @DeviousLab
Backport PR #42318: PERF/REGR: symmetric_difference revert most of #41833
State: closed

Backport PR #42318

Labels:
name: Performance
description: Memory or execution speed performance
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
xref https://github.com/pandas-dev/pandas/pull/42318#issuecomment-871255858
```
       before           after         ratio
     [f34376d1]       [ea963bf8]
     <1.3.x>          <backport-of-pr-42318-on-1.3.x>
-      83.0±0.6ms       48.0±0.2ms     0.58  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'symmetric_difference')
-      1.11±0.01s         643±10ms     0.58  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'symmetric_difference')
-      83.5±0.8ms       47.8±0.6ms     0.57  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'symmetric_difference')
-       102±0.9ms       58.1±0.4ms     0.57  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'symmetric_difference')
-       103±0.4ms       58.2±0.8ms     0.57  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'symmetric_difference')
-      1.14±0.01s          643±3ms     0.56  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'symmetric_difference')
-      7.14±0.1ms      3.97±0.08ms     0.56  index_object.SetOperations.time_operation('datetime', 'symmetric_difference')
-      5.18±0.1ms      2.85±0.04ms     0.55  index_object.SetOperations.time_operation('int', 'symmetric_difference')
-      30.4±0.6ms       16.0±0.4ms     0.53  index_object.SetOperations.time_operation('date_string', 'symmetric_difference')
-      28.7±0.4ms       14.7±0.2ms     0.51  index_object.SetOperations.time_operation('strings', 'symmetric_difference')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
removed MultiIndex symmetric_difference also
> removed MultiIndex symmetric_difference also

Huh, i thought i had done that in the other one.  LGTM.
BUG: `Styler.to_latex` now doesn't manipulate the `Styler` object.
State: closed

- [x] closes #42320 
- [x] tests added / passed
- [x] whatsnew entry not needed if included in 1.3.0 (although will be needed for 1.3.1)


Labels:
name: Bug
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
> [ ] whatsnew entry (probably for 1.3.1)

you can put in 1.3.0 for review purposes and move later. (after #42312)

although there seems to be conda-incubator/setup-miniconda@v2 issues today which could slow down the ability to release
> > [ ] whatsnew entry (probably for 1.3.1)
> 
> you can put in 1.3.0 for review purposes and move later. (after #42312)
> 
> although there seems to be conda-incubator/setup-miniconda@v2 issues today which could slow down the ability to release

if thats the case then not technically necessary for 1.3.0 (it fixes a feature that was added for 1.3.0)
@jreback ok for 1.3 (we have issues with #42304 that's blocking release)
@meeseeksdev backport 1.3.x
thanks @attack68 
Something went wrong ... Please have  a look at my logs.
@attack68 

shouldn't the following render the output as Latex in a notebook?

```
df = pd.DataFrame([[1, 2.2, "dogs"], [3, 4.4, "cats"], [2, 6.6, "cows"]],
                  index=["ix1", "ix2", "ix3"],
                  columns=["Integers", "Floats", "Strings"])
s = df.style.highlight_max(
    props='cellcolor:[HTML]{FFFF00}; color:{red};'
          'textit:--rwrap; textbf:--rwrap;'
)

from IPython.display import display, Latex
display(Latex(s.to_latex()))
```
@jreback I don't use LaTeX much in a notebook but I believe Jupyter uses MathJax to render and Mathjax does not support all LaTeX environements including tabular, so this just displays as text in a box.
Revert "Revert "REF: move shift logic from BlockManager to DataFrame""
State: closed

Reverts pandas-dev/pandas#42317 which was a revert to backport for 1.3.0 release, this puts those changes back on master.

does not close perf issue as will still exist on master

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
@jbrockmendel do you want to merge this or see if this can be combined with a fix for #42246
> see if this can be combined with a fix for #42246

Yah let's keep this open with the milestone so we don't forget it, try to get a perf fix before merging.
closing to clear queue
BUG: not dropping scalar-indexes MultiIndex levels
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

There are a bunch of issues about inconsistent dropping of MultiIndex levels: #12827, #18631, #10552, #42102, #41416, none of which are closed by this.

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
hmm this doesn't appear to be user facing?
> hmm this doesn't appear to be user facing?

in test_loc_getitem_drops_levels_for_one_row_dataframe this causes us to drop levels that are retained in master
Revert "REF: move shift logic from BlockManager to DataFrame"
State: closed

Reverts pandas-dev/pandas#40536

Discussed in #42246, we can revert for 1.3 without breaking anything, then un-revert in 1.4 and have some time to sort out the perf trouble.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
```
(pandas-dev) simon@T3630:~/pandas/asv_bench (revert-40536-ref-shift)$ asv compare v1.3.0.dev0 e496d4aa22df656adfab14ec216e66198732e1c5 |grep frame_methods.Shift.time_shift
       6.05±0.1ms      5.51±0.03ms     0.91  frame_methods.Shift.time_shift(0)
-        540±30μs         478±10μs     0.88  frame_methods.Shift.time_shift(1)
```
Thanks @jbrockmendel
REF: remove drop_level kwarg from MultiIndex._get_loc_level
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry



Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
yeah this is a pretty nasty piece of code .....
> yeah this is a pretty nasty piece of code .....

yah, lots of small steps to improve it coming up, after which there are some API design decisions we'll need to pin down e.g. #39775
Auto backport of pr 42293 on 1.3.x
State: closed

backport of pr #42293

Labels:
name: Performance
description: Memory or execution speed performance
name: Regression
description: Functionality that used to work in a prior pandas version
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
@jbrockmendel this is failing
Thanks @jbrockmendel
ENH: `Styler.bar` extended to allow centering about the mean, value or callable
State: closed

This refactors `Styler.bar` to allow more flexible arguments to align bars, and an additional keyword `props` akin to other builtin methods that allow some customisation, demonstrated in the new user guide (image below)

- [x] closes #26070
- [x] tests added / passed
- [x] whats new

**All previous tests pass with minor formatting alterations.**

As a follow-on I should improve the original testing for this function - it is sporadic and unstructured.

![Screen Shot 2021-06-29 at 14 53 56](https://user-images.githubusercontent.com/24256554/123800804-f2f42c00-d8e9-11eb-970a-31468d7a29fa.png)


Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
> looks fine. anything need updating in the doc-string?

have also done the changes to the doc string and added a `versionadded` or `versionchanged` tag for 1.4.0
thanks @attack68
PERF/REGR: IntervalIndex.intersection, PeriodIndex.get_loc
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Should fix #42240, #42247, not checking the box because I jumped the gun last time.

Labels:
name: Performance
description: Memory or execution speed performance
name: Period
description: Period data type
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
looks good can u run the asvs on these to see the change
backporting in any event
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 17ecb56c219d8d01399faffcffa8f059ecf6fd45
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42293: PERF/REGR: IntervalIndex.intersection, PeriodIndex.get_loc'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42293-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42293 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.3.x
@jbrockmendel needs a manual backport
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 17ecb56c219d8d01399faffcffa8f059ecf6fd45
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #42293: PERF/REGR: IntervalIndex.intersection, PeriodIndex.get_loc'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-42293-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #42293 on branch 1.3.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REGR: ExcelWriter fails when passed kwargs
State: closed

- [x] closes #42286
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Seeing some open resource test failure in excel writer, but not sure why. Checking if they occur on CI too.

Before this PR, I didn't notice that only xlsxwriter actually utilizes the kwargs/engine_kwargs. This makes the non-xlsxwriter tests slightly awkward - we pass the kwargs and just make sure the call to ExcelWriter doesn't fail because I don't see anything else to check.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Excel
description: read_excel, to_excel

Comments:
The open resource failures were from `test_kwargs_deprecated`, which is an invalid test. The try-except in that test catching the TypeError, which was supposed to be raised due to the invalid kwarg, was instead covering up this regression. The fix for the regression in this PR then made the test in question fail when creating the workbook for xlsxwriter, leading to the open resource error.

The fix for this is to break up the test for each engine, supplying valid kwargs (particularly, for xlsxwriter). Since this was already done in this PR, I've simply removed the invalid test.
thanks @rhshadrach very nice. is there an issue to remove this for 1.4?
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
> is there an issue to remove this for 1.4?

For 2.0 - yes, it's part of #30228
DOC: Only include numpy.std note for std, not all ddof methods
State: closed

I noticed that this note was mistakenly added to all methods that use `num_ddof_doc`, e.g. including [`sem`](https://pandas.pydata.org/docs/reference/api/pandas.Series.sem.html), which is misleading - this statement is only true for `std`. This PR makes sure we only add the note in that case.


Labels:
name: Docs

Comments:
Sorry for the delay and thanks for the PR. Could you post some screen shots of the doc changes after these changes, especially a method with something after `Notes` since that newline was removed?
# var (no Note)
![image](https://user-images.githubusercontent.com/675055/125475383-905f826f-f4af-4de1-af9d-2a2ed7086e75.png)

# std (with Note)
![image](https://user-images.githubusercontent.com/675055/125475430-55d511cb-e229-47ca-8b95-2052501a0f75.png)

Thanks @TheNeuralBit
BUG: get_indexer_non_unique with np.nan
State: closed

- [x] closes #35392
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

One of the tests is copied from #35498

Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
I think in order to be consistent, one could use `_lib.hashtable.PyObjectHashTable` (instead of Python's `set`  and `dict`) - it has the "expected" behavior when it comes to nan-floats and can also handle more complex cases (like `(float("nan),)` or `complex(0, float("nan"))`) out-of-the-box.

The downside is, that one has to misuse a table for a set (#39799 would fix that).
@realead IIUC what you're suggesting is viable medium-term but likely not short-term?  i.e. is it worth doing a temporary fix like this PR without what you're describing?
I must confess, I assumed the solution would be within reach. 

But there are some stumbling blocks:

Panda's `PyObjectHashTable` doesn't perform reference counting

https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/hashtable_class_helper.pxi.in#L1126-L1138

 so using it as set/dict with temporary objects is problematic. 

Also `PyObjectHashTable` maps objects to int64 and not `pyobject`, which is at least a problem for performance. Maybe it is worth to introduce `PandasSet` and `PandasDict` which would have the same handling of nans as pandas algorithm. But until then a fix like that is probably best we can do.
> But until then a fix like that is probably best we can do.

Makes sense, thanks for taking a look.

> Maybe it is worth to introduce PandasSet and PandasDict which would have the same handling of nans as pandas algorithm

I'll defer to you on this, as you've mentioned using other khash functionality before.  I'm a bit wary about the increasing build size.
closing in favor of #35498
DEPR: treating dt64 as UTC in Timestamp constructor
State: closed

- [ ] xref #24559
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Once enforced this will match the DatetimeIndex/DatetimeArray/Series behavior

Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
pretty sure we had a long discussion about this, can you link the issue (PR looks fine)
to_timedelta docs warning on default behaviour
State: closed

- [x] closes #34633
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Have added clarification to the user guide on the default behaviour of `to_timedelta` when a string is passed.

Labels:
name: Docs

Comments:
Hi @mroeschke - I've made this small documentation contribution which appears to work fine but I'm seeing timeout errors in the CI jobs. Do you have an opinion on how to fix these errors or whether this is an issue?
No need to worry about the CI failures @TAJD. It looks unrelated to your PR. 
Thanks @TAJD
TST: move custom business month tests to own file (#27085)
State: closed

- [ ] xref #27085
- [x] tests added / passed (no new tests added; all tests pass)
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Move custom month business cases from the `test_month.py` file into their own `test_custom_business_month.py` file per the current pattern in the `tseries.offsets` tests that split offset business and/or custom subclasses into their own files with ample cases (see `test_custom_business_hour.py`)

There is much refactoring work here to do that could be linked to this issue (#27085) going forward, but the simplest pieces that probably belong under this issue are simple organization of existing tests into smaller, logical files that make it easier for contributors to find a tests' home.

I'm a new contributor to the codebase so place let me know if I misunderstood the scope or what you all were thinking with this issue. Happy to adjust.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets

Comments:
can you merge master
@github-actions pre-commit
@datapythonista Thank you for spending the time to approve this work. I'm really hoping to get more involved in open source, but I have a lot to learn.

I fixed the piece of the process I overlooked (isort) and re-merged master. Afterward, I had some CI checks fail that seemed unrelated to the changes that I made (errors while running pytest on `pandas/tests/io/fast_parquet.py`).

I was able to replicate these errors locally and merging master again just now did not resolve them. How would you like for me to proceed?
I'm in my phone now, can't check easily myself, but in general the best after what you did is to check the last build of master, and see if it's failing. If it's not, it's likely that the problem is in your branch. If master is failing, you can just wait for someone else to fix it, or if you prefer, you can also have a look, and see if you can fix it in a separate PR.
Thanks for working on this @felixDulys
Backport PR #42268: PERF: IntervalIndex.intersection
State: closed

Backport PR #42268

Labels:
name: Performance
description: Memory or execution speed performance
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
Thanks @jbrockmendel for the backport.
TST: Added test for upcasting to object when concatinating on categorical indexes with non-identical categories
State: closed

This PR simple adds a test for upcasting to object when concatinating on categorical indexes with non-identical categories.

- [x] closes #17629
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
One of the azure pipelines failed, because it timed out after 60 min. Is this something that just happens sometimes or does it indicate there is something wrong with the PR?
thanks @P-Tillmann very nice!
DOC GH29075 close database connections after creation
State: closed

- [x] closes #29075 
- [ ] whatsnew entry

I've expanded on the use of a context manager to handle the closing of the database connection after the connection has been opened. I've also included a warning at the end of the section highlighting problems with leaving the database connection open. I think these additions tackle the docs problem associated with this issue.

Whilst psycopg2 is supported through sqlalchemy it appears to not be explicitly supported in its own right. I linked to the section in the sqlalchemy docs which discusses what happens when you use a Python context manager to handle connecting to a database. Hopefully users will now have a better idea on how or why database connections are being made in the recommended manner and can access the relevant documentation quickly.

Labels:
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
thanks @TAJD 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.