BrandonFors/pandas-github-issues huggingface dataset:
CI/TST: change skip to xfail #35660
State: closed

Followup of #35660 

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @VirosaLi 
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 3c87b019b63ebff3c56acb6402fba5bc6f99671f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #35672: CI/TST: change skip to xfail #35660'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-35672-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #35672 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
ENH: Enable short_caption in to_latex
State: closed

- [x] closes #36267
- [x] tests added / passed
- [x] passes black pandas
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry


Enable short_caption for ``DataFrame.to_latex`` by expanding the meaning of kwarg ``caption``.
Optionally ``caption`` can be a ``Tuple[str, str] = full_caption, short_caption``.

The final caption macro would look like this:
```
\caption[short_caption]{full_caption}
```


Labels:
name: Enhancement
name: API Design
name: Needs Discussion
description: Requires discussion from core team before further action
name: IO LaTeX
description: to_latex

Comments:
For some reason CI / Checks (pull_request) failed with the typing validation.

```
Run source activate pandas-dev
mypy --version
mypy 0.730
Performing static analysis using mypy
pandas/core/generic.py:3032: error: Argument 1 to "DataFrameFormatter" has incompatible type "NDFrame"; expected "DataFrame"  [arg-type]
Found 1 error in 1 file (checked 1030 source files)
Performing static analysis using mypy DONE
##[error]Process completed with exit code 1.
```

But I never touched DataFrameFormatter myself. Can you help?
Please put this PR on hold before the PR related to full refactoring of ``latex.py`` is merged https://github.com/pandas-dev/pandas/pull/35872 (unless abandoned).
It will be easy to implement the changes of the present PR if https://github.com/pandas-dev/pandas/pull/35872 is merged first.
I guess, I need to add release notes into Enhancements section, as @simonjayhawkins highlighted. Would it be v1.2.0?
Correct - 1.2 whatsnew
@jreback, @WillAyd, can you please check the changes I have made after your review? Thank you!
I resolved conflicts with the master branch. It seems that there is an issue with the new style docstring (https://github.com/pandas-dev/pandas/pull/33633). ``Returns`` section gets over indented (as a result of running ``python scripts/validate_docstrings.py pandas.DataFrame.to_latex``).
@jreback, @WillAyd, @simonjayhawkins, @toobaz, ping. Is it good to go?
@jreback all checks are good except Windows py38_np18, which failed to create anaconda environment.
All checks are good except Windows:

```
##[error]We stopped hearing from agent Azure Pipelines 8. 
```
thanks @ivanovmg
Backport PR #35633 on branch 1.1.x (BUG: DataFrame.apply with func altering row in-place)
State: closed

Backport PR #35633: BUG: DataFrame.apply with func altering row in-place

Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
doctest failure occurs on other PRs e.g. https://github.com/pandas-dev/pandas/pull/28509#issuecomment-671720219 so merging
BUG: Styler cell_ids fails on multiple renders
State: closed

- [x] closes #35663 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@jreback this is a very minor alteration to my previously merged commit. Are you able to review? Should go to 1.1.1, where a bug fixes line already exists in the 'whats changed' file to address this `cell_ids` arg.
changes complete @jreback .. ping
thanks @attack68 
@meeseeksdev backport 1.1.x


@meeseeksdev backport 1.1.x



@meeseeksdev backport 1.1.x
Moto server
State: closed

Changes moto for s3 tests from monkeypatched/mocking to server mode. This allows aiobotocore exceptions to raise correctly, needed for tests to pass; the change is required by the upcoming async release of s3fs, but also works for old sync (botocore) version. A release of s3fs without this change would break pandas tests.

Note: since I now need to pass storage_options to the various s3 calls in the tests to make them pass (giving the endpoint of the moto s3 server), I needed to also plumb storage_options through the excel IO, which was previously missing, and update the excel tests.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
Good to go.
Does something like this need a whatsnew, when it only affects the tests and won't be seen by users?
I could add whatsnew about the excel integration, although it's pretty niche.

Sorry about the long commit list, it contains the original list of commits from #35381 (i.e., wasn't squashed here, only in master).
@jreback : went green again
I didn't understand "comment on the impl" - but I did move the imports and add a sentence to whatsnew for excel.
lgtm. @TomAugspurger if any comments.
I see one environment where there's moto but not flask, which results in skips like https://travis-ci.org/github/pandas-dev/pandas/jobs/717300240#L4633.

- ci/deps/azure-37-locale.yaml

Also, at least some of the tests are being skipped for other builds.

- https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=40289&view=logs&j=4b4d380c-92b5-58cf-5b80-2c6c9b1d5805&t=e685736b-3705-5348-f648-7de631559230&l=2417 (moto too old)
- azure-38-locale: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=40289&view=logs&j=b1c7b65e-b3ce-541a-7fd5-29b4ba56ce18&t=f832e1c9-f124-52e9-664c-6040860879fb&l=2159 (no s3fs)
- azure-37-locale: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=40289&view=logs&j=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c&t=99777837-3d45-5d3a-4778-aac6f4679b04&l=2217 (no s3fs)
- travis-37-cov: https://travis-ci.org/github/pandas-dev/pandas/jobs/717300242#L4113 (moto too old)

I think we should take the files that have s3fs and ensure that they have new enough versions of moto and s3fs. That would be

* ci/deps/azure-37-slow.yaml
* ci/deps/azure-windows-37.yaml
* ci/deps/travis-37-cov.yaml
* ci/deps/travis-37.yaml
@martindurant the old pyarrow seems to be coming from defaults. Perhaps only conda-forge has new enough packages for us.


So maybe make sure that all the pyarrows have `>=0.12.0` (or whatever you need). That'll get us what we need as long as `conda-forge` is one of the channels in the env. 
Interestingly, travis-37-locale doesn't have pyarrow listed at all (but the newest version on defaults is 0.16.0)
Thanks!
BUG: GH-35558 merge_asof tolerance error
State: closed

- [x] closes #35558
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@simonjayhawkins Sorry, I made a mistake for the test code. It was fixed.
Thanks @ikedaosushi 
@meeseeksdev backport 1.1.x
Refactor tables latex
State: closed

- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As suggested [here](https://github.com/pandas-dev/pandas/pull/35284#issuecomment-665273834), utils fonctions which begin or end tables / tabulars / longtables environments could be merged. 

Labels:
name: Refactor
description: Internal refactoring of code
name: IO LaTeX
description: to_latex

Comments:
  @toobaz 

I had to change one test to have the same behaviour between tables and longtable, their was no newline between the caption and the label when both were provided, which is fine for latex but I guess it's better to match the behaviours
thanks @SylvainLan very nice.
BUG: Support custom BaseIndexers in groupby.rolling
State: closed

- [x] closes #35557
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke 
@meeseeksdev backport 1.1.x
REF/PERF: Move MultiIndex._tuples to MultiIndex._cache
State: closed

Currently, the heavy-to-calculate ``MultiIndex.values`` attribute is cached in ``MultiIndex._tuples``. It would be more dogmatic to store it in ``MultiIndex._cache`` IMO, which is what this PR does.

This has the added benefit of ``._values`` getting copied over to new copies of the MultiIndex, so also gives a performance boost in cases where copying is needed:

```python
>>> n = 100_000;
>>> df = pd.DataFrame({'a': ['a', 'b'] * int(n / 2), 'b': range(n), 'c': range(20, n + 20)})
>>> mi = pd.MultiIndex.from_frame(df)
>>> mi.values # also caches mi._values in mi._cache
array([('a', 0, 20), ('b', 1, 21), ('a', 2, 22), ...,
       ('b', 99997, 100017), ('a', 99998, 100018), ('b', 99999, 100019)],
      dtype=object)
>>> %timeit mi._shallow_copy().values
22.8 ms ± 3.43 ms per loop  # master
34.6 µs ± 997 ns per loop  # this PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex
name: Clean

Comments:
lgtm. can you rebase as merged your other MI PR; ping on green.
Rebased.
thanks @topper-123
DOC: Add specific Visual Studio Installer instructions
State: closed

- [ ] further improves #28316 - Currently the 'Contributing to pandas', 'Installing a C compiler' section advises to install Build Tools for Visual Studio 2017 but doesn't specify which components to install.

Notes:

- The first time I installed the Build Tools it didn't seem to get everything needed, and I had to search around for a while before I could figure out what I'd missed.
- The advice at [python.org](https://wiki.python.org/moin/WindowsCompilers#Microsoft_Visual_C.2B-.2B-_14.2_standalone:_Build_Tools_for_Visual_Studio_2019_.28x86.2C_x64.2C_ARM.2C_ARM64.29) is more specific, so I have added that in here. I'm fairly sure that's the minimum required, correct?


Labels:
name: Docs

Comments:
I'm actually still having issues building `pandas` in a fresh Windows VM with just "Windows Universal CRT SDK" and "MSVC v140". I get the error:

> fatal error C1083: Cannot open include file: 'basetsd.h'

Searching that error, many people recommend getting the latest "Windows 10 SDK". I also found that I was having errors trying to run `cl.exe` until I installed the "C++ Build Tools core features". I'm going to continue to test in the VM until I find the combination which works and makes the most sense.

It seems like the instructions could indeed be more specific, if this much effort is required to set up a working development environment.
> I'm going to continue to test in the VM until I find the combination which works and makes the most sense.

In the end I seemed to require these components (starting from the assumption of using MSVC v140):

![image](https://user-images.githubusercontent.com/15697539/90313837-f3463b80-df52-11ea-8b8f-c7cec2b7bbc0.png)

I also had to copy `rc.exe` and `rc.dll` from

> C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x86

to

> C:\Program Files (x86)\Windows Kits\10\bin\x86

as per https://stackoverflow.com/a/45319119/336001.

With this setup, I was able to get `pandas` to build (with a retry as a few objects failed to compile the first time around). However, I'm not sure I really understand why this particular setup is working. Can anyone weigh in on how to validate a correct Visual Studio Build Tools install?
I've made a final change to just suggest installing the "C++ build tools" workload. I've just tried this in a fresh VM and `pandas` builds without any issues.
Also, can you squash to a cingle commit please.
Thanks @bdforbes
BUG: RollingGroupby with closed and column selection no longer raises ValueError
State: closed

- [x] closes #35549
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke 
@meeseeksdev backport 1.1.x
DOC: Mention NA for missing data in README
State: closed


Labels:
name: Docs
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> (actually also as NaT in other data types)

Good point @jorisvandenbossche, should I add that here?
Yeah good to add NaT here as well.
Thanks @dsaxton !
ENH: Make explode work for sets
State: closed

- [x] closes #35614
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> impls and tests look fine, the only issue i have is one of ordering. sets are un-ordered, unlike lists and dicst so this could in theory change between runs?

That's true, the order of the output won't be well-defined and could change between runs
Maybe for the test we just use a set with 1 element so its deterministic, and in the docs we can highlight for sets order is not guaranteed.
> Maybe for the test we just use a set with 1 element so its deterministic, and in the docs we can highlight for sets order is not guaranteed.

Good catch on updating docs. For the tests I'm thinking we'd want multiple elements to show that it actually expands the result (I could add a singleton set case as well to show that it becomes a scalar)? To deal with non-deterministic order we can sort the output.
this looks fine, can you merge master and merge on green.
BUG: DataFrame.apply with func altering row in-place
State: closed

closes #35462
closes #35634 

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
#35634 is another example 
This does not address  #35517 or #35518
thanks @jbrockmendel 
@meeseeksdev backport 1.1.x
REF: _cython_agg_blocks follow patterns similar to _apply_blockwise
State: closed

Follows #35535 

This isn't yet identical to apply_blockwise, because both this and apply_blockwise have some odd behaviors that need to get standardized before they get combined into a BlockManager method, but this is a move in that direction.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
thanks
TST: add test for agg on ordered categorical cols
State: closed

- [x] closes #27800
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
> this is adding a lot of code, can you simplify / parameterize things.

Sure. Planning to add a fixture for the input dataframe and use it across the six test cases.  I feel parameterizing the tests will have a big decorator function on top which might become hard to understand/maintain. Let me know your thoughts.
thanks @mathurk1 very nice
DEPR: Deprecate inplace param in MultiIndex.set_codes and MultiIndex.set_levels
State: closed

Deprecates ``inplace`` parameter in ``MultiIndex.set_codes`` and ``MultiIndex.set_levels``.

Allowing inplace ops in indexes makes them more complicated wtr. caching etc. After ``inplace`` is removed, ``MultiIndex`` will be immutable, except the ``names`` attribute, similarly to other index classes.

Labels:
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
Updated.
thanks @topper-123
CI: Linux py36_locale failures with pytest DeprecationWarning
State: closed

xref #35620

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Linux py36_locale is passing. will merge shortly (and backport) to get ci green unless any objections.
@MeeseeksDev backport to 1.1.x
Backport PR #35473 on branch 1.1.x (REGR: Fix conversion of mixed dtype DataFrame to numpy str)
State: closed

Backport PR #35473: REGR: Fix conversion of mixed dtype DataFrame to numpy str

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
test failures unrelated, see #35620
DOC: compression support for file objects in to_csv
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I made a mistake in #35129: `to_csv` supports the compression argument for file objects in binary mode (not for file objects in text/non-binary mode).

Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @twoertwein ping on green.
@jreback ping
Doc notes for core team members
State: closed


Labels:
name: Docs

Comments:
Something went wrong ... Please have  a look at my logs.

 It seem that the branch you are trying to backport to  does not exists.
> Something went wrong ... Please have a look at my logs.
> 
> It seem that the branch you are trying to backport to does not exists.

lol, trigger the bot with my comment :->
thanks @TomAugspurger
Backport PR #35582 on branch 1.1.x (BUG: to_timedelta fails on Int64 Series with null values)
State: closed

Backport PR #35582: BUG: to_timedelta fails on Int64 Series with null values

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
known flaky test so merging

```
=========================== short test summary info ===========================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 72705 passed, 1743 skipped, 1054 xfailed, 58 warnings in 748.75s (0:12:28) =
```
Backport PR #35578 on branch 1.1.x (BUG: df.shift(n, axis=1) with multiple blocks)
State: closed

Backport PR #35578: BUG: df.shift(n, axis=1) with multiple blocks

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
test failure on Windows py37_np18 unrelated

```
INTERNALERROR> AssertionError: ('pandas/tests/base/test_unique.py::test_unique_bad_unicode[idx_or_series_w_bad_unicode0]', <WorkerController gw1>)
INTERNALERROR> assert not 'pandas/tests/base/test_unique.py::test_unique_bad_unicode[idx_or_series_w_bad_unicode0]'
Skipping uploading of coverage data.
```

will be good to merge if other runs pass
> will be good to merge if other runs pass

macOS py36_macos was canceled so restarted
REF: Simplify Index.copy
State: closed

Avoid the use of the ``Index.set_names`` in ``Index.copy``, which is a slightly awkward method to use here (it calls ``_shallow_copy`` + is longwinded).

This is archieved by tightening up the checks in ``_validate_names``.

Labels:
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
I`ve added tests.
lgtm ping on green.
Hello @topper-123! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-08-08 15:43:16 UTC
Updated.
thanks @topper-123
BUG: validate index/data length match in DataFrame construction
State: closed

- [x] closes #33437
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
ping on green.
ping
hmm this didn't backport 
@MeeseeksDev backport to 1.1.x
Ensure file is closed promptly in case of error
State: closed

Fixes #35566.  Replaces #35567.

Labels:
name: Bug
name: IO SAS
description: SAS: read_sas

Comments:
Not quite sure how the test failure is going to work in this case:  previous issues (e.g. #13932) mention ResourceWarnings but I don't have time to set up a full local build environment to duplicate the tests. @jreback do you have any insight from a few years back?
copied the milestone from #35567. feel free to amend as appropriate.
Hello @rxxg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-05 11:01:11 UTC
Added the What's New note.
I also added a try/except to ensure correctness under more circumstances, but it raised two PEP8 flags (which I silenced).  Happy to discuss my justification.
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
I'm fine with this implementation, but curious: wouldnt the idiomatic way to do this be with a contextmanager?
BUG: NaT.__cmp__(invalid) should raise TypeError
State: closed

- [x] closes #35046
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @jbrockmendel