BrandonFors/pandas-github-issues huggingface dataset:
DOC: remove use of head() in the comparison docs
State: closed

This helps to clarify the examples by removing code that isn't relevant. Added a dedicated section to the SAS, SQL, and Stata pages.

This builds on https://github.com/pandas-dev/pandas/pull/38933; ~~will rebase and mark as ready for review once that's merged. In the meantime, the last commit is the one that can be reviewed.~~ Thanks!

- [ ] ~~closes #xxxx~~
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
Ready for review! Added a few more fixes+improvements to the pull request — see individual commits.
thanks @afeld
ENH: Improve numerical stability for groupby.mean and groupby.cumsum
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
thanks @phofl very nice
DOC: improve shared content between comparison pages
State: closed

This pull request does a few things between the SAS and Stata pages, in separate commits:

- Makes the headings match, where it makes sense for them to
- Create more shared includes, as a follow-up to https://github.com/pandas-dev/pandas/pull/38887
- Improves some wording and ensures more methods are linked in the comparison includes

The motivation here is that I'm working on adding the other sections to the Comparison to Spreadsheets page, and want to ensure they're consistent.

---

- [ ] ~~closes #xxxx~~
- [x] tests added / passed
- [ ] ~~passes `black pandas`~~
- [ ] ~~passes `git diff upstream/master -u -- "*.py" | flake8 --diff`~~
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
nice, ping on green.
thanks @afeld
BUG: rank_2d raising with mixed dtypes
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
thanks @mzeitlin11 keep em coming!
TST: Replace pytest.xfail
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Part of #38902.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @rhshadrach
TST: stricten xfails
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=51664&view=logs&j=eab14f69-13b6-5db7-daeb-7b778629410b&t=ce687173-08c6-5301-838d-71b2dda24510 look legit
ENH: Add support to import optional submodule and specify different min_version than default
State: closed

- [ ] closes #38888
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

cc @jreback, @arw2019 

Labels:
name: Build
description: Library building on various platforms

Comments:
thanks @lithomas1 

ok this is not actually running anywhere, can you use this wherever we actually use a min_version
also pls create an issue as described above.
TST: 26807 split pandas/tests/tseries/offsets/test_offsets.py into multiple smaller test modules
State: closed

This is to address xref #26807 specifically for pandas/tests/tseries/offsets/test_offsets.py . I tried to get all the new modules below about 1000 lines and break it up in logical ways.

Other than moving code, I have changed some list construction for pytest parameterizing to be a tiny bit more compact by changing a bunch of sequential `append`s to a single list literal. Other than that no changes.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets
name: Code Style
description: Code style, linting, code_checks

Comments:
can you confirm before / after tests are the same number
> can you confirm before / after tests are the same number

Good question! At the time you wrote this I had 25 duplicate tests due to having 2 test classes inherit from common.Base with the same _offset. But now that I have fixed that, they have exactly the same number.
thanks @moink
TST: GH30999 Add placeholder messages to pandas/tests/io/test_sql.py and remove test for numexpr < 2.6.8
State: closed

This is my attempt to finally finish off #30999

In pandas/tests/io/test_sql.py, there is a whole test class skipped. It looks like xref #20536 is supposed to address that, but no one has commented there since March 2018, so I don't think that's going to be fixed any time soon. I noticed that there were other tests in the same module with `match="<insert message here>"` so I decided to put it in the two tests that I can't figure out the correct error message for.

In pandas/tests/computation/test_compat.py there was an if statement that the numexpr library is at least 2.6.8. I tried to set up an environment with a lower version but conda couldn't resolve the dependencies. That test isn't running in the CI (xref #38876) and that test was last touched in a substantive way in 2016. I think that portion of the test is no longer required.

Not sure that I actually addressed these correctly, but I made an attempt so we could have a conversation about it in a more concrete way.

- [x] xref #30999
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @moink 

we can close the issue when we have a precommit check fully in place
Remove Python2 numeric relics
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Modernize the code to Python 3 by removing Python 2 numeric hacks, such as:

- `a / b` of two integers now always returns a float, so no need to convert `a` and/or `b` to float (or to use float literals)
- `a // b` of two integers returns an integer, use it instead of `int(a / b)`
- `math.ceil`, `math.floor`, and `round` return integer, so no need to convert the result to integer
- `1e6` is a float, converting it to `int(1e6)` is slower than `10 ** 6` (or `1_000_000`)


Labels:

Comments:
> Nice, thanks @eumiro !
> 
> Did you use some tool to find these, or was it just inspection?

Thank you. It was manual search&replace (with at least one typo as you could see, thank you, fixed, repushed), because every case is different. Usually it is a good idea to search for `int(`, `float(`, `round`, `math.`, `\de\d`, `\d\.0\b`, etc.
BUG: Fixed regression in rolling.skew and rolling.kurt modifying object
State: closed

- [x] closes #38908
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Alternative would be np.copy before releasing gil. In this case we would have to touch values_copy twice in case of ``min_val - mean_val > -1e5``

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Window
description: rolling, ewma, expanding

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
CI: enable parallel testing on arm64 build #36719
State: closed

- [x] closes #36719



Labels:
name: CI
description: Continuous Integration
name: ARM
description: aarch64 architecture

Comments:
It took about 19min to setup the environment and cache...
make sure it's actually using the workers in setup.py (-j)

and u can use 4
looks like it completed (with some failure)

mark the longest tests with arm_slow and try more workers 
I did a simple test on my Travis account. 

<details>

```yml
os: linux

dist: bionic

language: python

python: 3.8

arch:
  - arm64
  - arm64-graviton2

addons:
  apt:
    packages:
      - libsnappy-dev

branches:
  only:
    - main

before_install:
  - |
    git clone https://github.com/pandas-dev/pandas.git
    cd pandas
install:
  - python -c 'import os,sys,fcntl; flags = fcntl.fcntl(sys.stdout, fcntl.F_GETFL); fcntl.fcntl(sys.stdout, fcntl.F_SETFL, flags&~os.O_NONBLOCK);'
  - python -m pip install --no-deps -U pip wheel setuptools
  - python -m pip install cython numpy python-dateutil pytz pytest pytest-xdist hypothesis
  - python setup.py build_ext -j4
  - python -m pip install -e . --no-build-isolation --no-use-pep517

script:
  - pytest -n 4 -m 'not slow and not network and not clipboard' pandas --junitxml=test-data.xml
```

</details>

It only took about 7 min to setup and 21 min to run the test. Let's see if using 4 cores would cut the test time to 20 min range.
let start by limiting the directories it is running 

we do this for the windows builds (for reference)
> let start by limiting the directories it is running
> 
> we do this for the windows builds (for reference)

I did some tests locally. The config that affects the runtime the most is the distribution algorithm. We were using `loadfile`, which group tests by file. It's extremely slow on arm machine. I changed it to the default value `no`, which assigns tests to workers one by one. Now, the test only took 1131.13s.
wow this looks great

ok can u open an issue for the failing arm tests themselves, xfail them and then make this build a required one
> wow this looks great
> 
> ok can u open an issue for the failing arm tests themselves, xfail them and then make this build a required one

I xfailed those tests. But they still failed on CI. I tested them with py38 and numpydev on M1 mbp. The rolling test passed without xfail. But the other two tests passed with xfail. 
hmm do we need to disable the cache for arm?
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
<samp>
Commenter does not have sufficient privileges for PR 38905 in repo pandas-dev/pandas<br>

</samp>
The test failure in py37 macOS build is caused by the --dist=no.
note that travis is currently not running at all :-< working on a credit issue there
let's close for now as travis does not look like its coming back
ENH: Use Kahan summation to calculate groupby.sum()
State: closed

- [x] closes #38778 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This simplifies the op example.

I think 
```
series = Series([1e16, 99, -5e15, -5e15])
series.sum()
```
is dispatched to numpy? The result is wrong too. ``100.0``

Mean doesn't work either. I will look through the functions to determine which need Kahan summation too and open an issue to track these.

Labels:
name: Groupby

Comments:
We dispatch to numpy in case of 32bit systems, which returns the wrong result
@jreback green
thanks @phofl
TST: strictly xfail
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Continuation of #38881. This handles cases where replacing pytest.xfail is clear, opened #38902 for the handful of remaining cases.

In the case where `pytest.xfail` is used for a test which has been yet to be implemented, I've added ~~`assert False` to maintain the current behavior of counting it as an xfail. Maybe there is a better alternative here.~~ `raise NotImplementedError`.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
In #38881 you used request.applymarker which i guess is equivalent to request.node.add_marker.  is that an Officially Supported way of doing this?  What we've been doing works, but AFAICT isn't something pytest advertises
@jbrockmendel Yes, they are equivalent. Under the hood, `applymarker` is just the line `self.node.add_marker(marker)`.

Docs for applymarker:

https://docs.pytest.org/en/stable/reference.html#pytest.FixtureRequest.applymarker

> This method is useful if you don’t want to have a keyword/marker on all function invocations.

I take this to mean we are using it in the intended way.
great. conflict if you can merge master. also do we have an issue to check for *not* using `pytest.xfail` at all in precommmit?
@jreback Merged master, will ping on green. Issue is #38902 
thanks @rhshadrach
Validate axis args
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is a follow up PR on #37924 @jreback cheers

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks @tonyyyyip - just for reference, as the linked PR has lots of comments, this follows up on https://github.com/pandas-dev/pandas/pull/37924#discussion_r550812455
thanks @tonyyyyip
doc fix for testing.assert_series_equal check_freq arg
State: closed

This fixes `check_freq` arg docs for `assert_series_equal` like #38471 did for `assert_frame_equal`.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs

Comments:
thanks @kylekeppler 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 69a4ba2f2edf62b4720c538aa104030f969e4f09
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #38893: doc fix for testing.assert_series_equal check_freq arg'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-38893-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #38893 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Backport PR #38471 on branch 1.2.x (DOC: fixes for assert_frame_equal check_freq argument)
State: closed

Backport PR #38471


Labels:
name: Docs

Comments:
@kylekeppler where were the merge conflicts that prevented the autobackport from working?
> @kylekeppler where were the merge conflicts that prevented the autobackport from working?

nvm, i see now. I was focussing on the release notes and not the versionadded tag
CLN: add typing for dtype arg in directories core/indexes and core/strings (GH38808)
State: closed

Follow on PR for #38808

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @avinashpancham
DOC: create shared includes for comparison docs, take III
State: closed

From original pull request (https://github.com/pandas-dev/pandas/pull/38771):

> This will help ensure consistency between the examples.

- [ ] ~~closes #xxxx~~
- [x] tests added / passed
- [ ] ~~passes `black pandas`~~
- [ ] ~~passes `git diff upstream/master -u -- "*.py" | flake8 --diff`~~
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
Tests pass! 🎉 
thanks @afeld 

pls have a look at the built docs and issue a followup PR if needed.

OT: pls comment on your open PRs / merge master for them to see what we need to merge (IIRC some were WIP)
Yep yep, they look how I'd expect 👍 

- https://pandas.pydata.org/docs/dev/getting_started/comparison/comparison_with_sas.html
- https://pandas.pydata.org/docs/dev/getting_started/comparison/comparison_with_stata.html
- https://pandas.pydata.org/docs/dev/getting_started/comparison/comparison_with_sql.html
CLN: add typing for dtype arg in core/arrays (GH38808)
State: closed

Follow on PR for #38808

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @avinashpancham
TST: Add test_groupby_std_on_nullable_column.
State: closed

- [x] closes #35516
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Groupby
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Hello @GabrielSimonetto! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-01-01 23:37:48 UTC
There's still an error in the CI but it seems to be unrelated to my changes
Thanks @GabrielSimonetto ,

Looks like the original issue isn't fixed yet, I think the original assessment that it just needed a test may not have been correct

Do you want to try fixing the issue as well (which may be a bit harder), or shall we close? Lots of other good first issues are open
yeah closing this. the original is NOT fixed on master and needs a more comprehensive fix. thanks @GabrielSimonetto (happy to have you work on that or on any other issues, you may want to search for the good first issue)
No problem (:

Since there is a problem involving NA and groupby, I am pretty sure it's a task beyond my reach, but I am glad I could push the discussion forwards!
CLN: Use signed integers in khash maps for signed integer keys
State: closed


- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

The goal of this PR is to make this comment obsolete: https://github.com/pandas-dev/pandas/blob/1fc5efd76ea1f85979fa2364a292d85482e338aa/pandas/_libs/src/klib/khash.h#L619-L624

see also [this discussion]( https://github.com/pandas-dev/pandas/pull/37920#discussion_r527191378) for even more details.

The issue is:

- the current way (inherited from khash-original) is too clever (and suprising)
- it is wrong as the (unsigned) keys are converted back to signed values (which is implementation defined behavior) for example here: https://github.com/pandas-dev/pandas/blob/1fc5efd76ea1f85979fa2364a292d85482e338aa/pandas/_libs/hashtable_func_helper.pxi.in#L110

Using signed integer, one must take into account that hash-function with signed integers might have undefined/implementation defined behavior, thus we cast signed integers to unsigned counterpart in the (64bit-) hash function now.

Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
this looks fine. I assume that nothing in the current impl breaks because of this (e.g. converting int -> uint is safe as far as our tests go). any perf implications?
@jreback I expect no perf changes and ran asv only for a subset of tests ( `-b hash_functions -b ^algorithms -b ^indexing`) and there were no changes (albeit `hash_functions.NumericSeriesIndexing` seems to be somewhat flacky on my machine - but I have observed this behavior in other PRs as well).
thanks @realead
TST: strictly xfail mid-test
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Using `pytest.xfail` when a test is running will immediately stop the test and will not xpass if test would otherwise succeed. It doesn't seem to be documented anywhere, but an alternative would be to use the request fixture. In this demo, I've added an xfail that runs successfully, making the test fail (xpass).

If this looks like a good idea, I can make a tracking issue to replace `pytest.xfail` with the method used here.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
a fair amount of the xfails are already like this

sure i think we should be consistent and change then all
@jreback Not certain, but I think you may be referring to using the `pytest.mark.xfail` decorator. The decorator has limitations, in the case shown here it can't be used with the fixture `transformation_func`. 

A search for "applymarker" turns up empty across all our tests.
not at all search for

```           
mark = pytest.mark.xfail(
                reason="BUG: it must be pd.Series([1, 1, 3, 4], dtype=np.int16"
            )
request.node.add_marker(mark)
```
for example

```
~/pandas$ grep -r add_marker pandas | wc
      49      99    4128
```
Thanks, `request.applymarker` is the same as `request.node.add_marker`, I'll stick to add_marker for consistency.
yeah i think @jbrockmendel started putting these in recently. we should change all occurernces, update the docs and put in a pre-commit hook (once all fixed).
TST: add missing loc label indexing test
State: closed

- [x] xref #38824
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
there are some linting issues. you don't to open a new PR, just push to this one.
thanks @vangorade
BUG: silently ignoring dtype kwarg in Index.__new__
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

ATM we handle object but not others, xref #21311

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
nice cleanup. needs a rebase. should we consider deprecating `dtype=` keyword in the constructor?
> should we consider deprecating dtype= keyword in the constructor?

my hope/goal is for `Index.__new__`, `Series.__init__`, and pd.array` to share most of their code, so probably not