BrandonFors/pandas-github-issues huggingface dataset:
Implement pd.NA checking function
State: closed

- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This may already exist in some form, but if not I think it could be useful to have a fast function for checking specifically for the presence of pd.NA (e.g. if we're looking at an arbitrary array and don't know if we'll have access to a _mask attribute, or if we're handling some other data type that uses pd.NA for missing values but doesn't have this attribute).

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> isna already does this

What I was thinking here was a way to get a mask for _only_ pd.NA (so not including np.NaN, None or other missing value indicators like isna does).  The main motivation was trying to think of a way to fix https://github.com/pandas-dev/pandas/issues/31990 without writing the equivalent of a Python for loop to do this check
related helper function ive wanted is `is_matching_na`
> What I was thinking here was a way to get a mask for _only_ pd.NA (so not including np.NaN, None or other missing value indicators like isna does). The main motivation was trying to think of a way to fix #31990 without writing the equivalent of a Python for loop to do this check

Can't you use the mask that the StringDtype has to check this? 
Closing since it looks like others aren't in favor of the idea. I do think some kind of "dtype agnostic" way of checking for NA could be helpful later
REF: _convert_to_index_sliceable as DataFrame method
State: closed

It really doesnt make sense as anything other than a method.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
>  belongs as a method on index itself if anything.

It's specifically DataFrame-only behavior.  I'd like to deprecate it (#31476), but failing that this is effectively a function with `self` as the first arg, which is an antipattern.
mothballing to clear the queue
PERF: Cythonize `from_nested_dict`
State: closed

I did try to cythonize this function, but the problem is that it also accepting a ```collections.OrderedDict```, which cannot be ```cdef``` as ```dict```, but have to ```cdef``` as ```object```.

---

I don't see much of a performance boost, I have ran the full benchmark suite and ```asv``` says that the ```BENCHMARKS NOT SIGNIFICANTLY CHANGED.```

Maybe just better to remove the ```TODO``` note, thoughts?

Labels:
name: Performance
description: Memory or execution speed performance
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
I don’t think the OrderedDict is needed here anymore. It’s likely a leftover from when Pandas supported python < 3.6 when normal dicts weren’t ordered.

Can you try to convert the use of OrderedDict to a normal dict? That will probably give some speedup on your Cython impl.
ASV benchmarks:

```
       before           after         ratio
     [c6c53671]       [94977b2b]
     <master>         <TODO-cythonize>
-        1.52±0ms      1.22±0.08ms     0.80  series_methods.NanOps.time_func('prod', 1000000, 'int8')
-     1.11±0.01ms         833±70μs     0.75  series_methods.NanOps.time_func('sum', 1000000, 'int8')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Not sure if it's related to the change.
> I don’t think the OrderedDict is needed here anymore. It’s likely a leftover from when Pandas supported python < 3.6 when normal dicts weren’t ordered.
> 
> Can you try to convert the use of OrderedDict to a normal dict? That will probably give some speedup on your Cython impl.

Agree I started removing our usage of OrderedDict in https://github.com/pandas-dev/pandas/issues/30469 think I have a local branch with a bunch more fixed. Will submit soon. 
> Thanks. I think a little strange to implement this in lib since it relies on frame methods that are implemented in frame.py - not sure if that's a blocker yet just thinking out loud

I don't really know where to put it, what you had in mind?


> Thanks. I think a little strange to implement this in lib since it relies on frame methods that are implemented in frame.py - not sure if that's a blocker yet just thinking out loud

this is a strange comment

lib holds cython code 
> this is a strange comment
> 
> lib holds cython code

Yea to clarify my point is that the code moved is just really looping over `df.items()`. I think this just fragments the code without really offering an improvement (at least from latest ASV results posted)
@MomIsBestFriend if we cant find a performance bump here, better to just remove the TODO comment and move on.
Closing.  @MomIsBestFriend if you want to re-open to remove the comment let us know.
TST: Don't use deprecated version of read_html
State: closed

Usage of function read_html is marked as deprecated when used with
positional arguments. In case when we do not explicitly check the
warning message with assert_produces_warning we should use non-deprecated
version.

This will resolve #33397

To identify candidates to change I modified decorator to raise an `Exception`

```diff
--- a/pandas/util/_decorators.py
+++ b/pandas/util/_decorators.py
@@ -294,6 +294,7 @@ def deprecate_nonkeyword_arguments(
                     "Starting with Pandas version {version} all arguments of {funcname}"
                     "{except_args} will be keyword-only"
                 ).format(version=version, funcname=func.__name__, except_args=arguments)
+                raise Exception("keyword argument test")
                 warnings.warn(msg, FutureWarning, stacklevel=stacklevel)
             return func(*args, **kwargs)
```

Related test build https://dev.azure.com/piotrnielacny/piotrnielacny/_build/results?buildId=2&view=ms.vss-test-web.build-test-results-tab

- [x] closes #33397
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] ~whatsnew entry~

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
LGTM - @LTe can you put an x in the closes checkbox so the issue is closed when merged
@alimcmaster1 checked 
cc @TomAugspurger 
thanks @LTe
BUG: Add unordered option to pandas.cut (#33141)
State: closed

Added tests and error message when there are no labels and `ordered=False`.

Issue: Pandas cut raises error if labels are non-unique (Closes #33141)

- [x] closes #33141 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: cut
description: cut, qcut

Comments:
- Updated the documentation in `pandas.cut`.
- Added an example.
- Run the documentation validation script.
- Also added a new line in bugs -> reshaping. Should a new line about the new functionality be added in the `Enhancements` section?
cc @pandas-dev/pandas-core 
Updated:
            labels = Categorical(
                labels,
                categories=labels if len(set(labels)) == len(labels) else None,
                ordered=ordered,
            )


Updated:

```python
            labels = Categorical(
                labels,
                categories=labels if len(set(labels)) == len(labels) else None,
                ordered=ordered,
            )
```

instead of:
```python
            labels = Categorical(
                labels,
                categories=labels if ordered else None,
                ordered=ordered,
            )
```

since we need to also handle the case of unordered labels. We just need to avoid passing duplicates labels to categories and also raise an error when we are using `ordered=True` and  there are duplicate labels. 
thanks @mabelvj very nice!
Added option to ignore file paths
State: closed

This is needed both for #32942 and to #33394 to ignore the ```pandas/tests``` directory

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
If you merge master test failures will be resolved
Sorry, I got into other obligation in the past week and a half, I'll try to finish what I started as much as I can.
can you merge master
closing as stale, if you want to continue pls ping.
Send None parameter to pandas-gbq to set no progress bar
State: closed

- [x] closes #33360 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
@TomAugspurger Request for review.
Do we need to set the min version as mention in https://github.com/pandas-dev/pandas/issues/33360#issuecomment-610412310
@alimcmaster1 need help in understanding the reason for build failures, should we just re-run it ?
> @alimcmaster1 need help in understanding the reason for build failures, should we just re-run it ?

Sure - if you merge master this should now be fixed. 
@jreback Green
@sumanau7 please see my comments above. We need to change the min supported before doing this https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html#optional-dependencies it’s currently 0.8.0 you are proposing 0.12.0 right? 
> @sumanau7 please see my comments above. We need to change the min supported before doing this https://pandas.pydata.org/pandas-docs/stable/getting_started/install.html#optional-dependencies it’s currently 0.8.0 you are proposing 0.12.0 right?

@alimcmaster1 Have done that change, can you please verify if these are the only places where this change is required
> @alimcmaster1 Have done that change, can you please verify if these are the only places where this change is required

You will also need to update `_optional.py`
Note current default behaviour of pandas `read_gbq` is tdqm progress bar. This change makes the default no progress bar right?. Do we want to be consistent with pandas_gbq and use tdqm by default? 

(From @TomAugspurger comment here https://github.com/pandas-dev/pandas/issues/33360#issuecomment-610401113 sounds like we should have same default :) )
This looks pretty much there to me. (Just need to decide on this [point](https://github.com/pandas-dev/pandas/pull/33477#issuecomment-613743313))

@sumanau7 looks like some of the tests are failing (logs on longer available) mind taking a look? - https://github.com/pandas-dev/pandas/pull/33477/checks?check_run_id=588645024
Looks like this build job it failing - (not publishing back to PR) https://github.com/pandas-dev/pandas/pull/33477/checks?check_run_id=588645024


@sumanau7 could you update as per Jeff’s comments? Thanks!
thanks @sumanau7
REF: unstack
State: closed

cc @jreback this moves a small amount of the unstack logic up the call stack, which is worthwhile, but the big wins available are if you can help me figure out the questions in inline comments.

Labels:
name: Refactor
description: Internal refactoring of code
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
looks fine. you can try experimenting if you try to unstack on a non-sorted index and raising and see what breaks.
BUG: Series.__setitem__[listlike_of_integers] with IntervalIndex
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Changes make sense.  Just needs a `whatsnew` I think.
yep if you can add a whatsnew
updated with whatsnew
updated + green (finally)
Thanks @jbrockmendel
REF: Simplify __getitem__ by doing positional-int check first
State: closed

This gets rid of the last non-test usage of `Index.get_value`, so that can be deprecated, xref #19728

This focuses only on `__getitem__`; `__setitem__` is tougher for a few reasons, including #33469.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks
DOC: timezone conversion example added to pandas.Series.astype doc #33399
State: closed

…3399

- [x] closes #33399 

Have added an example in the documentation page of pandas.Series.astype for datetime type with timezone.

Labels:
name: Docs

Comments:
Hello @venkateshdatta1993! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-11 18:56:24 UTC
Thanks @venkateshdatta1993! 
> Thanks @venkateshdatta1993! 

Thanks @mroeschke for guiding me through this.
setup.py Simplifications
State: closed

Just a smattering of things that I doubt anyone is using


Labels:
name: Build
description: Library building on various platforms
name: Clean

Comments:
The cython coverage is worthwhile, There’s an issue about running it nightly.
now with an actual keyboard: I periodically do a run with cython coverage enabled to see what parts of the cython code can be trimmed.   Moreover, ours is IMO the clearest example for other projects to use on how to enable cython coverage.
Sounds good - I’ll scale this back on next iteration

Sent from my iPhone

> On Apr 11, 2020, at 2:41 PM, jbrockmendel <notifications@github.com> wrote:
> 
> ﻿
> now with an actual keyboard: I periodically do a run with cython coverage enabled to see what parts of the cython code can be trimmed. Moreover, ours is IMO the clearest example for other projects to use on how to enable cython coverage.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Closing for now to keep queue down; can reopen later
BUG: None converted to NaN after groupby first and last
State: closed

- [x] closes #32800
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This PR solves #32800. In #32124 `not checknull()` was introduced to `groupfirst()` and `grouplast()`. `None` is not caught in this conditional and it's being converted to `nan` because of that.

I'm a newbie and not sure whether I should add whatsnew entry or not. Should I?

Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks!
CI: fix lint issue
State: closed

https://github.com/pandas-dev/pandas/runs/577244968

Linting .py code
##[error]./pandas/core/internals/blocks.py:11:1:F401:'pandas._libs.tslibs.Timedelta' imported but unused
Linting .py code DONE

Labels:
name: CI
description: Continuous Integration

Comments:
LGTM
oh is master broken
merge when ready
Thanks @simonjayhawkins
TST: concat(..., copy=False) with datetime tz-aware data raises ValueError: cannot create a DatetimeTZBlock without a tz
State: closed

…n version 0.24.1 but passes on current version

- [x] closes #25257
- [x] tests added / passed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
Hello @SurajH1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-12 10:18:23 UTC
I'm not sure why the  Publish benchmarks artifact error is popping up on Travis-CI tests. Any help would be appreciated @jreback 
can you merge master
@SurajH1 i've pushed a few changes to the branch of this PR. I notice you raised the PR from master. In future you should create a seperate branch, see https://pandas.pydata.org/docs/dev/development/contributing.html#creating-a-branch
Thanks @SurajH1 and @simonjayhawkins !
REF: de-nest Series getitem/setitem checks
State: closed

ATM the first checks we do in `Series.__getitem__` and `__setitem__` are for `get_loc` with some not-totally-straightforward try/except logic.

This moves the slice case and the com.is_bool_indexer case up, and aligns the ordering of the checks between getitem/setitem.

Getting the com.is_bool_indexer case out from the try/except in setitem is particularly nice.

That said, this _may_ not be what we want to do.  This will very slightly improve performance in the cases moved up, and equally slightly harm performance in the scalar case.  AFAIK the current ordering is not based on optimizing one case over the others.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
> but about the is_bool_indexer, I am not so sure

yah, im torn on this.  I think following #33404 i can improve the try/except blocks to allow us to retain the current ordering while also getting the de-nesting we get here
can you rebase following #33404 merge with any changes you are planning
mothballing until after the next pass
BUG: Fix ValueError when grouping by read-only category (#33410)
State: closed

- [x] closes #33410
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Groupby

Comments:
thanks @erik-hasse
DOC: Fix heading capitalization in doc/source/whatsnew - part4 (#32550)
State: closed

Add exceptions to the list in 'scripts/validate_rst_title_capitalization.py'

- [ ] Modify files in doc/source/whatsnew: v0.14.0.rst, v0.14.1.rst, v0.15.0.rst, v0.15.1.rst, v0.15.2.rst, v0.16.0.rst, v0.16.1.rst, v0.16.2.rst, v0.17.0.rst, v0.17.1.rst
- [ ] Modify files in scripts: validate_rst_title_capitalization.py

Labels:
name: Docs
name: Clean

Comments:
can you merge master and ping on green.
@jreback 
Merged
Im not sure to understand why there are unsuccessful tests here
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks!
ENH: Add prod to masked_reductions
State: closed

Adding prod to /core/array_algos/masked_reductions.py and using them for IntegerArray and BooleanArray. This also seems to offer a decent speedup over nanops like the other reductions:

```python
# Branch

[ins] In [3]: %timeit arr.prod()  # arr = pd.Series([None, 0, 1, 2] * 10_000, dtype="Int64")
102 µs ± 379 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

[ins] In [5]: %timeit arr.prod()  # arr = pd.Series([0, 0, 1, 2] * 10_000, dtype="Int64")
82.6 µs ± 752 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

# Master

[ins] In [4]: %timeit arr.prod()  # arr = pd.Series([None, 0, 1, 2] * 10_000, dtype="Int64")
291 µs ± 6.23 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

[ins] In [6]: %timeit arr.prod()  # arr = pd.Series([0, 0, 1, 2] * 10_000, dtype="Int64")
78.6 µs ± 2.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Can you also add prod to the existing whatsnew note about this?
Hmm, looks like we are hitting an overflow issue in /tests/extension/base/reduce.test_reduce_series. This is interesting because I think the current implementation is only passing this test "by accident" since the input gets cast to float64 whenever it contains NA values (without this casting we would still get an overflow, which would be the case if the test input happened not to have any missing data): https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/integer.py#L571

Is this test actually correct (i.e., if an integer input would overflow, do we "expect" the output we would get if it was first cast as float)? https://github.com/pandas-dev/pandas/blob/master/pandas/tests/extension/base/reduce.py#L19
@dsaxton shot in the dark maybe algorithms.checked_add_with_arr has something useful?
I suppose this is just the test that was not correct. Also for int64 dtype (default numpy one), we let it silently overflow instead of casting to float:

```python
In [1]: a = np.arange(1, 101)  

In [2]: pd.Series(a).prod()   
Out[2]: 0

# no missing values -> use int algo from numpy
In [3]: pd.Series(a, dtype="Int64").prod()     
Out[3]: 0

# on this branch, also this returns 0
In [4]: pd.Series(list(a) + [None], dtype="Int64").prod()     
Out[4]: 93326215443944102188325606108575267240944254854960571509166910400407995064242937148632694030450512898042989296944474898258737204311236641477561877016501813248
```

So certainly having it differ depending on the presence of missing values or not as the current implementation gives is clearly a bug.

Hello @dsaxton! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-04-11 16:42:43 UTC
thanks @dsaxton
BUG: Timedelta == ndarray[td64]
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

xref #33346

Between the two of these, we should be able to get rid of this kludge: https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals/managers.py#L629

IIRC there were some test_coercion xfails related to that kludge, will be worth revisiting.

Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
> do we have tests for NaT (object and inside the ndarray and 0-dim?

no, i have the 0-d case in my notes for a follow-up
thanks
BUG: `weights` is not working for multiple columns in df.plot.hist
State: closed

- [x] closes #33173 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
thanks @charlesdong1991
BUG: Fix bug when concatenating Index of strings
State: closed

- [ ] closes #33425
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
thanks @dsaxton
Support downcasting of nullable arrays
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Addressing GH #33013. Supports downcasting of nullable dtypes by transferring elements in extension array to ndarray to avoid TypeError thrown by np.allclose. Enhancement documented on v1.1.0 in the "other enhancements" section. Unit test written in test_to_numeric.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
can you merge master and address comments
@yixinxiao7 is this still active? Can you merge master and address comments?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Closing as stale.
remove flake8-rst cap
State: closed

looks like this was capped a ways back in #24756 - tried locally and didn't see any issues so may be fine to lift the cap

@datapythonista 

Labels:
name: CI
description: Continuous Integration

Comments:
Looks like it's still needed: https://github.com/pandas-dev/pandas/pull/33432/checks?check_run_id=574620238#step:6:47

CC: @FHaase 
Yep looks like an oversight on my end. Tabling this for now