BrandonFors/pandas-github-issues huggingface dataset:
CLN: avoid _internal_get_values in groupby.generic
State: closed

xref #27165, #27167

Labels:
name: Clean

Comments:
Looks good - are we just getting rid of `internal_get_values` entirely? 
> are we just getting rid of `internal_get_values` entirely

Thats the goal, xref #27165, #27167
Thanks @jbrockmendel
CLN: avoid values_from_object in Series
State: closed

xref #32422, #32419

Labels:
name: Clean

Comments:
thanks
@jbrockmendel another problem with this PR is that you enabled "mean" for Period dtype (but only for DataFrames), while we had long discussions before (when initially adding mean support for datetimelikes) that ended in not supporting mean for period (-> https://github.com/pandas-dev/pandas/pull/24757/)
Opened a PR for that in the mean time: https://github.com/pandas-dev/pandas/pull/33758
BUG: Fix DataFrame.apply(..., raw=True) not calling with raw array
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/32423
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Seems to pass. :tada: Where do I put the whatsnew?
thanks @kernc
Fix BUG: overflow on pd.Timedelta(nanoseconds=) constructor
State: closed

Add regression test

- [x] closes #32402
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
@jreback care to take a look?
thanks @roberthdevries very nice
TYP: enforce annotation on SingleBlockManager.__init__
State: closed

cc @simonjayhawkins my impression from yesterday's threads was that mypy should be catching the wrong-type being passed in `SingleBlockManager.__init__`, did I misunderstand something?

Also (and I know ive asked before) is there a nice way to annotate the return type as "same type as self"?  I tried `pandas._typing.T` but that didnt do it.

Parts of this are repeated in several BlockManager methods, should probably be made into a helper function, pending discussion on above.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
sorry @jbrockmendel didn't get round to this today. will look tomorrow.
thanks
ENH: Categorical.fillna allow Categorical/ndarray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #32414.

cc @TomAugspurger ATM the new test implemented in tests.arrays.categorical.test_missing is failing on the ndarray case for reasons that I dont fully grok.  Are you familiar with this?

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
I think the failing test case is consistent with accidentally using a code for a value? Perhaps test with a Categorical and ndarray containing something other than integers to see if that's the case.
Looks like the problem is that ATM Categorical.fillna is altering in-place for Series `other`
gentle ping; this is a blocker for tightening types (and avoiding unwanted code patterns) in Block.putmask and Block.where
thanks, i think we need a whatsnew note as well, pls do in a followon
BUG: fixes plotting with nullable integers (#32073)
State: closed

- [x] closes #32073
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Hello @jeandersonbc! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-03 18:33:32 UTC
Is this still useful?
@jeandersonbc can you update and address comments.
@jeandersonbc closing as stale. ping if you want to continue.
Properly handle missing attributes in query/eval strings
State: closed

Consider this script:

```python
import pandas as pd

pd.eval("pd.thing")
```

Currently it raises an error like this:

```
  File "/home/alex/work/pandas/pandas/core/computation/expr.py", line 640, in visit_Attribute
    raise ValueError(f"Invalid Attribute context {ctx.__name__}")
AttributeError: 'Load' object has no attribute '__name__'
```

Adding `__class__` to that line changes the error to the more sensible:

    ValueError: Invalid Attribute context Load

Re-raising the original error gives what's really needed:

```
  File "/home/alex/work/pandas/pandas/core/computation/expr.py", line 631, in visit_Attribute
    v = getattr(resolved, attr)
  File "/home/alex/work/pandas/pandas/__init__.py", line 260, in __getattr__
    raise AttributeError(f"module 'pandas' has no attribute '{name}'")
AttributeError: module 'pandas' has no attribute 'thing'
```


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
I don't know what the failure "[codecov/patch — 0% of diff hit (target 50%)](https://codecov.io/gh/pandas-dev/pandas/compare/4018550c796f9b23369267565b8ab8c87b9d14ad...f392c60f8a83c26d2cb3e4d854b9a3ba3b44511d/changes)" means. Am I supposed to write tests for this change?
can you add a whatsnew note for 1.1, in the Other bug fix section
thanks @alexmojaki will merge on green.
Presumably that docs failure is for unrelated reasons?

    no theme named 'pandas_sphinx_theme' found (missing theme.conf?)

> Presumably that docs failure is for unrelated reasons?
> 
> ```
> no theme named 'pandas_sphinx_theme' found (missing theme.conf?)
> ```

I _think_ fixed by #32840. can you merge master
[1 failed test in 1 job in 1 build: `numpy array values are different (2.0 %)`](https://travis-ci.org/github/pandas-dev/pandas/jobs/670723319?utm_medium=notification&utm_source=github_status)

Again, unrelated.
Can you merge master again and re-push? Should be fixed on master
The last build succeeded 5 hours ago, but github hasn't been updated. I've never seen that before.
Thanks @alexmojaki
CLN: remove is_period_arraylike, is_datetime_arraylike
State: closed

There are better alternatives

Labels:
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
needs a rebase. are these public? if so need to deprecate (unfortunately) and provide a shim.
> are these public?

no
@WillAyd any idea why mypy would be complaining here (and i think on many other PRs) but not locally?
Can you merge master? @simonjayhawkins had a recent fix for mypy
@jbrockmendel can you merge master
rebased+green
Add missing newline
State: closed

The line ">>> idx.isin([1, 4]) array([ True, False, False])" is was not rendered properly, because it was missing a newline above it.

https://pandas.pydata.org/docs/reference/api/pandas.Index.isin.html

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Hello @davidwales! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-03 02:21:11 UTC
Great thanks @davidwales
CLN: Avoid unnecessary values_from_object
State: closed

xref #27165.

Labels:
name: Clean

Comments:
What are the cases where `asarray` and `values_from_object` would give a different result?
> What are the cases where asarray and values_from_object would give a different result?

They should only ever be distinct for EA dtypes, none of which is the case here
great. yeah this was for compatibility when we removed Series from subclassing ndarray (like 0.13 or so), so would be nice to actually completely remove `values_from_object`
> so would be nice to actually completely remove values_from_object

Yah, I got tripped up the other day by `PeriodIndex.values` and am now tilting at windmills to get rid of a few patterns like this.
Added figsize for autocorrelation_plot and lag_plot (#31650)
State: closed

- closes #31650
- Passed pandas.tests.plotting.test_misc.py
- Now creating new axes with "fig, ax = plt.subplots(1, 1, figsize=figsize)"
- Also changed plotting.lag_plot regarding the same issue
- Figure size is only changed if no axis parameter is given



Labels:
name: Visualization
description: plotting

Comments:
@jqmviegas do we have tests that cover all of the changed code?  codecov seems to think not
> @jqmviegas do we have tests that cover all of the changed code? codecov seems to think not

@jbrockmendel  Added checks using the _check_plot_works(...) with the figsize keyword but I'm not sure if that does the trick. Unsure how to improve testing regarding this issue.
Running coverage.py indicates that everything is covered in the methods I worked on.
@jbrockmendel I'm almost sure that this is the result of the CI running the fast tests. The tests for the methods I worked on are decorated with @pytest.mark.slow. I was able to reproduce the results of codecov/patch running the following:

`pytest --cov=pandas --skip-slow pandas/tests/plotting/test_misc.py`

This returns:

```
Name                                        Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------
...
pandas/plotting/_matplotlib/misc.py           269    167    38%   63-69, 113, 118-196, 202-256, 261-303, 332, 340-342, 344-348, 388-402, 406-437
```

Running the full tests with:

`pytest --cov=pandas pandas/tests/plotting/test_misc.py`

Returns:


```
Name                                        Stmts   Miss  Cover   Missing
-------------------------------------------------------------------------
...
pandas/plotting/_matplotlib/misc.py           269     52    81%   63-69, 113, 187, 202-256, 332, 340-342, 344-348
```
No missing lines include the functions autocorrelation_plot or lag_plot. 
@charlesdong1991 can you take a look; i think you know the plotting code better than i do
@jbrockmendel , @charlesdong1991 - Changed the tests to ensure `figsize` changes to what we set. Additionally improved the docstrings and added change to 1.1 rst. CI failing due to other issues I think.
can you merge master and update to comments
closing as stale if u want to continue pls ping
DOC: change link to contributing guide in README.md
State: closed

xref https://github.com/pandas-dev/pandas/pull/32068#issuecomment-593362224

Labels:
name: Docs

Comments:
LGTM
Working on the actual fix now, see #32530
Thanks @datapythonista 
> Can you leave the dev. for now? (until we have a proper place for the dev docs) The link you are changing it to now are the stable docs

updated.
DOC: Reorganize Getting Started documentation pages
State: closed

- [x] closes #32308 

Issue Related:
- `getting_started/index.html` removes `10 minutes to pandas`, `Essential basic functionality` and `Intro to data structures` from the toctree and moves them as links in `user_guide/index.html`
- `getting_started/tutorials.html` now only references community tutorials
- the pandas cheatsheet is now referenced at the bottom `Tutorial` section of `getting_started/index.html`

Misc:
- Lowercases `Pandas` to `pandas`
- Some rephrasing


Labels:
name: Docs

Comments:
My sphinx-foo is a bit poor.

I see I'm getting these doc build warnings

```
2020-03-02T07:41:43.8295810Z /home/runner/work/pandas/pandas/doc/source/getting_started/10min.rst: WARNING: document isn't included in any toctree
2020-03-02T07:41:43.8296699Z /home/runner/work/pandas/pandas/doc/source/getting_started/basics.rst: WARNING: document isn't included in any toctree
2020-03-02T07:41:43.8297521Z /home/runner/work/pandas/pandas/doc/source/getting_started/dsintro.rst: WARNING: document isn't included in any toctree
```

And we have a note in the docs

```
.. If you update this toctree, also update the manual toctree in the
   main index.rst.template
```

But still unsure what I need to change in `index.rst.template`
cc @jorisvandenbossche 
@mroeschke for the warnings, I think you need to add a toctree to `tutorials.rst` with those (10min, etc.) documents included. Otherwise they won't be included in the output.
Or alternatively put `:orphan:` on the top of the rst page that is not included in any toctree but we still want to include in the output, which will silence the warnings as well.

However, I think we want to include it in a toctree for better visibility?
Yeah, I think including it is better. I think putting it in an `:orphan:` page also kinda breaks "next" and "previous" links.
I am not sure about moving 10min to the user guide. In any case, based on its name I would say it belongs in the getting started section.

cc @datapythonista are you OK with moving dsintro.rst and basics.rst back to the user guide? (since you put them in the getting started section when the split was done)
Planning on merging tomorrow if there are no further comments.
Plotting Int64 columns with nulled integers (NAType) fails #32073
State: closed

- [x] closes #32073
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Thanks @jeandersonbc 

However, before this PR can be accepted, you need to write tests / make sure you don't break existing ones - see [contributing to the code base](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#contributing-to-the-code-base):

> pandas is serious about testing and strongly encourages contributors to embrace test-driven development (TDD). This development process “relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test.” So, before actually writing any code, you should write your tests. Often the test can be taken from the original GitHub issue. However, it is always worth considering additional use cases and writing corresponding tests.
> 
> Adding tests is one of the most common requests after code is pushed to pandas. Therefore, it is worth getting in the habit of writing tests ahead of time so this is never an issue.
Thanks for the feedback @MarcoGorelli! I'm wondering whether the example provided in the related issue (first comment in https://github.com/pandas-dev/pandas/issues/32073) would be sufficient to be added as a test in the `TestDataFramePlots` (test_frame.py) class. I started the discussion on tests on the issue yesterday (although here might be more appropriated).
well, I'm going to close this pull request and work in a single branch as this was not the first time that I messed up after updating my branch
BUG: Fix rolling functions with variable windows on decreasing index
State: closed

- [x] Closes https://github.com/pandas-dev/pandas/issues/32385
- [x] tests updated & passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
cc @mroeschke are you familiar with this code? 

I can confirm that indeed the current output (and tested behaviour) is wrong
Only familiar from a refactor standpoint. I didn't write the original window bounds calculations.
@leftys can you respond to https://github.com/pandas-dev/pandas/pull/32386#pullrequestreview-368778011? Trying to get 1.0.2 out soon.
> @leftys can you respond to [#32386 (review)](https://github.com/pandas-dev/pandas/pull/32386#pullrequestreview-368778011)? Trying to get 1.0.2 out soon.

@TomAugspurger done
thanks @leftys very nice!
@meeseeksdev backport 1.0.x
TYP/CLN: Optional[Hashable] -> pandas._typing.Label
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM
The only consideration point is that `Label` at least from a reader point of view was supposed to signal something that would actually be used as a Label for a row / column. Some of the uses here blur that distinction and simply leverage it because it can be used as the key for a mapping

Do you see any value in trying to maintain that distinction?
can you be specific on the offending changes. where Label is used for an index name, I don't see that as being different from an row/column name.
@WillAyd any specific objections?  i dont see any problem here

@simonjayhawkins needs rebase
@simonjayhawkins what do you think about reverting the `attrs` annotation and calling this a win?
will do. we have flakey ci atm but will give it a go. 

(we have a fix for ci pending in #32449 but if we merge @SaturnFromTitan doesn't get credit (see #32457) or could just revert #32281 for now instead.)
PR #32068 follow up - Print merged df result in doc
State: closed

- [x] follow up to PR #32068 which closed #12550  - small fix - see https://github.com/pandas-dev/pandas/pull/32068#discussion_r386029773



Labels:
name: Docs

Comments:
@simonjayhawkins  just made the change and rendered it locally.

![Screenshot 2020-02-29 at 15 15 08](https://user-images.githubusercontent.com/16509490/75610178-b7372f80-5b06-11ea-85dd-50c2aafa081c.png)

Thanks @ryankarlos
CLN: Some code cleanups in pandas/_libs/parsers.pyx
State: closed

There are __a lot__ of cdef unused variables in ```pandas/_libs/parsers.pyx``` this PR is covering *some* of the unused variables.


Labels:
name: Clean

Comments:
@MomIsBestFriend can you merge master?
~~can you merge master once again, we lost azure coverage; ping on green.~~
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
> /azp run

That's very cool, didn't know it's possible.
Great thanks @MomIsBestFriend 
nice
Silence warnings when compiling pandas/_libs/parsers.pyx
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This is getting rid of this warning:
```
pandas/_libs/parsers.c: In function ‘__pyx_f_6pandas_5_libs_7parsers_10TextReader__get_header’:
pandas/_libs/parsers.c:9313:27: warning: ‘__pyx_v_data_line’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 9313 |   __pyx_t_5numpy_uint64_t __pyx_v_data_line;
```

Labels:

Comments:
Thanks @MomIsBestFriend 
@jbrockmendel pls label and version PRs be fore merging
CLN: some code cleanups to pandas/_libs/missing.pyx
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend
CLN: Replaced "bool_t" with "builtins.bool"
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@simonjayhawkins I have to be honest I didn't make this changes with Type annotations in mind, I simply saw ```bool_t```  and didn't knew what it was, and thought this changes will not cause future confusion.

And for your question, which approach is the best practice, I really don't know, is there anyone at ```mypy``` that we can ask?
looking at the solution in https://github.com/ahawker/ulid/pull/26, does that look clearer?

i.e. we define aliases in pandas._typing that use the same name with capitalization.

so the alias for str is Str, bool is Bool etc.

I also like this as it is consistent with dict and Dict (although for a different reason, Dict can accept type parameters)

@WillAyd 
These might not be required with recent updates to the semantic analyzer so can try that, but otherwise I don't think there's any real benefit to changing
> These might not be required with recent updates to the semantic analyzer so can try that, but otherwise I don't think there's any real benefit to changing

worth giving it a try, but we should probably keep the "alias trick" as "a good defensive solution"
I agree with Will.  The bool_t definition also has a comment telling us why it is there, which the builtins.bool doesnt
I think that perhaps we could move the alias to pandas_typing and add a better explanation and ensure consistency.

in `pandas\core\indexes\base.py` we have `str_t = str` and in `pandas\core\dtypes\dtypes.py` we have `str_type = str`, neither of which have comments to explain.

in https://github.com/ahawker/ulid/pull/26 the alias is in a shared hints module (equivalent to our pandas_typing)

and looks like...
```
#: Type hint that is an alias for the built-in :class:`~str` type.
Str = str
```

I think we should do that for bool here and str in the other two modules mentioned.

( and agree on _t vs _type vs Capitalization)
Do we support ```float16```?
> Do we support `float16`?

#32449 should fix that.
Would have to search but fairly certain what we have now is the suggested approach by Guido / mypy folks. 

Do any of these. alternatives have a clear benefit? If not I think should just close

> Do any of these. alternatives have a clear benefit? If not I think should just close

No, there is not a clear benefit.

---

Closing, thank you all for the notes :)
TST: Removed import of itertools
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
Thanks @MomIsBestFriend
Changed kind parameter from integer to int, Added example
State: closed

- [x] closes [Fix PR06 error in pandas.arrays.SparseArray](https://github.com/pandanistas/pandanistas_sprint_ui2020/issues/1)
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

```
################################################################################
################################## Validation ##################################
################################################################################

3 Errors found:
        Parameter "sparse_index" has no description
        Parameter "index" has no description
        See Also section not found

```

Labels:
name: Docs

Comments:
Great thanks @mackarelfish !
DOC: Update the pandas.DatetimeIndex docstring
State: closed

- [x] closes https://github.com/pandanistas/pandanistas_sprint_ui2020/issues/7
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Output of ```python scripts/validate_docstrings.py pandas.DatetimeIndex```
```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
	No examples section found
```


Labels:
name: Docs

Comments:
Thanks for the improvements to this docstring @Hori75
Fixing RT02 pandas.Index.dropna and PR08 pandas.Index.fillna
State: closed

- [x] closes https://github.com/pandanistas/pandanistas_sprint_ui2020/issues/17
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @zaki-indra