BrandonFors/pandas-github-issues huggingface dataset:
DOC: unpin numpydoc, fix validation script #39688
State: closed

- [x] closes #39688



Labels:
name: Docs
name: Dependencies
description: Required and optional dependencies
name: Stale

Comments:
The breaking change was from https://github.com/numpy/numpydoc/pull/302. 

After modifying our script according to the new API. There is an AttributeError error:

```
pandas.Series.array <property object at 0x1199e99f0>
Traceback (most recent call last):
  File "scripts/validate_docstrings.py", line 438, in <module>
    main(
  File "scripts/validate_docstrings.py", line 384, in main
    return print_validate_all_results(
  File "scripts/validate_docstrings.py", line 328, in print_validate_all_results
    result = validate_all(prefix, ignore_deprecated)
  File "scripts/validate_docstrings.py", line 298, in validate_all
    doc_info = pandas_validate(func_name)
  File "scripts/validate_docstrings.py", line 241, in pandas_validate
    result["examples_errs"] = doc.examples_errors
  File "scripts/validate_docstrings.py", line 161, in examples_errors
    for test in finder.find(self.raw_doc, self.name, globs=context):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/pandas-dev/lib/python3.8/site-packages/numpydoc/validate.py", line 135, in name
    return '.'.join([self.obj.__module__, self.obj.__name__])
AttributeError: 'property' object has no attribute '__module__'
```
@fangchenli is this actionable?
cc @MarcoGorelli @simonjayhawkins 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
This PR has appeared to have gone stale. Feel free to reopen if you're still interested in working on this.
[ArrowStringArray] PERF: Series.str.get_dummies
State: closed

planning to remove the padding code from _wrap_result eventually, but until then we can skip it when we return a integer array from get_dummies

adding tests and benchmarks as precursor to potential changes to _wrap_result  #41372

have a working implementation for ArrowStringArray using pyarrow native functions but is slower than object fallback, so am leaving that for a followup.

```
       before           after         ratio
     [4ec6925c]       [091b0b02]
     <master>         <get_dummies>
-      2.58±0.02s         655±10ms     0.25  strings.Dummies.time_get_dummies('arrow_string')
-      2.58±0.03s          643±9ms     0.25  strings.Dummies.time_get_dummies('string')
-      2.59±0.07s          638±7ms     0.25  strings.Dummies.time_get_dummies('str')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
great
DOC: Complete first sentence in DataFrame.hist (#41421).
State: closed

Complete sentence and mark column and figsize as optional parameters, following suggested fix.

- [ ] closes #41421
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
thanks @mdhsieh
CI: Pin jinja2 to version lower than 3.0
State: closed

- [x] xref #41450 

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
cc @jreback greenish, failure unrelated
Thanks @phofl !
@meeseeksdev backport 1.2.x
We should backport to keep ci on 1.2.x passing
Deprecate inplace=True for reset_index
State: closed

- [ ] xref #16529
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Per the discussion yesterday, I just wanted to clarify that putting in a deprecation warning (such as thing one) in v1.3.0 gives enough time to remove `inplace` in v2.0.0 if, as is likely, there is not another minor version between v1.3.0 and v2.0.0. Will it be OK to change it to a `FutureWarning` in a patch release of v1.3?

If this is OK, then I'm running a little sprint on Saturday for PyLadiesLondon, and putting in more of these warning will make for a good sprint issue

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @jreback and @phofl for your reviews, have updated accordingly

Does this seem right? If so, I'll use it as a template on Saturday and we can power through the rest of the non-controversial ones from the list
> i am not sure if we can go from DeprecationWarning to FutureWarning in a patch release

That's my concern exactly, and this feels like the kind of change that would be well-suited to a major release... @jreback thoughts on this? Sorry if I missed it from the call yesterday
> > i am not sure if we can go from DeprecationWarning to FutureWarning in a patch release
> 
> That's my concern exactly, and this feels like the kind of change that would be well-suited to a major release... @jreback thoughts on this? Sorry if I missed it from the call yesterday

right the purpose of showing a DeprecationWarning only rn is to give folks a long lead time & library authors to update. We would still need to flip this in a minor release, e.g. 1.4 to FutureWarning, then remove in 2.0.
Clear, thanks - if the next non-patch release after 1.3 will be 2.0 in December/January, then wouldn't that give people plenty of time?

Furthermore, it's painfully common to see people ignore or even silence warnings (e.g. every other notebook on Kaggle), and so while no amount of warning will get them to update their code, seeing a major release might

I've merged master and updated to `FutureWarning` here, just to drive the conversation forwards, though of course will revert to DeprecationWarning if you prefer
This one isn't on the "non-controversial" list from #16529, so perhaps it's best to close it for now and to just set its default arguments as keyword-only - would be great if we could get agreement on that, but let's take the discussion forward in #16529
I think a potential deprecation of `inplace` should be done with a DeprecationWarning first. It's widely used, so users will see a flood of warnings from libraries they wouldn't be able to do anything about themselves. 

(and also as you said, `reset_index` is not a clear candidate for deprecation, as it can potentially be done in place)
Good points - closing for now then, thanks for your input!
BUG: Raise ValueError if names and prefix are both defined
State: closed

- [x] closes #39123
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
thanks @phofl
BUG: resample.apply with non-unique columns
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Resample
description: resample method

Comments:
i guess a bug, an you add a whatsnew. lgtm otherwise, ping on green-ish.
ping
sweet.  once the other groupby stuff goes in, i think this was the last bugfix before we're ready for some nice de-duplication
DOC: Improve examples of df.append to better show the ignore_index param
State: closed

Change index to [0, 2] to stress that ignore_index=True resets the index of both dataframes

- [x] closes #41407 
- [x] tests passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Test Output: 

```
################################################################################
################################## Validation ##################################
################################################################################

1 Errors found:
	Return value has no description
```



Labels:
name: Docs

Comments:
Anything to fix? Sounds not like it's coming from my changes, does it? 

`Sphinx parallel build error:
nbsphinx.NotebookError: UndefinedError in user_guide/style.ipynb:
'nbformat.notebooknode.NotebookNode object' has no attribute 'tags'`
No, see #41450 
Sorry pressed the wrong button, this was not on purpose
Btw, if you merge maser, the CI should be fixed.
Another btw, feel free to add the `Return` description, since the validation is complaining that one is missing, and should be very easy to add.
thanks @t0asty
DOC: freeze old whatsnew #6856
State: closed

xref #6856



Labels:
name: Docs

Comments:
As Jeff says, this is unfortunately more complex than simply replacing the directives. See those:
```
.. ipython:: python
   print('hello world')

.. code-block:: python
   >>> print('hello world')
   hello world
```

The reason we use the ipython directive is that we don't need to write the `>>>` or the output, just the code. So, to do this, we need to add those two to every example. :\

If we want to do this, what I'd recommend is to use the master issue to keep track of all the files to edit. And then increasingly create issues for each individual file, and label them as "good first issue". Maybe you can work on the first file yourself, so there is a reference PR for the other issues. But this is a lot of work, and too much for a single person in my opinion. And we surely don't want to review PRs with more than one file updated IMHO.
closing this as being done in pieces
BUG: Series.str.extract with StringArray returning object dtype
State: closed

test changes needed in #41372 revealed an issue with the current implementation. fixing for completeness and to add release note.

the dtype for an empty DataFrame from  .str.get_dummies looks iffy so marking this as draft until further investigation.

Labels:
name: Bug
name: Strings
description: String extension data type and string data

Comments:
> the dtype for an empty DataFrame from .str.get_dummies looks iffy so marking this as draft until further investigation.

I can't recall why I originally changed that test in #41372 as there are no columns in the result so the dtype is irrelevant. have removed dtype on the .str.get_dummies test instead.
looks good.
BUG: convert dtypes copies column_index names
State: closed

- [x] closes #41435 
- [x] tests added / passed

tests may be in wrong place.

solution may be the easiest, but it maybe not the best or most robust.


Labels:

Comments:
> solution may be the easiest, but it maybe not the best or most robust.

id suggest doing something like `self._mgr.apply(soft_convert_objects)`
this solution should be probably be deferred then, until more time to understand and develop
TST: Added regression test
State: closed

- [x] closes #41382
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype

Comments:
Thanks for the test @ShaharNaveh
TYP: define `subset` arg in `Styler`
State: closed

This works but someone with more knowledge might be able to point out a better definition of the type for `Subset`,

 i.e. here `Subset = Union[slice, Sequence, Index]`

It types all the `subset` args in `Styler`

Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Styler
description: conditional formatting using DataFrame.style

Comments:
cc @simonjayhawkins if any comments
thanks @attack68 

if you can create a followon issue to try to integrate this to the rest of pandas; e.g. move Subset to _typing and then *use it* in .dropna, and so on (e.g. where ever we use a subset argument), may need to add a todo list
> This works but someone with more knowledge might be able to point out a better definition of the type for `Subset`,

would/could/should this be the same as PositionalIndexer in pandas._typing
REG: quantile with IntegerArray/FloatingArray
State: closed

- [x] closes #39771
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Not present in any released version, so no release note needed as long as before 1.3

Labels:
name: quantile
description: quantile method
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
let's move this off the milestone for now
updated as suggested; since its a regression we should try to get this in for the RC
lgtm. needs a whatsnew note?
> needs a whatsnew note?

no, the regression didnt make it into any released version
cool, do we have an issue for discussion about this (eg. the api enhancement / consolidation of the ea_compat quantile methods) ? if not can you create one
REF: remove name arg from Grouping
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Groupby

Comments:
can you rebase
rebased + green
BUG: always strip .freq when putting DTI/TDI into Series/DataFrame
State: closed

ATM when we put a DatetimeArray/TimedeltaArray/DatetimeIndex/TimedeltaIndex (from here on, just "DTI") into a Series or DataFrame, we drop its .freq for BlockManager-DataFrame cases, but not the other three cases (BlockManager-Series, ArrayManager-DataFrame, ArrayManager-Series).

The long-term behavior is definitely going to always drop the freq (more specifically, DTA/TDA won't _have_ freq, xref #31218).  So this PR standardizes always-dropping.

cc @mroeschke this should unblock your window PR

cc @jorisvandenbossche bc this touches ArrayManager

Labels:
name: Datetime
description: Datetime data dtype

Comments:
looks fine, can you rebase & add a user facing note. ping on green.
ping
[ArrowStringArray] TST: parametrize str.extractall tests
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
thanks @simonjayhawkins 
for future PRs of this sort, it would be somewhat easier to review if you did variable renaming (which I heartily approve of, and in fact would ask you to go one step further and rename `s` to `ser` to avoid 1-letter variable names) in a pre-cursor PR
I think changes I may need to be made to _wrap_result for the expand path in the string accessor so this affects split/rsplit, extract/extractall, partition/rpartition and get_dummies. I want to make sure that all dtypes are tested before that change is made.

so these test parameterisation PRs are 'blockers' for the next step of the pyarrow implementation. So not keen to do precursor to the precursors, so can maybe leave out the other cleanups and just parameterise on dtype? 
we should maybe discuss in today's dev meeting, but I'm getting concerned that I won't get this done for 1.3.

the rc is only a few weeks away and if it doesn't make the rc it'll have to wait for 1.4

In the original PR adding the dtype #27949 there was some discussion on why the StringDtype was even needed and the gist, as I understand it, is that it was added so that the api could be established/tested ready to swap out the memory data model in the future. 

#35169 has an updated discussion where having a parameterised dtype has been proposed (but AFAICT not agreed), leading to breaking api change for users.

so far all the changes merged for ArrowStringArray have the same behaviour as StringArray (I hope). There is draft PRs, that I don't want to merge yet if a) performance using native pyarrow is slower than object fallback, or b) the behaviour of ArrowStringArray is different from StringArray.  Doing this may preclude the simple swap out memory model option. 
> So not keen to do precursor to the precursors, so can maybe leave out the other cleanups and just parameterise on dtype?

that would have the same effect vis-a-vis making review easier
[ArrowStringArray] REF: extract/extractall column names
State: closed


Labels:
name: Refactor
description: Internal refactoring of code
name: Strings
description: String extension data type and string data

Comments:
thanks @simonjayhawkins 

wouldn't object to moar tests
the column naming already seems to be extensively tested in the extract/extractall tests with named groups, unnamed groups, some unnamed groups, one group, two groups etc.

this is a precursor to #41372, same for extractall and then the dispatch to pyarrow kernels and perf opt of the object fallback. will add tests where there appear to be gaps. (but I don't think DataFrame column names or Index/Series names is one of those gaps)
BUG: able to create nonexistent Timestamp
State: closed

- [X] closes #41072 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Timezones
description: Timezone data dtype
name: Stale

Comments:
Upon seeing Github Actions CI, we seem to have a couple of issues with other test cases that invoke methods that eventually invoke the creation of a new timestamp in which one of their timestamps is nonexistent. For one of these, a `nonexistent = 'shift_forward'` parameter in `tz_localize` isn't accounted for in `__new__` and in another, nonexistent isn't directly accounted for in any parameters. A possible way to resolve this would be adding a `nonexistent` parameter to `Timestamp` or potentially to `__new__` that defaults at "raise", but I haven't had a lot of time to think through the feasibility of that. Let me know how you think we should address this.
> A possible way to resolve this would be adding a nonexistent parameter to Timestamp or potentially to __new__ that defaults at "raise", but I haven't had a lot of time to think through the feasibility of that. Let me know how you think we should address this.

We're pretty reticent to add new args to `Timestamp.__new__`, i think always raising is fine
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but it appears that it has gone stale and needs to address the reviews. Going to close due to inactivity but let us know if you'd like to continue and we can reopen.
Bug in read_csv and read_excel not applying dtype to second col with dup cols
State: closed

- [x] closes #35211
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: IO Excel
description: read_excel, to_excel

Comments:
FYI this breaks parsing with a non-dict dtype.
Could you provide an example?
Documentation for read_csv says dtype may be "a type name _or_ dict".
`pandas.read_csv("test.csv", engine="python", dtype="str")` and `pandas.read_csv("test.csv", engine="python", dtype=str)` now error because this code's new calls to .get assume a dict. 
Your test file has duplicate columns too? Will look into this later 
> Your test file has duplicate columns too? Will look into this later

Indeed it does, good eye. That's the unfortunate reality when non-programmer people make documents.

Here's my test file:
[test.csv](https://github.com/pandas-dev/pandas/files/6657781/test.csv)

But of course
```
A,B,B
1,1,1
```
shows it too.
I should have filed an issue first. I've done that now: https://github.com/pandas-dev/pandas/issues/42022
BUG: fix TypeError when looking up a str subclass on a DataFrame with…
State: closed

… DatetimeIndex (#37366)

- [X] closes #xxxx
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
thanks @mortada
BUG: Rolling.__iter__ includes on index columns in the result
State: closed

- [x] closes #40373
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
`self._on` has freq="D" with arraymanager and not with blockmanager, bc in BlockManager we drop the freq when putting a DTI into a block.
Thanks for the explanation @jbrockmendel. Would you consider that a "bug" in BlockManager then?
> Would you consider that a "bug" in BlockManager then?

No.  I think its overwhelmingly likely that the BM behavior will eventually be shared.
Okay I'm going to punt on this until the AM & BM behavior aligns. Probably not worth having a work around for either case.
ill move aligning these up my priority list and ping you when it happens
No worries. This is just a nice to fix issue so no real priority.
looks fine some ci/checks are failing (maybe just rebase)?
Rebased and green.
DOC: Adding Cylon under ecosystem/out of core
State: closed

- [x]  Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This PR adds a link and a description about Cylon to the ecosystem page.  

Github: [https://github.com/cylondata/cylon](https://github.com/cylondata/cylon)
Web: [https://cylondata.org/](https://cylondata.org/)


Labels:
name: Docs

Comments:
CI still red for trailing spaces, `git diff upstream/master` should highlight them if your editor doesn't.
Thanks, @datapythonista that did the trick!
thanks @chathurawidanage
REF: groupby remove _selection_name
State: closed

We use self._selected_obj and self._obj_with_exclusions in different places with no clear distinction as to when to use which.  I _think_ in pretty much all cases we really want obj_with_exclusions, am moving towards pinning that down.

The only non-trivial thing here is getting rid of pinning _selection in [here](https://github.com/pandas-dev/pandas/compare/master...jbrockmendel:cln-selection_name?expand=1#diff-f1ec980d06b0b54c8263f663f767636c3f5921f04f8a7ee91dfc71a2100b05cdL326).  We have no tests that reach that.  I can cook up a case that gets there, but haven't found any cases where removing it actually changes anything.

Labels:
name: Groupby

Comments:
great