BrandonFors/pandas-github-issues huggingface dataset:
BUG: inconsistent name-retention in Series ops
State: closed

- [x] closes #33930
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As a follow-up I think we can remove a bunch of name-based tests from tests.arithmetic.

Labels:
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
great. can you add a whatsnew note and ping on green. i would specific mentions of broken operators is ok (or is this *all* binary ops)?
ping
ENH: allow for top and mid-level assignment to DataFrames with MultIndex columns #7475
State: closed

- [x] closes #7475 (related to #35727)
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This allows for one to assign to the top and mid-levels of DataFrame columns. To do so, it adds another fallback execution path to `__setitem__`. The logic is:
- In `__setitem__`, if the other existing paths don't apply, there is a single key, and the columns are a MultiIndex, pass to `_setitem_multilevel`. Here, assign if the key exists in the columns. Otherwise, use the new function `append_block` to assign multiple columns at once.
- In the existing `_setitem_array`, as a fallback, if the length of the assigning key equals the length of the top level of the value columns, recurse these back onto `__setitem__`. This allows for higher level list assignments.

I've also added some tests of the functionality in various cases. Any comments much appreciated.

Labels:

Comments:
Thanks for the comment @arw2019. Just added in some new tests for that exception path. I believe there is full coverage now. The only weird thing I found was that `indexing/multiindex/test_insert.py` expects assigning one column to a multiindex with insufficient key levels to work regardless, so I commented out what would have been a conflicting test on my side.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Sorry for the delay here, finally brought this up to date again. Had to change a few things to work well with the changes in #38459. Basically this is now a sub-path of `_set_item_frame_value` that kicks in when the `key` is not in the existing columns and when it is not a fully specified column.
closing as stale, if you want to continue working, please ping.
ENH: Add headers paramater to read_json and read_csv
State: closed

- [x] closes #36688
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This adds the option to specify headers when reading a csv or a json file from an URL in Python3.
Let me know if new tests are needed.

Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: IO JSON
description: read_json, to_json, json_normalize
name: Stale

Comments:
Hello @Antetokounpo! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-03 17:20:18 UTC
I'm not sure we would add new arguments to the function signatures like this. There has been some updates to the original issue and I think would prefer to just document how this can be handled instead of changing these functions - can you adjust the PR accordingly?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Closing in favor of #37966
BUG: Segfault with string Index when using Rolling after Groupby
State: closed

- [x] closes #36727
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The segfault was caused by ``self.obj.index.asi8=None`` when Index is a string Index. ``self._on.asi8`` solves that issue.

Additionally I noticed, that ``obj.index`` was already sorted, so the insert of ``extra_col`` mixed up the order. We should use ``self.obj.index``.

I will add a whats new after #36689 is merged.

cc @mroeschke 

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @phofl! I suspect we may need to be using `self._on` in a lot more places as I don't think our test coverage of the `on` keyword is really extensive 
hah  you already mentioned

> I will add a whats new after #36689 is merged.
> I will add a whats new after #36689 is merged.

@phofl 



thanks @phofl very nice
needs a release note
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 3a63fedefe9c4f06c66242c07a211e1bbcd0f596
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36753: BUG: Segfault with string Index when using Rolling after Groupby'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36753-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36753 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jreback  Thanks.

Sorry was busy today. Should I open another PR for a Release Note? cc @simonjayhawkins
Backport PR #36706 on branch 1.1.x (CI: npdev new exception message)
State: closed

Backport PR #36706: CI: npdev new exception message

Labels:
name: CI
description: Continuous Integration

Comments:
test_basic_series_frame_alignment failing 
restarting ci, seen before e.g. https://github.com/pandas-dev/pandas/pull/35543#issuecomment-674777443
travis failures not seen b4 so restarting also

```
=========================== short test summary info ============================
FAILED pandas/tests/test_downstream.py::test_pandas_gbq - AttributeError: mod...
FAILED pandas/tests/util/test_show_versions.py::test_show_versions - Attribut...
ERROR pandas/tests/io/test_gbq.py - AttributeError: module 'google.protobuf.d...
ERROR pandas/tests/io/test_gbq.py - AttributeError: module 'google.protobuf.d...
= 2 failed, 73402 passed, 1363 skipped, 1050 xfailed, 6 xpassed, 211 warnings, 2 errors in 1105.01s (0:18:25) =
```
Standardize cast_str behavior in all datetimelike fill_value validators
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This will in turn allow us to simplify/de-duplicate a bunch of code since we now pass cast_str in all cases.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
can you actually just remove this parameter entirely now? (e.g. its always true), or other cases?
> can you actually just remove this parameter entirely now? (e.g. its always true), or other cases?

yes, planned for the next pr
actually, since i need to rebase anyway, might as well now
lgtm ping on green
ping
@jreback gentle ping; this is a blocker for long-awaited code de-duplication
BUG: DTI/TDI.equals with i8
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Closing while i troubleshoot slow-test failures
sure
TYP: some more static definitions of methods for DatetimeIndex
State: closed

xref #32100, https://github.com/pandas-dev/pandas/issues/31160#issuecomment-701244665

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
web and doc fails
```
2020-09-30T14:45:19.4780323Z /home/runner/work/pandas/pandas/pandas/core/indexes/datetimes.py:docstring of pandas.DatetimeIndex.rst:122: WARNING: autosummary: stub file not found 'pandas.DatetimeIndex.to_julian_date'. Check your autosummary_generate setting.
2020-09-30T14:45:19.4786910Z /home/runner/work/pandas/pandas/pandas/core/indexes/datetimes.py:docstring of pandas.DatetimeIndex.rst:122: WARNING: autosummary: stub file not found 'pandas.DatetimeIndex.isocalendar'. Check your autosummary_generate setting.
```
Thanks @WillAyd and @jbrockmendel for the review.

I'll do two commits, one to remove the decorator factory and another to remove the actual decorator. I think the static approach that I did originally is clearer, especially considering the handling of the different return types.

I raised this PR in connection with https://github.com/pandas-dev/pandas/issues/31160#issuecomment-701244665 to be more like https://github.com/pandas-dev/pandas/issues/31160#issuecomment-699193016


> I think the static approach that I did originally is clearer,

and more error prone. Looks like made an error originally for `to_period`. but looks like can use `_simple_new_` for the different return types instead of the using the `Index` constructor, so maybe small perf improvements.

I think adding the doc decorator to each method could also be error prone, but at the same now allows substitution so could have separate html docs for Index and array methods.

LGTM
sure
TST: honor encoding in read_fwf for memory-mapped files
State: closed

- [x] closes #23254
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry: not needed

`encoding` was already working for memory-mapped files in `read_fwf`.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv
name: IO Fixed Width
description: read_fwf

Comments:
Hello @twoertwein! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/tests/io/parser/test_read_fwf.py`](https://github.com/pandas-dev/pandas/blob/66666cb1ee59daff22a8dc79cddbab387ea21627/pandas/tests/io/parser/test_read_fwf.py):

> [Line 640:1](https://github.com/pandas-dev/pandas/blob/66666cb1ee59daff22a8dc79cddbab387ea21627/pandas/tests/io/parser/test_read_fwf.py#L640): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace


@twoertwein rebased this in the browser and i think introduced an issue, can you rebase and ping on green. thanks!
thanks @twoertwein
TST: read binary file objects with read_fwf
State: closed

- [x] closes #18035
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry: not needed

`read_fwf` has already supported reading binary file objects. Added the test case from the issue.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv
name: IO Fixed Width
description: read_fwf

Comments:
thanks @twoertwein
DOC: Format more code blocks
State: closed

Actually, contrary to https://github.com/pandas-dev/pandas/pull/36700#issuecomment-700957831 there is quite a lot more clean up to do (so maybe could make for some good first issues @simonjayhawkins), it's just that other docs contain blocks that blacken-docs can't parse (e.g., raw output or ipython code containing magic commands). A workaround is to comment them out, parse the doc, and then uncomment (the commented blocks would then have to be edited manually).

Is it possible to render blocks like this as pure code in rst which are allowed to raise?

```python
   .. code-block:: python

       >>> pd.DataFrame(np.random.randn(10, 2)).to_hdf('test_fixed.h5', 'df')
       >>> pd.read_hdf('test_fixed.h5', 'df', where='index>5')
       TypeError: cannot pass a where specification when reading a fixed format.
                  this store must be selected in its entirety
```

Labels:
name: Docs

Comments:
> Actually, contrary to [#36700 (comment)](https://github.com/pandas-dev/pandas/pull/36700#issuecomment-700957831) there is quite a lot more clean up to do (so maybe could make for some good first issues @simonjayhawkins ), it's just that other docs contain blocks that blacken-docs can't parse (e.g., raw output or ipython code containing magic commands). A workaround is to comment them out, parse the doc, and then uncomment (the commented blocks would then have to be edited manually).
> 
> Is it possible to render blocks like this as pure code in rst which are allowed to raise?
> 
> ```python
>    .. code-block:: python
> 
>        >>> pd.DataFrame(np.random.randn(10, 2)).to_hdf('test_fixed.h5', 'df')
>        >>> pd.read_hdf('test_fixed.h5', 'df', where='index>5')
>        TypeError: cannot pass a where specification when reading a fixed format.
>                   this store must be selected in its entirety
> ```

yes you can add ```:okexcept:``` though to be honest it doesn't look as nice I think (maybe there is a way to limit the traceback)
thanks @dsaxton
CI: disable ARM build
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #36719

Labels:
name: CI
description: Continuous Integration

Comments:
could use some help with the config.  tried to match the older version, but looks like it isnt running the allow-fail build
> could use some help with the config. tried to match the older version, but looks like it isnt running the allow-fail build

just leave the original AND the new and it will work; e.g. it is a copy that's *also* in allow_failures.
EA: tighten TimedeltaArray._from_sequence signature
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #36718

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
I am not sure this actually makes sense. Based on the discussion in #36718 (but we should first futher discuss that), we would rather make `_from_sequence` the non-strict version, so the other way around as you did here (now, it's only naming, so easy to change later of course)
@jorisvandenbossche there isn't a whole lot of discussion in #36718.  did you mean #33254?
> there isn't a whole lot of discussion in #36718. did you mean #33254?

Yes, indeed, sorry for the typo ;) 
Will take a look at your comment over there now!
[BUG]: Rolling selected too large windows with PeriodIndex
State: closed

- [x] closes #34225
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

``asi8`` of PeriodIndex is not in nanoseconds, so we can not set ``self.window=freq.nanos``

Labels:
name: Bug
name: Window
description: rolling, ewma, expanding

Comments:
Test failure seems unrelated?
@jbrockmendel 
Should we check, if the divison result is convertible to an integer?
can you rebase
@jreback done
thanks @phofl
DOC: Update roadmap for completions
State: closed

Moves "Improved documentation" to completed section.

I don't think any others can be marked as done yet, though "Numba-accelerated operations" may be getting close (is that right @mroeschke?).

Labels:
name: Docs
name: Roadmap
description: A proposal for the roadmap.

Comments:
Regarding Numba, pretty much complete in reference to the `and in groupby and window contexts` line in that section.
Will need to see if it's still relevant for the non groupby and window contexts
ci/../doc/source/development/roadmap.rst:193:Heading capitalization formatted incorrectly. Please correctly capitalize "Completed Items" to "Completed items" 

Fixed.
great
CLN: private funcs in concat.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Refactor/cleanup ``_get_empty_dtype_and_na`` in ``pandas/core/internals/concat.py``
Extract functions, add typing.


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
Thanks @ivanovmg
TYP: Ignore remaining mypy errors for pandas\tests\*
State: closed

- [ ] closes #28926
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


These are the only 3 errors outstanding in pandas\tests\* that are stopping us closing #28926

```
pandas\conftest.py:301: error: List item 0 has incompatible type "Type[Index]"; expected "Type[PandasObject]"  [list-item]
pandas\tests\window\conftest.py:79: error: List item 0 has incompatible type "ParameterSet"; expected "Sequence[Collection[object]]"  [list-item]
pandas\tests\window\conftest.py:330: error: List item 15 has incompatible type "ParameterSet"; expected "Sequence[Collection[object]]"  [list-item]
```

these appear to be from a mypy inference issue. It maybe best to `# type: ignore[list-item]` just these lines and remove the `ignore_errors=True` from setup.cfg that applies to the whole file for all error codes.

if the mypy error is resolved in a future release, the `warn_unused_ignores = True` in setup.cfg means that we will be alerted and the ignores can be removed.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @simonjayhawkins
CI unpin flake8, only run flake8-rst in pre-commit
State: closed

- [ ] closes #34150 (maybe?)
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Turns out this can be simplified - no need for a fork/mirror, we can just pin `flake8-rst` and `flake8` in `additional_dependencies` and run it as a local hook
The new pip dependency resolver will become default soon https://discuss.python.org/t/announcement-pip-20-2-release/4863. I tested it earlier. It got stuck if `flake8` is pinned using `<` sign. 

Also
```
The conflict is caused by:
    The user requested pyflakes>=2.2.0
    flake8 3.7.9 depends on pyflakes<2.2.0 and >=2.1.0
```

So we probibly need to merge this before pip 20.3 release.
can you merge master and ping on green.

Is there a way to have an option in code_checks.sh that actually runs the appropriate pre-commit hook? that way we have a single source of truth, and you can actually still run them (of course have to *avoid* running these on the CI from the script as well).
Sure, done

> Is there a way to have an option in code_checks.sh that actually runs the appropriate pre-commit hook? 

Have opened #36956 for this
lgtm. can you merge master. merge on green (just to be sure nothing broken since this).
thanks @MarcoGorelli
CLN: Remove unnecessary rolling subclass
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
Does the affect Rolling._constructor?
lint complaint about importing private _Window
> Does the affect Rolling._constructor?

Thanks for the catch, added the property directly to `Rolling`
thanks @mroeschke
EA: Tighten signature on DatetimeArray._from_sequence
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref #33254

There are a few more steps to get DatetimeArray._from_sequence down to the ideal behavior, this is just trimming the signature down to what its supposed to be.

_from_sequence_not_strict could use a better name

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
thanks
CLN: use standard method to set unhashable object
State: closed

#20589 mentioned lgtm. So I decided to test it. And it found this potential improvement. There is even a detailed reference with code examples. https://lgtm.com/rules/1780095/ Very interesting.

Just trying to see if their suggestion works.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ci checks is failing as well
> ci checks is failing as well

`pandas/core/generic.py:1767: error: Incompatible types in assignment (expression has type "None", base class "object" defined the type as "Callable[[object], int]")  [assignment]`

If __hash__=None is the desired way to indicate unhashable object, this could be a mypy issue.

xref https://github.com/python/mypy/issues/4266
close for now. It doesn't work with mypy.
CI: remove duplicated code check #36642
State: closed

- [x] closes #36642


Labels:
name: CI
description: Continuous Integration

Comments:
I wonder if it's also possible to move flake8-rst into pre-commit, perhaps as a local hook. Maybe @MarcoGorelli knows?
xref #34150 flake8-rst is not under active development anymore. It requires flake8<3.8.0
Is flake8-rst the only thing blocking us from upgrading flake8? If so, I'd say let's fork it, run it in pre-commit (so it's in its own environment), and no longer pin flake8. Then later we can replace it with a different check (if necessary), but for now this would ensure the same checks will be run

EDIT
----
taken forward in  #36722
@simonjayhawkins I think should back port? 
Sorry to backtrack, but TBH I'd appreciate if we could wait a little before merging this.

I noticed in #36978 that isort, when run in pre-commit, wasn't picking up `.pxd` files (which the newest version of isort supports), even though they're picked up when running `isort` from the terminal*. So first, I think that a version of #36978 which passes both code_checks and pre-commit should be in.

*this is because isort's `pre-commit-hooks.yaml` specfiies `types: [python]`, while `identify` picks up `.pxd` as `cython`. 
> @simonjayhawkins I think should back port?

I _think_ better to keep the backport without the newer style checks. If we release 1.2 in November, 1.1.4 maybe the last in the 1.1.x Series. so maybe not that many more regressions to backport now anyhow.
Hi @fangchenli 

We now have two separate hooks for isort:
```
    -   id: isort
        name: isort (python)
    -   id: isort
        name: isort (cython)
        types: [cython]
```
so IMO it's OK to remove it from `code_checks`. Can you fetch and merge upstream/master?
Can you merge master to get CI green?
thanks @fangchenli