BrandonFors/pandas-github-issues huggingface dataset:
WIP [ArrayManager] API: setitem to set new columns / loc+iloc to update inplace
State: closed

xref indexing work item of #39146

This actually started as trying to get the tests passing for `tests/frame/indexing`, but while doing that I needed to make decisions on what the behaviour should be for setitem / loc / iloc assignment (so related other ongoing discussions about this). 

Very much a draft, but a good way to see what we actually want / the consequences of those choices.

cc @jbrockmendel 

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Stale

Comments:
This has some overlap with #39510; would that solve this issue?
@jorisvandenbossche is #40380 helpful for this?  if not im going to mothball it
Sorry for the slow reply here. I need to look into it again, but in any case, a part of the test changes of this PR was broken off and is already in master (#40323,  #40325). 

Another part of the code changes here were related to making loc and iloc update in place (https://github.com/pandas-dev/pandas/issues/39584)

I need to look again at #40380 to know if it would be helpful for this. 

(I generally plan to look at the indexing code starting next week, for the copy-on-write exploration)


This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
this is quite old, happen to reopen if actively worked on.
TST/REF: de-duplicate SetitemCastingEquivalents
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
needs a rebase
rebased + green
BUG: DataFrame.append with timedelta64
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

AFAICT several of the existing tests are just wrong, xref #39122 cc @jorisvandenbossche 

Fixes (but havent yet added a test for) https://github.com/pandas-dev/pandas/issues/39037#issuecomment-756748197



Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Can you summarize the behaviour that is changed / bug that is fixed?
> Can you summarize the behaviour that is changed / bug that is fixed?

```
df = DataFrame({"a": pd.array([1], dtype="Int64")})
other = DataFrame({"a": [np.timedelta64("NaT", "ns")]})
result = df.append(other, ignore_index=True)

>>> result
      a
0     1
1  <NA>
>>> result.dtypes
a    Int64
dtype: object
```

```
df = DataFrame(columns=["a"]).astype("Int64")
other = DataFrame({"a": [np.timedelta64(1, "ns")]})

>>> result = df.append(other, ignore_index=True)
ValueError: Wrong number of dimensions. values.ndim != ndim [1 != 2]
```
@jorisvandenbossche helpful comments, thank you.  i found a nice way to retain the None/nan behavior while fixing the original bug: supplementing JoinUnit.is_na with a check that it is a compatible NA.
@jorisvandenbossche i think your comments have all been addressed, lmk if i missed anything
@jorisvandenbossche gentle ping before your day ends
Can you add a whatsnew?
will do, thanks
CI: ipython tab completion tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
thanks!
@meeseeksdev backport 1.2.x
ENH: Styler css Str optional input arguments as well as List[Tuple]
State: closed

- [x] closes #39563 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
this is ready for review..
@jreback changes made, with some better typing throughout the module.

one failed test is unrelated.
thanks @attack68 keep em coming!
STYLE use absolute imports
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

The code style guide reads

> 
> Imports (aim for absolute)
> ==========================
> 
> In Python 3, absolute imports are recommended. Using absolute imports, doing something
> like ``import string`` will import the string module rather than ``string.py``
> in the same directory. As much as possible, you should try to write out
> absolute imports that show the whole import chain from top-level pandas.
> 
> Explicit relative imports are also supported in Python 3 but it is not
> recommended to use them. Implicit relative imports should never be used
> and are removed in Python 3.
> 
> For example:
> 
> ::
> 
>     # preferred
>     import pandas.core.common as com
> 
>     # not preferred
>     from .common import test_base
> 
>     # wrong
>     from common import test_base

yet this isn't checked (nor followed).

So I've made a little package which does this - it's pretty simple and light ([link to source](https://github.com/MarcoGorelli/abs-imports/blob/main/abs_imports.py)), just standard Python libraries (`ast.parse`). Timing on my laptop:
```console
$ time pre-commit run abs-imports --all-files
abs-imports..............................................................Passed

real    0m1.102s
user    0m10.050s
sys     0m0.373s
```

I asked about doing this on the gitter channel and got the response

> in general we go for absolute imports, the exception being in places where that creates circular imports (generally init files)

I didn't find that this created any circular imports (at least, running `./test_fast.sh` worked fine), but `__init__` files can always be excluded if necessary (`exclude: __init__\.py$`)

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
> I didn't find that this created any circular imports

i think this has been an issue in some of the cython code

Does this affect import speed?
> > I didn't find that this created any circular imports
> 
> i think this has been an issue in some of the cython code

So if this is just run on Python files, it should be OK?

> Does this affect import speed?

I didn't notice any difference from several runs of `python -X importtime -c "import pandas"`
thanks @MarcoGorelli very nice.
Added an example for infer_freq
State: closed

Added an example for infer_freq function. The initial index is built using a fixed number of periods to avoid encoding frequency in this index explicitly.


Labels:
name: Docs
name: Frequency
description: DateOffsets

Comments:
Hello @Remit! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-30 18:15:00 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Also this PR should target the master branch instead
let's close this in a couple of days if no response
@Remit I changed the target branch to master, but it will require you to rebase with master
> @Remit I changed the target branch to master, but it will require you to rebase with master

@mroeschke  thanks! I did the rebase, but it seems like some checks fail for the current build. Does not look to be connected with my commit
thanks @Remit 

the failures on default are known
> thanks @Remit
> 
> the failures on default are known

thanks Jeff! Got it.
Fixed comment for pandas.unique
State: closed

Timed pd.unique vs np.unique with 1_000 and 100_000 sequences lengths. In first case np.unique was faster, whereas in the second case pd.unique won.
Hence, I propose to make the doc a bit more accurate.

![image](https://user-images.githubusercontent.com/1339818/106612351-519ac100-6569-11eb-9388-1acfc39ef3a1.png)


Labels:
name: Docs

Comments:
thanks @Remit 
@jreback This was merged directly to 1.2.x and milestoned 1.3

revert on 1.2.x and open PR against master?

oh prob an accident
@Remit if you'd like to re-submit this against master.
#39643
closed by https://github.com/pandas-dev/pandas/pull/39643
> @Remit if you'd like to re-submit this against master.

Ok, did so. Thanks!
Backport PR #39492: DOC: Document how encoding errors are handled
State: closed

Backport PR #39492

Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
```
=========================== short test summary info ============================
FAILED pandas/tests/arrays/categorical/test_warnings.py::TestCategoricalWarnings::test_tab_complete_warning
FAILED pandas/tests/frame/test_api.py::TestDataFrameMisc::test_tab_complete_warning[DataFrame]
FAILED pandas/tests/frame/test_api.py::TestDataFrameMisc::test_tab_complete_warning[Series]
FAILED pandas/tests/indexes/test_base.py::TestIndex::test_tab_complete_warning
FAILED pandas/tests/resample/test_resampler_grouper.py::test_tab_complete_ipython6_warning
= 5 failed, 145676 passed, 4115 skipped, 1098 xfailed, 1 xpassed, 6 warnings in 1275.90s (0:21:15) =
```
CI: numpy deprecation warnings
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 8c35fcd3483e2c8b9b4f96db3c9f898a7fd5d71c
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39551: CI: numpy deprecation warnings'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39551-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39551 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
Backport PR #39546 on branch 1.2.x (CI/TST: update exception message, xfail)
State: closed

Backport PR #39546: CI/TST: update exception message, xfail

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
hmm 3.8_locale failing with an odd one: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=53435&view=logs&j=b1c7b65e-b3ce-541a-7fd5-29b4ba56ce18&t=46ecc253-e38f-5abc-2ea7-addca6b44d0a

but ignoring for now.
CI/TST: update exception message, xfail
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
hmm this is failing the 32-bit build  (though likley to pass all others)
ive got some troubleshooting to do here.  if you're starved for things to review, #39454 #39453 #39446 are collectively blockers for reshape bugs/refactor, #39508 #39502 #39495 are test improvements, and #38992 is a blocker for fleshing out tests.  #39163 is not at all trivial, but i think will lead to appreciably simpler code once it goes in.
> ive got some troubleshooting to do here. if you're starved for things to review, #39454 #39453 #39446 are collectively blockers for reshape bugs/refactor, #39508 #39502 #39495 are test improvements, and #38992 is a blocker for fleshing out tests. #39163 is not at all trivial, but i think will lead to appreciably simpler code once it goes in.

heheh, will look. more trying to have a green CI  :->
green, booyah
nice!
@meeseeksdev backport 1.2.x
CI: pin numpy<1.20
State: closed

- [x] closes #39541
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
great!
merging this: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=53418&view=logs&j=eab14f69-13b6-5db7-daeb-7b778629410b&t=ce687173-08c6-5301-838d-71b2dda24510 still failing but can do after
@meeseeksdev backport 1.2.x
TYP: blocks
State: closed


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
new mypy errors with numpy 1.20

```
pandas/core/internals/blocks.py:252: error: Incompatible return value type (got "Union[ndarray, ExtensionArray]", expected "ndarray")  [return-value]
pandas/core/internals/blocks.py:1538: error: Value of type variable "ArrayLike" of "putmask_inplace" cannot be "Union[ndarray, ExtensionArray]"  [type-var]
pandas/core/internals/blocks.py:1886: error: Item "dtype[Any]" of "Union[dtype[Any], ExtensionDtype]" has no attribute "na_value"  [union-attr]
pandas/core/internals/blocks.py:1972: error: Argument 1 to "can_hold_element" has incompatible type "Union[dtype[Any], ExtensionDtype]"; expected "dtype[Any]"  [arg-type]
pandas/core/internals/blocks.py:2039: error: Incompatible return value type (got "Union[ndarray, ExtensionArray]", expected "ndarray")  [return-value]
pandas/core/internals/managers.py:925: error: Argument 1 to "get_values" of "Block" has incompatible type "Union[ExtensionDtype, str, dtype[Any], Type[object], None]"; expected "Union[dtype[Any], ExtensionDtype, None]"  [arg-type]
```

This is not a criticism. This has been the situation every time numpy types are added. more so recently with many PRs targeted at getting dtype function arguments typed.

Just wanted to highlight that it is preferable to get #39513 resolved soon. please weight in there with your thoughts.
CLN: localising scope of functions in `Styler`
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

`Styler` has a mixture of function placements, e.g. outer scope, or as class methods or inner scope of class methods.

This PR has moved two outer scope functions to inner scope of class methods, where they are only used / relevant for that section, to be more consistent with the rest of the class.



Labels:
name: Clean
name: Styler
description: conditional formatting using DataFrame.style

Comments:
i would rather go the other way. slimming the logic in the class and pulling out function to the module level scope.
@jreback ok, ill close this and if I get a chance submit PRs that try to do that instead.
CI: pin numpy for CI / Checks github action
State: closed

failures with numpy 1.20

https://github.com/pandas-dev/pandas/pull/39517/checks?check_run_id=1803471267

xref #39513 and https://github.com/pandas-dev/pandas/pull/38379#discussion_r567753143

conda-forge should pick up 1.19.5 with these changes.

Labels:
name: CI
description: Continuous Integration

Comments:
Should probably backport these changes until #39513 and https://github.com/pandas-dev/pandas/pull/38379#discussion_r567753143 are fixed and then revert this pin on the backport branch
thanks @simonjayhawkins 
@meeseeksdev backport 1.2.x
CLN: remove unnecessary name in stack_arrays
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@jorisvandenbossche any clues on the new arrow breakage?
numpy 1.20 is incompatible with pyarrow < 3.0 if installed with wheels. I would need to look in the exact details of how the environment is created to check how to solve it though. But in general builds with older pyarrow should also have older (<1.20) numpy.
lgtm pls rebase
ENH: Add I/O support of XML with pandas.read_xml and DataFrame.to_xml…
State: closed

… (GH27554)

- [X] closes #27554 
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry
- [X] io entry
- [X] install entry

To Do List
- [ ] Add `parse_dates` feature for `read_xml`.
- [ ] Add tests for `storage_options`. 
- [ ] Add tests for `ParserError`, `OSError`, `URLError`, etc. 
- [ ] Add `xpath_vars` feature to pass `$` variables in `xpath` expression. See [lxml xpath() method](https://lxml.de/xpathxslt.html#the-xpath-method).
- [ ] Add `xsl_params` feature to pass values into XSLT script. See [lxml stylesheet parameters](https://lxml.de/xpathxslt.html#stylesheet-parameters).
- [ ] Add `iterparse` feature for memory efficient parsing of large XML. See [etree iterparse](https://docs.python.org/3/library/xml.etree.elementtree.html#xml.etree.ElementTree.iterparse) and  [lxml iterparse](https://lxml.de/3.2/parsing.html#iterparse-and-iterwalk).


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
just a note here. we can merge this as long as it passes all tests and is mostly complete. meaning can certainly have a followup issue to tackle small things / corrections / xfails.
cc @twoertwein if you have some time to review
cc @pandas-dev/pandas-core if anyone has time to review
thanks @ParfaitG really nice. please issue PRs for followups when you can. You *may* want to move the list into an issue for tracking.
Thanks @twoertwein for your tremendous help on I/O side!
REGR: Reading csv files with numbers with multiple leading zeros losses a lot of precision
State: closed

- [x] closes #39514
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Trying this change out to see what the tests say. It fixes the current issue I experienced.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@CalebBell if you can merge master and update
Hi Jeff,
My apologies, but this PR seems to break other aspects of CSV parsing. I only proposed this as a PR for the CI, but it failed. I was able to quickly find the code point where the issue is, but fixing this the `right` way will take more work than I can put in right now.

Sincerely,
Caleb
thanks @CalebBell no problem., pls come back on this or other issues!
CLN: Fix userguide deprecation warning new numpy
State: closed

This should get the ipython directives passing

Labels:
name: Docs
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Dependencies
description: Required and optional dependencies

Comments:
I don't think we will want to pin numpy on the backport branch for ci, so should probably backport this.
thanks @phofl 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
BUG: DataFrame.stack not handling NaN in MultiIndex columns correct
State: closed

- [x] closes #39481
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thanks @phofl
Remove repeated keywords (#39503)
State: closed

- [x] closes #39503
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
Thanks @Rasori - merging as the CI failures are unrelated
TYP: Add annotation to assert_frame_equal and assert_series_equal (GH26302)
State: closed

Incremental typing PR for #26302

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking
name: Stale

Comments:
@WillAyd I looked into `NoReturn` since Python should be >= 3.7 for v1.2.0, but it gave me the mypy error shown below. So I used `None` as I normally do. If you have any hints on how to make `NoReturn` work then LMK

Mypy error when using `NoReturn`: `Implicit return in function which does not return  [misc]`
@jreback could you have a look at my comments? Thanks!
cc @simonjayhawkins if you can have a look.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as stale. @avinashpancham please ping if you want to pick this up again.
DOC: add example to insert (#39313)
State: closed

- [x] closes #39313
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry - no 


Labels:
name: Docs

Comments:
Thanks @nofarm3 - the example looks good, but I get a couple of warnings when trying to build it locally (`python make.py --single pandas.DataFrame.insert`):
```
reading sources... [100%] reference/api/pandas.DataFrame.insert                                                                                                                                           
/home/marco/pandas-marco/pandas/core/frame.py:docstring of pandas.core.frame.DataFrame.insert:58: WARNING: Unexpected indentation.
/home/marco/pandas-marco/pandas/core/frame.py:docstring of pandas.core.frame.DataFrame.insert:59: WARNING: Block quote ends without a blank line; unexpected unindent.
```


@MarcoGorelli Thanks Marco, fixed.
I think the current failures are not related to my change.
REF: Move agg helpers into apply
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Refactor
description: Internal refactoring of code
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
looks good. pls rebase; 32-bit failing oddly
thanks @rhshadrach
ENH: Enable parsing of ISO8601-like timestamps with negative signs using pd.Timedelta (GH37172)
State: closed

- [x] closes #37172 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Timedelta
description: Timedelta data type

Comments:
Hello @avinashpancham! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-02-05 22:04:18 UTC
thanks @avinashpancham