BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix MultiIndex column stacking with dupe names
State: closed

- [x] closes #36353
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
thanks @dsaxton yeah this must have been some leftover code from when the warning was removed.
@meeseeksdev backport 1.1.x
BUG: Python Parser skipping over items if BOM present in first element of header
State: closed

- [x] closes #36343
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
@gfyoung if you'd have a look
@jreback's comments notwithstanding, this looks good.  Nice catch @asishm !
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @asishm 
nice. I *think* we might have some other BOM/EOM issues in the tracker if you'd have a look (and maybe this will close them).
@jreback 

I searched for 'bom' and only found #31609 which can be closed - it talks about the same solution as this PR. not really sure what else to search for. (eom doesn't lead to anything)
k thanks @asishm yeah there might be some others but IIRC those are for the c-parser
DOC: Added docstring for storage_options GH36361
State: closed

DOC: Added docstring for storage_options GH36361

- [x] closes #36361
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @kshivi99! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-14 18:07:13 UTC
Thanks @kshivi99
DOC: move release note for #36175
State: closed

xref https://github.com/pandas-dev/pandas/pull/36175#issuecomment-691994652

Labels:
name: Docs

Comments:
test failures. restarting ci

```
=========================== short test summary info ===========================
FAILED pandas/tests/plotting/test_groupby.py::TestDataFrameGroupByPlots::test_series_groupby_plotting_nominally_works
ERROR pandas/tests/io/json/test_pandas.py::TestPandasContainer::test_to_s3 - ...
= 1 failed, 75229 passed, 1809 skipped, 1180 xfailed, 6 xpassed, 67 warnings, 1 error in 1769.60s (0:29:29) =
```
Fix documentation for new float_precision on read_csv
State: closed

- [x] closes https://github.com/pandas-dev/pandas/pull/36228#discussion_r487587534
- [ ] tests added / passed
     N/A
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
     N/A

A followup to PR #36228 to get the documentation string right for version changed. (was in wrong place, missing blank line)



Labels:
name: Docs

Comments:
/azp run

<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jreback @jbrockmendel this is all green - just a doc change so it can be merged
Thanks @Dr-Irv
DOC: Example for natural sort using key argument
State: closed

- [x] closes #36286 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
/azp run
<samp>
Commenter does not have sufficient privileges for PR 36356 in repo pandas-dev/pandas<br>

</samp>
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @erfannariman
ADMIN: Auto-label PRs for review
State: closed

Not certain this will work, but nonetheless I think something like this would be helpful (analogous to "Needs Triage" for issues)

Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
At what point does the "Needs Review" tag get removed?
> At what point does the "Needs Review" tag get removed?

I think it'd be the judgement of whoever is doing the reviewing, but initial thought is to take it down as soon as initial feedback is given (so the label means something like "not yet looked at"; so you could untag this one 😄). I'm imagining a way of making it easier to do "visual triage" when looking at the PR list (and also to have meaningful exemptions if the linked stale PR action gets merged).
That makes sense. I might even think of going further, to untag if the PR "has legs". That is, if it appears to be going in the right direction, even if more refinement is needed.
I'm happy with this in principal since sometimes I look at the PR list for PRs without comments but that misses PRs where the author has added additional comments outside the OP.
sure
ENH/BUG: consistently cast strs to datetimelike for searchsorted
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

ATM we are pretty inconsistent in how we handle strings for (DTI|TDI|PI|Series).searchsorted.  The underlying EAs will cast scalar strings but not Sequene[str].  The Indexes will raise on both scalars and sequences.  Series follows the EA and casts "2016-01-01" but not `["2016-01-01"]`.

This makes us consistently cast in all cases.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
sure, i guess need a whatsnew for this
whatsnew added + green
sure
FYI im planning PRs to make cast_str=True in _all_ of the validator method in that file for internal consistency (and ideally to end up with fewer methods)
PERF: styler uuid control and security
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

`Styler` uuid is randomly generated from 16bytes, or 128bit entropy, which is poorly formatted for data transmission over the web.
`Styler` uuid1 is super-ceded by uuid4 which is more network secure.

This PR addresses the two above items by switching uuid method, and then coding the default entropy to 5 characters (20bit) which should be more than sufficient to avoid HTML table collision on a webpage, but make substantial data transfer savings for large tables.

uuid length remains configurable. 

Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
lgtm. @WillAyd if you have any remaining comments.
Thanks @attack68
REF: use check_setitem_lengths in DTA.__setitem__
State: closed

Turning this into a two-line check will make it easy to add to `PandasArray.__setitem__` and `Categorical.__setitem__`, at which point we'll be able to move these methods up to `NDArrayBackedExtensionArray.__setitem__`

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@jbrockmendel can you add a test for the setitem case with zero-length indexer and non-zero length value (as it is clearly untested ..)
updated per suggestions
Thanks!
BUG: iloc.__setitem__ with DataFrame value, multiple blocks, non-unique columns
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
does this directly close an issue?
> does this directly close an issue?

No, stumbled on this bug yesterday
ERR: Cartesian product error
State: closed

- [x] closes #31355
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Added a test to trigger the reported error using code from the issue. An if statement was included to return a more helpful error message - borrowing the solution from the issue.

Replacement of [this](https://github.com/pandas-dev/pandas/pull/36169) pull request.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @TAJD
[TST]: Groupy raised ValueError for ffill with duplicate column names
State: closed

- [x] closes #25610
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Was fixed in the past. Added test to avoid regression


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-13 20:59:25 UTC
great let's merge on green.
The timeout was probably unrelated?
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
thanks @phofl
PERF: StringArray construction
State: closed

Currently, when constructing through `Series(data, dtype="string")`, pandas first converts to strings/NA, then does a check that all scalars are actually strings or NA. The check is not needed in cases where we explicitly already have converted.

Performance example:

```python
>>> x = np.array([str(u) for u in range(1_000_000)], dtype=object)
>>> %timeit pd.Series(x, dtype="string")
357 ms ± 40.2 ms per loop  # v1.1.0
148 ms ± 713 µs per loop  # after #35519
26.3 ms ± 291 µs per loop  # after #36304
12.6 ms ± 115 µs per loop  # this PR
```

xref #35519, #36304 & #36317.

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
rebase/lint checks
I'd prefer not to have a public argument that allows people to create an invalid object.

> The check is not needed in cases where we explicitly already have converted.

Can you say a bit more / give an example where we're doing the conversion / validation twice? Keep in mind that even if we know that we have all strings, we want to normalize all the NA values to `pd.NA`.
> Can you say a bit more / give an example where we're doing the conversion / validation twice? 

`_from_sequence` converts the input to an object array of all strings (or nans), and then `__init__` validates that the input are all strings (which we know is the case, as `_from_sequence` already ensured the conversion)
Right, Thanks.

Thoughts on refactoring `_from_sequence` to use `object.__new__(StringArray)` and then manually initializing by setting `_ndarray` and `_dtype` (and skipping the conversion / checks we don't need?) My preference would be to keep this out of the public API given the relative immaturity of StringArray.
I'll look into this tonight or tomorrow.
I’ve made a new version.
thanks @topper-123
PERF: constructing string Series
State: closed

Avoid needless call to `lib.infer_dtype`, when string dtype.

Performance example:

```python
>>> x = np.array([str(u) for u in range(1_000_000)], dtype=object)
>>> %timeit pd.Series(x, dtype=str)
344 ms ± 59.7 ms per loop  # v1.1.0
157 ms ± 7.04 ms per loop  # after #35519
22.6 ms ± 191 µs per loop  # after #36304
11.2 ms ± 48.6 µs per loop  # after this PR
```

Similar speed-up is possible for `pd.Series(x, dtype="string")`, but requires some refactorng of `StringArray`, so I'll do that is a seperate PR.

xref #35519 & #36304.

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
Updated.
thanks @topper-123
BUG: Don't overflow with large int scalar
State: closed

- [x] closes #36291
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Series
description: Series data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
thanks @dsaxton 

if you can run the construction asv's just to confirm no regression would be great
@meeseeksdev backport 1.1.x
@jreback Ran series_methods.py asvs (which includes Series construction) and performance improved (probably a false positive):
```
     <master>         <integer-overflow-series>
-         137±4μs          117±3μs     0.85  series_methods.NanOps.time_func('std', 1000, 'boolean')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
REF: Back IntervalArray by array instead of Index
State: closed

The benefit I have in mind here is that we could back it by a single 2xN array and a) avoid the kludge needed to make `__setitem__` atomic, b) do a view to get native types for e.g uniqueness checks, c) possibly share some methods with NDarrayBackedExtensionArray.

Also just in principle having EAs not depend on Index is preferable dependency-structure-wise.

cc @jschendel 

Labels:
name: Interval
description: Interval data type

Comments:
AFAICT remaining test failures are on from arrow
cc @jorisvandenbossche are the pyarrow failures here something we can address on our end?
It are our own tests (testing the conversion that is implemented in pandas itself), so that's something that need to be fixed in this PR.

Looking at the failure, it seems that setting a missing value in an IntervalArray does not/no longer set a missing value in both left and right arrays. But looking at the changes in `__setitem__` in the diff, I don't directly see why that would be the case.
Nice idea, +1 to changing the storage.

We shouldn't change `IntervalArray.left` from an index to ndarray without warning. I'm fine with just continuing to return an Index. Or if we really want to change it we can deprecate `IntervalArray.left` in favor of `IntervalArray.left_array`.
i think comments have been addressed, LMK if i missed anything
this is marked POC, but i assume want to merge it?

> this is marked POC, but i assume want to merge it?

Yes, updated the title.
any more comments here?
> This should improve the performance of IntervalArray.__setitem__, right? That could be added in the release notes.

```
In [2]: ii = pd.IntervalIndex.from_breaks([0, 1, 2, 3, 4]) 
In [3]: ia = ii._data                                                           
In [4]: val = ia[0]                                                             

In [5]: %timeit ia[-1] = val                                                    
38.4 µs ± 1.74 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- master
23.9 µs ± 335 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)  # <-- PR
```

note added to whatsnew.
updated per comments
PERF: get_dtype_kinds
State: closed

Upshot: avoid doing lots of non-performant is_foo_dtype checks.

This fixes a little more than half of the performance hit in the benchmark discussed in #34683.

Labels:
name: Performance
description: Memory or execution speed performance
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
sure, do we have sufficient bencharmarks for this?
ENH: Support MultiIndex columns in parquet (#34777)
State: closed

- [x] closes #34777
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Support MultiIndex for columns in parquet format by updating value column names check to handle MultiIndexes.

Labels:
name: Enhancement
name: IO Parquet
description: parquet, feather

Comments:
Hello @hweecat! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-19 01:05:10 UTC
@jreback Since the enhancements work on pyarrow 0.15.0, we could leave the min pyarrow version as >= 0.15.0 for this pull request.

I've added a whatsnew note on enhancements in 1.2 under "Other enhancements" - feel free to let me know if a more detailed whatsnew note is needed. :)

P.S. Tests are failing after I did a rebase to update my branch.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Test failures look unrelated, looks related to https://github.com/pandas-dev/pandas/pull/37818

```
fsspectest = <pandas.conftest.fsspectest.<locals>.TestMemoryFS object at 0x7fb2b99e5e50>
extension = 'xls'

    @pytest.mark.parametrize("extension", ["xlsx", "xls"])
    def test_excel_options(fsspectest, extension):
        df = DataFrame({"a": [0]})
    
        path = f"testmem://test/test.{extension}"
    
>       df.to_excel(path, storage_options={"test": "write"}, index=False)

pandas/tests/io/test_fsspec.py:133: 
```
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@hweecat small comments, can you merge master and ping on green.
thanks @hweecat and @charlesdong1991 
all credits to @hweecat very nice job!
Any reason why this explicitly raises errors for non-string multi-indicies?
PERF: creating string Series/Arrays from sequence with many strings
State: closed

Improves performance of `pandas._libs.lib.ensure_string_array` is cases with many string elements.

Examples:

```python
>>> x = np.array([str(u) for u in range(1_000_000)], dtype=object)
>>> %timeit pd.Series(x, dtype=str)
344 ms ± 59.7 ms per loop  # v1.1.0
157 ms ± 7.04 ms per loop  # v1.1.1 and master
22.6 ms ± 191 µs per loop  # this PR
>>> %timeit pd.Series(x, dtype="string")
357 ms ± 40.2 ms per loop  # v1.1.0
148 ms ± 713 µs per loop  # v1.1.1 and master
26.3 ms ± 291 µs per loop  # this PR
```

#35519 is the cause of the improvement from 1.1.0 to 1.1.1.

Together with #35519 this PR means that the overhead of working with strings in pandas has gotten considerably smaller for cases when we many times instantiate new `Series`/`PandasArrays` with dtype `str`/`StringDtype`, i.e. probably quite often.

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
nice.  Does this have/merit an asv?
There are ASVs for both `str`and `"string"`in `asv_bench/benchmarks/strings.py`.
thanks @topper-123 very nice!
REGR: Fix IntegerArray unary ops regression
State: closed

- [x] closes #36063
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@jreback @simonjayhawkins Should this change be backported?
will do it after merged the two open ones. if meeks... fails can't retrigger and need to do it manually
@meeseeksdev backport 1.1.x
[DOC]: Improve drop docs
State: closed

- [x] xref#36293

Improved the docstring a bit in case of multi-index and no level given.

Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Stale

Comments:
Before merging we should address the inconsistency betweend unique MultiIndexes, where something like df.drop(index="test") works and non unique MultiIndexes, where the same does not work (#36293)
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
I dont' think we actually want to change the docks, rather fix the bug. closing.
BUG: Index.union() inconsistent with non-unique Indexes
State: closed

- [x] closes #36289 
- [x] closes #31326
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

The else-block was pretty buggy in case of non unique indexes (was dependent on order of indexes and if one was subset of the other). The function in the try block worked perfectly, so I figured it would be way safer to let that function compute the new index and sort it based on the input indexes. If there is a faster way to achieve the resorting, I would be happy to change the code block.

We could delete the second sort block with the RunTimeWarning too, but then we would no longer get a Warning when incomparable objects should be sorted.

Labels:
name: Bug
name: Index
description: Related to the Index class or subclasses

Comments:
Hello @phofl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-03-02 20:43:35 UTC
ping @jbrockmendel ?
are test failures unrelated?
Somehow I do not get emails when pipelines fail since a few weeks...

Failure was related. Seems like something changed for CategoricalIndex, but is fixed now. Previous behavior was confusing
Added a test for the concat issue since the other pr was already merged
Unfortunately I had to change the implementation :( The previous try had two main difficulties:

- nans where counted twice, because of the *outer_join*
- Same element duplicated in both was blown up (This is an issue with the situation on master too. 
```
idx1 = Index([0, 0, 1])
idx2 = Index([0, 0, 1, 2])
idx1.union(idx2)
Int64Index([0, 0, 0, 1, 2], dtype='int64')
```

opposed to 
```
idx1 = Index([0, 0, 1])
idx2 = Index([0, 0, 1])
idx1.union(idx2)
Int64Index([0, 0, 1], dtype='int64')
```

The nan issue were caused through my change.
The implementation on master had problems with duplicates in other, which were unique in self. Non-monotonicity of at least one of both and ``sort=False`` is important.

```
idx1 = Index([2, 1, 0])
idx2 = Index([0, 0, 1, 2])
idx1.union(idx2, sort=False)
Int64Index([2, 1, 0], dtype='int64')
```

The solution handling cases with duplicates in both and duplicates in other if sort is False and index is not monotonic.
I would like to get rid of the align call, but I am not quite sure how to do this. Currently I am getting the maximum number of occurrences from Both Series, but to do this they have to be aligned?
can you merge master. let me look again.
not sure of status here, @phofl if you can merge master
@jreback merged master, last commit is for unused import.

This should fix inconsistencies with duplicates in Index.union
merge master as well
Done
thanks @phofl this took quite some effort, keep em coming!
> thanks @phofl this took quite some effort, keep em coming!

seriously, way to go
Thx, feels good to get it done
Py39 Support - Remove Deprecated Unicode Length Check
State: closed

- [X] closes #36279
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Unicode
description: Unicode strings

Comments:
huh cythonize error
/azp run

<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
lgtm. are the CI failures unrelated? we should genreally backport these 3.9 changes if at all possible.
@WillAyd can you update
Also removed pre per https://github.com/pandas-dev/pandas/issues/36296#issuecomment-693106050
closing as superceded by https://github.com/pandas-dev/pandas/pull/36393