BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix duplicate entries in LaTeX List of Tables when using longtable environments
State: closed

- [x] closes #34360
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO LaTeX
description: to_latex

Comments:
this should target 1.2 as there was a large refactor in this code (which u already did), just move the note 
@ivanovmg if you'd have a look.
> @ivanovmg if you'd have a look.

@jreback, this looks good to me. However, if there are other modifications to longtable in future, we may want to redefine the method ``get_result`` for ``LongTableBuilder``, where we specify more specific order of the items. Note that this PR suggests to include header in the middle separator. This is not very obvious for the reader. But the author left the comment why it is there, so it is OK for now.
lgtm ping on green.
thanks @jeschwar
[BUG]: Fix ValueError in concat() when at least one Index has duplicates
State: closed

- [x] closes #36263
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

If obj_labes Index has duplicates and they are not removed from new_labels before redindexing, they are multiplied. So we would get a way too big index.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
also pls merge master
@jreback merged master
Moved it to algorithms and added a benchmark. I looked through indexing and the with index associated modules but could not find anything, which would help here. The check for uniqueness is not necessary from a technical standpoint, but it should save some time for unique indices.
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@phofl can you resolve merge conflicts
Done
Example:

We have the 2 DataFrames
```
df1 = DataFrame([1, 2, 3, 4], index=[0, 1, 1, 4], columns=["a"])
df2 = DataFrame([6, 7, 8, 9], index=[0, 0, 1, 3], columns=["b"])
```

When looping over ``df1`` (df2 case is similar)
In this case ``new_labels`` before calling ``make_duplicates_of_left_unique_in_right`` will be
```
new_labels=Index([0, 0, 1, 1, 3, 4])
```

while ``obj_labels`` will be
```
obj_labels=Index([0, 1, 1, 4])
```

Simply reindexing would return an indexer which would duplicate the pair ``[1, 1]`` again: ``np.array([0, 0, 1, 2, 1, 2, -1, 3])``. (taking every ``1`` twice)
The new function takes the duplications of the 1 in ``new_labels`` away.

``make_duplicates_of_left_unique_in_right`` returns 
```
new_labels=Index([0, 0, 1, 3, 4])
```

which leads us to the desired reindexing result ``np.array([0, 0, 1, 2, -1, 3])``
Removed the test_concat file, probably got in through merging master the last time
thanks @phofl very nice
Thx, will try to use this for the Index.union with duplicates problems
reverted here: https://github.com/pandas-dev/pandas/pull/38654
DOC: Fixed type hints for GroupBy.{any, all}
State: closed

The current documentation says the return value of GroupBy.all and GroupBy.any is `bool`.

```
    def any(self, skipna: bool = True):
        """
        Return True if any value in the group is truthful, else False.

        Parameters
        ----------
        skipna : bool, default True
            Flag to ignore nan values during truth testing.

        Returns
        -------
        bool
        """
```

However, the actual returned type is DataFrame or Series (with the same shape as `GroupBy.sum()`).

```
In [25]: df
Out[25]:
     0    1    2    3
0  1.0  0.0  0.0  0.0
1  0.0  1.0  0.0  0.0
2  0.0  0.0  1.0  0.0
3  0.0  0.0  0.0  1.0

In [26]: df.groupby(0).any()
Out[26]:
         1      2      3
0
0.0   True   True   True
1.0  False  False  False

In [27]: type(df.groupby(0).any())
Out[27]: pandas.core.frame.DataFrame
```

This PR fixes the wrong type hints for `GroupBy.{any, all}`.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Groupby

Comments:
That is a bit short imho. Could you maybe write something like here ? https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.groupby.GroupBy.sum.html
Okay. I wrote some drafts for `GroupBy.all()` documentation. Which one do you think better?

1. Computed all of values within each group.
    - just replaced `sum` by `all` in GroupBy.sum documentation.
2. Computed all() of values within each group.
    - added parentheses to make it clear that `all` represents somewhat function (that is, it's not a pronoun).
3. DataFrame or Series of boolean values, which is True if all elements are True within each group, False otherwise.
    - wrote whole description.

If you have any alternative, plz tell me.
Hello @ytakashina! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-09 00:30:11 UTC
Edited the description for the return value of GroupBy.{any, all} after receiving @rhshadrach's suggestion.
Seems like a clear improvement over the status quo.  @rhshadrach are you happy here?
@ytakashina Can you merge master? I think it will fix CI.
@rhshadrach CI Fixed!
Thanks @ytakashina!
TST: #31922 assert no segmentation fault with numpy.array.__contains__
State: closed

- [ ] closes #31922
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] new test cases


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Could you merge master again? Should fix the CI failures
thanks @ylin00 
see comments on the issues. this appears to be failing on older numpies; its ok to skip those in the CI.
Concatenating rows with Int64 datatype coerces to object
State: closed

- [x] closes #24768
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Issue was fixed in the past.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
The test itself looks good just not sure on location - @jreback should this go in pandas/tests/extension/test_integer.py::TestReshaping or is that intended for something else?
I added it there, because directly above is a similar test. If we move the new test, we should move that one too.
nah this looks like a good location. thanks @phofl
ENH: add peek function
State: closed

- [x]  closes #18691
- [x] tests added / passed
- [ ] passes black pandas
- [ ] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry

added peek, which is essentially a concatenation of head and tail functions. The reason I add, is because often for sorted data, you are interested in looking at both the head and tail simultaneously to see the min/max range.

Labels:
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
Hello @Quetzalcohuatl! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-10 22:26:09 UTC
This has been discussed previously in #18691 - not sure we want to add this to the API
yeah -1 on adding this. We *already* already pretty much show this when doing a repr.
thanks @Quetzalcohuatl plently of formatting issues if you would like to tackle.
@jreback We have `head` and `tail` even though you can _already_ already show that by doing `.iloc[n:]`. I think this would be a very useful option to add instead of having to type the very long `with pd.option_context('display.max_rows'=50)`... I can never remember the phrase to use, and this would be a quick and easy way to do it. Even in the linked issue someone made a function `def show` that they use in every notebook. Nobody wants to be copy pasting the repr every time they want to show different amounts of rows. Sometimes you only want head(1) and tail(1), sometimes you want head(25) tail(25) which will fit within the default 50 rows displayed in Jupyter Notebook. Please reconsider.
@Quetzalcohuatl see the linked issue - happy to take an example function (see @TomAugspurger example) / doc string update 

this is useful but we cannot have everything
REF: implement Categorical._validate_listlike
State: closed

I found it really confusing why we used slightly different dtype-comparisons in a bunch of different places (e.g CategoricalIndex.get_indexer vs CategoricalIndex.get_indexer_non_unique).  AFAICT It's just a matter of optimizations accruing over time without a helper method to keep them in sync. 

Labels:
name: Categorical
description: Categorical Data Type
name: Clean

Comments:
i doubt this makes any perf difference, but maybe worth checking.
if you can check perf, ping when ready
```
In [3]: dti = pd.date_range("2016-01-01", periods=1000).repeat(2)                                                                                                                 
In [4]: ci = pd.CategoricalIndex(dti)                                                                                                                                              

In [5]: %timeit res = ci.get_indexer_non_unique(dti)                                                                                                                              
8.17 ms ± 96.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
7.8 ms ± 122 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- PR

In [8]: dti2 = dti.copy(deep=True)                                                                                                                                                
In [9]: np.random.shuffle(dti._data._data)                                                                                                                                        
In [10]: ci2 = pd.CategoricalIndex(dti2)                                                                                                                                          

In [12]: %timeit res = ci.get_indexer_non_unique(ci2)                                                                                                                             
4.06 ms ± 78.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <-- master
991 µs ± 3.31 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR
```
thanks @jbrockmendel
Fix issue #36271 to disambiguate json string
State: closed

pd.read_json() fails currently for strings that look like fsspec_url and contain "://". adding another condition to fix this at least in most cases

- [X] closes #36271
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Stale

Comments:
Hello @tbachlechner! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-11 15:02:53 UTC
Does this close an issue? Can you add a test?
@jreback yes, this ought to be caught, pd.read_json() is claimed to accept a json string, but it fails for any json string containing `://`- which is a lot of json strings. For now we pinned the version pandas==1.0.5. I'm busy shipping products, but I'll see if I get to a fix at some point. The underlying issue though is that `is_fsspec_url()` should not be returning `True` when it is not a URL, removing json is just a hack that probably does not fix other problems that likely exist.
I agree that `is_fsspec_url()` should be better, and clearly is failing for this JSON case. 
Does the regex I suggested earlier ( `"^\s*[\[{]"` ) not suffice?
mypy complaint
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
are there opinions about moving `read_json(json_string)` into a function like `parse_json(json_string)` for an API-breaking 1.2+ release (are there more `read_` functions that would need `read_` vs `parse_`)? Something like:
```py
def parse_json(json:str, **kwargs) -> DataFrame:
    buffer = StringIO(json)
    data = read_json(buffer, **kwargs)
    buffer.close()
    return data
```
and a deprecation warning for 1.2 when passing a json string to `read_json`.

edit:
`read_excel`  has the same 'issue', it accepts bytes.
> are there opinions about moving read_json(json_string) into a function like parse_json(json_string)

IIRC there was a discussion at some point of making a `read_jsons` mirroring the stdlib's `json.load` vs `json.loads` (or maybe the discussion was about pickle?)
it wouldn't be unreasonable to deprecate passing a str (that is not a path) to read_josh and instead only accept a StringIO

i don't think this is much of any issue for pickle as it needs bytes and so str always means a file

excel i think is the same

only read_josh accepts str as an actual transmission format (read_csv as well of course is similar but i think we are restrictive there already)
closing as stale. if you would like to continue, pls ping.
CI: xfail failing parquet test
State: closed


Labels:
name: CI
description: Continuous Integration

Comments:
is there an issue ref for this?
> is there an issue ref for this?

no
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 dc1c849f0cf4f1b8c5602a54f5231f6b57d1d913
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36272: CI: xfail failing parquet test'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36272-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36272 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@WillAyd changing the milestone and not backporting. this test is not failing on 1.1.x
@jbrockmendel OK by you?
yep
Do we have an issue to track this? Could anyone debug if it just the test setup (moto or something?) that was causing us to return empty dataframes, or do we have an actual issue?
FIX: allow sorting by tuple of column names
State: closed

- [ ] closes #36268
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Turns out that ``by=(some, tuple)`` is reserved to sorting by multilevel columns.
Thus, these changes are not to be implemented.
CLN: pandas/io/parsers.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Slight clean-up of ``pandas/io/parsers.py``.

Labels:
name: IO CSV
description: read_csv, to_csv
name: Clean

Comments:
thanks @ivanovmg nice!
BUG: fix isin with nans and large arrays
State: closed

Does a np.isnan if nan is given to isin and we have a large enough array to trigger the `np.in1d` path
- [x] closes #22205
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Seems tests are failing, but it seems unrelated to this?
> Seems tests are failing, but it seems unrelated to this?

Yes, those are unrelated
Thanks @Hanspagh, can you add a release note for 1.1.3?
Changes as requested, let mere know if there is anything else needed
Updated as requested
Fixed
Done
I can do a rebase if needed?
> I can do a rebase if needed?

git pull will normally get the changes from the commit suggestion locally.

to update the PR with the latest changes on master see https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#updating-your-pull-request
thanks @Hanspagh 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 aed64e85eb17edb0e55013868b1aa4e44e977a36
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36266: BUG: fix isin with nans and large arrays'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36266-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36266 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
https://github.com/pandas-dev/pandas/pull/36385#issuecomment-694275310 could have been responsible for not being able to auto backport
Resample fix dst transition
State: closed

Fix bug when resampling with DST transition.

- [x] closes #35219
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Timezones
description: Timezone data dtype
name: Resample
description: resample method

Comments:
thanks @Flix6x very nice. happy for you to tackle other resample issues (or any others!)
ENH: consistently cast strings for DTA/TDA/PA.__setitem__
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

ATM we cast strings for listlike inputs but not for scalars.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
I guess this is technically a bug fix on DTA, maybe add a note; ping on green.
ping
thoughts on if we want to support this in Series.__setitem__ etc?  that's pretty easy, just define DatetimeLikeBlock._can_hold_element to wrap DatetimeLikeArray._validate_setitem_value xref #36226
CLN: _wrap_applied_output
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Groupby
name: Clean

Comments:
Changing `v` to `first_not_none` is a clear improvement.  the other rearrangement im fine either way
@jbrockmendel: Thanks for the review. Regarding the rearrangement, I thought it was better to separate "not a series" and "is a series" cases. Before they were (needlessly IMO) intertwined.
thanks @rhshadrach yeah groupby has built up over the years to be long / complicated functions, thanks for working on this.
BUG: na parameter for str.startswith and str.endswith not propagating for Series with categorical dtype
State: closed

- [x] closes #36241
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I parametrized dtype in existing tests. Should I create a separate test instead?

Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: Categorical
description: Categorical Data Type

Comments:
thanks @asishm 
@meeseeksdev backport 1.1.x
DOC: Fixed a broken JSON Table Schema link
State: closed

Fixed a link to JSON Table Schema specification hosted on https://specs.frictionlessdata.io/table-schema/

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Great thanks
DOC: Update numeric.py
State: closed

changed set to list as it is more appropriate

- [x] closes #36239 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Since this is now a duplicate of the other PR I think it can be closed?
> Since this is now a duplicate of the other PR I think it can be closed?

How long should it take to get published in the release? 29 September ??
The changes in the other PR would likely be reflected in 1.2 which is slated for end of November, but I don't know if that's a firm date. Closing this as a duplicate of the other.