BrandonFors/pandas-github-issues huggingface dataset:
BUG: iter with readonly values, closes #28055
State: closed

@TomAugspurger wanted to get this in under the wire?

- [x] closes #28055 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Sure.
Fixed the merge conflict.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 bdcab11e234a34be47e4395173b1893033629a5a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #28074: BUG: iter with readonly values, closes #28055'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-28074-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #28074 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: Update whatsnew
State: closed


Labels:
name: Docs

Comments:
Ignoring codecov failure.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 7f6ba6b353e63e004c293cfa4b7d3a23bda4cc5a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #28073: DOC: Update whatsnew'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-28073-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #28073 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
TST: non-strict xfail for period test
State: closed

This was XPASSing on the MacPython builds:
https://travis-ci.org/MacPython/pandas-wheels/jobs/574706922

Labels:

Comments:
LGTM.

Darn though; I'll look at the macpython build and see if the condition can be made more strict
If it's Python 3.5 only it may not be worth digging too much. I'm not sure how long we'll support it once 3.8 is out.
Remove alias in pandas docs for numpy.random.randn
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@datapythonista , Hi Marc
Ok Marc

On Wed, 21 Aug 2019, 10:56 PM Marc Garcia, <notifications@github.com> wrote:

> *@datapythonista* commented on this pull request.
>
> Thanks for the change . Can you replace the randn in the release notes of
> 0.10.0 by np.random.randn too please?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/28068?email_source=notifications&email_token=AKDK7XDMPM3CNJPH22443JLQFW22VA5CNFSM4IOKX2QKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCCJWTYQ#pullrequestreview-278096354>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AKDK7XCDX4QPV5HODH3WTOLQFW22VANCNFSM4IOKX2QA>
> .
>

Superseded by #28082
REF: do extract_array earlier in series arith/comparison ops
State: closed

With this, the middle third of _arith_method_SERIES and _comp_method_SERIES are array-specific and can be refactored out (separate step) to become a) block-wise implementation for DataFrame and b) PandasArray implementation.

This also simplifies the NullFrequencyError handling nicely.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
@jreback gentle ping; this turns out to be a blocker(ish) for fixing #28080.
Types added
Can you expand on `keep_null_freq`? It seems strange to have that specific of an option in a general method like `dispatch_to_extension_op`. When / why do we sometimes convert that to a TypeError?
> Can you expand on keep_null_freq? It seems strange to have that specific of an option in a general method like dispatch_to_extension_op. When / why do we sometimes convert that to a TypeError?

If we have a Series that is dt64, dt64tz, or td64, `ser+1` should always raise TypeError.  But because of the way we wrap/unwrap for dispatch_to_extension_op, we end up operating on a DTA/TDA, which will raise NullFrequencyError on `arr+1`.  In these cases we need to catch NullFrequencyError and re-raise as TypeError.

The exception is when we have a Series[int] and we're adding/subtracting a DTA/TDA/DTI/TDI, in which case NullFrequencyError is the correct thing to raise.
So the complexity comes from having different rules for `op(Series[datetime], int)` and `op(DatetimeArray, int)`? Can / should we align that behavior instead?
> Can / should we align that behavior instead?

The DTA/TDA/DTI/TDI/Timestamp behavior has been deprecated, now just need to be patient
So in the future `keep_null_freq` won't be needed? In that case, +1 on the changes here, if we could add a TODO to remove it once the deprecation is enforced.
sounds good
thanks @jbrockmendel 

yeah the ``keep_null_freq`` stuff is a bit hard to follow, though you have documented it, so I think ok
CI: disable codecov
State: closed

xref #27602

Labels:

Comments:
Give me ~15 minutes here. Doing some debugging of our config.
This may be unnecessary if https://github.com/pandas-dev/pandas/pull/28070 passes.
I’m AFK for a bit.  If codecov problem is resolved, feel free to close this.
No luck just targeting master (400 error again).

Just disabling.
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 cf6b09d94d648b4bc3783eb5fff8b3241600528a
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #28065: CI: disable codecov'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-28065-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #28065 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
DOC: fix punctuation in Timestamp/Timedelta docstrings
State: closed

- [ ] xref #27977
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Add periods to Timestamp and Timedelta doctrings for the following attributes and methods:
```
pandas.Timestamp.resolution
pandas.Timestamp.tz
pandas.Timestamp.ceil
pandas.Timestamp.combine
pandas.Timestamp.floor
pandas.Timestamp.fromordinal
pandas.Timestamp.replace
pandas.Timestamp.round
pandas.Timestamp.weekday
pandas.Timedelta.ceil
pandas.Timedelta.floor
pandas.Timedelta.isoformat
pandas.Timedelta.round
```

I did not find the method for `pandas.Timestamp.isoweekday` this in the Timestamp source code. Was this deprecated ? @datapythonista 

Labels:
name: Docs

Comments:
Looks like this is failing `test_nat_doc_strings` in `pandas/tests/scalar/test_nat.py` - @sparalic can you take a look at that and update accordingly?
@WillAyd I am not sure what's happening now. I fixed the errors associated is mismatching of overlapping methods per the CI logs . However, it still seems to be failing. 
@sparalic the tests are failing because we have apparently tests that check that the docstring of `Timestamp.some_method` and `NaT.some_method` is the same (to ensure they keep being consistent). 
So you will need to make the same changes (adding the periods) for `NaT` as well. You can find those in `pandas/_libs/tslibs/nattype.pyx`
@sparalic see my last comment about how to get this PR passing CI, if you have some time to update it, that would be welcome!
@jorisvandenbossche I am sorry for the delay. I found the culprits and updated them all. I think it should be all set now.
BUG: timedelta64(NaT) incorrectly treated as datetime in some dataframe ops
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@TomAugspurger thoughts?  lots of ops stuff in the works, getting really close to fixing the perf problem
> Could a user hit this from normal code? (i.e. do we need a release note?)

Yes.  e.g. the test this implements incorrectly raises TypeError on master.  Will update with release note.
@jreback gentle ping
Sounds good - merge away

Sent from my iPhone

> On Sep 18, 2019, at 9:46 AM, jbrockmendel <notifications@github.com> wrote:
> 
> @jbrockmendel commented on this pull request.
> 
> In pandas/core/ops/__init__.py:
> 
> > @@ -499,8 +499,19 @@ def column_op(a, b):
>          # in which case we specifically want to operate row-by-row
>          assert right.index.equals(left.columns)
>  
> -        def column_op(a, b):
> -            return {i: func(a.iloc[:, i], b.iloc[i]) for i in range(len(a.columns))}
> +        if right.dtype == "timedelta64[ns]":
> +            # ensure we treat NaT values as the correct dtype
> +            # Note: we do not do this unconditionally as it may be lossy or
> +            #  expensive for EA dtypes.
> +            right = np.asarray(right)
> +
> +            def column_op(a, b):
> +                return {i: func(a.iloc[:, i], b[i]) for i in range(len(a.columns))}
> not sure if there is a perf difference
> 
> Might be worth looking at using iat instead of iloc for perf
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or mute the thread.

lgtm merge on green 
@WillAyd gentle ping
Thanks @jbrockmendel
BUG: retain extension dtypes in transpose
State: closed

I'll have to look through the issues to see what this closes.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> but can you check the output of

It looks like df.dtypes[0] == df.dtypes[1] == df.T.dtypes[0] == df.T.dtypes[1].  Is this the desired behavior?
I believe so.
Could you add a test case for unordered categories with differing orders? https://github.com/pandas-dev/pandas/pull/28048#pullrequestreview-280471818
will do
also see if you can find any issues this solves (e.g. likely Datetime w/timezone transpose we have a couple of issues IIRC) & needs a whatsnew note.
Closing to clear the queue.  The Right Way to do this will be to allow 2D EA.
reopened and rebase
closing in favor of #30091.
Backport PR #27991 on branch 0.25.x (DataFrame html repr: also follow min_rows setting)
State: closed

Backport PR #27991: DataFrame html repr: also follow min_rows setting

Labels:

Comments:
Hmm, no idea why this is failing here. cc @TomAugspurger 

There are some errors related to the change:

```
__________________ TestDataFrameFormatting.test_repr_min_rows __________________
[gw0] linux -- Python 3.7.4 /home/travis/miniconda3/envs/pandas-dev/bin/python
self = <pandas.tests.io.formats.test_format.TestDataFrameFormatting object at 0x7f1cf575e850>
    def test_repr_min_rows(self):
        df = pd.DataFrame({"a": range(20)})
    
        # default setting no truncation even if above min_rows
        assert ".." not in repr(df)
>       assert ".." not in df._repr_html_()
E       TypeError: argument of type 'NoneType' is not iterable
```

That should only happen (the result being None) if `get_option("display.notebook_repr_html")` gives False. Not sure why that would not be the default here (which is hardcoded to True).

But, there are also a bunch of repr tests failing that are not related to the html repr .. 
Hmm strange. I'm able to reproduce on 0.25.x.
OK, the return value of `formatter.to_html` changed on master recently (https://github.com/pandas-dev/pandas/pull/27598), and to avoid to need to backpart that PR as well, just pushed a small fix here.
DOC: Add punctuation to IntervalArray docstrings
State: closed

- [ ] xref #27979
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Added missing period to IntervalArray docstrings as suggested by @datapythonista for the following attributes and parameters:
```
pandas.arrays.IntervalArray.left
pandas.arrays.IntervalArray.right
pandas.arrays.IntervalArray.closed
pandas.arrays.IntervalArray.mid
pandas.arrays.IntervalArray.length
pandas.arrays.IntervalArray.is_non_overlapping_monotonic
pandas.arrays.IntervalArray.from_tuples
pandas.arrays.IntervalArray.set_closed
pandas.arrays.IntervalArray.to_tuples
```

Labels:
name: Docs

Comments:
This was my fault. I did not switch branches. Let me know if I need to fix this again.
(failure is unrelated)
CLN: small ops optimizations
State: closed

@jorisvandenbossche you've mentioned some ops optimizations; any suggestions for things to add here?

Labels:
name: Clean

Comments:
Do you have an idea how much those changes matter? 

(anyway, even if it is almost negligible, I think eg checking the actual dtype with the `is_..._dtype` functions is better (more explicit) than passing the "container that has a dtype" so an improvement anyway)

> you've mentioned some ops optimizations; any suggestions for things to add here?

Not directly related to the ones you are doing here. I need to look it up again (I described it in the issue about the slowdown), but it was mainly about the way we iterate through the columns and how to reconstruct the DataFrame/Series back in the end, not so much the ops itself.
> Do you have an idea how much those changes matter?

Haven't measured these specifically, but I know the is_foo_dtype checks get about a 2x speedup from passing dtype #27224
reverted sparse changes, except for one cosmetic thing that annoyed me
updated to change `if is_extension_array_dtype(right) and not is_scalar(right)` to reverse the order of the two checks.  if we happen to have a str, the `is_extension_array_dtype(right)` can be fairly expensive
Thanks @jbrockmendel
Add support to names keyword in Index
State: closed

- [x] closes #19082
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: API Design
name: Needs Discussion
description: Requires discussion from core team before further action
name: Index
description: Related to the Index class or subclasses

Comments:
Hey @WillAyd I did some changes related to what you've commented, that being said, I'm not a 100% sure that I've understood your second comment.
To clarify my comment it looks like code has been added here to validate the `names` argument when called as part of the Index constructor. However, there also appears to be validation as part of the `MultiIndex` constructor (at least from an end user perspective - haven't dug deep in code).

Would there be a way to align those?

With that said I'm a little hesitant on this change. I understand the goal but it feels weird to have both a `name` and `names` argument as part of the constructor. Have to think through in more detail
> With that said I'm a little hesitant on this change. I understand the goal but it feels weird to have both a name and names argument as part of the constructor.

I'd say that's an issue with the `Index` constructor's API. They're both valid arguments, this just updating the signature / docs to reflect that :)
> They're both valid arguments, this just updating the signature / docs to reflect that :)

I assume that's just because the Index constructor appears to accept anything though, right? Not necessarily that it's valid.

```python
>>> pd.Index([], name="foo")
Index([], dtype='object', name=['foo'])
>>> pd.Index([], names=["foo"])
Index([], dtype='object')
>>> pd.Index([], something="foo")
Index([], dtype='object')
```
It's used if you take the MultiIndex route

```python
In [5]: pd.Index([(1, 2), (3, 4)], names=['a', 'b'], tupleize_cols=True)
Out[5]:
MultiIndex([(1, 2),
            (3, 4)],
           names=['a', 'b'])
```
Sorry I think I went in a circle but coming back to it...I don't see how having both `name` and `names` makes sense from an API perspective. I get we want to do that to achieve symmetry in our inheritance but from an end user perspective this feels like a step in the wrong direction
closing; we want to deprecate names I think, this is going backwards.
DOC: Change document code prun in a row
State: closed

- [x] closes #28026 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @EunSeop, can you fix the CI please, seems like your change is making something fail.
![image](https://user-images.githubusercontent.com/6908807/63394923-e9036700-c3fc-11e9-822d-9e6fe3a2032b.png)
Seems working correctly
@datapythonista strange thing is that this error was not visible (and thus catched) on Azure ..
BUG: rfloordiv with fill_value, closes#27464
State: closed

- [x] closes #27464, closes #26793
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:

Comments:
Also addresses #26793, though needs test for that
test added for #26793
CLN: timeseries in plotting clean up
State: closed

`format_timedelta_ticks` appears both in `timeseries.py` and `matplotlib.converter.py`, and there is no test. Then i look through the code base, it is only used in `matplotlib.converter.py` for `TimeSeries_TimedeltaFormatter`. So I remove the one in `timeseries.py` and add test for it.

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Visualization
description: plotting
name: Clean

Comments:
do you prefer to remove `pos`? @jbrockmendel or parametrize it?
fancy a merge? :D  @TomAugspurger 
Seems OK to me though not super familiar with this part of the code base. Looking at the comment preceding your change, is _format_coord something we can get rid of as well (OK as separate PR - just curious)
lgtm. can you merge master and ping on green.
ping @jreback 
Thanks @charlesdong1991
TST: fix flaky xfail
State: closed

xfail in more specific circumstance

- [x] closes #28011
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Unreliable Test
description: Unit tests that occasionally fail

Comments:
SGTM. https://github.com/pandas-dev/pandas/pull/27715 was backported, so we'll backport this as well.
BUG: Avoid duplicating entire exploded column when joining back with origi…
State: closed

…nal frame

- [x] closes #28005
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Hello @MarcoGorelli! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-15 23:06:12 UTC
Thanks for your help Marc, really appreciate it!
@MarcoGorelli looks like a git merge issue. Can you try fetching and merging master, and then repushing? Let us know if you get stuck.
@TomAugspurger sorry to chase you up on this - I've resolved the merge conflict, is there anything else you'd like changed?
Thanks!
DOC: Corrected file description in read_fwf()
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
Changing the name looks good, but can you please remove the link to Oracle? That document is not a spec or description of the format, it's a description of how to open fixed width files with Oracle Enterprise Data Quality, and I don't think we want to have that. It's ok to have no link if there is no good page to link to.

Thanks!
@datapythonista Link removed. Can you merge?
Needs to pass CI first.
@TomAugspurger , @datapythonista CI Tests passed, please merge!
@isantolin would be really appreciated if you could stop telling us when to merge. Thanks
[CI] Add pip dependence explicitly
State: closed

Close #27374. Get rid of the conda warning.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
@makbigc can you add it to the other CI deps too? If it makes sense to have it for that build, I guess it makes sense to have it in all them (and also `environment.yml`).

Thanks!

CC: @TomAugspurger 
Agreed with doing it for all the env files.
Added in other builds. And pip is already added in environment.yml.
https://github.com/pandas-dev/pandas/blob/e118b1d6b23d1d829c3db05505ef4873f0314e99/environment.yml#L55
Sorry I wasn't very specific @makbigc, but I meant to add it to the dependency files where pip is being used, we don't need it in the files where there are no pip dependencies. For what you say I guess it was only missing in the file you initially changed. Sorry about that.
It doesn't matter. I haven't done a lot of work.
`azure-macos-35.yaml`  is the only file which doesn't specify pip dependence explicitly but use pip to install other packages. 
Thanks!