BrandonFors/pandas-github-issues huggingface dataset:
BUG: #10445 cannot add DataFrame to empty Panel
State: closed

closes #10445 

I based this bugfix off how DataFrame handles the analogous situation ‒ being intialized to empty and then having a series added (cf. [DataFrame._ensure_valid_index](https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L2167))


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
a more generic fix to this would be nice. IOW, put rename `DataFrame._ensure_valid_index` to `_ensure_valid_axes` and put in `core/generic/NDFrame` (and then `DataFrame/Panel.__setitem__` can call it; Series doesn't need it).

then you can use the properties of the object to make sure that all of the not `info_axis` are valid

you iterate over the `.axes` (and skip the info_axis). For a frame this will yield (0), and a panel (1,2)

Right, good call.  I'll do that this week.

@jreback, so, there might an issue with merging the functionality of both DataFrame's `_ensure_valid_index` and Panel's `_ensure_valid_major_minor_axes` (which was part of this PR).

The DataFrame needs to case the incoming value to `Series` while the Panel one needs to cast to `DataFrame`.

At first I was worried about introducing a circle import, but in fact, I see `generic.py` is already circularly importing `pandas.core.frame` (through the `import pandas as pd` line).  I guess I could special case the two options, but that would be kind of convoluted right?  If I'm special casing for the 2 cases, I might as well keep the original two functions separate.

use `_constructor_sliced` that's what its there for. We rarely want to special case if at all possible.

Hmmm... I'm struggling with this one.

Attempt 2 to move this into `NDFrame` is no dice.  You said above I could use the non-info axes to reindex and "For a frame this will yield (0), and a panel (1,2)".  But actually, the axes I need to reindex is (1) for a DataFrame and (1,2) for a Panel...   

Now that would not be hard to do by just doing a for loop and starting from 1 and incrementing up, but that seems like a convoluted way to tackle this. 

Any ideas?  Moving the functionality up to the base class might save some lines of code and is certainly clever, but keeping the reindexing in the derived classes wins in terms of simpleness.  Am I missing an advantage of moving to the base class?

@Garrett-R can you update according to above comments.

@jreback, I was kind of struggling with moving the functionality up to the base class as per my last comment.  I have time to keep trying (EDIT: meant to say: "I don't have").  If you think it's better to have the more general solution, we should probably close this PR and let the next person do it.

@Garrett-R will leave this open if you find the time to come back to it.

closing for now, if you want to update according to comments, pls reopen

@jreback, I apologize for not responding.  I probably won't end up getting back to it.  Sorry and thanks for your time!
DOC: Add warning for newbs not to edit auto-generated file
State: closed

@jreback, here is the proposed changed we talked about in [this PR](https://github.com/pydata/pandas/pull/10337#issuecomment-111822584) for other fellow newbs that make the same mistake I did.

You said "and more importantly prob a note in `internals.rst` about how/what to change", but I didn't understand that.  `internals.rst` is not auto-generated, right?  So, why would I add a note there?  Perhaps you wanted a note to appear in `internals.html`?  That seems unnecessary since if the user has that file on their machine, they must have intentionally generated it...


Labels:
name: Docs

Comments:
This is where `internals.rst` ends up in the generated docs: http://pandas.pydata.org/pandas-docs/stable/internals.html

That said, I think the warning in the generated file serves this purpose more directly. Warnings in docs tend to get ignored anyways :).

@shoyer, hmmm.. maybe I'm confused.  So are you saying it would be good to have a comment in `internals.html` warning new devs that `internals.html` is auto-generated and they should edit `internals.rst` instead?  (similar to what I just did for `generated.pyx`)

If that's the case, we should probably have a similar comment in all the [generated] HTML files, right?

@Garrett-R No, I think @jreback was thinking you could have the warning "generated.pyx is generated from make_generated.py" in the internals documentation. None of the documentation HTML pages are checked into git, so there's no danger we could get a pull request editing them directly.

HOWEVER, I actually disagree with @jreback on this one. I think the warning you have here is sufficient.

Ah, I see.  Yeah, I agree with you.  I think having the source code warning in `generated.pyx` should be enough.  It's already kind of embarrassing that I didn't realize that `generated.pyx` was generated.

no the issue is that I don't think it's obvious that you need to Python generate_code.py

that instruction should live maybe in the contributing docs as well as in generated.pyx 

Oh yeah, that's true, it's definitely not obvious that you need to run `python2 generatre_code.py` (I had to be told).  

The only situation you'd ever need to know this is if you're actually trying to modify `generated.pyx`, so one option would be to just include it these instructions in both `generated.pyx` and `generate_code.py` (which I've now done), but not add extra noise to the contributing docs.

hmm this should be compat but I guess that's another issue

ok then on the py2

Yeah, it would be nice to have it compatible.

merged via https://github.com/pydata/pandas/commit/758ca05e2eb04532b5d78331ba87c291038e2c61

thanks!
Attempt to fix issue #10366 encoding and categoricals hdf serialization.
State: closed

closes #10366 . 

Probably not quite the right approach but want to run travis. Tests are a bit weak at this point (just testing for non-exceptions). Encoding issues might be relevant to other types besides categoricals (but categoricals raise exceptions when strings to mangled to non-uniqueness).


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Also, let's make a store in 0.16.2. and try it on the current to see if back-compat can be rescued. (but do this after you have it completely working; you might have to try things to 'detect' the encoding, e.g. try to re-encode in 'utf-8' and some such).

I haven't resolved the python2 issue (TypeError: [unicode] is not implemented as a table column) so have put a skip in the test for PY2.

In Python 3.4.3, have successfully created a store in 0.16.2 and read it using the current branch.

@jreback I've squashed. Let me know if there are other comments/concerns. 

can you rebase / update?

Rebased and squashed.

@cottrell can you rebase and update according to my last

@jreback I have rebased and commented above that I did not get very far with the _convert_string_array suggestion you made.

ok will take a look thxs

replaced by #10889
DOC: fixed docstrings for StringMethods ljust and rjust
State: closed

Very minor fix but `str.ljust()` and `str.rjust()` have broken links in the docstrings. 

You can see in http://pandas.pydata.org/pandas-docs/version/0.16.2/generated/pandas.Series.str.ljust.html#pandas.Series.str.ljust that it should be linking to `str.ljust()` instead of `str.right()` (which doesn't exist) 


Labels:
name: Docs

Comments:
thank you sir!
DOC: GH10414 Missing example in NA values in GroupBy
State: closed

closes #10414 


Labels:
name: Docs
name: Groupby

Comments:
@harshnisar see the contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

pls pull current master and rebase. I know its only a trivial change, but this is in conflict as master changed a tiny bit in this section a couple of days ago.

@jreback

Done :)

Looks good now! @harshnisar Last question: can you squash the three commits into one? (see end of this section: http://pandas.pydata.org/pandas-docs/stable/contributing.html#committing-your-code)

@jorisvandenbossche  I know by now, this pull request is more of an effort sink for you than a contribution by me. Won't happen from next pulls.

@harshnisar no, no problem! And thanks!
ENH: Simplify using read_hdf for HDF files with one dataset
State: closed

Allow read_hdf to be used without a key when a single pandas object
is stored in a HDF file.  Raises if multiple pandas objects found.


Labels:
name: API Design
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Small simplification to read_hdf when the HDF file is trivial.

``` python
df = pd.DataFrame([[1,2],[3,4]])
df.to_hdf('test.h5', 'df')
del df
df = pd.read_hdf('test.h5')
```

minor doc point. ping when pushed/green.

@jreback ready

thanks!
BUG: Bug fix implement __reduce__/__setstate__ for Period pickle support
State: closed

Fix Period pickle issue https://github.com/pydata/pandas/issues/10439


Labels:
name: Period
description: Period data type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
can you add some tests pls. you can use `self.round_trip_pickle(..)`

if you want to be adventurous, you could edit `pandas/io/tests/generate_legacy_pickles.py`, adding in an example `Period/PeriodIndex`. Then run this under 0.16.2! which will generate some older pickles. These can be committed and automatically tested.

added a test method in tseries.test.test_period.py

can you update / squash

done, please let me know if what I did was correct.

On Wed, Aug 5, 2015 at 5:19 PM, Jeff Reback notifications@github.com
wrote:

> can you update / squash
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10441#issuecomment-128152118.

ok, if you can squash would be gr8. ping when green.

ok I think I did it, but im not quite sure what to do, I still see the
entire commit history on github but did put in a squashed commit.

On Thu, Aug 20, 2015 at 11:00 AM, Jeff Reback notifications@github.com
wrote:

> ok, if you can squash would be gr8. ping when green.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10441#issuecomment-133041214.

superseded by #10866
BUG GH10425 test_categorical big-endian fix
State: closed

BUG: Changing test_categorical to compare with native byte ordering instead of comparing with little-endian one

closes #10425


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
can you add a whatsnew note in 0.17.0. just push (and squash) and lmk.

I hope now is OK.
Thank you!

Regards,
Jurica

merged via https://github.com/pydata/pandas/commit/7e9026db746d2762c615b32e4e0ddf4cf1cc61e5

thanks!
BUG: GH10365 in interpolate_1d when method is piecewise_polynomial
State: closed

Starting point for closing #10365.
I only have the trivial case. I think this method (and perhaps other methods as well) needs #10383 for it to be useful. 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Added tests for `krogh` and `barycentric`.

can you rebase

Maybe I should just keep the tests for krogh and barbycentric? Piecewise polynomial needs higher order derivatives to be passed in for it to be useful -- otherwise it's just linear interpolation; not sure what a good way would be. 

@kawochen what's the status of this?

Still don't have a good way to pass in the higher order derivatives.  Will close if for now.
BUG: closes bug in stack when index is not unique
State: closed

closes https://github.com/pydata/pandas/issues/10417


Labels:
name: Bug
name: Groupby

Comments:
can you rebase

@jreback rebased

thanks!

@jreback _my_ version of travis build of pandas fails (https://travis-ci.org/behzadnouri/pandas/builds), do u know why is that and how to fix it? 

it seems that pandas have recently made some changes to travis setup, and for whatever reason those changes are not picked up on my side. is there anything that i have to do manually?

hmm, odd, we did remove `pandas/version.py`, but that's should cause an issue on travis. you sure you are based on remote master?

merged version passed ok: https://travis-ci.org/pydata/pandas/builds/74690863

**my** hook of travis fails, not pydata;
i.e. this one: https://travis-ci.org/behzadnouri/pandas/builds
not this: https://travis-ci.org/pydata/pandas/builds

i have rebased couple of times, and `origin/master` is identical to `upstream/master` but still it fails.

no I know. that's a bit odd. I wonder if travis is doing something like migrating infrastructure (and yours is migrating).

@behzadnouri, would the `factorize()` in `pandas.core.algorithms` not work here?

I'm trying to rebase #9023 (which I think addressed the issue of this PR, among other things), and am wondering what was the reason for defining a local `factorize()`.

a categorical is created with factorize
this just takes advantage of a unique index

So it's purely a speed optimization? If so, can it be incorporated in to `pandas.core.algorithms.factorize()`? It seems rather confusing to define a new function of the same name and similar behavior (at least I was confused).

no this is a special case

you normally DO want to factorize even unique indices except in some certain cases

FWIW, I deleted the specialized local `factorize()` as part of rebasing #9023 (for no particular reason other than that I didn't understand why it was necessary), and all tests pass: https://travis-ci.org/pydata/pandas/builds/77865221.
Removed scikit-timeseries migration docs from FAQ
State: closed

Issue #10281


Labels:
name: Docs

Comments:
LGTM, thanks!

Indeed, looks good!  
Only, I think, when we merge this, the scikit-timeseries website should ideally also be updated: http://pytseries.sourceforge.net/ (it still mentions it is the 'plan' to port it to pandas)

Because it is still downloaded quite a lot: only looking at the last of year and from sourceforge, it has 2000 downloads (or is it downloaded from travis ci or the like?)

I agree

If people really are interested in migration they will hit the pandas page, and eventually ask on the tracker so I don't think that link is ok

Anyway, we ourselves can't change that website. So merging this. Thanks @ihoegen !

@pierregm I don't know if we can contact you this way, but: the warning notice on the http://pytseries.sourceforge.net/ website is a little out of date. Would it be possible to update this?
BUG: GH9907 generate_range when start/end has higher resolution than offset
State: closed

To close #9907


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
@jreback any follow up needed for this one?

looks good just want to give a manual later today

merged via https://github.com/pydata/pandas/commit/d62f02b22441e23b4e06c91163f1eb63d3a7092f

thanks!
DOC: DataFrame Properties
State: closed

Added explanations for `axes` and `shape`


Labels:
name: Docs

Comments:
could  you reconcile this with the same properties in `core/series.py`, & `core/generic.py` (we would normally use the `Appender` decorator, but I don't think it works on properties.

Just want to make sure that these descriptions are all consistent.

Updated with changes.

pls squash, otherwise looks good.

can you rebase/squash. 

merged via https://github.com/pydata/pandas/commit/873bbc2ada4f37a51a397cdb3bf3301e522dd236
TST: Use unicode literals in string test
State: closed

Follow-up of #10397. Fix some ugly unicode tests. Shall I fix followings also?
- Remove `compat.u()` completely, because the escaped literal is different from normal unicode literals internally.
- Remove `compat.callable` brought back in v3.2 ([some code](https://github.com/pydata/pandas/blob/master/pandas/core/frame.py#L2307) doesn't use `compat.callable` already)


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unicode
description: Unicode strings

Comments:
should we change ALL usages of `u()`, not even sure how much work that would be (is it even worth it?)

@jreback I think we should not use `u()` further, but no need to change existing impl which currently works. If OK, let me merge this.

ok, that sounds fine. I didn't really search the docs, but if you find any reference saying to use `u()` let's clear that up.

merge when ready

Thanks. I found no description in the doc, but in wiki. Merging this and fix wiki.
- https://github.com/pydata/pandas/wiki/Code-Style-and-Conventions
BLD: remove support for 3.2, #9118
State: closed

closes #9118 


Labels:
name: Build
description: Library building on various platforms

Comments:
woohoo!

unicode literal will be nice :).

gr8! I'll simplify some unicode tests like below after this is merged. 
- https://github.com/pydata/pandas/blob/master/pandas/tests/test_strings.py#L750

Nice!
PERF: parse and timedelta ops improvements, #6755
State: closed

closes #6755 

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
dtype_infer_timedelta64_1                    |   7.0527 | 119.7317 |   0.0589 |
timedelta_convert_string                     |  13.3094 | 108.9200 |   0.1222 |
timedelta_convert_string_seconds             |  18.2626 |  85.0617 |   0.2147 |
dtype_infer_timedelta64_2                    |   9.4047 |   9.4063 |   0.9998 |
timedelta_convert_int                        |   0.1300 |   0.1260 |   1.0315 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [da92dc0] : PERF: timedelta and datetime64 ops improvements
Base   [d8a2f30] : Check for size=0 before setting item
Fixes #10193

```


Labels:
name: Performance
description: Memory or execution speed performance
name: Timedelta
description: Timedelta data type

Comments:
This is a cython conversion of the timedelta parsing routines. Was much more involved that it looked, as the original was using regexes, this is a string parsing state machine in cython.

However, I think this could actually be done in c (or maybe using `char *` pointers in cython), rather than the list build up. This could be another 10x perf improvement on this (my benchmark is the c parser for ISO datetimes).

if any of the c gurus are interested, pls comment!
Add nlargest/nsmallest for DataFrame
State: closed

Closes #7359


Labels:
name: Enhancement
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
is there an open issue in pandas for this?

can you show the usage in the top section (and maybe an example in doc-string)?
maybe also add to the docs (I think its in basics.rst)

Issue is #7359

@jreback added a whatsnew, doc example and docstring example

should this have nan for values that are not the nlargest/smallest? (eg use where rather than loc)

i'm not sure what you mean. can you give an example?

Was thinking of the utility of this, but your impl is better / more intuitive (as it give the correct ordering, not just the subset).

so nvm.

```
In [19]: df.select_dtypes(include=[np.number]).apply(lambda x: Series.nlargest(x,3))
Out[19]: 
    a   c
1  10   2
2   8 NaN
3  11   3
4 NaN   4

In [20]: df.nlargest(3,'a')
Out[20]: 
    a  b   c
3  11  c   3
1  10  b   2
2   8  d NaN

In [21]: df.nlargest(3,'a').reindex_like(df)
Out[21]: 
    a    b   c
0 NaN  NaN NaN
1  10    b   2
2   8    d NaN
3  11    c   3
4 NaN  NaN NaN
```

give a nice squash as well

lemme dust off the ol' rebase

What is the status on this?

@cpcloud needs to update for comments
otherwise lgtm

@jreback @jorisvandenbossche @mrocklin ok 2 merge on passing?

lgtm. just add to API.rst as well

@cpcloud you have some doc build failures in basics.rst (you can see it in the logs of the third travis build). Needs a `pd.` before Series and DataFrame

For the rest, lgtm!

@jorisvandenbossche thanks!

@cpcloud thanks!
BUG: closes bug in reset_index when index contains NaT
State: closed

closes https://github.com/pydata/pandas/issues/10388


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: MultiIndex

Comments:
thanks!
BUG: DataFrame.plot raises ValueError when color name is specified by multiple characters 
State: closed

Derived from #9894. Passing color name with multiple characters results in `ValueError`. Below is the bahavior of current master.

```
# OK
df = pd.DataFrame(np.random.randn(3, 3))
df[0].plot(color='green')
# single green line

# OK
df.plot(color=['green'])
# triple green lines

# NG
df.plot(color='green')
# ValueError: to_rgba: Invalid rgba arg "e"
# -> This should be triple green lines
```

If passed str can be parsed as both single color and color cycle, following error will be raised.
- _"'green' can be parsed as both single color and color cycle. Specify each color using a list like ['green'] or ['g', 'r', 'e', 'e', 'n']"_

Currently, there is no color name which can meet above condition (thus cannot tested).
- http://matplotlib.org/examples/color/named_colors.html


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
cc @TomAugspurger @jorisvandenbossche 

Why is the third case not OK? I would only rasie the error if the length is correct (corresponds with the number of columns, as only then it is ambiguous?)

Ah, I meant NG(raise error) in current master. The fix allows to draw 3rd case with 3 lines in green.

> I would only rasie the error if the length is correct (corresponds with the number of columns, as only then it is ambiguous?)

Currently, the logic only checks whether the input can be parsed as both single color and color cycle, without comparing input length and the number of columns. Because `color='rg'` for 10 columns `DataFrame` is valid as color cycle.

ah, yes :-)

Yes, for the length, you can ignore my comment, as now if the length is shorter it is regarded as a cycle that gets repeated, I see now

@TomAugspurger Can you check this when you have a time?

@TomAugspurger @jorisvandenbossche Looks OK? Let me merge this if there is no objection until this weekend.

Looks ok to me!

:+1: 

Voila!

Thanks. Will back to #9894.
BUG: fix multiple columns as primary key in io.sql.get_schema (GH10385)
State: closed

Closes #10385


Labels:
name: Bug
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Apparantly there was also an bug in the fallback version, it could only handle single character column names ...

@artemyk time we clean this up :-) (I will try to look at your ideas on get_schema and others next week!)

@jreback any remarks?