BrandonFors/pandas-github-issues huggingface dataset:
BUG: stacking with MultiIndex column with some unused level uniques fail...
State: closed

...s #3170


Labels:

Comments:
Using the example from #3170, seems to be a rogue column: :S ?

```
In [15]: df.unstack('ID').resample('W-THU').stack('ID')
Out[15]: 
Params        NAME  VAR1
DATE       ID           
2013-01-03 1   NaN  69.0
           2   NaN  69.0
2013-01-10 1   NaN  76.0
           2   NaN  73.5
```

oops you're right. i'll put in a fix

One last thing, ID stack loses the `column.name`:

```
In [14]: df.unstack('ID').resample('W-THU').stack('Params')
Out[14]: 
ID                  1     2
DATE       Params          
2013-01-03 VAR1    69  69.0
2013-01-10 VAR1    76  73.5

In [15]: df.unstack('ID').resample('W-THU').stack('ID')
Out[15]: 
               VAR1
DATE       ID      
2013-01-03 1   69.0
           2   69.0
2013-01-10 1   76.0
           2   73.5
```

I think because:

```
new_columns = new_columns - drop_cols
```

drops the column name (i.e. 'Params' in the case of the example). Maybe save it with:

```
cols_name = new_columns.name
new_columns = new_columns - drop_cols
new_columns.name = cols_name
```

Will need to add that name to `xp` in the test, since `assert_frame_equal` tests `column.name` now :).

(Probably `-` should keep the indexes name...)

Also: thanks for fixing this! :)

good catch :) thanks!

I'll fix it later today. 

I'm going to make include these two commits in a PR with some name functionality of - (for MultiIndex and Index), and will fix the name bit. Hope that's ok :)
TST: fix for tseries/test_period for py3k
State: closed


Labels:

Comments:
I think [this fails](https://travis-ci.org/jreback/pandas/builds/5876358) py2 now... ?

i know....something else just showed up as a bug in spares....worknig ....

498cc4c7fd96b042633d2fb6a75417299b5bf333 broke it because the SparseFrame wasn't renaming (if needed) in the `.columns`....so I think your fix (and my fix for sparse) are ok now....

maybe should revist this somehow?

It was broke [before my commit](https://travis-ci.org/pydata/pandas/builds/5860234)... I think we should just back away... slowly.

(It was something in [here](https://github.com/pydata/pandas/compare/f70e4823a67c...ea6fdc34822f).)
BLD: Cache 2to3 artifacts
State: closed

This is the logical followup to the cython cache hack.

Using both, I put together a server on Redhat Openshift to serve as 
network cache storage, and a travis build now takes 1.5 minutes bare,
and up to 4 minutes for a FULL_DEPS build.

The traffic used up by each build is significant, so the travis bit can't
be put on master. the 2to3 cache will shave another couple of minutes 
off local iterations, though.

https://travis-ci.org/y-p/pandas/builds/5873760


Labels:

Comments:
superceded by #3358
TST fix broken tests (which were using fancy asserts, broke travis py2.6...
State: closed

... and py3)

Master is not passing travis.

For py2.6: cleared up assertGreater etc. which don't work in py2.6 :(.
For py3: one test is broken in py3 (not sure how to fix):

```
ERROR: test_period_constructor (pandas.tseries.tests.test_period.TestPeriodProperties)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_4b61349-py3.3-linux-x86_64.egg/pandas/tseries/tests/test_period.py", line 102, in test_period_constructor
    self.assert_(i1 != i4)
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_4b61349-py3.3-linux-x86_64.egg/pandas/tseries/period.py", line 132, in __eq__
    raise ValueError("Cannot compare non-conforming periods")
ValueError: Cannot compare non-conforming periods
```


Labels:

Comments:
@wesm I think this was something in [this bunch](https://github.com/pydata/pandas/compare/f70e4823a67c...ea6fdc34822f)?

@hayd if this fixes these errors merge it!

It fixes the py2.6 ones. Strangely there is still one py3 error :S

Will merge and leave the final error for someone else :)
CLN: refactor core/index and tseries/index,period to have their format, to_native_types methods consistent
State: closed

- ENH: add support for Period formatting (as a native type), add `equals` comparison method
         `__eq__` et al added by others
- TST: add period support to makeCustomDataFrame for testing purposes


Labels:

Comments:
@changhiskhan @y-p any problem with this?

`Index` `format_native_types` does nan handling but corresponding `Multindex` code doesn't,
is that intentional?

not on purpose 
I am not sure that we allow nan in a mi though? (it messes with the sortedness)
I am sure we can construct one though

yeah, I've never seen one, but..

``` python
In [15]: df=mkdf(10,3,r_idx_nlevels=2)
    ...: df.index.values[0]= (np.nan, 'R_l1_g0')
    ...: pd.DataFrame(df.values,index=df.index.values)
    ...: 
Out[15]: 
                       0     1     2
(nan, R_l1_g0)      R0C0  R0C1  R0C2
(R_l0_g1, R_l1_g1)  R1C0  R1C1  R1C2
(R_l0_g2, R_l1_g2)  R2C0  R2C1  R2C2
(R_l0_g3, R_l1_g3)  R3C0  R3C1  R3C2
(R_l0_g4, R_l1_g4)  R4C0  R4C1  R4C2
(R_l0_g5, R_l1_g5)  R5C0  R5C1  R5C2
(R_l0_g6, R_l1_g6)  R6C0  R6C1  R6C2
(R_l0_g7, R_l1_g7)  R7C0  R7C1  R7C2
(R_l0_g8, R_l1_g8)  R8C0  R8C1  R8C2
(R_l0_g9, R_l1_g9)  R9C0  R9C1  R9C2


```

that is printing wrong then
should be NaN

I'll put a fix in - still should be illegal

perhaps less synthetic:

``` python
In [25]: df=mkdf(10,3,data_gen_f=lambda r,c: np.nan if randint(0,2) > 0 else randint(0,100))
    ...: df
Out[25]: 
C0       C_l0_g0  C_l0_g1  C_l0_g2
R0                                
R_l0_g0      NaN      NaN       68
R_l0_g1       91       74      NaN
R_l0_g2        6      NaN       70
R_l0_g3       96      NaN      NaN
R_l0_g4      NaN      NaN       37
R_l0_g5      NaN       66       12
R_l0_g6      NaN      NaN      NaN
R_l0_g7      NaN       14       26
R_l0_g8      NaN      NaN      NaN
R_l0_g9      NaN       91      NaN

In [26]: df.set_index([df.columns[0],df.columns[1]])
Out[26]: 
                 C_l0_g2
C_l0_g0 C_l0_g1         
Na      Na            68
91      74           NaN
6       Na            70
96      Na           NaN
Na      Na            37
        66            12
        Na           NaN
        14            26
        Na           NaN
        91           NaN
```
BUG: Append the empty frame with columns, #3121
State: closed

From [issue3121](https://github.com/pydata/pandas/issues/3121)
### Bug

```
In [1]: import pandas as pd

In [2]: df1 = pd.DataFrame(columns=['test'])

In [3]: df2 = pd.DataFrame()

In [4]: df1.append(df2)

ValueError: need at least one array to concatenate
```
### Reason

numpy cannot concatenate nothing:

```
In [10]: import numpy as np

In [11]: np.concatenate([])

ValueError: need at least one array to concatenate
```
### Fix

Return an empty numpy array if we find that the arrays to be concatenated are empty

```
# return the empty np array, if nothing to concatenate, #3121
    if not to_concat: return np.array([], dtype=object)
```
### Test

Test case `test_append_empty_frame` is added in `test_frame.py`


Labels:

Comments:
This bug fix looks great, but what are the other datetime tests at the end of the file? (or am I missing something?)

Sorry for accidentally adding some duplicated test cases. Have removed them.

Awesome!
WIP,CLN: com.ReprMixin everywhere, cls defines __unicode__ and py2/3
State: closed

str,bytes,repr work.

reduces code duplication and boilerplate.
still need to replace 

```
class Foo(com.ReprMixin,object)
```

by

```
class Foo(PrintableObject)
```

lots of churn, it'll wait till 0.12.


Labels:

Comments:
@y-p in 0.12 can do this before my #2915 where I moved all these routines to generic (as part if PandasObject)

but I'll just rebase after u merge and take your changes

I think we'll do a "big change" merge period right after 0.12, some things have 
accumelated.

This is trivial to fixup compared to your refactor. after you...
ENH add time to DatetimeIndex
State: closed

This adds a convenience property (`.time`) to DatetimeIndex, implementation is not very clever, but it means you can extract an array of time objects similar to `.hour` et al.

Thoughts?


Labels:

Comments:
can you put up a usecase?

In [this SO question](http://stackoverflow.com/questions/15640038/map-a-pandas-dataframe-indexd) someone is trying to plot by time of the day, and this seems like a reasonable thing to do. _I think I have seen quite a few similar cases, and have been confused that it doesn't exist before._

week/weekofyear/dayhour/minute/nanosecond work in this way...

Also I think if time is available for Timestamp it would be consistent to have it in DatetimeIndex... Saying that, there are a few others also not in DatetimeIndex but in Timestamp, perhaps they should be available as well?  (or maybe it's just strange that the above things work in this way...). Hmmm.

I saw that question....but sort of asking whether it might be worthwhile to give back a Timestamp like object here (e.g. a different sub-class, rather than a datetime), or is this overkillish...?

It's a fair point, the other attributes return integer arrays, this would be an object and the implementation won't be efficient on big DatetimeIndexes.

I don't really think it's a problem in the use cases I was be thinking of (e.g. plotting), and you can groupby (or something...) to do "time of day" analysis... (?)

At the moment when you subtract a timestamp from another you get a datetime.timedelta... The delta would be an easier thing to do something like this with, since probably modulo "nanoseconds in a day" is hopelessly naive to get the time... (it probably is overkillage anyways.)

actually I was sort of thinking of timedeltas
in 0.11 we r fully supporting timedelta64[ns] types
not yet a scalar that wraps timedelta64 nor a dedicated index (it's a int64 for now)
just search for timedeltas on issues

in any event I am not that is what u r after here 

see #3009 for example

Cherry-picked. Thanks
BUG: Fix the rename function for Series and DataFrame, #3165
State: closed

### Bug 1: After `rename`, the `name` of the `index` is missing:

```
In [1]: import pandas as pd

In [2]: data = [1, 2]

In [3]: index = pd.Index(['a', 'b'], name='name')

In [4]: df = pd.DataFrame(data, index=index)

In [5]: df 
Out[5]: 
      0
name   
a     1
b     2

In [6]: df.rename({'a': 'c'})
Out[6]: 
   0
c  1
b  2
```

Both `Series` and `DataFrame` has the same problems. The reason is that we didn't set the `name` of the `index` while reconstructing the `DataFrame/Series` by `rename` .
### Bug 2: `rename` cannot work for the `MultiIndex` case, the `index` becomes tuples, the `names` of the index are gone

```
In [1]: import pandas as pd

In [2]: data = [1, 2]

In [3]: index = pd.MultiIndex.from_tuples([('a', 'b'), ('c', 'd')], names=['name1', 'name2'])

In [4]: df = pd.DataFrame(data, index=index)

In [5]: df
Out[5]: 
             0
name1 name2   
a     b      1
c     d      2

In [6]: df.rename({'a': 'e'}) 
Out[6]: 
        0
(a, b)  1
(c, d)  2
```

The missing name issue is same as Bug1. The original `rename` function convert the dict to the mapper function and map each tuple but not each index, and doesn't reconstruct it as `MultiIndex`, so it becomes a `Index` with some tuples.
### Fix

I've fixed the above two bugs, the `rename` will also pass the `name` of the `index` too, the `rename` mapper will go through each index for the `MultiIndex` case now.
I also added some test cases in `test_series.py` and `test_frame.py`


Labels:

Comments:
This looks great!

@waitingkuo you beat me too it! looks good

Hope this might help : )

Please add an entry to RELEASE.rst
DOC release notes add assert_frame_equal checks index and column names
State: closed

Appended to release notes re #2964.  
I put it in "API changes" (not sure if that is optimal but I wasn't sure where else it would fit).


Labels:

Comments:
hold off on merging for just a bit

going to put up a PR to fix index name propagation

so @hayd u might be able to some of the checks back on 

This PR is just a DOC update and only changes `release.rst`.

Excellent to your PR, there are a few index/column name bugs/features to work through. Let me know which you are hitting and perhaps I can help on some others.

Weirdly, travis is broken on LOCALE build (presumably it is in master too):

```
FAIL: test_to_csv_moar (pandas.tests.test_frame.TestDataFrame)
```

@y-p I _think_ it's one of your last commits fails on travis. (Or did you just fix it?)

I think it's been there for a while and just didn't register. I just added
slow tests to LOCALE_OVERRIDE a couple of days ago on jeff's prompting.
Pretty sure I just pushed a fix for it 5 minutes ago, skipping some tests 
until  https://github.com/pydata/pandas/pull/3173 gets merged. float precision issues.
Index out of bounds error raised for negative indices.
State: closed

Accessing out of bounds negative timeseries indices silently return nonsense values.


Labels:

Comments:
@da415 can you put an example of this change?

also just check that performance is unchanged
e.g. run `test_perf.sh`

Looks like the cython (no) bounds-checking monster has roared again.
https://github.com/pydata/pandas/issues/3028 , https://github.com/pydata/pandas/pull/3029

I get a segfault rather then junk data.

``` python
In [6]: s=mkdf(10,5,r_idx_type='dt').C_l0_g0
   ...: s
Out[6]: 
R0
2000-01-03    R0C0
2000-01-04    R1C0
2000-01-05    R2C0
2000-01-06    R3C0
2000-01-07    R4C0
2000-01-10    R5C0
2000-01-11    R6C0
2000-01-12    R7C0
2000-01-13    R8C0
2000-01-14    R9C0
Freq: B, Name: C_l0_g0, dtype: object

In [7]: s[0]
Out[7]: 'R0C0'

In [8]: s[-1]
Out[8]: 'R9C0'

In [9]: s[-11]
** segfault **
```

#3029 doesn't seem to catch this either, @stephenwlin?

I think this is very easy to fix by putting some minimal error checking in Series.**getitem** , and using `indexing._maybe_convert_indicies` (like how we are handling `take` which is user facing), I think the perf impact will be quite minimal then, rather than trying to catch this in cython (which is very general)

In fact, this is maybe an excuse to do some more unification on series/frame **getitem**/**setitem**, @stephenwlin made them nice and user friendly......prob not too hard to 'fix' series

sounds good.

not sure will have time to get out before 0.11...shall we push to 0.12? @wesm 

@da415 I believe that your solution fails the tseries/test_resample test..can you confirm?

@jreback Apologies, the earliest I can confirm this failure is Tuesday 02/04.

@da415 sorry...didn't mean that...I tried your solution and it fails test_resample (on your commit); this was for sure a problem all along! thanks for catching it. I was just trying to see if maybe I am seeing something different.

I fixed it directly in the segfaulting function and added test case. thanks
BUG/CLN: Exception in HDFStore are now ValueError or TypeError
State: closed

A table will now raise if min_itemsize contains fields which are not queryables


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
this solves the bug (update 4): http://stackoverflow.com/questions/15488809/how-to-trouble-shoot-hdfstore-exception-cannot-find-the-correct-atom-type?noredirect=1#comment22113979_15488809
ENH: Use xlrd >=0.9.0 for both xls/xlsx, sidesteps GH1629
State: closed

Ssers with xlrd<0.9 will get an error message.
unlike openpyxl, xlrd parse() leaves type inferencing to
TextReader, which looks at the first values only, but this
was already the behaviour for .xls files, so small price to pay.

Removed much code duplication, and xlrd 0.9.0 added py3
support (courtesy of @takluyver, Thanks!), though that's not
tested yet.

Also, closes #1629


Labels:

Comments:
The bool issue might be related to type sniffing behaviour raised in #3171 

@wesm, yay or nay on upping min ver of xlrd to 0.9.0? how about 0.12 with some 
added testing for py3 support?
WIP: fixes to rolling_window functions with center=True
State: closed

#3155, #2953


Labels:

Comments:
@bmu, still interested in your help on testing this.

Just updated my fork of pandas. How can I pull your branch into my repository?

```
git remote add y-p https://github.com/y-p/pandas.git; 
git fetch y-p; 
git checkout y-p/center_window -b center_window
```

thanks, will post my comments in #2953.

push to 0.14 for further review, amazing this is mergeable.....

guess moments code hasn't been touched in a while

The tests suite for this was a nightmare to get working, and so I've
never trusted these changes. Throwing the issue back in the queue.
DOC: Add more detailed description of commit prefixes
State: closed

I was unfamiliar with these particular prefixes when I just contributed, so I thought it might be helpful for contributors to have some general guidelines for when to use them.  I honestly just guessed at some of them, so if they are wrong please feel free to correct them.

However, I looked on the mailing list and didn't see any guidelines there either, so I think it would be nice to enumerate them here.


Labels:

Comments:
sold.
ENH: Pass **kwargs through to matplotlib .scatter()
State: closed

Allow for more configurability to making scatter plots.  Most of the other plotting functions already do this, just giving .scatter_plot() the same power.  Works for both single scatters and grouped scatters.

I didn't make a test because anything that would be tested would be matplotlib specific, but I'd be happy to do so if you'd like (test_graphics.py: TestDataFramePlots.test_scatter is a good spot).


Labels:

Comments:
Merged. thanks.
ENH: added numexpr support for where operations
State: closed

Added support for numexpr to do where operations, so this will have an
effect on most boolean indexing operations themselves
(inital commit would speedup ops like `df>0`)

this PR is in effect speeding up `np.where`

`df[(df>0)&(df2>0)]` (100k x 100 columns)

`no_ne` is no numexpr at all
`st` is single threaded

This operation is restricted to int64/float64 dtypes
(others would be upcasted, which we could deal with, but
not now)

note: _the above operation could (and should) be much faster
if done as a single operation, but for now this is basically
4 calls to numexpr (3 boolean, then the where), but that's
for another day_

```
----------------------------------------------------------------
Test name                      | target[ms] | base[ms] |   ratio
----------------------------------------------------------------
frame_multi_and                       128.5169   385.2890     0.3336
frame_multi_and_st                    215.7741   417.0649     0.5174
frame_multi_and_no_ne                 397.3501   401.0870     0.9907
```


Labels:

Comments:
any issues?

I didn't test, but obviously the more the better.
PERF: GH2121 groupby transform
State: closed

closes #2121

Two items were causing slowness

1) using apply for each group (which in this case is equivalent to directly calling
the function on the group). the function that is testing is `fillna` which we have defined as a function of a data frame so its ok here to use the direct function call

I create a slow_path/fast_path with the first group determining the path, not sure this is super robust, but it is a significant source of slowness

2) at the end of the groupby the concetated object has a reindex_like, this is way slow, replacing by sort_index is much faster (this is a multi-index), again not of the robustness, but all tests pass

This is a comparision of the bench/bench_transform.py (supplied in #2121)

The apply_by_group DOES include the sort_index (which is necessary for correctness)

```
In [2]: %timeit apply_by_group(grouped, f_fillna)
1 loops, best of 3: 2.11 s per loop

In [3]: %timeit grouped.transform(f_fillna)
1 loops, best of 3: 2.14 s per loop
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
@y-p had a brain freeze

@y-p bigger issue is correctness.....I don't have a good intuitive feel for the groupby reindexing/sorting at the end....all the tests pass..but.....

@y-p I made the wrapper change, but didn't really impact resuls, not sure if there are actually a lot of calls in this case (or maybe optimized away somewhere)

I don't think large group counts are the common case. probably worth a vbench though

@y-p in this bench there were 2000 groups, not a lot I guess

yep.

Actually, for 2x on a corner case, I'd peel off even the original lambda.

@y-p unfortunately cannot peel off the 2nd level lamba because its needed in the apply (only in the slow path though)...fast path fixed up...good news is that the 2nd level lambda is sometimes optimized away in any event (in apply)

Could you add a simple vbench (preferable that takes 300ms or less per iteration) that can be used to track the performance of this?

done

```
-----------------------------------------------------------
Test name                 | target[ms] | base[ms] |   ratio
-----------------------------------------------------------

groupby_transform             494.3950  1367.9831     0.3614

Target [89a0c5d] : PERF: added vb_suite test for groupby_transform 
Base   [05a737d] : TST: skip problematic xlrd test (essentially 0.10.1)
```

should I take out the bench/bench_transform.py ?
TST: test for GH2623, object in frame upconverting a datetime64[ns] in another column
State: closed

this is tests only at #2623 is covered by other PRS


Labels:

Comments:
You can feel free to push commits like this directly.

yep, makes sense

if you merge branches manually and push, please use `merge --no-ff`, to ensure a merge commit
that packages up related commits into a well-defined block. no need if it's something small.

how is that different from pushing to my local branch then auto merging? which generates the merge commit
we do want the merge commit , right?

no difference, it's just the command line equivalent of the greenbutton.
for a PR with serveral coupled commits, I prefer a merge commit yes.
But I often push singles straight to master.

ok that's great thanks
ENH: extend selection semantics on ordered timeseries to unordered
State: closed

closes #2437
closes #2787
closes #3070


Labels:
name: Bug
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
related #3070  
Selection semantics should be a property of the underlying index type, not the data 
container type wrapping it.

thanks....pretty straightforward to wrap up #3070 in this

going for a 3-fer

do it in seperate branches and use an octopus merge ftw.

what's an octopus merge?

[this](https://secure.flickr.com/photos/sermoa/3425577867/)

moved to other PR's
DOC: GH3105 better error message on ndarray construction
State: closed

closes #3105


Labels:
name: Docs

Comments:
I never liked that error message, will be glad to see it die.

Most likely I am missing something, but are these tests asserting anything? Should each test include something similar to:

```
self.assert_(detail.message.startswith("Shape of passed values"))
```

.

_(I'm afraid I can't help mentioning)_:

```
tuple([len(index),len(columns)]) == (len(index),len(columns))
```

yah...prob should be asserting....

what' s the 2nd part of your message?

Looks like python3 doesn't give .message attribute to ValueError... hmmm.

Second part is simply that: `required_shape = tuple([len(index),len(columns)])` could read `required_shape = (len(index),len(columns))` :)

``` python
try:
    raise ValueError("foo")
except (Exception), detail:
    assert(str(detail).startswith("foo"))
```

@y-p @hayd thanks....just figured that out!

also was missing some cases, so did a bit of refactoring to make it consistent

@hayd Andy on a side note, you are very patient with people on stackoverflow!

@jreback Thanks! I try not to think about it... :)

anymore comments on this....merging soon...

looks fine to me.

@jreback I have only just looked at the google groups page... right back at you!

TOTD