BrandonFors/pandas-github-issues huggingface dataset:
ENH Assert_frame_equal check_names to True
State: closed

This is a follow up to #2962, where I added a `check_names=False` argument to `assert_frame_equal`, this defaults it to `True`. That is, when asserting two DataFrames are equal, it compares `.columns.names` and `.index.names`, unless explicitly passing `check_names=False`.

I've added in `check_names=False` where it wouldn't make sense to check names. I've added a TODO comment (and `check_names=False`) where I think it probably _should_ work but isn't... This was in the following cases:

```
to_csv
reindex
reset_index
groupy-a-get
pop
join (dataframe to a series)
from_xls (when setting index as first column)
```

i.e. these appear to drop column names  (`df.columns.names`). Whether this behaviour is correct / desired in these cases, I'm not sure, but here they all are. :)

_Note: By changing the A1 cells to 'index', I had all but one test (labelled "read xls ignores index name ?") in `test_excel.py` working locally (with `check_names=True`) , however Travis was less convinced  throwing:_

```
InvalidFileException: "There is no item named 'xl/theme/theme1.xml' in the archive"
```

_so I reverted to the previous xls and xlsx, and added back in `check_names=False`. I suspect LibreOffice wasn't saving it properly...?_


Labels:

Comments:
@wesm @changhiskhan @jreback Do you think this is ok to merge in?

I'm being a bit wary of changing the behaviour of assert_frame_equals if you're not onboard. (Since, it could lead to test passes merging into test fails, if assert_frame_equals is used in a test to frames with distinct column/index names.)

any tests failing?

@jreback Nope it's "good to merge"..

@hayd so you basically set `check_names=False` with a comment to check it.....

I guess maybe make a new issue refing this one....so you (or someone) can follow up in those cases.....they seem like the _should_ work.....

The original PR #2962 does just that (sets it to False), this second one sets it True...

I had to go in and fixed those calls to assert_frame_equals where it wasn't working, either by setting check_names=False, changing the name (of an expected/result) or flagging it with TODO (and setting check_names=False) if I thought it could be a bug.

Then I (or someone) can then follow them up :)

This looks fine to merge to me. I think it'll help catch bugs

Sorry about that, I've a line in release notes in the PR above.
BUG keep name after DataFrame drop, add check_name to assert_dataframe_equal
State: closed

This should fix #2939. (redo of pull-request #2961, which I screwed up).


Labels:

Comments:
I want to default `check_names` to True but at the moment this causes some errors (_some_ are just test expected not including name...), some may be bugs. Work in progress... I'll put it in another branch.

Update: It looks like there'll be a couple of similar (name-based) bugs...
BUG: timezone offset double counted using date_range and DateTimeIndex.append (fixes #2906, #2938)
State: closed

fixes #2906, #2938


Labels:

Comments:
Does your patch fix a similar bug for concat in 10.1?

In [17]: from pandas import *

In [18]: dates_a = [Timestamp('20130219 13:00', tz='US/Eastern'), Timestamp('20130220 13:00', tz='US/Eastern')]

In [19]: dates_b = [Timestamp('20130222 13:00', tz='US/Eastern'), Timestamp('20130223 13:00', tz='US/Eastern')]

In [20]: a = Series(range(2), dates_a)

In [21]: b = Series(range(2), dates_b)

In [22]: concat([a, b])
Out[22]: 
2013-02-19 18:00:00-05:00 0
2013-02-20 18:00:00-05:00 1
2013-02-22 18:00:00-05:00 0
2013-02-23 18:00:00-05:00 1

Note that the index of Out[22] is wrong (time shifted by 5 hours).

probably not, this was specific to date range generation...let me take a look

this ok to merge?

Looks fine to me-- I haven't looked to verify correctness but no tests had to be changed.

yeah, it should be good
BUG: Fix tz handling in date_range when start arg has tz, #2926
State: closed

This commit should fix [issue 2926](https://github.com/pydata/pandas/issues/2926).


Labels:

Comments:
tests please.

I tested with `nosetests pandas` and everything was OK. Do you want some sort of output?

Sorry, I meant you've fixed a bug which isn't caught by the current test suite,
so it's important to add a test that fails before the fix and demonstrates that your
change indeed corrects the bug. It'll also prevent future regressions if someone
modifies that part of the code.

The tests covering `date_range` live at `pandas/tseries/tests/test_daterange.py`,
please add a new test there.

Hi Moisan,

I am using the stable release 10.1. It's not easy for me to test the latest commit right now. Could you check if your patch fixes the following:

(1) issue #2906 date_range bug?

(2) I found a related bug when using concat:

In [17]: from pandas import *

In [18]: dates_a = [Timestamp('20130219 13:00', tz='US/Eastern'), Timestamp('20130220 13:00', tz='US/Eastern')]

In [19]: dates_b = [Timestamp('20130222 13:00', tz='US/Eastern'), Timestamp('20130223 13:00', tz='US/Eastern')]

In [20]: a = Series(range(2), dates_a)

In [21]: b = Series(range(2), dates_b)

In [22]: concat([a, b])
Out[22]: 
2013-02-19 18:00:00-05:00    0
2013-02-20 18:00:00-05:00    1
2013-02-22 18:00:00-05:00    0
2013-02-23 18:00:00-05:00    1

Note that the index of Out[22] is wrong (time shifted by 5 hours).

Hi xdong,

No my patch does not fix either of these issues. However I do believe they are related to each other. I tried to force the timezone in issue 2906 and found another bug and opened issue 2926. I'll go back to issue 2906 soon :).

Cherry-picked, thank you!
CLN: refactor of frame.py/panel.py to move common code to generic.py
State: closed

- Refactor of frame.py/panel.py to move common code to generic.py
  all axis creation and manipulation code is now common (except for Series)
  - added _setup_axes to create generic NDFrame structures
  - moved methods (some methods moved from series as well)
    - from_axes,_wrap_array,axes,ix,shape,empty,swapaxes,transpose,pop
    - **str**,**bytes**,**repr**
    - **iter**,keys,**contains**,**len**,**neg**,**invert**
    - convert_objects,as_blocks,as_matrix,values
    - **getstate**,**setstate** (though compat remains in frame/panel)
    - **getattr**,**setattr**
    - _indexed_same,reindex_like,reindex,align,where,mask
    - filter (added axis argument to selectively filter on a different axis)
    - reindex : changed a fair amount
    - truncate (moved to become part of PandasObject)
  - API Changes in Panel which make Panel more consistent with DataFrame
    - swapaxes on a Panel with the same axes specified now return a copy 
    - support attribute access for setting
    - filter supports same api as original DataFrame filter
- API change in that reindex called with no arguments will now return a copy of the input object (e.g. `df.reindex()`)

_performance is comparable to the existing codebase_

Reasoning for this change:
1. this is in preparation for a change in series to sub-class NDFrame, rather than ndarray directly, with proper support for BlockManager. _this will also theoretically allow a way to do `integer NaNs`, via a mask in the IntBlock_
2. preparation for adding SparseBlocks to BlockManager
3. future changes/additions in indexing will be more straightforward, e.g. adding an integer like `ix` can be done, hopefully in one place
4. future API compatibiliy across objects (e.g. won't have to add methods to both Frame and Panel, and there API can be pretty consistent)


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
@wesm any issues with this? 

sorry to butt in, might it be a good idea to merge this right _after_ 0.11? 
it's a big change.

that sounds right
pushed till 0.12

new one at some point
BUG: nanops.var produces incorrect results due to int64 overflow (fixes #2888)
State: closed

fixes #2888


Labels:

Comments:
as a side note, apparently the tests as they were would never fail for certain ops, because they were comparing (e.g.) `Series.std()` to `np.std()`, but the latter was just calling the former anyway. (maybe this is new behavior since the tests were created.)

anyway, the tests are doing the right thing now and all pass

ok to merge on this?

yep
BUG: Fix broken YearBegin offset. Closes #2844.
State: closed

This should close #2844. Good but unsettling news is it wasn't an implementation bug. The logic for handling months in the YearBegin offset just wasn't written. It was hard-coded to always be Year, 1, 1.


Labels:

Comments:
FYI, rebased.

This breaks some unit tests.  Can you take a look?

Yeah, I don't think the expectations for AS were correct in those tests.

Thanks Skipper!
BUG: Series construction from MaskedArray fails for non-floating, non-object types
State: closed

Currently Series constructor is throwing exceptions or ignoring the mask for non-floating, non-object MaskedArray input:

``` python
In [1]: from numpy.ma import MaskedArray as ma
In [2]: from pandas import Series, DataFrame
In [3]: from datetime import datetime as dt

In [4]: mask = [True, False]

In [5]: Series(ma([2, 0], mask=mask, dtype=int))
(exception)

In [6]: Series(ma([True, False], mask=mask, dtype=bool))
Out[6]: 
0     True
1    False

In [7]: Series(ma([dt(2006, 1, 1), dt(2007, 1, 1)],
   ...:        mask=mask, dtype='M8[ns]'))
(exception)
```

After fix:

``` python
In [1]: from numpy.ma import MaskedArray as ma
In [2]: from pandas import Series, DataFrame
In [3]: from datetime import datetime as dt

In [4]: mask = [True, False]

In [5]: Series(ma([2, 0], mask=mask, dtype=int))
Out[5]: 
0   NaN
1     0
Dtype: float64

In [6]: Series(ma([True, False], mask=mask, dtype=bool))
Out[6]: 
0      NaN
1    False
Dtype: object

In [7]: Series(ma([dt(2006, 1, 1), dt(2007, 1, 1)],
   ...:        mask=mask, dtype='M8[ns]'))
Out[7]: 
0                   NaT
1   2007-01-01 00:00:00
Dtype: datetime64[ns]
```

The upcasting will only be done if necessary (i.e. if the mask has any `True` elements):

``` python
In [8]: Series(ma([2, 3], mask=[False, False], dtype=int))
Out[8]: 
0    2
1    3
Dtype: int32

In [9]: DataFrame(ma([2, 3], mask=[False, False], dtype=int))
Out[9]: 
   0
0  2
1  3
```

This latter is a minor change to the previous behavior of the DataFrame constructor, which was upcasting correctly but was doing it even if the mask was all `False`: I've changed it to only upcast when necessary because this is more consistent with how masks are treated elsewhere.

Also made some PEP8 line length fixes


Labels:

Comments:
can u rebase to master and see if this still merges cleanly?
I had merged a tiny change that might conflict
otherwise ok to merge?

everything looks ok to me
BUG/TST: Yahoo Finance (a foundational resolve of GH: #2847,  #2853)
State: closed

Resolve issues in `pandas.io.data.py` and `test_yahoo.py`.
- BUG: Fixed decode issue in `get_data_yahoo()` for python 3.3 (resolves #2847)
- ENH: Updated `_calc_return_index()` function to properly handle groups of multiple stocks that do not share the first valid price on the same date (i.e., stocks whose IPO's dates exist within the query date range). Ensures that all stock's first values in `ret_index` are set to 1, regardless of actual initial starting date.
- TST: Modify test suite for yahoo finance functions in `test_yahoo.py` to accurately test queries to yahoo finance (resolves #2853/#2847)


Labels:

Comments:
Hey @y-p, apologies for the wait on the fix. I'm not sure about my rebase with the branch, let me know if I can address anything!

Thanks!

You removed the sanity checks, it's better to leave them in place and merely
add to them.

Not sure about removing @slow, because they are in fact slow. 
Better keep test_fast.sh true to it's name.

I believe decode() uses the terminal encoding when no arg is passed in.
That's utf8 on most boxen but not all. notably non-english speaking windows.
So you should specify an encoding, yahoo returns utf8 doesn't it?

Good tip on the sanity checks. I do like those, and I've already put them back in. 

About the `@slow`, is 6.5 seconds slow for all 4 of the tests in `test_yahoo.py`? I substantially reduced the amount of data being queried in the background and that's the average time they take to complete now.

I believe the encoding is utf8, and I've added this and it seems to run fine, passing with flying colors. 

Let me know about the `@slow` and I'll push the changes in single commit, thanks!

thanks, cherry-picked, 44b249534ff0c0df237d484cebc5be2d76974dd8 and friends.

Great, thank you @y-p. Nice commit messages, btw. Helpful to learn from. Oh, and, what was your thoughts about 4 tests in 6.5 seconds? Would like to know for future reference.... Thanks!

The tests included in `not @slow` average about 20ms each. So by comparison
these are slow.  4 seconds may not be a long time, I just opted for keeping the status-quo.

Gotcha. Thanks for the info!
BUG: issue in get_dtype_counts for SparseDataFrame (introduced by dtypes)
State: closed

- added tests for GH #2873  (commented out)


Labels:

Comments:
Noticed intermittent failures in vbench `unstack_sparse_keyspace`.

https://travis-ci.org/pydata/pandas/builds/4808659
https://travis-ci.org/pydata/pandas/builds/4822850

I have seen this error too
i doesn't have to do with this pr
(as it has happened 'sometimes' on several of PRs )
must be something specifc in the vbench maybe?

ok then. I'm guessing some randomly generated data is sometimes causing 
a non-unique index. will look when I get a chance.
ENH: implement Block splitting to avoid upcasts where possible (GH #2794)
State: closed

- provides an implementation of IntBlock splitting that can avoid upcasting (to float) these blocks if a boolean condition does not affect the column, closes GH #2794)
- bugfix in internals.Block.putmask; was upcasting always on an int; and upcasting to float, (now float64)

```
In [6]: df = pd.DataFrame(1,columns=['a','b'],index=range(3))

In [7]: df.ix[1,'b'] = 0

In [8]: df
Out[8]: 
   a  b
0  1  1
1  1  0
2  1  1

In [9]: df.dtypes
Out[9]: 
a    int64
b    int64
Dtype: object

In [10]: df[df>0].dtypes
Out[10]: 
a      int64
b    float64
Dtype: object

In [11]: df[df>0]
Out[11]: 
   a   b
0  1   1
1  1 NaN
2  1   1
```


Labels:

Comments:
@y-p I obviously merged this (to catch up with some changes in master); the merge commit is there, but doesn't 'affect' things....is this correct?

it depends on what master contained at the time you merged. If this gets merged in a month
after many changes, yes it could create problems. just rebase or use reset and "git add -i/-p"
to make new commits.

so i just rebase and pick out my 3 or so commits (just like normal)....what about the 'merge' changes? just leave them out in the rebase?

@y-p thanks...i had done a strigiht `git pull` instead of `git pull --rebase` ....fixed

I sometimes do this, to fix that sort of thing:

```
git checkout -b temp_branch this_branch # work in sandbox
git merge upstream # sync
git reset upstream/master # leave tree as is, reset HEAD to master
git status # show uncommited changes
while remaining_changes:
    git add -p # select chunks to include in commit
    git commit -m "message"
echo voila
```

also, git has a tweak to always rebase:

```
$ git config branch.autosetuprebase always
$ git config branch.*branch-name*.rebase true
```

http://stevenharman.net/git-pull-with-automatic-rebase

you r git-fu master

me just going to turn on auto rebase
ENH: Added notes about changes in Options class to what's new for 0.11
State: closed

BUG: Also fixed a bug in the puts section for Options.get_forward_data.


Labels:

Comments:
Thanks, would it be possible for you to rebase this on top of master and 
get rid of the merge commit? 

It can create problems, as mentioned in `CONTRIBUTING.MD`.

See new pull request  #2884.

I didn't want to have to worry about rebasing so I just made a new branch and grabbed these commits.
BUG: Make DataFrame not hardcode it's own constructor in the code. Issue #2859
State: closed

BUG: Make DataFrame not hardcode it's own constructor in the code. Issue

Changed all occurrences of `return DataFrame` to `return
self._constructor` and changed the latter to return `self.__class__`
rather than `DataFrame` by default.

Also made static methods like `DataFrame.from_dict` work on the
`cls` passed to them rather than on hard-coded `DataFrame`.

This is a minimal quick win on the cases described in Issue #2859.
A formal approach to make classes friendly to subclassing might still be
needed, see discussion in the case.


Labels:

Comments:
Quick "hand merge". This won't solve the issue but helps in some cases
BUG: automatic color cycling when plotting Series without style or color argument (fixes #2816)
State: closed

It should fix #2816 (and #2827 which is a duplicate).


Labels:

Comments:
Could maybe @wesm or @changhiskhan have a quick look at this one? I seem to remember the promise this would be fixed for the next release on the mailing list.

More than happy to change anything as needed.

Also I seem to have enabled travis-ci after pushing this branch to my fork so that this branch was never tested. If that makes things easier and there is an easy way  to force travis-ci to run on this branch let me know.

There's a failing test:

```
λ nosetests ./pandas/tseries/tests/test_plotting.py:TestTSPlot.test_secondary_legend

======================================================================
FAIL: test_secondary_legend (pandas.tseries.tests.test_plotting.TestTSPlot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user1/src/pandas/pandas/tseries/tests/test_plotting.py", line 832, in test_secondary_legend
    self.assert_(len(colors) == 4)
AssertionError: False is not true
```

If it's fixed, I'll merge this.

Hmm I thought I ran the tests and checked but apparently not. I'll take a look.

Plot tests are usualy skipped due to slowness, you have to ask for them.

They were added to travis just a few days ago (tacked on to the LOCALE_OVERRIDE build).
or you can just do:

```
nosetests ./pandas/tseries/tests/test_plotting.py
```

To be honest, not sure whether that's the best way to do it but the problem was that the left and right axis have independent color cycles which matters when you plot timeseries without any color or style arguments. In the failing test case you ended up having two blue and two green lines instead of the expected blue, green, red, cyan lines.

Any better work-around I am interested.

Good enough for me.
Can you add a commit with a note in RELEASE.rst?

This fix works great.

merged

great, thanks for this and sorting out the merge conflict.
Doc: Add documentation for the classes parameter of the to_html method
State: closed

It seems that there is no documentation about the classes parameter in `to_html`.

Current output of documentation in IPython:

```
Docstring:
to_html-specific options
bold_rows : boolean, default True
    Make the row labels bold in the output

Render a DataFrame to an html table.

 Parameters
 ----------
 frame : DataFrame
     object to render
buf : StringIO-like, optional
    buffer to write to
columns : sequence, optional
    the subset of columns to write; default None writes all columns
col_space : int, optional
    the minimum width of each column
header : bool, optional
    whether to print column labels, default True
index : bool, optional
    whether to print index (row) labels, default True
na_rep : string, optional
    string representation of NAN to use, default 'NaN'
formatters : list or dict of one-parameter functions, optional
    formatter functions to apply to columns' elements by position or name,
    default None, if the result is a string , it must be a unicode
    string. List must be of length equal to the number of columns.
float_format : one-parameter function, optional
    formatter function to apply to columns' elements if they are floats
    default None
sparsify : bool, optional
    Set to False for a DataFrame with a hierarchical index to print every
    multiindex key at each row, default True
justify : {'left', 'right'}, default None
    Left or right-justify the column labels. If None uses the option from
    the print configuration (controlled by set_printoptions), 'right' out
    of the box.
index_names : bool, optional
    Prints the names of the indexes, default True
force_unicode : bool, default False
    Always return a unicode result
```


Labels:

Comments:
thanks!
PEP8-compliant HDFStore documentation
State: closed

(This is to be applied after a PR from https://github.com/jreback/pandas/tree/pytables_update7, as it is based on it)

This is my first PR and I am hoping to get right the github worfklow to start with 'real contibutions'. Any hints on how to PR better very welcome.


Labels:

Comments:
@jreback when you get a chance can you skim the changes to the docs here?

I think this is fine to merge in as is (with usual fixes to change 0.10.2 -> 0.11 for RELEASE and whatsnew)

cherry-picked @meteore's commits
Creating DatetimeIndex from empty array with datetime64 dtype raises IndexError
State: closed

using pandas from master:

``` python
import numpy as np
import pandas as pd
pd.DatetimeIndex(np.array([], dtype='datetime64'))
```

I get the following stack trace:

``` python
/data/python/pandas/pandas/tseries/index.pyc in __new__(cls, data, freq, start, end, periods, copy, name, tz, verify_integrity, normalize, **kwds)
    229             else:
    230                 if data.dtype != _NS_DTYPE:
--> 231                     subarr = tslib.cast_to_nanoseconds(data)
    232                 else:
    233                     subarr = data

/data/python/pandas/pandas/tslib.so in pandas.tslib.cast_to_nanoseconds (pandas/tslib.c:14151)()

IndexError: invalid index
```

This happens because pandas.tslib.cast_to_nanoseconds expects the array to be not empty to get the datetime64 unit of the first element.

One way to fix it would be to check whether `data` is empty before trying the conversion. 

It would be nice to have a better error inside `pd.tslib.cast_to_nanoseconds` saying that it expects a non-empty array as well.


Labels:

Comments:
thanks!
v0.10.1 bug fixes not displaying properly on website, PDF
State: closed

The Bug Fixes list on the ["What's New" page](http://pandas.pydata.org/pandas-docs/stable/whatsnew.html) is a bit broken, as is the corresponding section of the PDF.

I've added some missing indentation, which fixes the issue for me.


Labels:

Comments:
Thanks for noticing and fixing.

cherry-picked via 89f2cd1
BUG: Period/DateTime slicing and setitem fixes (fixes #2782, #2788, #2789)
State: closed

fixes #2782

simple fix...can't see any reason why the original code was doing what it was, especially since indices are immutable

added test coverage in test_series.py


Labels:

Comments:
also fixes #2788, with test coverage in test_series.py

also fixes #2789, with test coverage (make sure to rebuild extensions)

Cherry-picked with a tiny tweak. Thanks a lot for the PR!