BrandonFors/pandas-github-issues huggingface dataset:
Fix issue #4496: tslib.tz_convert(vals, tz1, tz2) may raise an IndexErro...
State: closed

Bug-fix for Issue #4496 Bug: tslib.tz_convert(vals, tz1, tz2) may raise an IndexError exception.


Labels:

Comments:
couple of things, please do the following
- ~~add a test~~
- hook up travis CI
- add documentation to `doc/release.rst`, preferentially move the docstring in your test to release notes, but not necessary. if you just want to add a blurb that's fine too.

- hook up travis CI // done;
- add documentation to doc/release.rst // done (not sure whether such a long bug description is appropriate in release.rst);
- "You should rewrite this according to what the hour should be here, don't catch every exception: that could hide another bug." // That's reasonable. Thanks! Fixed.

@stenri can you rebase and hook up travis?

Surely. I rebased  tslib_tz_convert_trans_pos_plus_1__bugfix against upstream/master (and combined all my commits into a one commit).

@stenri can you squash into 1 commit? then will merge

Excellent! Done.

thanks!
BLD/TST: correct pytz version for tox
State: closed

Tox fails to build pandas without a specific version of pytz.


Labels:

Comments:
hm wonder if this should be reported to upstream pytz? i seem to remember this working without a version at some point...

Was failing for me, could be related to #4210?

ah forgot about that...i don't follow `pip` dev, maybe i should

FYI here's the error I get for attempting a clean build with tox without a version for pytz. 

It actually looks like tox is using pip 1.4 regardless of what version the local env is using (I've got pip 1.2 in the virtualenv I'm using). 

```
Downloading/unpacking pytz
  Could not find a version that satisfies the requirement pytz (from versions: 2009r, 2008b, 2009f, 2008c, 2007g, 2011g, 2005m, 2011e, 2007f, 2011k, 2007k, 2006j, 2008h, 2008i, 2011e, 2008a, 2009e, 2006g, 2011j, 2010l, 2005m, 2008i, 2005k, 2008g, 2007c, 2007i, 2009l, 2009r, 2006j, 2011k, 2007d, 2006p, 2009i, 2009u, 2007i, 2009f, 2010g, 2008h, 2009a, 2007g, 2011e, 2006p, 2012b, 2010k, 2005r, 2007f, 2009l, 2009p, 2008c, 2009j, 2008g, 2010g, 2010h, 2011h, 2010k, 2007c, 2007d, 2011d, 2009l, 2011c, 2008a, 2005m, 2007k, 2009n, 2011d, 2010o, 2013b, 2012h, 2010e, 2012c, 2012d, 2012f, 2011n, 2011b, 2011j, 2008c, 2012j, 2007k, 2009f, 2009d, 2010e, 2010b, 2013b, 2011d, 2009p, 2008h, 2005r, 2009i, 2009n, 2009a, 2010k, 2008g, 2006g, 2008b, 2012c, 2009i, 2007g, 2012c, 2010h, 2011n, 2012g, 2007d, 2008a, 2009u, 2012g, 2010o, 2006p, 2010b, 2009u, 2012d, 2011k, 2012f, 2009a, 2007f, 2011h, 2010l, 2009j, 2011g, 2009g, 2009g, 2005r, 2011c, 2012g, 2009g, 2012d, 2009j, 2010o, 2007c, 2010g, 2006g, 2009d, 2010h, 2005k, 2006j, 2010b, 2009n, 2011g, 2011c, 2012b, 2009e, 2009d, 2011j, 2007i, 2012j, 2010l, 2009r, 2012h, 2010e, 2009p, 2008i, 2012f, 2009e, 2012b, 2011h, 2005k, 2008b, 2013b, 2011n, 2012j, 2004b)
No distributions matching the version for pytz
```

```
(py27)kieran@smallbang ~/work/git/pandas $ pip --version
pip 1.2.1 from /home/kieran/work/pyenv/py27/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg (python 2.7)
(py27)kieran@smallbang ~/work/git/pandas $ .tox/py27/bin/pip --version
pip 1.4 from /home/kieran/work/git/pandas/.tox/py27/lib/python2.7/site-packages (python 2.7)
```

yep i get that too i just remember the 2012 pytz versions working w/o this...

Also I'm using the latest tox

```
(py27)kieran@smallbang ~/work/git/pandas $ tox --version
1.5.0 imported from /home/kieran/work/pyenv/py27/lib/python2.7/site-packages/tox/__init__.pyc
```

i also am really :-1: on this not removing of build directories when i try to update my env packages

thanks @Komnomnomnom
FIX: resample to single group with custom function
State: closed

Using master code, when re-sampling to a frequency that would result in a single group the use of custom aggregation functions do not work as expected. The included changes fix this, and should fix #3849 too.

``` python
In [17]: import pandas as pd

In [18]: import numpy as np

In [19]: mysum = lambda x: x.sum()

In [20]: rng = pd.date_range('2000-1-1', '2000-1-10', freq='D')

In [21]: ts = pd.Series(np.random.randn(len(rng)), index=rng)

In [22]: ts.resample('M', how='sum')
Out[22]: 
2000-01-31    0.985236
Freq: M, dtype: float64

In [23]: ts.resample('M', how=mysum)
Out[23]: 
2000-01-31    0
Freq: M, dtype: float64
```


Labels:

Comments:
@jtratner take a look?

@Komnomnomnom can you add a release notes.....?

and add in tests for #3849
much appreciated

Done and done (I presume you meant #3849 ... ?)

yep..that's what I meant!
BUG: off-by-one when doing network test repeats
State: closed

Imagine num_runs is two, then runs will take on the values [0, 1] and runs < 
num_uns will always be False. Hence any genuine Exception the code might
raise will be silently ignored and just printed.


Labels:

Comments:
thanks for catching this, can you hook up travis ci

Not sure what to do about the Travis error though.

Think it has been failing all along, but unnoticed.

this is marked as issue #4427 (the failing test), which was reported.....

My guess is, it should look like  46f6c43 

I see, well in that case, let's just ignore the fix I made?

@esc your fix is good; was just linking the issue that needs to be fixed (your are fixing the test that was not catching this issue) thxs!

Doesn't fail on `master` though :confused: 

That test needs prob just needs to come out...it really just needs to check that the reader 'works', rather than comparing a value which gets revised (and of course they don't tell you that!)

I'm +>9000 on removing that test!

@esc why don't u put a skip test in the failing test that you uncovered

btw - the original PR is a nice object lesson for why tests are important for correct code :sweat_smile:

And now I get this one:

https://travis-ci.org/esc/pandas/jobs/9918210

```
======================================================================

ERROR: test_string_select (pandas.io.tests.test_pytables.TestHDFStore)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0_127_g46f6c43-py2.7-linux-x86_64.egg/pandas/io/tests/test_pytables.py", line 2424, in test_string_select

result = store.select('df',Term('x!=none'))

File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas-0.12.0_127_g46f6c43-py2.7-linux-x86_64.egg/pandas/io/pytables.py", line 475, in select

raise KeyError('No object named %s in the file' % key)

KeyError: 'No object named df in the file'
```

its ok unrelated 2 u

So I am now skipping the unstable test. Problem is that Python 2.6 `unittest` module has no `skip`. What could/should I use as an alternative? Also the `pytables` test mentioned above is still failing. BTW: can/should I rebase this onto master?

yes rebase onto master

@esc just `raise nose.SkipTest`.

@esc how's this coming?

Okay, worked out how to skip the test properly. gonna rebase and then it'll be ready to merge.

Rebased onto current master, please merge.

@esc can you add a skip test to test_fred_nan and test_fred_parts? Then we can merge this. Going to open an issue that they need to be fixed.

Merged (I added two more skip tests for Fred). Thanks!

Thanks for adding the missing skips and for merging!
BUG/ER: Stricter testing of 'monotocity' when reindexing with ffill or bfill.
State: closed

closes #4484

Reversing an index and forward filling succeeds but produces misleading data:

``` python
>>> dr = pd.date_range('2013-08-01', periods=6, freq='B')
>>> df = pd.DataFrame(np.random.randn(6,1), index=dr, columns=list('A'))
>>> df['A'][3] = np.nan
>>> df.reindex(df.index[::-1], method='ffill')
                   A
2013-08-08  2.127302
2013-08-07       NaN
2013-08-06       NaN
2013-08-05       NaN
2013-08-02       NaN
2013-08-01       NaN
```

This change enforces "monotocity" (well actually weakly increasing) and causes a ValueError to be raised.


Labels:

Comments:
Can you also add an example of how to do this in the missing_data.rst ? (and maybe a pointer from the cookbook)?

also need release notes (put in the API section), reference the issue #4484

This is the solution (but yours is nice on error reporting!)

```
In [11]: >>> df.reindex(df.index[::-1]).reset_index().ffill().set_index('index')
Out[11]: 
                   A
index               
2013-08-08  1.112043
2013-08-07  0.250710
2013-08-06  0.250710
2013-08-05 -0.499677
2013-08-02 -0.327536
2013-08-01 -1.774846
```

also pls hook up travis to this (see contributing.md)

@prossahl ok...this looks good (ignore the failure on travis)...that's a weird error that is fixed in master

merge?

and I added the example I put above to the cookbook (will be there later today)

I have moved this to a PR #4642.

ok...close this one then?

Will do. Over to GH4642.
BUG: raise on frozenset construction by Series
State: closed


Labels:

Comments:
1. Argument from simplicity - Pandas data structures are ordered and the user is expected to provide an ordered (or keyed) data structure as input. It's simple and logical, I don't think there's a need to change it.
2. Argument from necessity - What's the problem this actually fixes? that the user has to pass in `list(values)` rather then `values`? meh.
3. Argument from consequence - Slippery slope, once you introduce set to the `series` ctor you break consistency (and an established pandas rule of thumb) with dataframe and panel.  a small break in 
   consistency can spread far and wide across a codebase when later attempts try to regain it. This may be FUD, but I just don't think it's worth it.

I'm -1 on this.

Your argument about consistency is the most convincing to me, but the original thought was: if `list` can take a `set` why can't `Series`? That would make `Series` more consistent with Python, but of course breaks it with pandas. I agree with you that we should keep consistency with the latter.

I agree with 1. and 2.

I don't feel strongly about this .... I've never actually run into the issue before....

At the very least this should raise on `frozenset` objects with the same error.

cc @hayd since he might have a stronger opinion than I about this.

Not really strong... the list argument is all I really have.

I think passing in any iterator should be ok... :s

```
In [11]: s = set([2])

In [12]: pd.Series(iter(s))
Out[12]: <setiterator at 0x104559280>

In [13]: pd.Series(iter(list(s)))
Out[13]: <listiterator at 0x104554b90>

In [14]: pd.Series(i for i in s)
Out[14]:
0    2
dtype: int64
```

so don't see why iterables should be so different, and why we should stop it (just cos we can't test it) but... meh.

_Interested to see if/when dictionaries have different ordering for different python versions!_

Good example. That's actually my 3rd point in action: a while ago generators were accepted
into the ctor (to save users the anguish of `list(gen)`) and now you're showing the inconsistency
_it_ generated as an (reasonable) argument to support even more stuff.

I think it's fine to support iterators as you suggest (spilled generator milk, so to speak), but that means `iter(set)` _not_ `set`,
so that's really a separate issue your example raises.  `iterat-or` and an `iter-able` are 
different beasts I believe, am I wrong?

re different ordering for different python versions, how about different ordering in the _same_ 
python version courtesy of hash randomization?

```
λ python3.3 -c 'print(dict(a=1,b=2))'
{'b': 2, 'a': 1}
λ python3.3 -c 'print(dict(a=1,b=2))'
{'a': 1, 'b': 2}
```

@y-p nice example.

yep iterables and iterators are different, `iter(list('abc')) != list('abc')`

@hayd also _generators_ are yet another type of _iterable_ which is why you get the output above

woah! that _is_ a nice example :) I'm sold.

Maybe error could mention "first use list or sorted" ?

@jreback good 2 go after passing?

thought u closed this? obviously not...ok
BUG: values array for non-unique, interleaved cols incorrect
State: closed

I came across this problem when I was doing some work for #4362. When a frame with mixed dtypes including Timestamps has dupe column labels the values array returned is different, as illustrated below (master branch):

``` python
In [3]: df = pd.DataFrame([[pd.Timestamp('20130101'),3.5],[pd.Timestamp('20130102'),4.5]], columns=['x', 'x'], index=[1,2])

In [4]: df
Out[4]: 
                    x    x
1 2013-01-01 00:00:00  3.5
2 2013-01-02 00:00:00  4.5

In [5]: df.values
Out[5]: 
array([[1356998400000000000L, 3.5],
       [1357084800000000000L, 4.5]], dtype=object)

In [6]: df = pd.DataFrame([[pd.Timestamp('20130101'),3.5],[pd.Timestamp('20130102'),4.5]], columns=['x', 'y'], index=[1,2])

In [7]: df
Out[7]: 
                    x    y
1 2013-01-01 00:00:00  3.5
2 2013-01-02 00:00:00  4.5

In [8]: df.values
Out[8]: 
array([[datetime.datetime(2013, 1, 1, 0, 0), 3.5],
       [datetime.datetime(2013, 1, 2, 0, 0), 4.5]], dtype=object)
```

The included changes should fix the problem. This is my first time messing about in pandas internals so any feedback is appreciated!

Possibly related to #4377


Labels:

Comments:
This looks good, interestingly in 0.12 this was an assertion error (so it's good it's tested!):

```
AssertionError: All items must be in block items
```

I think @jreback fixed the assertion error in #4388. 

let me look at this a sec

@Komnomnomnom this looks good....nice catch

Awesome, thanks :-)
BUG: Fix Panel instance variable namespace issue GH3440
State: closed

The change is minor but so easy I thought I'd submit it: If you switch the list comprehensions to generators, it sidesteps leaking the temporary variables into the class variable scope.

This will be microscopically slower, but they are just class variables so it really doesn't matter.
fixes #3440

https://travis-ci.org/danbirken/pandas/builds/9836459

```
import numpy as np
from pandas import DataFrame, Panel

df0 = DataFrame(np.zeros([3,4]))
df1 = DataFrame(np.ones([3,4]))

p = Panel({'a':df0, 'b':df1})

In [1]: p.a
Out[1]:
   0  1  2  3
0  0  0  0  0
1  0  0  0  0
2  0  0  0  0

In [2]: p.b
Out[2]:
   0  1  2  3
0  1  1  1  1
1  1  1  1  1
2  1  1  1  1
```


Labels:

Comments:
@danbirken can u add a test? thx

There you go

Travis: https://travis-ci.org/danbirken/pandas/builds/9843789

@danbirken sweet thanks!

@jreback will this be fixed in #3482?

I think this fix is fine

@danbirken thanks!
PERF: enhance HDFStore Table writing performance
State: closed

Took out all of the cython code, turns out you can assign directly to numpy recarrays per
column if you put in the correct shape yourself 

```
Invoked with :
--ncalls: 5
--repeats: 5
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
write_store_table_mixed                      |  45.6208 | 121.5352 |   0.3754 |
write_store_table_panel                      |  59.4140 |  99.5456 |   0.5969 |
write_store_table                            |  40.2264 |  65.8978 |   0.6104 |
write_store_table_dc                         | 143.3910 | 174.9424 |   0.8196 |
read_store_table_wide                        |  16.8930 |  17.5310 |   0.9636 |
query_store_table_wide                       |   9.6692 |   9.7564 |   0.9911 |
write_store_table_wide                       | 117.8906 | 118.8232 |   0.9922 |
read_store_table_mixed                       |   4.7604 |   4.7478 |   1.0026 |
query_store_table                            |   4.2646 |   4.2320 |   1.0077 |
read_store_table                             |   2.3678 |   2.3178 |   1.0216 |
read_store_table_panel                       |  20.9388 |  20.3526 |   1.0288 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [d11041d] : PERF: enhance HDFStore Table writing performance
Base   [aca1a42] : Merge pull request #4437 from cpcloud/fix-astype-calls

```


Labels:

Comments:
Jeff this looks fantastic for writing tables. Does this enhancement also apply when appending to an existing table?

I cant wait to give this a shot once 0.13 is out. :+1: 

http://pandas.pydata.org/pandas-build/dev/

there are are binaries out (iirc you use windows)

this works for appending / writing (it's the same operation )
TST: pandas/util/testing : test improvements and cleanups
State: closed

A few things
1. Make with-statement-capable `assertRaises` and `assertRaisesRegexp` (which I want to use later in a cleanup of some of the parser tests).
2. Earlier introduction of `assert_attr_equal` changed it so that, if one object had an attribute that was `None` and the other did not have that attribute at all, it would pass. This is now fixed.
3. combines `assert_panel_equal` and `assert_panel4d_equal`, because they are both doing the same thing. Now it's just `assert_panelnd_equal` with specific assert functions for each.
4. Switched unnecessary `check_index_freq` check in `pandas.io.tests.test_pytables:TestHDFStore.test_index_types` to be `check_series_type`, (because none of the tests ever produce objects with `freqstr`)
5. Removed `check_index_freq` argument and corresponding code from `assert_series_equal` since it's not being actively used anywhere in the test suite


Labels:

Comments:
the freq checks are really to avoid having a separate method to check index vs index and datetime index vs datetime index, as the only the latter has a freq

@jreback Actually, none of the tests pass if you force the frequency check to be explicit. None of the objects produced have a `freqstr` attribute in that test.

right - even a datetime index does not necessarily have a freqstr 

@jreback to be really clear, all I mean is that making it so that both objects have to have a freqstr is fine for the entirety of the test suite, it's just that one test that was doing it wrong (probably because check_freq was added and it didn't update its positional arguments).

@jreback at least in 2.7, it looks like that test is the only place in the entire test suite that checks index frequency.

it doesn't need to be explicitly test I think
because equals on 2 datetime indexes checks it

@jreback yeah, I'm just running this build right now - https://travis-ci.org/jtratner/pandas/builds/9791919 - which puts an assert False under the check_freq part of assert_series_equal and also checks that none of the functions in that one test case have freqstr. If it doesn't, are you okay with removing that argument and branch from assert_series_equal?

@jreback if you're okay with this, I'll squash it together - just wanted to leave them separate for now in case you didn't want a part of the PR.

this is all ok

that test was probably just thrown together (obviously should have kw args)

go ahead and I'll merge

@jreback does this need to go in docs? if so, since this is small, which file?

@jtratner what do you mean go in docs?

meant to say release notes. I thought you all had said something about
wanted to move small fixes out of one doc (release.rst? v0.13.0.rst?)

oh....what I meant was that if you say fixing bug xyz, just put the reference in the release.rst. If its a feature (or important bug fix) or you just think most users should know about it in whatsnew then go ahead and put it in there TOO. But for the most part, no point in duplicating in BOTH release.rst and whatsnew (e.g. v0.13.0).

merge?

yes.

thansk!
DOC: lrange, lzip --> list(range and list(zip
State: closed

Internally, definitely want to use `lzip` and `lrange`, but for clarity I think it's better to just use the written out versions of them, especially since they aren't really key components of pandas. (and generally would not expect someone working through an example to care about compatibility issues).

This only changes docs.

cc @cpcloud


Labels:

Comments:
+1

merge?

ready to go for me

thank you sir!
TST: better assertion messages on test failures (GH4397)
State: closed

closes #4397

a start at least


Labels:

Comments:
@cpcloud 
cc @jtratner

good?

what about a `name` attr check for indices?

and maybe just pep8 this sucker while ur at it ...

I think that's handled separately (needs check_names to be true), maybe after @jtratner PR we can turn it on in full

oh ok

oh i see it's there nvm

peped

nice llooks good glad to have some more info here!

you are prob right....feel free to put this in....

my main reason for having this is the problem of

`assertion failed....left.dtype == right.dtype`...but have no idea what happened
providing kwargs to create_index at store.append()
State: closed

This should enable the creation of a 'full' index at append time.


Labels:

Comments:
Trying to solve #4424 

I am not so sure this is a good idea. The problem is we usually want to create indexes on multiple columns. But optlevel/kind would usually be applied to a single column (e.g. optlevel=9,kind='full' which can ONLY be done on a single column)...I think the workaround is good

Agreed. Is closing this the same as rejecting the PR? Go ahead and reject / close.

yep...thanks
FIX: doc imports
State: closed

Bunch of doc build cleanups

closes #4418.


Labels:

Comments:
Jinja2 from pip doesn’t work for python 3 so it's a moot point. 

i should say it doesn't work for py3.2

also not sure how to deal with the relative import issue i'll try the abs import

arg

http://stackoverflow.com/questions/16981921/python-3-relative-imports

Maybe put an `__init__.py` in doc and then import with doc at top level.

or just do sphinxext and then just `from sphinext.blahblah import`

i got it to build....issue is that conf.py lists plot_directives from matplotlib but then pandas includes its own.... not sure what that's about

now i'm fixing the slew of python3 incompatible doc examples

wow almost all were fixed by this single line

``` python
from pandas.compat import lzip as zip
```

could make it explicit with these two lines:

``` bash
perl -pi -e "s/zip/lzip/g" ./doc/**/*.py
perl -pi -e "s/from pandas.compat import lzip as zip/from pandas.compat import lzip/" ./doc/**/*.py
```

so it's explicit. added terminating slash

arg these compat issues are SO annoying

bottom line for now is that the docs need to be able to be built on py2.7 which they can't be right now

@cpcloud just revert my edits to the docs (i.e., back before the merge commit from jreback), dump everything but the docs/*.py changes and commit it.  They worked before that, if we're only building on 2.7, then who cares about compatibility?

I think it would be:

```
git checkout 338f3406fadc6d114accbc94fac6c7ba075a10f5 -- docs/**/*.py docs/*.py
```

this fixes the doc building??

not quite...still need to add the changes to examples for py3 compat

ohhh :-/

@cpcloud after you fix this, maybe we should add docs build to Travis on 2.7 so that changes to docs to stop them from building don't happen again.

Also, I apologize for not catching this - I just didn't think to run make docs. 

no problem...happy that i can now run py3 except for docs!

add to slow build or "production"? i think production since that has `scikits.timeseries`

also getting some use out of `vim` `grep` which i should probably use more often

actually `Ack` as it turns out is _much_ better

@cpcloud are you using the vim ack plugin? very useful.  That + Ctrl-P is really helpful.

i love ctrl+p!

like emacs ido-mode

i feel like this is attack of the killer `range`s!

i'll add the doc build to this PR

ok whew got them working ... fixes on the way

yeah so i don't think the doc build on travis is a good idea

would have to install R which requires a whole boatload of packages (55 MB d'load) which will be breathtakingly slow when travis is running in ~~1990~~ 28.8K mode

@jreback ok 2 merge?

all docs fixed??? crazy

ok