BrandonFors/pandas-github-issues huggingface dataset:
DOC: Adding Panel docs to API, and shortening line length in panel.py
State: closed


Labels:

Comments:
In it goes. 

If your editor supports it, try to eliminate dangling whitespace at the end of lines,
it drives us emacs users crazy :)

05a6f8e.

Again, you're welcome to work on anything you like.  But if you want to take a dip into
code while still helping clean up the docs, maybe #3338 would be a good choice.

Thanks again.

@changhiskhan , can you please do a website docs refresh when you get a chance?
DOC: Adding parameters address issue sub-points 8 and 10 - 24 (issue #2916)
State: closed


Labels:

Comments:
We're glad to have contribs on whatever you like to work on, 
but If it suits you:

http://pandas.pydata.org/pandas-docs/stable/api.html#panel

Could use some TLC and is a good, systematic way of getting to know the lib.

The general "cheat sheet" issue has been in limbo for a while, with several
attempts not really hitting the mark.

Thanks, merged.
DOC: ref / val caveat, point at pandas methods
State: closed

This in part addresses #3340.

I added a few comments in the doc that point users ad using the pandas
at, iat, loc, iloc, etc. methods and included an example similar to
the one exposed in #3340 that addresses some of the reference / value
intricaies encountered with pandas and numpy objects.

To make things clear I selected a rather verbose style.
Let me know if this is over the top ;-)


Labels:

Comments:
looks good, I would add the notes using the rst `note` directive, makes them stand out (with the lines around them)...my 2c

like this

```
.. note::

   The recent addition of ``.loc`` and ``.iloc`` have enabled users to be quite
   explicit about indexing choices. ``.ix`` allows a great flexibility to specify
```

@jreback thanks for your pointer. Just pushed another iteration over the text with .rst directives.

next tip:rebase these to a single commit (you don't always want to do this, but in this case since
these are just changes it makes sense)

`rebase -i <commit>`

use the commit 1 previous to your first commit
you can also do `HEAD~4` which in this case will get you that commit

then either squash (s) / fixup (f) the commits to form a bigger one (s preservers the commit message, f does not)

then

`git push myfork branch_name -f`

will replace all of these commits with a new one.... (w/o the f it just appends new commits)

@jreback here we go ;-)

@dengemann , can you wrap that text to match the rest of the document?
That's easier to revise in a text editor later on.

Yeah, I forgot about about that, sorry. Hold on.

On Sat, Apr 20, 2013 at 11:54 AM, y-p notifications@github.com wrote:

> @dengemann https://github.com/dengemann , can you wrap that text to
> match the rest of the document?
> That's easier to revise in a text editor later on.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3399#issuecomment-16701562
> .

Wrapped, rebased.

@dengemann I moved the view vs copy example to consolidate with the section we had for it; I think should actually expand this example to show a proper assignment using .loc/.iloc (using your example), want to do this?

>  I moved the view vs copy example to consolidate with the section we had for it;

Thanks!

> I think should actually expand this example to show a proper assignment using .loc/.iloc (using your example), want to do this?

Sure, I might find some time tonight.
ENH: speed up testing when --processes arg is given to test_fast.sh
State: closed

I know this is a stupid one liner but...this allows the pytables fixtures to be shared, but not split (because PyTables is not reentrant [only reading is thread-safe]), during a test run with multiple processors. I get a 3x speed up (75 secs to 25 secs) just by adding this single class variable.


Labels:

Comments:
Whoa travis is failing to install pandas...

@y-p I see you are getting similar errors on your travis build as well.

nope, all sweetness and light there. are you updated to latest master? there was a glitch
with a recent PR that touched the travis scripts.

Yep I am and I was when I pushed that...weird. [This](http://travis-ci.org/cpcloud/pandas/builds/6459979) shows that one of your builds failed. I think I can see it because I'm still tracking your fork.

checking.

The log looks like a pypi hiccup which failed some dep installs including cython which caused it
to crash and burn.

Looks ok now:
https://travis-ci.org/y-p/pandas/jobs/6460587

I was actually hoping that waiting on IO would make parallelism worthwhile,
to speed up travis with one thread. But i see no noticeable change.

Should I close this? Seems not worthwhile anymore.

I'm more then glad to make things faster if it's possible, but I just can't repro on
my local machine OR travis, so if this isn't faster in practice - yep, close.
ENH/CLN: make Timestamp repr valid python code, like datetime does.
State: closed

In the same vein as #3038 , make `repr()` output valid python code where feasible. I find this not
being the case inconvenient for example when issues contain code examples, 
and the data needs to be recreated by hand rather then merely copy pasted.

```
In [2]: zs=[Timestamp('99-04-17 00:00:00',tz='UTC'),
   ...: Timestamp('2001-04-17 00:00:00',tz='UTC'),
   ...: Timestamp('2001-04-17 00:00:00',tz='America/Los_Angeles'),
   ...: Timestamp('2001-04-17 00:00:00',tz=None)]
   ...: for z in zs:
   ...:     print repr(z)
   ...:     assert eval(repr(z)) == z
Timestamp('1999-04-17 00:00:00+0000', tz='UTC')
Timestamp('2001-04-17 00:00:00+0000', tz='UTC')
Timestamp('2001-04-17 00:00:00-0700', tz='America/Los_Angeles')
Timestamp('2001-04-17 00:00:00', tz=None)
```

Misgivings?


Labels:

Comments:
looks ok to me

Great ENH.
multiindex xs slicing bug fix
State: closed

#2903 solves the case for the first level but using the same variables from that issue, the following code raises a `TypeError` for the second level:

``` python
import pandas as pd
from numpy.random import rand
columns = pd.MultiIndex.from_tuples([('a', 'foo'), ('a', 'bar'), ('b', 'hello'), ('b', 'world')], names=['lvl0', 'lvl1'])
df = pd.DataFrame(rand(4, 4), columns=columns)
df.xs('foo', level='lvl1', axis=1)
```

This pull request fixes that.


Labels:

Comments:
looks ok to me

Added a vbench for xs,  looks fine.
CLN: DataFrame move doc from __init__ to cls
State: closed

This, in part addresses our discussion from #3337.
This PR moves the doc string  from **init** to the more common class location.
(cf.. numpy, rest of pandas classes that have a doc string)
Besides STY / consistency  this would allow for more directly accessing the doc string in a debugging (pdb) session: 

``` Python
print mydf.__doc__
```


Labels:

Comments:
Thanks. For consistency's sake, we'll hold off on this until the rest of the major classes 
get the same treatment: Series, Panel, *Index...

@y-p one approximation to what is to be regarded as 'major classes' might be looking what we find in the pandas name space:

These are the classes for which **doc** is not directly under the class statement, below the rest of the classes.

``` Python
In [11]: [(k, c.__doc__) for k, c in pd.__dict__.items() if k[0].isupper() and c.__doc__ is None]
Out[11]: 
[('WidePanel', None),
 ('SparseTimeSeries', None),
 ('Panel4D', None),
 ('SparseSeries', None),
 ('Timestamp', None),
 ('Period', None),
 ('Series', None),
 ('DateRange', None),
 ('NaT', None),
 ('Panel', None),
 ('TimeSeries', None),
 ('Int64Index', None)]

# here the ones for which __doc__ is under the class statement
In [12]: [k for k, c in pd.__dict__.items() if k[0].isupper() and c.__doc__ is not None]
Out[12]: 
['SparseArray',
 'Categorical',
 'ExcelWriter',
 'DataFrame',
 'Index',
 'SparseList',
 'PeriodIndex',
 'Factor',
 'SparseDataFrame',
 'Term',
 'MultiIndex',
 'HDFStore',
 'DateOffset',
 'SparsePanel',
 'ExcelFile',
 'TimeGrouper',
 'DatetimeIndex']
```

I could do the same for the missing cases listed above. Wdyt?

Yeah, that'd be good.
That's pretty inconsistent alright.

The remaining cases are such that constitute pure factory classes employing **new** or have entirely missing doc strings.
Would it make to add empty doc strings in such cases?

``` Python
In [6]: sorted([(c.__module__, k) for k, c in pd.__dict__.items() if k[0].isupper() and c.__doc__ is None])
Out[6]: 
[('pandas.core.daterange', 'DateRange'),
 ('pandas.core.index', 'Int64Index'),
 ('pandas.core.panelnd', 'Panel4D'),
 ('pandas.core.series', 'TimeSeries'),
 ('pandas.sparse.series', 'SparseTimeSeries'),
 ('pandas.tslib', 'NaT'),
 ('pandas.tslib', 'Timestamp')]
```

Btw. is there a 101 around on how to activate Travis-CI for ones fork?

http://about.travis-ci.org/docs/user/getting-started/

`NaT` and `DateRange` don't really need a docstring, the rest do. 
Marking this as 0.11, since there's still a few days until 0.11 final,
and I'll put the missing docstrings together till then.

Thanks.

Pushed directly to master with some added docstrings.

I believe this is your first commit to pandas. good job! :+1: 

Thanks @y-p ! ;-)
PERF: fixed int64 indexing perf issue when conversion to int64
State: closed

fixes an issue I when using non-int64 Int64Indexes (which is very rare) anyhow

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
timeseries_large_lookup_value                |   0.0213 |   2.5117 |   0.0085 |
Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [0933ba3] : PERF: fixed int64 indexing perf issue when conversion to int64
Base   [8f92d9a] : Merge PR #2881 and fix unit test

```


Labels:

Comments:
fixes this bench in #3326, I think was introduced in #3322
BUG: ensure index casting works even in Int64Index
State: closed

```
df = DataFrame({'A' : [1.1,2.2,3.3], 'B' : [5.0,6.1,7.2]}, index = [2010, 2011, 2012])
df.index = df.index.astype(int)
df.ix[2010]
```

was failing - fixed up

from the mailing list
https://groups.google.com/forum/?fromgroups#!topic/pydata/ClJA5ldNQNQ


Labels:

Comments:
Thanks Jeff
BUG: GH3311 Dataframe where with a datetimelike was not correctly selecting
State: closed

closes #3311


Labels:

Comments:
@jreback I think this PR introduced some behavior (definitely codified it in a test) that is incorrect.  From the test:

```
        df = DataFrame(dict(A = date_range('20130102',periods=5), 
                            B = date_range('20130104',periods=5),
                            C = np.random.randn(5)))

        stamp = datetime(2013,1,3)
        result = df[df > stamp]
```

But if you just look at the `df  > stamp`:

```
>>> df > stamp
       A     B     C
0  False  True  True
1  False  True  True
2   True  True  True
3   True  True  True
4   True  True  True
```

Consistency with `Series`, `Index`, and scalars would require comparison with the "C" column to raise `TypeError`.  Also the reverse inequality _also_ comes back as all-True for the "C" column:

```
>>> df < stamp
       A      B     C
0   True  False  True
1  False  False  True
2  False  False  True
3  False  False  True
4  False  False  True
```

So the question: OK to make this raise?  Or is there a compelling reason for this to be different?
TST/CLN: legacy pickle testing
State: closed

CLN:
- standardized testing data to live under tests/data (for that set of tests),
- added method `util/testing.get_data_path` to give you this path

TST:
- added `io/tests/test_pickle.py` and `io/tests/generate_legacy_pickles.py`
  to generate legacy pickle files and provide comparsions
  just run `python path_to_generate_legacy_pickles.py`
  
  it will create a version of pickle for that python version, arch, and system, and store in `pandas/io/tests/data/legacy_pickle/VERSION`
- sample pickles for windows-64/2.7/0.10.1 and linux/2.7/0.10.1 and linux/2.7/0.11.0 added
- need to run this on various combinations to generate the reference library
- allows backwards compatibility testing for these pickles


Labels:

Comments:
anybody know if py3 is supposed to be able to read py2 pickles?
(this is a pickle I created in py2.7 reading in 3.3)

Do we have to unicode any strings in the data first?

```
======================================================================
ERROR: test_read_pickles_0_10_1 (pandas.io.tests.test_pickle.TestPickle)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_a8cdf20-py3.3-linux-x86_64.egg/pandas/io/tests/test_pickle.py", line 42, in test_read_pickles_0_10_1
    self.compare(vf)
  File "/home/travis/virtualenv/python3.3_with_system_site_packages/lib/python3.3/site-packages/pandas-0.11.0.dev_a8cdf20-py3.3-linux-x86_64.egg/pandas/io/tests/test_pickle.py", line 26, in compare
    data = pickle.load(fh)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 14: ordinal not in range(128)
```

Turns out this does work, but you have to jump thru some hoops, so I updated the main pickle load method
Apparently numpy saves the dtype strings as strings and not unicode (I guess for compat)

from core/common

```
def load(path):
    try:
        with open(path,'rb') as fh:
            return pickle.load(fh)
    except:
        with open(path,'rb') as fh:
            return pickle.load(fh, encoding='latin1')
```

does the other way round work?

no, can't read a py3 pickle in py2 (its protocol is 3, which py2 doesn't know about), raises a ValueError (which I am now catching in the comparators). I added a py3 pickle as well. Cross platform is fine as well. 

Merged. thanks!
BUG: DataFrame.hist passes keyword arguments to mpl.hist; closes #3296
State: closed

This is a fix for https://github.com/pydata/pandas/issues/3296 .

Now keyword arguments to `plotting.grouped_hist` are passed to `matplotlib.Axes.hist`. This allows you to create e.g. normalized, cumulative, or log scaled histograms::

```
df = pd.DataFrame(np.random.randn(500, 2), columns=['A', 'B'])
df['C'] = np.random.randint(0, 4, 500)
df.hist(column=['A'], by=df.C, normed=True)
```

PS: the guideline says I should enable Travis-CI on my fork - is there a tutorial how to do that? thx!


Labels:

Comments:
there might be an issue with backward compatibility. Since the current master version ignores the keyword arguments the following snippet is valid:

```
df.hist(column=['A'], foo='bar')
```

In this PR, the above line will raise an AttributeError since `matplotlib.Axes.hist` does not support the argument `foo`.

That last is fine IMO.
BUG: fix unsafe dtype conversion in Series
State: closed

fixes assignment to smaller itemsizes on dtype upcasting


Labels:

Comments:
@wesm this should fix it. FYI once series inherits from NDFrame this magic is completely gone (as its a simple block change, which is easy to do even inplace)

@wesm this new version is cleaner

Shoehorning `Series.where` into working for the `__setitem__` case isn't totally satisfactory but this will work. At some point this all will need to get cleaned up

@wesm already set to clean up in 0.12. where is moved to core/generic.py and a dtype change is simple because just make a new block...coming soon

@y-p sorry....:)
BUG: test case showing why assigning to dtype is unsafe
State: closed

@jreback this is a test case that shows why assigning to `dtype` is unsafe, namely that the dtype item sizes have to be compatible obviously =P Could you take a look at this and fix? This came up when I booted the windows box back up because Windows uses int32 by default.

There is a secondary bug here that the test case in question probably _should not_ cause the dtype of the resulting Series to change from int64 to float64, agreed?


Labels:

Comments:
moved to #3292
BUG: suppress error raise by nonnumeric columns when plotting DataFrame ...
State: closed

...#3108

suppressing AttributeError here so you can plot the DataFrame and automatically exclude non-numeric columns. @y-p @wesm @jreback any thoughts on 1) whether it should raise instead of exclude and 2) whether catching the AttributeError here is too dangerously broad?


Labels:

Comments:
maybe add an option `raise_on_error=True` as the default for plotting? (we use this option syntax else, mostly internally but use can set it)

@jreback done. By default it will raise and the users should see a helpful error msg that includes a hint to set "raise_on_error=False" to suppress the exception.

I guess the warning makes this sort of ok, but the exception is "AttributeError" not "ValueError" ,
it's a very round-about way to catch the fact that the user is trying to plot non-numerical data 
and I think it might prove brittle. 

+1 for catching it higher up the stack, rather the 50 fathoms down mpl with a seemingly 
unrelated error.

why don't we do a get_numeric_data and just avoid this issue?
though I guess user will be wondering why there string data is not plotting

Merge?
ENH: add to_series() method to Index and subclasses GH3275
State: closed

#3275

``` python
In [10]: df=mkdf(10,5,r_idx_nlevels=2)

In [11]: df.index
Out[11]: 
MultiIndex
[R_l0_g0  R_l1_g0, R_l0_g1  R_l1_g1, R_l0_g2  R_l1_g2, R_l0_g3  R_l1_g3, R_l0_g4  R_l1_g4, R_l0_g5  R_l1_g5, R_l0_g6  R_l1_g6, R_l0_g7  R_l1_g7, R_l0_g8  R_l1_g8, R_l0_g9  R_l1_g9]

In [12]: # new method
    ...: df.index.to_series()
Out[12]: 
R0       R1     
R_l0_g0  R_l1_g0    (R_l0_g0, R_l1_g0)
R_l0_g1  R_l1_g1    (R_l0_g1, R_l1_g1)
R_l0_g2  R_l1_g2    (R_l0_g2, R_l1_g2)
R_l0_g3  R_l1_g3    (R_l0_g3, R_l1_g3)
R_l0_g4  R_l1_g4    (R_l0_g4, R_l1_g4)
R_l0_g5  R_l1_g5    (R_l0_g5, R_l1_g5)
R_l0_g6  R_l1_g6    (R_l0_g6, R_l1_g6)
R_l0_g7  R_l1_g7    (R_l0_g7, R_l1_g7)
R_l0_g8  R_l1_g8    (R_l0_g8, R_l1_g8)
R_l0_g9  R_l1_g9    (R_l0_g9, R_l1_g9)
dtype: object

In [13]: # so now this is possible
    ...: df[df.index.to_series().map(lambda x: "4" in x[0])]
Out[13]: 
C0              C_l0_g0 C_l0_g1 C_l0_g2 C_l0_g3 C_l0_g4
R0      R1                                             
R_l0_g4 R_l1_g4    R4C0    R4C1    R4C2    R4C3    R4C4
```


Labels:

Comments:
sandbox?

I say put in
it's mostly internal

Fine by me

Follow up on ml?

sure
ENH: update DataFrame to_latex for nicer typesetting
State: closed


Labels:

Comments:
Is this merge-able or does it need a unit test?

There doesn't seem to be a unit test for the to_latex method right now, but I can add one in the to_html style (comparing strings) if you'd like.

Also, I forgot to mention this requires the users to have the \usepackage{booktabs} in their latex source. So I should probably add that to the doc string.

OK thanks. This isn't official enough functionality to be that sensitive. 

Hi wes, I am using the to_latex method and found the \usepackage {booktabs} bug. Its harder to find with LaTeX and i was using matplotlib too. Probably missed the doc string.
BUG: pprint_thing should pprint and limit nested sequences in dicts GH3251
State: closed

#3251

**before**

``` python
pd.options.display.max_seq_items= 5
print pd.core.common.pprint_thing(dict(foo=range(100)))
foo = range(5)
foo[0] = foo
print pd.core.common.pprint_thing(dict(foo=foo))
{'foo': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]}
{'foo': [[...], 1, 2, 3, 4]}
```

**After**:

``` python

In [16]: pd.options.display.max_seq_items= 5
    ...: pd.core.common.pprint_thing(dict(foo=range(100)))
    ...: 
Out[16]: u'{foo: [0, 1, 2, 3, 4, ...]}'

In [17]: foo = range(5)
    ...: foo[0] = foo
    ...: pd.core.common.pprint_thing(dict(foo=foo))
    ...: 
Out[17]: u'{foo: [[[[...], 1, 2, 3, 4], 1, 2, 3, 4], 1, 2, 3, 4]}'
```


Labels:

Comments:
moved to 0.12 to be merge right along with #3038, to minimize possible test 
breakage for users due to a formatting flicker between merging this (change)
and that (change back, in another way).

preemptive merger !

hey what now?

Jeff humor :)

I'm trying to drag all y'all to a sane "merge window" type workflow,
and f*\* A if i'm not gonna lead by example.

I guess you meant to call me "mergerer!",  what a fun word to pronounce, rhymes well too.

Great work on 0.11 Jeff!

U2 @y-p
TST: more robust tests in HDFStore
State: closed

should close #3069
will print additional output if this somehow fails, which I have never seen (except on travis)


Labels:

Comments:
Doing str(e) won't tell the exception type that was raised (IOError?)
and if the test doesn't fail, ti'll be difficult to find the build that contains the
debugging lead...

I know but I have never seen it fail except a couple of times on Travis
and their is no reason it should fail (if the tests before didn't) except some sort of weird low level Pytables failure
which I suppose is possible

Yeah, it looks solid. All the same, a failing test is trying to tell us something. it might
be travis wonkiness, it might be a data corruption bug. hemstringing the test
might end up hiding a real issue.

is there a way in nose to easily show a warning of some kind?

From the annoying nose of the "inplace=True" patch, I'm pretty sure
warnings from the warnings module are not suppressed by nose.

If you're going for a looping test run looking for the error.... I already did a 1000
and came up empty.

there must be a very rare underlying error in PyTables that gets passed up which I am trapping and returning a top-level exception (related to IO)...
BUG: GH3216 Upcast when needed to DataFrame when setitem with indexer
State: closed

closes #3216

BUG: GH3216 Upcast when needed to DataFrame when setitem with indexer

The following would raise previously

```
In [8]: df = pd.DataFrame([{"a": 1}, {"a": 3, "b": 2}])
In [9]: df['c'] = np.nan
In [10]: df.ix[0,'c'] = 'foo'

In [11]: df
Out[11]: 
   a   b    c
0  1 NaN  foo
1  3   2  NaN

In [12]: df.dtypes
Out[12]: 
a      int64
b    float64
c     object
dtype: object
```

ENH: On a mixed DataFrame, allow setting with indexers with 
          ndarray/DataFrame on rhs (this was disallowed in the code previously)

```
In [8]: df = DataFrame([[1,2,'foo'],[3,4,'bar']],columns=['A','B','C'])
In [9]: df2 = df.copy()
In [10]: df2.ix[:,['A','B']] = df.ix[:,['A','B']]+0.5
In [11]: df2
Out[11]: 
     A    B    C
0  1.5  2.5  foo
1  3.5  4.5  bar

In [12]: df2.dtypes
Out[12]: 
A    float64
B    float64
C     object
dtype: object

In [13]: df.dtypes
Out[13]: 
A     int64
B     int64
C    object
dtype: object
```


Labels:

Comments:
@wesm this cleans up some of the setitem_indexer code (still a bit spaghetti but lots of cases to handle)....

Yo jeff, one blank line between commit message subject and body. I thank you.

huh ?

When you write your commit messages, please insert a blank line between the first, subject, line
and the rest of the commit message. it confuses all kinds of git tools, as well as `git log`.

ahh didn't know that 
will do

no worries.