BrandonFors/pandas-github-issues huggingface dataset:
Make sure not to use bottleneck when it is not installed.
State: closed

I don`t have bottleneck, this little change allows me to import trunk again
nosetests running fine


Labels:

Comments:
Sorry about that. I changed it to a slightly more EAFP:

```

    try:
        bn_func = getattr(bn, bn_name)
    except NameError:
        bn_func = None
```
ENH: closing issues #395, #449, #570, #571
State: closed

Fix column string formatting; index and header options to to_string; decorator for docstrings; floating point prettier printing. Also added to release notes & documentation. Address #395, #449, #570, #571


Labels:

Comments:
This is great. You have some trailing whitespace problems-- is your editor configured to strip trailing whitespace? I think we can set git to do it automatically on commit. 

The new string formatting looks awesome. There's a tiny buglet with padding zeros on scientific notation:

```

            A          B          C          D        
2000-01-03 -1.656e-08 -7.652e-08  9.046e-08  1.231e-07
2000-01-04  2.767e-08 -3.075e-08  1.148e-08  1.398e-08
2000-01-05 -1.505e-07 -6.278e-08  1.732e-08 -9.14e-080
2000-01-06 -7.624e-09 -4.579e-08  9.167e-08  2.895e-08
2000-01-07  1.238e-07  2.228e-07  6.408e-08 -2.168e-07
2000-01-10 -9.959e-08 -7.705e-09  6.522e-08  8.063e-08
2000-01-11  1.75e-070  8.02e-080  4.244e-08 -8.221e-08
2000-01-12  1.178e-08  1.204e-09  1.877e-07  2.385e-07
2000-01-13  7.682e-08  1.259e-07  1.135e-07 -1.111e-07
2000-01-14  4.326e-09  1.043e-08 -8.295e-08  6.293e-08
2000-01-17 -2.4e-0700 -1.156e-07  1.866e-08  1.331e-07
2000-01-18  8.905e-08 -1.822e-07  6.108e-08 -1.147e-07
2000-01-19 -1.088e-07  2.057e-08  1.323e-07 -1.07e-070
2000-01-20 -4.258e-08 -5.991e-08 -1.944e-07 -8.385e-08
2000-01-21  1.297e-07  9.573e-08  4.32e-080  7.103e-09
2000-01-24 -1.045e-07  5.521e-08  4.563e-08 -1.843e-08
2000-01-25 -1.461e-07 -7.573e-09  1.283e-07  1.451e-07
2000-01-26 -1.063e-07 -1.561e-07  1.223e-07  1.332e-07
2000-01-27  9.289e-08  9.192e-08  6.909e-08  5.926e-08
2000-01-28  5.496e-08  5.334e-08 -3.28e-070 -1.086e-07
2000-01-31 -7.841e-08  3.568e-08  3.921e-08 -1.512e-07
2000-02-01 -1.123e-07  6.324e-08  1.773e-07 -3.672e-08
2000-02-02 -1.42e-070 -1.12e-070  1.044e-07 -2.402e-09
2000-02-03  7.559e-08  8.136e-08  2.631e-08  1.09e-070
2000-02-04 -2.969e-08  4.038e-08  1.433e-07  2.184e-09
2000-02-07  1.106e-07 -4.719e-08  8.871e-08 -7.917e-08
2000-02-08 -7.816e-08 -3.934e-08 -8.182e-08 -1.172e-07
2000-02-09  1.354e-07 -1.268e-07 -5.352e-08  1.918e-08
2000-02-10 -1.104e-07 -2.125e-07 -2.127e-07  3.835e-08
2000-02-11  6.909e-08  1.969e-07 -1.518e-07 -7.226e-09
```

Can fix it separately after merging this-- the rebase was somewhat terrifying but appears to have gone through cleanly

OK I think I got that buglet licked here:

https://github.com/wesm/pandas/commit/94fbd7853756812c3cdcfe9637e0f1463bfb8dce

and off to bed :)
implemented description property
State: closed

Hi,

This is a basic implementation of description and unit properties in DataFrame. 
https://github.com/wesm/pandas/issues/39

I used dict to store both description and unit. I thought about using tuples but dict would be better if a user wants to modify individual descripition/unit. 

One can directly access description/unit, and also DataFrame.info() will print out those if they exist.

I haven't touched at the storing part yet so it is very rudimentary now (no way to save the descriptions), but I wanted to know what you think. Please let me know.

And also, is this (topic branch) right way to submit a pull request? I'm still learning about git. :)

Happy new year!
-Joon


Labels:

Comments:
Yes this is correct git usage :) 

The problem with this is that as soon as you do anything with the DataFrame you've created the column metadata is going to be lost. Probably the right way to do it would be to store the information _inside the column index_ itself. So it's a bit of an involved undertaking. I will take a closer look at your patch and think about how much work it would be to do 

That makes sense. Let me know!

Hey sorry this PR has been open for so long. I'm going to close it until more work is put into a deeper solution for tracking metadata. It's a pretty big project I think
more changes to setup.py
State: closed

Maybe consider just making people use either `setuptools` or `distribute`?


Labels:

Comments:
I have had ongoing problems with setuptools's interactions with Cython. If you can figure out how to get setuptools to play nice with my Cython extensions, then I will do it. Otherwise I can't accept the patch

Is it just setuptools in particular, or distribute also?   Distribute seems to work right on OSX (I don't have a windows machine to test it on).  Is this different than your experience?

It seems this is 'known fixed' in distribute:  https://bitbucket.org/tarek/distribute/issue/195/cython-build_ext-support-broken-when-using

So, big picture, and my goals here:
- I want setup.py to be as friendly as possible when failing 
- I want pip install to get as much as possible right, without too much headache (the numpy bit is a bit of a headache!)

If that's not possible, then maybe a 'suggested install style' should go out with the package... something like:

try:
    import Numpy, dateutil, .....
except:
   print "these onese failed.... here's how to install them"

With a lot of notes saying (in code) why we don't use the 'install_requires' bits.  

Perhaps as an alternative, it might be worth cribbing from `lxml`'s setup process...

https://github.com/lxml/lxml

Clearly, this can wait :)

That's a good idea. Wouldn't mind trying to get this done for 0.7.0

I can confirm that things are borked under setuptools in osx, and they do work fine under distribute.  Frustrating stuff!  

here's an open issue related to the setuptools bug

https://github.com/wesm/pandas/issues/192

Cherry-picked and merged per #578
Test coverage improvements
State: closed

- a few modules up to 100%
- a few small bugfixes on non-obvious codepaths


Labels:

Comments:
Not sure how you got a merge commit in this but...

Very strange. Not sure what I did - maybe rebase master before pushing
my pre-rebase changes to remote origin, possibly, could that do it?

On Jan 1, 2012, at 8:50 PM, Wes McKinney
reply@reply.github.com
wrote:

> Not sure how you got a merge commit in this but...
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/wesm/pandas/pull/562#issuecomment-3326901

Must have been a merge issue / conflict while rebasing. Dunno
Autodep
State: closed

still need numpy for setup, but at least we get the right version of `python-dateutil`


Labels:

Comments:
`install_requires` is invalid with regular distutils. Is there a way to check if pip/distribute is in use?

Sorry for the delay here!

As with usual in packaging, there are a million solutions :)  
- turn it off, and let people figure this stuff out themselves.  Something like "if there is no setuptools, then tell them eveyrthing they need to get going"
- just require one of `setuptools` or `distribute`.  It think this is the case for most people.  

Look for some inbound code... I think this is an improvement...
various HDFStore fixes regarding tables
State: closed

commit d1f439c: added optional keyword argument 'itemsize' to HDFStore.put and HDFStore.append to control string length; default 255

it sets the string length of strings in the index and columns for the table (rather than the previous behavior, which was to take the max string length from the longest index/column string in the dataset that is used to create the table). Default is 255.

commit 2b13848: nicer error message for store.put('ccc', DataFrame(['hi'], ['a'], ['col']), table=True); also fixes a bug in d1f439

commit 2bce685: before this commit, the following would discard column1:
store.put('eee', DataFrame([0], ['a'], ['column1'])); store.append('eee', DataFrame([1], ['a'], ['column2'])); store['eee']
even with this commit, strings such as column names which are longer than "itemsize" and which are .appended in the future will be silently truncated. I recommend that a warning be raised when this happens, but i didn't add that myself because i couldn't easily find the place where the truncation occurs, or how to extract the current string length.

i'm having trouble building so i won't be able to try writing unit tests. however, here are tests for these commits:

from pandas import *; store = HDFStore('test.h5')

store.put('ccc', DataFrame(['hi'], ['a'], ['col']), table=True); 
 # should raise "TypeError: At this time, Pandas HDFStore tables only support floating point values."

store.put('fff', DataFrame(['0'], ['a'], ['column1'])); store.append('fff', DataFrame([1], ['a'], ['column2'])); 
 # should raise "ValueError: Can only append to Tables"

store.put('fff', DataFrame([0], ['a'], ['column1']), table=True); store.append('fff', DataFrame([1], ['a'], ['column2'])); store['fff']
 # should yield
 #    column1  column2
 # a  0        1  
 #   (this was true even before the present commits)

store.append('j', DataFrame([0], ['a'], ['column1'])); store.append('j', DataFrame([1], ['a'], ['column2'])); store['j'] 
 # should yield:
 #    column1  column2
 # a  0        1  
 #   (this was true even before the present commits)


Labels:

Comments:
The test suite (pandas/io/tests/test_pytables.py) fails after applying these changes. There are also no unit tests for the new functionality added / fixed. Could you get the tests to pass and write a couple of tests exploring the things you worked on? Otherwise I won't be able to merge this for a while until I can find the time to do it myself

Secondly...slight git faux pas-- the changes are in your master branch. you'll want to clean up your master this is done (or move this work to a branch and fix up master right away)

Just pinging you to see if you want to follow through with these changes (and getting the test suite to pass)

Ping

oops, sure, not sure when tho :(

On Thu, Feb 23, 2012 at 11:46:25AM -0800, Wes McKinney wrote:

> Ping
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/515#issuecomment-4143638

Closing this PR-- if you get a chance to return to this at some point please let me know
Yahoo quote
State: closed

I've added a get_quote_yahoo function to io.data.py. The function returns a DataFrame of the current yahoo data for a set of symbols


Labels:

Comments:
Cherry-picked `3715dd2`. once I get all your pull requests merged you'll want to fix your master branch so be exactly wesm/master

Yes, was going to do this, but got totally confused by git :(. Cost me an
hour. Think I'll just refork the whole thing.

On Thu, Dec 8, 2011 at 12:35 AM, Wes McKinney <
reply@reply.github.com

> wrote:
> 
> Cherry-picked `3715dd2`. once I get all your pull requests merged you'll
> want to fix your master branch so be exactly wesm/master
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/wesm/pandas/pull/463#issuecomment-3055459
Minor fix for simpler indexing syntax in array of subplots.
State: closed

`plt.subplots()` returns a numpy array, so it's OK to use the more compact/familiar/efficient `a[i,j]` indexing syntax instead of `a[i][j]`.


Labels:

Comments:
Good catch, I indeed thought it returned a list of lists but an ndarray makes more sense 

On Sun, Dec 4, 2011 at 6:39 PM, Wes McKinney
reply@reply.github.com
wrote:

> Good catch, I indeed thought it returned a list of lists but an ndarray makes more sense

Thanks for the quick merge.  Loving pandas :)
Using pandas with PyQt
State: closed

I've written a simple data model for showing pandas DataFrame in a QDataTable. It is all in the sandbox, qtpandas.py. This file can be run stand-alone as an example.


Labels:

Comments:
Also, why instead of a setter use an instance attribute?

```
def __init__(self, df, parent=None):
    super(DataFrameModel, self).__init__(parent)
    self.dataframe = df
```

Do you want me to merge this or are you planning to do some more work here?

I'll redo this later on. Just skip it, it's sandbox anyway.

On Thu, Dec 8, 2011 at 12:37 AM, Wes McKinney <
reply@reply.github.com

> wrote:
> 
> Do you want me to merge this or are you planning to do some more work here?
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/wesm/pandas/pull/435#issuecomment-3055479

I make these merges and send you instructions to fix up your git branch. It's not so hard =P
Bugfix for issue 419 and .gitignore fixup
State: closed

Two different things fixed.


Labels:

Comments:
thanks, ralph. i cherry-picked that last commit and add all `*.cpp` files to the .gitignore. you'll want to revert to before your changes above and then pull wesm/master so you don't have that merge commit...annoyances of maintaining a clean repo in git. alternately make a pull request next time out of a branch
more momery efficient describe() for DataFrame
State: closed

reindex() in describe() in DataFrame was making copy of the data. I changed describe() so it processes one column at a time instead of get all columns with reindex(). It seems to be a better implementation (much less memory usage).


Labels:

Comments:
hi Joon, thanks, cherry-picked this into master. in the future can you make pull requests out of a feature branch? if you try to pull/merge wesm/master now you are going to generate a merge commit which will makes things a bit hairy. you may wish to git reset --hard to a commit before your change, then pull wesm/master

Of course. Thank you!
Handle masked arrays in DataFrame constructor
State: closed

If passing a masked array to DataFrame constructor, my expectation was that the masked entries would end up as NAN's in the DataFrame.  This does not appear to be the case.  The commits in the pull request try toe address that.

as always, comments welcome.

thanks


Labels:

Comments:
This is great-- thanks for making sure it's well-tested

I'll make an issue to investigate handling of other dtypes. like int or boolean maskedarrays will need special handling. not that important at the moment
DataFrame.to_html()
State: closed

DataFrame.to_html() -- Render a DataFrame to a html table.


Labels:

Comments:
This is great. Someone was just asking me about this (conversion to HTML) the other day

i made some fixes here for formatting non-MultiIndex etc. so the output looks like the console output:

https://github.com/wesm/pandas/commit/3af4e33b34affea9384a1f7a8be2ead069c7549d
Issues #194 (cov function) and #92 (multi-column sort)
State: closed

Added:
1. Series.cov
2. DataFrame.cov
3. test_cov in test_series.py
4. test_cov in test_frame.py

Modified:
1. DataFrame.sort
2. DataFrame.sort_index
3. test_sort in test_frame.py


Labels:

Comments:
In the future you should make these commits in a feature branch instead of master-- I'll have to cherry-pick these commits but then you should clean up your master branch to not have any merge commits-- after I take your commits you should reset to a state prior to this work then pull wesm/master...git-fu =) 

You should also set your e-mail address in your global git config 

OK I cherry picked and amended your commits and merged them in. Thanks a lot for the PR-- I'll make a few little edits and see if there's a more performant way to do multi-column sorting.

Also pls stick to 80 chars per column / other pep8 conventions
Vectorize pandas.Series.apply
State: closed

When profiling my code I noticed that numpy.vectorize was a bit faster than pandas.Series.apply.  

Execution Times:
numpy.vectorize: 3.3 seconds
pandas.Series.apply: 4.4 seconds.


Labels:

Comments:
hey, be careful to add features in branches and not the master branch :) I will take a look at the apply patch, looks like a no brainer. You may consider resetting your fork to be exactly wesm/master since you've got a couple of merge commits

Definitely a good idea for me to reset my fork.  Also, I guess the same reasoning could apply to Series.map. Unless you have some use case distinctions for map and apply in mind (besides map also accepting a dict).  It would be good to document this distinction as well :).

`np.vectorize` has some string handling problems (changing Series.map and apply to vectorize caused unit test failures). I'll patch this separately very quickly (maybe writing a Cython function) and make sure the speed is comparable

see above commit...added a new cython function that's about 25% faster than np.vectorize and gets the types right. thanks for pointing this out!
Add kind='line' or 'bar' argument to DataFrame.plot
State: closed

This is just a first stab at it, comments are very welcome, plus feel free to say you'd rather not add it at all.


Labels:

Comments:
Thanks, Dieter. I refactored the method, tweaked, the plot a little bit, and added some unit tests. Can see this being very useful-- too bad matplotlib isn't better as making these kinds of plots without a lot of tweaking

FYI you might want to fix up your master branch because I cherry-picked this commit on top of wesm/master
ENH: Added the level arguments to the series and frame math operations
State: closed

such as sum, etc.


Labels:

Comments:
Awesome, thanks. I did the DataFrame docstring templating stuff I mentioned here and updated the return types in the Series methods

https://github.com/wesm/pandas/commit/e775d09ff5ad652090b84c6bb4658469a534e588

I'll chase down the groupby error separately

fixed the bug in the above commit, turned out to be pretty simple

I just noticed that the DataFrame methods don't handle the `axis == 1` case, I'll fix that

Need to also test skipna=True vs. skipna=False

ok, I think I've got everything sorted in the above commit. Maybe a little test coverage remaining

I can write a couple more tests tomorrow if you'd like.

No worries, already added the additional tests. Seeing if I can get the release ready or almost tonight
ENH: Adding DataFrame.boxplot()
State: closed

Added DataFrame.boxplot() in similar way as DataFrame.plot() and DataFrame.hist().
Some syntactic sugar ;-)

Thinking about adding DataFrameGroupBy.boxplot(), now this call is transfered to DataFrame.boxplot().
But i would prefer the behavior to be different. When groupby is used i`m interested to see in a single plot boxplots for one particular column for all groupby keys and not in one plot to have boxpots for all columns for one groupby key. An input argument can be used to so both ways of boxplotting can be used.


Labels:

Comments:
I'll think about this a bit. There are two cases:
- Box plots of each column
- Box plot of one column, grouped by the values in another

I guess it would make sense to support both of these use cases in a single function. You could imagine a plot with multiple box plots generated off of multiple columns, would be pretty cool

So I'm skipping your patch and I hacked up a `DataFrame.boxplot` with a groupby and column selection option, it's very nice! See e.g. http://imgur.com/1oko5

take a look in the latest and try out the new boxplot function...let me know if there's anything else you would want to add

I noticed an inconsistency in the boxplot documentation. It's supposed to return an AxesSubplot instance, but at least without grouping, it returns None.

Just gave it a spin, this is what i had in mind and looks good.

some comments:
- As cswegger mentioned DataFrame.boxplot has no return call (pandas.tools.plotting.boxplot does), when i create boxplots i usually add two horizontal lines on the plot indicating the data boundaries, getting the `ax` as return from boxplot makes this easy. I did not try to use the `ax` input argument.
- Either i don`t understand the column argument or it has some issues. 
  _repo_/scripts/boxplot_test..py is where i started from:
  - df.boxplot(by=['indic', 'indic2'], fontsize=8, rot=90) -- works fine, this is what is in the test script
  - df.boxplot() -- ok
  - df.boxplot(column=['A']) -- i would expect this would only create a boxplot for column A, but boxplot for all coumns are generated
  - df.boxplot(by=['indic', 'indic2'], column=['A']) -- i woudl expect a boxplot similar as in the first example, but only for column A, it gives TypeError: unhashable type: 'list'
- I mainly use MultiIndex together with DataFrame. The by argument of Dataframe.boxplot works on columns, not on the index, workaround is to use DataFrame.delevel(), but then the indices are also boxplotted (if they hold numeric data). Deleveling also offers possibility to do groupby on multiple levels, which is not (yet :-) )possible on the MultiIndex itself.
- subplot layout improved (not always using a rectangular layout), which is nice. DataFrame.hist() could benefit from this also.
ENH: Adding argmin, argmax to Series and DataFrame.
State: closed

Hi Wess,

Pull request to let you know that i added argmin and argmax methods to Series and DataFrame.
I also included unit tests. It`s not that much work/code, but anyway an interesting enhancement i think.

some examples:
In [23]: s
Out[23]:
a    0.80015881463
b    0.983432892599
c    0.965122593504
d    0.81910825442
e    0.726239458506
Name: None, Length: 5

In [24]: s.argmin()
Out[24]: 'e'

In [25]: s[s.argmax()] == s.max()
Out[25]: True

In [26]: df
Out[26]:
               A       B       C        D       E
one two three
a   b   10    -0.6155  1.237   1.371    1.24   -0.7725
p   q   20     1.14   -0.3743  0.5251   0.2426  1.81
x   y   30     0.6626  2.004  -0.01327  1.587   0.3524

In [27]: df.argmax(axis=0)
Out[27]:
A    ('p', 'q', 20)
B    ('x', 'y', 30)
C    ('a', 'b', 10)
D    ('x', 'y', 30)
E    ('p', 'q', 20)
Name: None, Length: 5

In [28]: df.argmin(axis=1)
Out[28]:
a  b  10    E
p  q  20    B
x  y  30    C
Name: None, Length: 3

What do you think?

Wouter


Labels:

Comments:
I need to think about the consequences of changing `Series.argmin` and `Series.argmax` to return labels instead of integers as they currently do (since it's a subclass of `ndarray`). I can think of times where I've relied on this. Otherwise it looks good

This didn't merge cleanly so I did it by hand. Renamed the methods to `idxmin` and `idxmax`-- very useful, thanks. Did not do argmin/argmax to avoid overloading the ndarray functions. I don't have strong feelings about the names

I made some changes to NA-handling in these functions to be consistent with the other reductions, particularly when `skipna=False`:

https://github.com/wesm/pandas/commit/c1307b68d5cc16848be832db67937fd9e639800e
fixed #251
State: closed


Labels:

Comments:
Whoops, forgot to configure my username on a new git install. 

BTW in the future you'll want to do these changes in a branch instead of in your master branch. I'm going to merge these in but using a rebase which will confuse git if you try to pull in the upstream changes. so I would recommend resetting to before you made these commits, then pulling upstream 

merged into my master branch, thanks!
Support different delimiters for to_csv
State: closed

This pull requests implements delimter support into to_csv. A number of analysis tools expects TSV files rather than CSV files, and this is a simple solution to the problem.

The delimiter parameter is by default ",", hence there are no side effects with existing code.


Labels:

Comments:
thanks! merged it into master

i'm going to change the argument name to `sep` instead of delimiter to conform with `read_csv` and `read_table`