BrandonFors/pandas-github-issues huggingface dataset:
.describe() for series of objects
State: closed

See #210.

I don't know if `series.dtype == object` is the best way to check for non-numeric data, but it appears to work.

At present, I've used fields 'count', 'unique', 'top' (i.e. most common) and 'freq' (i.e. count of the most common). These seem like obvious things for a summary.

I also noticed a problem doing `df.describe()` if df has integer columns, so I copied some code from series to fix that.


Labels:

Comments:
this looks good and thanks for the min/max bugfix. will need to add a compatability function for Counter which is not present in Python 2.5 which I'm still supporting

Actually, checking the docs, 2.6 doesn't have Counter either. It's pure Python, so the simplest thing is probably to copy the 2.7 version and ship it in a module in util. Let me know if you want me to do that.

would you please? separate pull request since I already rebased and merged this one

OK, no worries.

Do you have an opinion on how NA (NaN/None) should be handled in the output of Series.describe? I should also take care with that in value_counts
Update docs on reading CSV files.
State: closed

This does the essential reworking of the documentation for `read_csv` and `read_table` for #227. I'm happy to do more doc updates.

I also think the autosummary API documentation isn't particularly good - it doesn't seem to handle classes at all. I'll see if I can copy what IPython uses (example output is [here](http://ipython.org/ipython-doc/stable/api/generated/IPython.testing.tools.html#tempfilemixin)).


Labels:

Comments:
I've removed the autosummary extensions, which ship with Sphinx since v 0.6.

Thanks Thomas. Merged this into master
Add simple attribute access to DataFrame columns
State: closed

This one might need some discussion, but I think it makes sense.

If a column name in a dataframe is a valid Python name and doesn't conflict with any DataFrame attributes, this lets you type `df.MyColumn` than `df['MyColumn']`. That's much handier for interactive use, especially because IPython will tab complete `df.MyColumn.<tab>`, while it won't by default tab complete `df['MyColumn'].<tab>`.

This doesn't interfere with accessing columns through the dictionary interface.


Labels:

Comments:
This seems useful-- it might even be possible to get tab completion but the overall computational cost (overriding `__getattribute__`) might be too great. Merged it into master. Thanks!

Thanks. I think there's a method by which objects can expose extra things for tab completion to IPython, but I that's a lower priority.

Just looking at the code, I think we'd need to register the DataFrame type with `IPython.utils.generic.complete_object` (see code [here](https://github.com/ipython/ipython/blob/master/IPython/utils/generics.py)). I'll get round to doing it at some point.

I created an issue about this:

https://github.com/wesm/pandas/issues/230
Use stdlib csv module to write csv
State: closed

I noticed while using pandas that data with commas in the fields wasn't getting escaped properly when written to CSV (data in biology often has bits of text as well as numbers). This causes the data to spread out over extra columns. This commit uses the `csv` module from the standard library to write out CSV 'properly'.

I've also added a simple test for this case.


Labels:

Comments:
Cool. I was actually made aware of this a while ago but as you can see forgot to fix it ;) I'm sure performance is better, too. 

I wouldn't expect a big performance difference, but it may be a bit faster, since I think csv.writer is implemented in C.
Python 3 compatibility
State: closed

No doubt there are ways this could be improved, but this gets it installing, importing, and _almost_ passing the test suite.

There's one test failure left, and I think that's something odd with numpy, specifically the `ndarray.searchsorted()` method. It returns seemingly random results when it's called in `panel.get_axis_dummies()` - but I can't replicate those results calling searchsorted on identical arrays interactively.

It doesn't add any failures to the Python 2 test suite (one test is failing for me in master), but I've only checked this with Python 2.7.


Labels:

Comments:
This is really great-- thanks for doing this. It's also impetus for me to get Python3+NumPy+SciPy installed on my machine for testing purposes. I will do that, run the test suiteu, and merge this in as soon as I can.

You're welcome!

Rebased this and merged it in. There are some test failures in what appear to be corner cases in NumPy trunk, I will have to investigate those some other time

Thanks Wes. For the record, I was testing with Numpy 1.6.1.

Thomas, can you help me figure out why the cython command stopped working once setuptools is introduced (for the purpose of running 2to3)? I'm talking about:

```
python setup.py cython
```

in the meantime, I kludged things so setuptools only gets imported in py3k, so no urgency

https://github.com/wesm/pandas/commit/5a330385479edbe715daedaceb59c3a222a84e0f

I'll have a look - I'd just been doing `python setup.py install`.

It appears to work for me with setuptools 0.6 from Ubuntu repos, or with distribute 0.6.19 as installed by virtualenv.

<pre>
abc
running cython
cythoning pandas/src/tseries.pyx to pandas/src/tseries.c
cythoning pandas/src/sparse.pyx to pandas/src/sparse.c
</pre>


(the `abc` is from a print statement I added just below the setuptools import)

that's prior to my kludge, right? i was running it on win32 to see if I can distribute 3.x binaries for the imminent next release. sigh, windows, the source of so much dev unhappiness 

Here's another not so awesome thing: on 3.1/3.2 on Win32 (haven't tested Win64 yet), the default csv writer writes not 1, but 2 line breaks per line. Specifying explicitly lineterminator='\n' for now which solves the problem (I think csv.reader interprets CRLF as \n\n which seems really bad)

I tweaked your kludge so it imported setuptools on both platforms (the `abc` is to check that it's getting imported).

With Windows, I think it writes `\r\n` by default. Unix-y programs might interpret that as two line breaks, but it's the platform default for Windows. Explicitly specifying the lineseparator is probably sensible.

I don't have Windows to hand, but I can test in Wine if you like.
ENH: improve DataFrame read_csv / to_csv for Index/MultiIndex
State: closed

2 issues for DataFrame.to_csv

1 ) If header and index are True, you should write something for an index header, even if index_level is None.
2) If given a MultiIndex, don't want to just call str() on each element.

Also fixed read_csv to take an iterable for index_col so that I could write tests.

I haven't looked at the similar methods in Series or the other to_\* methods yet. This is all I needed at the moment.

Aside: it looks like in test_frame setUp isn't a classmethod and is called for each test method. Not a big deal, but might be a little heavy in the test suite.


Labels:

Comments:
thanks a lot for doing this. after hemming/hawing i think it makes sense to have a default index header column. rebased into master

I think it would be a good idea to make it so a regular Index can have a name attribute. Will create a separate github issue
Minor change to csv reading
State: closed

Hi Wes

Firstly, congrats on such an amazing project! I love prototyping in python / numpy / ipython, but I always envy some of R's features. I tried pandas out about 9 months ago and although it was interesting, it seemed very rough around the edges. Now, however, it is looking really polished and I've been using it for prototyping and testing some trading models, and everything works extremely well. I hope it keeps growing, together with the integration with scikits statsmodels/timeseries and maybe even scikits.learn in future ...

Anyway, as I was starting to dive into the code, I came across the <code>read_csv</code> functions and noticed that there was full duplication in <code>read_table</code>. The <code>csv</code> module in python actually has full support for arbitrary delimiters, so there is no need for the duplication. Also, there is <code>csv.Sniffer().sniff(sample)</code> that attempts to sniff out the delimiter automatically. This commit tries to "magically" handle any arbitrary CSV file without needing to specify a separator, whether separated by blank spaces, tabs, commas, semicolons or other weird separators (I have a file at work with "^" separators :). If it doesn't work, one can fall back on specifying the separator (so <code>read_csv</code> looks more like <code>read_table</code>). In future it could make sense to simply have one <code>read_data</code> or <code>read_table</code> function.

Incidentally, the <code>csv.Sniffer()</code> also tries to sniff out other things like quote escaping and double quoting, but this commit effectively only uses it for the delimiter. If problems with quote / string escaping crop up with users one could always let the sniffer try to figure out the full dialect.


Labels:

Comments:
Rebased into wesm/master-- thanks for these changes, a big help!
Enh tests compat pytables 2.1
State: closed

unfortunately on Debian systems we still have pytables 2.1; but it still would be nice to have pandas' unittests to pass with skips but without an error or failure -- major incompatibility is absense of blosc compressor.   This changes make those tests conditional on the version of pytables being at least 2.2 to test blosc


Labels:

Comments:
Got it-- I can merge this later tonight. Are you OK with packaging the 0.4.0 release with your patches? Doesn't strike me that a bugfix release should be necessary.

sure -- I can absorb all those into a patch queue ;-)

> ## Got it-- I can merge this later tonight. Are you OK with packaging the 0.4.0 release with your patches? Doesn't strike me that a bugfix release should be necessary.
> 
> =------------------------------------------------------------------=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic

Cherry picked these commits. Thanks!
minor: just removed pasting of import of datetime into version.py
State: closed

so to avoid change of file while doing setup.py operations, removing pasting of the import
of datetime into version.py


Labels:

Comments:
Cherry-picked. Since you know more about git, how should I be merging small pull requests like this? Maybe having the merge commit isn't so bad

it is really up-to-you -- to cherry-pick or to merge (which IIRC should
close the pull request on github automatically whenever you push)

On Fri, 16 Sep 2011, Wes McKinney wrote:

> ## Cherry-picked. Since you know more about git, how should I be merging small pull requests like this? Maybe having the merge commit isn't so bad
> 
> =------------------------------------------------------------------=
> Keep in touch                                     www.onerussian.com
> Yaroslav Halchenko                 www.ohloh.net/accounts/yarikoptic
Added function to pull data from Yahoo! finance, FRED, and Ken French's website.
State: closed

The goal is to make it easy to pull in data off the web from various different sources using the same consistent framework.

As an example:

from pandas.io.data import DataReader
gs = DataReader("GS", "yahoo")


Labels:

Comments:
looks good to me. I'm not up on my git-fu but in the future I think you should do stuff like this in a branch so I don't pull your merge commits into pandas master. I'm going to cherry-pick these commits and close the pull request

OK wow git cherry-pick really doesn't like this. I'm still too new to git to know how best to do this. Pulling your branch makes the history look nasty so I'm just going to take your latest revision of that file and mark you as the author in a commit.

Alright, I'm just a n00b. the commits go top to bottom above, I was cherry picking in the wrong order. I just pushed to master-- you'll probably want to reset to a state prior to your first change (git reset --hard commit-sha1) and then pull from wesm/master

sorry for all the spam

Ha!  Thanks.

You're a n00b?  I just started really using git a week ago, so I think I have your thoroughly beat on that front.

Happy to do this however you see best in the future...I'll try to follow something like the scikit.learn guidelines (http://scikit-learn.sourceforge.net/dev/developers/index.html).  You might want to post something like that on the pandas docs eventually if more people start to meddle?
Added baseball.csv
State: closed

This seems to work. Was that what you had in mind? 


Labels:

Comments:
That's cool. I was thinking about including a bigger piece to write meaningful/interesting groupby examples. I haven't looked into the license for the baseball data, plyr is MIT license so I assume that data is just public domain but...

Didn't think about that. 

It comes from http://www.baseball-databank.org/

They say their data is open for use by individual researchers, but other uses require their permission. On the other hand, I read the MIT license as saying copying any part of the software and its documentation is fine, and the data was released with the software. So it seems like the only real question is if the MIT license should be included with the baseball.csv file. 

(The MIT license only says it needs repeating when 'substantial' parts of the software are copied.) 

Another potential solution is that I'm currently writing a little function to generate arbitrarily sized biostatistics-like data sets. I'd been intending it as a learning tool to let me explore pandas functionality for datasets more like what I'm used to (e.g., lots of categorical data), but it could be adapted to creating large tables of data for this as well. 

I included a little chunk (100 rows) in doc/data so we should be all set
Minor change to CleanCommand so build works with stdeb
State: closed

Hi Wes, i've been following your pandas work for a while, and regularly install it on my ubuntu based systems.  I generally use the stdeb (https://github.com/astraw/stdeb) package for creating .deb files from python packages.  Unfortunately, some of your recent changes in setup.py in pandas caused a problem with building using stdeb.  Essentially the problem is that debhelper tries to run "python setup.py clean -a" at some point, which doesn't work because the customized clean command does not have a "-a" option.  I just added a dummy argument "--all, -a" argument that has no effect.

No worries if you'd rather not merge it, just wanted to check.

thanks


Labels:

Comments:
Looks fine to me. Thanks for the patch!