BrandonFors/pandas-github-issues huggingface dataset:
console_encode() fails on unicode when using ipython-notebook
State: closed

When running in ipnb (linux) I get:

> > > pandas.common.console_encode(u"\u05d0")
> > > '?'

I traced this back to sys.stdin.encoding being set to None rather then 'utf-8' like in the shell.

Attached is a fix for your consideration. the test is just a stub since mocking is required
to reproduce the bug when testing in the shell instead of inside ipnb.


Labels:

Comments:
Thanks for looking into this. I'll take the first commit but won't take the latter without a unit test

72d014a breaks python 2. Will have to look into it another time

how do I reproduce the problem on python 2? 
i'll see if I can come up with something cleaner.

The unit test suite fails on python2

Added a unit test and a guard so that things behave properly when series.name is 
actually a tuple (didn't know that was an option).

test_series and test_format pass over here.

Thanks. cherry-picked these
Added options data functionality
State: closed

Added ability to fetch options data from yahoo! finance. With this there are a few 'helper' methods to get different things out of the data.

Answers issue #1739

Also cleaned up a bit of the /pandas/io/data.py file to comply better with PEP8.


Labels:

Comments:
Thanks for this!

Great functionality! Are there any plans to upgrade to BeautifulSoup4 (bs4) instead of version 3?

I would happily accept a pull request
Support for HTML class'es on DataFrame.to_html().
State: closed

I really like to use sorttable.js and dragtable.js.
I need to be able to customize the HTML classes.
I also added a default class, so that people can use css on all Pandas generated tables by default.
Default class is "dataframe"


Labels:

Comments:
I've also added support for formatting the index using the optional formatters argument.

I included to_latex() support. 
I've really, _really_ basic for now, but it works.
We can refine it later if some people want options. I pass the kw options through, so that'll be easy.
DOC: skipna not an argument of drop_duplicates
State: closed

From what I can tell, `skipna` keyword was both added and subsequently
removed in #1244. However, the docstring for DataFrame.drop_duplicates
still lists it as a parameter.

I remove this from the docs, so they accurately represent the code.
An alternative to this would be to put back the skipna functionality.

pinging @changhiskhan, since he was the one that worked on this.


Labels:

Comments:
Thanks Paul
Fixes bug in tools.scatter_plot
State: closed

```
In [1]: import pandas

In [2]: from pandas.tools.plotting import scatter_plot

In [7]: df = pandas.DataFrame(np.random.randn(5,2), columns=['a','b'])

In [8]: df
Out[8]: 
          a         b
0  0.032454  0.057020
1 -0.292223 -0.265303
2 -0.118943  1.315302
3 -2.754867  0.827289
4  0.497206 -1.360892

In [9]: df['c'] = [True, True, False, True, False]

In [10]: df
Out[10]: 
          a         b      c
0  0.032454  0.057020   True
1 -0.292223 -0.265303   True
2 -0.118943  1.315302  False
3 -2.754867  0.827289   True
4  0.497206 -1.360892  False

In [11]: scatter_plot(df, "a", "b", by="c")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/Users/johnnybrown/Desktop/src/pandas/pandas/<ipython-input-11-95d97e8283db> in <module>()
----> 1 scatter_plot(df, "a", "b", by="c")

/Users/johnnybrown/Desktop/src/pandas/pandas/tools/plotting.py in scatter_plot(data, x, y, by, ax, figsize, grid)
   1419 
   1420     if by is not None:
-> 1421         fig = _grouped_plot(plot_group, data, by=by, figsize=figsize, ax=ax)
   1422     else:
   1423         if ax is None:

/Users/johnnybrown/Desktop/src/pandas/pandas/tools/plotting.py in _grouped_plot(plotf, data, column, by, numeric_only, figsize, sharex, sharey, layout, rot, ax)
   1634     ravel_axes = []
   1635     for row in axes:
-> 1636         ravel_axes.extend(row)
   1637 
   1638     for i, (key, group) in enumerate(grouped):

TypeError: 'AxesSubplot' object is not iterable

```


Labels:

Comments:
 Thanks! you should `git reset --hard pydata/master` after a fetch since I cherry-picked the commit (pls try to avoid making pull requests out of your master branch)
Avoid extra calls to calendar.monthrange.
State: closed

daysinmonth isn't used beyond setting cal, which is only used if calendar_end or business_end. Putting the if check in improved performance by a factor of 10. Once all of the flags are false, they can never become true (because of the &= operations), and short-circuiting the rest of the loop in that case saved an additional 20% runtime in the provided example code (see #1686).

Much simpler than rewriting in Cython for now. monthrange is relatively expensive and was being called 3x per index datetime when doing a .resample("T"...) operation.

Thanks for the help from Erik Welch!


Labels:

Comments:
thanks, just cherry-picked this commit
ENH: add more possible bool values to read_csv #1295
State: closed

Added more possible bool values when parsing a file. New values include upper and lower case True/False and Yes/No.


Labels:

Comments:
Could you please add a unit test?

I merged this in. Hopeful it won't cause big problems

Woops, sorry, didn't realize I was doing that, will be more careful next
time.

On Wed, Sep 12, 2012 at 11:04 AM, Wes McKinney notifications@github.comwrote:

> I merged this in. Hopeful it won't cause big problems
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/1691#issuecomment-8496784.
Add vectorized string methods to Series
State: closed

I'd appreciate any feedback on the API (@takluyver and others), but this makes a lot of sense to me and I'm already finding my fingers falling right onto it. I swear I'm going to start doing more pull requests


Labels:

Comments:
API looks great

Hm. Should maybe raise an exception in repeat on numeric data, but how to do so without paying a high performance penalty...

Maybe a hack for now? f0afc3ba48

Nice, that works for me. Add a test for that and good to go?

yeah, I'm putting in a few more changes to handle unicode where the method was constructed with str.\* methods right now. Will push in a little bit

Thanks for the feedback. Merged up and closing relevant issues now
Document the behavior of idxmin when there are multiple rows containing the minimum value.
State: closed

I've added an example to the docs which demonstrates the behavior of idxmin() for a column where multiple rows contain the minimum value (and also the column contains a NaN value).  The current documentation is ambigous (to me) on this edge case.

From http://pandas.pydata.org/, a pull request seemed like the most appropriate way to contribute to the docs.  Please let me know if there is a better way or how I can improve the request.  And thanks for a great tool!


Labels:

Comments:
Thanks! cherry-picked in
Frame corr
State: closed

kendalltau usually returns a tuple but for kendalltau([1, 1, 1], [1, 1, 1]) it returns int, so _kendall in nanops.get_corr_func was failing

if empty arrays are passed in, kendalltau leads to max recursion depth exceeded in mergesort, so must check valid.any before valid.all case.


Labels:

Comments:
oops, branch got messed up. I'm going to redo
ENH: implement DataFrameGroupBy.boxplot(), close #1507
State: closed

I noticed this was marked for milestone 0.9.0, so a pull request iso push myself.
I know plenty of goodies are on it`s way for plotting, so this might be a temporary solution (it is not my intention to run in the way of @orbitfold).


Labels:

Comments:
Thanks. I think this doesn't do much harm while the plotting API is worked on
Probability plot
State: closed


Labels:

Comments:
Hi @orbitfold I won't accept this PR until we decide how to manage plotting code of this variety between statsmodels (e.g. the more mature `qqplot` in https://github.com/statsmodels/statsmodels/blob/master/statsmodels/graphics/gofplots.py) and pandas. Can you please engage the statsmodels developers on the mailing list to get the conversation started?

Hi @orbitfold just pinging you about this. Can you e-mail the statsmodels mailing list to bring up the overlap in functionality so we can make some decisions about how to proceed?

Done.

<CC'ing myself>

Would be good to finish the conversation on the statsmodels mailing list.

Bump

I've updated the function as per suggestions on the mailing list.

Is there something else I can/should do about this?

I had expected that discussion to end with a conclusion, preferably one a bit broader than just for this function. Something like:
- it is OK to duplicate some plotting functions between pandas and statsmodels, general idea is to use the same API where possible; keeping the one in pandas basic and the ones in statsmodels with more options.
- all the plot functions in pandas should return a figure like in statsmodels. It's a bit mixed now, sometimes an axis (or list of axes?), sometimes a figure.

I thought the conversation was concluded on the mailing list with your and @josef-pkt's stamp of approval "[pystatsmodels] probability plot"?

Your points seem to make sense to me.

Some worries that I'm not really going to champion since I first brought them up months ago.  It's going to cause some confusion for users, and I fear there will inevitably be PRs for pandas to enhance functionality for things like qqplot into areas we already cover. Ideally, I would think pandas would be used for visualization of data - not things that require estimation and inference - but, alas, it doesn't make sense to fight plots going into pandas and there's really not a clear cut line except that pandas can't have a reverse dependence on statsmodels.

@jseabold yes for this function, so the PR looks fine as is. However, what didn't get finished is (quoting Wes) "decide how to manage plotting code of this variety between statsmodels ... and pandas".

Besides cross-referencing in the See Also sections, I don't see much that can be done to avoid duplicate effort.

Oh, sure. I agree with your points then. Always return figs (this was from JDH advice). Also using intersphinx and See Also sections in the docs should hopefully help. (We just need to watch as new plotting functionality is added.)

@orbitfold closing this, if you want to restart this, pls reopen/do a new PR (as master has changed a lot)..thanks
Parallel coordinates plot
State: closed


Labels:

Comments:
A couple of comments and I'm going to make some quick edits.
- Best way to check for None is `obj is None` instead of `obj == None`. The latter will fail on many values of `obj`
- The fastest way to get the values in a row of a DataFrame is to use `irow`, `xs`, or indexing with `ix`. This function has issues with integer indexes (http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#api-changes-to-integer-indexing) that I'm fixing now

Also here's a more pandas-ian way for the following few lines:

``` python
    if cols is None:
        columns = data.drop(class_column, axis=1)
    else:
        columns = data[cols]
```

instead of 

```
    if cols is None:
        columns = [data[col] for col in data.columns if (col != class_column)]
    else:
        columns = data[cols] [data[col] for col in cols]
```

I see. This pattern is also in most other of my multi-variate data visualization routines.
Tsplot gaps
State: closed

Issue #1466


Labels:

Comments:
I don't think this fix works for irregular time series:

```
In [2]: rng = date_range('4/16/2012', periods=1000, freq='h')

In [3]: rng = rng[rng.hour < 7]

In [4]: ts = Series(np.random.randn(len(rng)).cumsum(), rng)

In [5]: ts.plot()
Out[5]: <matplotlib.axes.AxesSubplot at 0x2a63050>
```

This is great, thanks
Add parentheses to a print statement so it compiles under Python 3
State: closed

Just fixing a Python 2-style print statement in docs/make.py so it'll compile under Python 3. (Untested but I can't imagine how this would break anything)


Labels:

Comments:
Thanks, @diazona .

In case anyone reviewing this isn't aware, Python 2 treats this as a print statement with a single bracketed expression, so it won't change anything there.

I'll merge this in. But there's basically no chance the docs will build on Python 3 because it has to process the output/figures in the code examples in the docs using sphinxext/ipython_directive.py.

OK, thanks. I hadn't gotten a chance to test the patch so I wasn't sure if that was the only thing holding up the compilation on Python 3 or not.
Retry yahoo DataReader on 404 up to 3 times
State: closed

This should fix bug #826 that had yahoo randomly failing downloads.


Labels:

Comments:
Hey Paddy, thanks for the pull request! It would be nice to have two additional keywords, "retry_count" and "pause" to give further control for retrying. You can set the retry_count to 3 by default and pause to be 0.

I added retry_count and pause through DataReader.  Looking at the code it doesn't look like this will break any of the other ways that DataReader is called.  Are there testcases for the other uses?

I don't think so. I actually plan to deprecate the `DataReader` class in favor of independent functions for each data source

Merged / pushed this. A reminder that you should make changes in feature branches and not merge the upstream branch. You'll probably want to fetch upstream and do:

`git reset --hard upstream/master`
Autocorrelation plot
State: closed

http://pandasplotting.blogspot.com/2012/06/autocorrelation-plot.html


Labels:

Comments:
Cool. Could you be a bit more careful about not merging upstream into your feature branches? Makes merging the branch back to master a lot harder for me

Sure

Sorry, ignore my comment on the other PR, I see the same thing happened here

Not sure what happened, but it looks like this branch was not based off of pydata/master-- this resulted in very confused git (cherry-picking failed). I committed and pushed the changes by hand, but if it happens again I'll make you redo the PR / commits with a clean branch =)

Yes, sorry about that. I am not quite sure what happened myself, I am still a bit clumsy with git. I am used to mercurial. You shouldn't feel bad forcing me to resubmit pull requests if needed.
DOC: Indicate that to_csv and from_csv methods can handle string file pa...
State: closed

...th and file-like objects for io.

It also seems a good idea to align the argument names: _path_, _path_or_buffer_, _filepath_or_buffer_ are all three used among `to_csv()`, `from_csv()`, `read_table()` to indicate the same. I did not touch those since that is an api change.


Labels:

Comments:
thanks!
Add support for tox (http://tox.testrun.org/) and Travis CI (http://travis-ci.org/)
State: closed

These are testing tools.

Tox is run locally and can be used on code that is not even checked in. It runs the test suite on various Python versions.

Travis CI is a hosted CI system that can run tests whenever new commits are pushed to the GitHub repo.

You can see the Travis CI status for my fork at http://travis-ci.org/#!/msabramo/pandas -- it's failing at the moment -- it probably needs some tweaks as I am completely new to pandas and I'm probably not building it correctly.


Labels:

Comments:
This is cool. Would have to look closer to see why your builds are failing-- I have a local Jenkins that tests/builds on Windows and things are looking good right now
ujson fix compile error using clang on Mac OS Lion
State: closed

I had the compile error below when I built 0.8.0 beta1.  The build was on Mac OS X 10.7.3, using python.org's python version 2.7.3, numpy version 1.6.2, Apple clang version 3.1.

<pre>
clang: pandas/src/ujson/python/JSONtoObj.c
pandas/src/ujson/python/JSONtoObj.c:159:36: error: expression is not assignable
        ((char*)PyArray_DATA(ret)) = new_data;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
pandas/src/ujson/python/JSONtoObj.c:292:44: error: expression is not assignable
        ((char*)PyArray_DATA(npyarr->ret)) = new_data;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
</pre>


This patch changes those lines to use PyArray_BYTES instead, so that the assignment is to an lvalue.  The build then succeeds, and nose tests pass.  I also successfully tested on a linux box (Scientific Linux 6.0), running python 2.6.6, numpy 1.6.2, and gcc 4.4.4. (I don't have a windows box to test on)


Labels:

Comments:
This looks like it shouldn't be a problem as PyArray_BYTES is a supported API on 1.6 and 1.7+. Thanks for fixing this

This turned out to cause problems on other platforms, I'm committing a fix, can you verify that you can build with git master?

I confirm master builds and tests pass on my platform.  Thanks!
ujson remove link dependence on npymath
State: closed

npymath causing build woes on win32 (#1309 and #1360)

Removed half float support. `npy_isnan` and `npy_isinf` are macros defined in the npymath header so linking isn't necessary. 

Also, build fix for mingw32.

Tested on Python 2.7 on 64-bit OSX, 32-bit Ubuntu and 32-bit WinXP (VS 2008 and mingw32). 


Labels:

Comments:
Thanks, cherry-picked this, seems like a good idea to me
Get JSON extension to work with Python 3 #1319
State: closed

Python 3 fixes for ujson lib. All relevant tests pass (see below).

Tested on 64 bit OSX on Python 2.5, 2.6, 2.7, 3.1 & 3.2
Tested on 32 bit Ubuntu on Python 3.2

Apart from the define macros the handling for attribute / key names and the module initialisation code needed to be updated.

Some other notes:
1. I did not install scipy so the scipy tests failed but this should have no relevance for ujson
2. The `test_encodeBigEscape` in `test_ujson` seems to consume a lot of memory (or is more memory limited) in Python 3. It only passed in my Ubuntu VM when I upped its RAM to 2GB.
3. Several unicode tests now skipped for Python 2.6 (mostly the same as those skipped for 2.5). I didn't investigate this too much apart from checking that they failed with 2.6 for the original ujson library as well (they did).
4. `test_encodeListLongConversion` needs to pass `dtype=np.int64` to succeed for Python 3 on 32 bit. Under Python 3 numpy converts `int` to `np.int32` on 32 bit machines (whereas under Python 2 it's `long` -> `np.int64`).
5. I did not test on Windows :-/


Labels:

Comments:
Thanks for doing this. Have you/will you push the relevant parts upstream to ujson as well?

Yeah there's a few things I've been planning to send upstream apart from Python 3 support. (Wes' Python 2.5 changes, better datetime encoding & support for higher double precision).

Great!

If you look more closely at `encodeBigEscape`, the input string is being interpreted differently on Py2 vs Py3:

`\xc3\xa5` is a single UTF8-encoded character in Python 2:

(Pdb) print input[:10]
ååååå

It has this simple JSON encoding:

```
(Pdb) ujson.encode(input[:10])
'"\\u00e5\\u00e5\\u00e5\\u00e5\\u00e5"'
```

Py3:

```
(Pdb) input[:10]
'Ã¥Ã¥Ã¥Ã¥Ã¥'
(Pdb) input[0]
'Ã'
(Pdb) input[1]
'¥'

(Pdb) ujson.encode(input[:10])
'"\\u00c3\\u00a5\\u00c3\\u00a5\\u00c3\\u00a5\\u00c3\\u00a5\\u00c3\\u00a5"'
```

As a result the JSON version is twice as long in Python 3. Encoding a length 20MM string is a bit excessive for a unit test IMHO (resulting in ~120MM length JSON string, taking up ~ 500M, why your VM was crapping itself). I'm just going to trim down the size of the string to 1MM from 10MM. It will also cause problems for our CI system if not (which runs on a Windows VM)

That makes sense, thanks Wes.
ujson tabs -> spaces and plug some memory leaks
State: closed

The ujson source code was tab-intended, this commit converts the ugly tabs to (four) spaces.

Also fixed some memory leaks and some compiler warnings.

All tests pass, tested on 64 bit OSX and 32 bit Ubuntu with Python 2.7.


Labels:

Comments:
Cool, this looks good. I'm going to try to get this building with MSVC today so that things pass on Win32/Win64. Note: MSVC is very strict about C variable declarations (have to be the first thing to appear in functions). gcc is more loosey-goosey.

Ah I wasn't aware of that thanks, my main experience is with C++ and gcc.

There's also stupid crap like this:

http://stackoverflow.com/questions/7303740/error-c2057-expected-constant-expression

How important is the npymath dependency? I think it only affects halffloat? I am having an absolutely wretched time trying to get it to link on Windows 32-bit (works in win64)

Half float is probably not that important but off the top of my head I think the nan and infinity checking functions are from npymath too. 

http://docs.scipy.org/doc/numpy/reference/c-api.coremath.html

There might be alternatives I'm not aware of though?

I think the issue is that the official NumPy installers are built with mingw32 and not the VS2008 compiler (FML). As such the npymath.lib file is incompatible. Going to build NumPy 1.6.2 from source. I don't think I've ever hated Microsoft so much

Building NumPy from source with VS2008 did the trick. Fun times

Nice one. They do have a reputation to uphold after all:
- internet explorer - bane of web development
- windows - bane of python extension development
- msvc - bane of C development

On a brighter note I read today that Python is finally moving towards a new VS version http://blog.python.org/2012/05/recent-windows-changes-in-python-33.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+PythonInsider+%28Python+Insider%29

I've always built any extensions using minggw on my Windows VM but in future I might test my builds on msvc aswell, save others the pain ;)
Tread parsing of xlrd cell errors as NaN
State: closed

I can't test the unit tests as I don't have a build environment.


Labels:

Comments:
sry "Tread parsing" should have been "Treat parsing"

sry forgot to note closes #1213

This is great, thanks. BTW, I would recommend making pull requests from a branch other than master in the future (see http://pandas.pydata.org/developers.html)

don't know how I missed that, thx! Will do that in future.

---

 From: Wes McKinney reply@reply.github.com
To: ruidc ruidc@yahoo.com 
Sent: Monday, 14 May 2012, 16:01
Subject: Re: [pandas] Tread parsing of xlrd cell errors as NaN (#1225)

This is great, thanks. BTW, I would recommend making pull requests from a branch other than master in the future (see http://pandas.pydata.org/developers.html)

---

Reply to this email directly or view it on GitHub:
https://github.com/pydata/pandas/pull/1225#issuecomment-5690830
ENH: Allow different number of rows & columns in histogram plot
State: closed

It didn't seem necessary that a grid of histogram plots should be square.  For example, if you have 6 columns to plot, a 2x3 grid makes more sense than 3x3 with an empty row.  Likewise with 12 it should give you 3x4, and so on.

I added a simple way to allow this. The number of columns will not exceed the number of rows by more than 1 (e.g., you will not get a 2x5 grid if you have 10 columns).

For a quick test:

``` python
df = pandas.DataFrame(np.random.randn(200, 6))
df.hist()
```

Note: it might be desirable to allow more rows than columns for plots in portrait mode.  That could easily be a keyword option.


Labels:

Comments:
Sweet. This makes complete sense
ENH: Add support for converting DataFrames to R data.frames and matrices
State: closed

Finally I got around to implement this properly in pandas, and I added also a conversion to R matrices if need be. Notice that this doesn't handle MultiIndex and similar functionalities, but since I don't use them I'm not sure how they'd fit into R data frames.

The biggest drawback here is the use of an intermediate, but other solutions would probably involve hacking around the rpy2 internals (so far all the alternative solutions I found where worse, performance-wise).

Two (smallish) unit tests are included. This code has also been extensively used in production where I work. 

All other unit tests pass after this change.

closes #350


Labels:

Comments:
This is really great, thanks so much