ENH: Added method to pandas.data.Options to download all option data for...
State: closed

... a ticker.

Also added a few helper functions.  These functions could be applied to refactor some of the other methods.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
I fixed the issues you pointed out that weren't related to consistency with the rest of the module.

Any thoughts on the consistency issues? I'm happy to refactor the other methods if you think that would be helpful.

Hi, I don't know how to contribute changes but I noticed that the Chg column only has positive values. This is because yahoo only provides an class to display an up or down arrow to show the change. I modified the _unpack method to look inside the elements to find the appropriate class name and change the Chg to the appropriate sign.

```
def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    res = []
    for i in els:
        img = i.xpath('.//img')
        if len(img)>0:
            cls=img[0].xpath('.//@class')[0]
            if cls == 'neg_arrow':
                res.append(float(i.text_content())*(-1.0))
            else:
                res.append(i.text_content())
        else:
            res.append(i.text_content())
    return res
```

Nice catch strimp.  How about this?

```
def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    return [_parse_row_values(val) for val in els]


def _parse_row_values(val):
    ret = val.text_content()
    img = val.xpath('.//img')
    if img:
        if img[0].xpath('.//@class')[0] == 'neg_arrow':
            ret = float(ret)*(-1.0)
    return ret
```

Or this:

```
def _unpack(row, kind):
    els = row.xpath('.//%s' % kind)
    return [_parse_row_values(val) for val in els]


def _parse_row_values(val):
    ret = val.text_content()
    if 'neg_arrow' in val.xpath('.//@class'):
        ret = float(ret)*(-1.0)
    return ret
```

That works. Same results as my suggestion but much more compact and likely faster (avoids appending to a list). Thanks.

Another issue I was having with `lxml`. In the `_get_option_data` and `_get_expiry_months` methods, I believe your original code uses `lxml.html.parse` directly on the set `url` variable. I was having issues with that (potentially proxy although my `HTML_PROXY` path variable is set).

I simply used `urllib` to solve in both methods:

```
    url = 'http://finance.yahoo.com/q/op?s={sym}'.format(sym=self.symbol)

    try:
        from lxml.html import parse
    except ImportError:
        raise ImportError("Please install lxml if you want to use the "
                          "{0!r} class".format(self.__class__.__name__))
    try:
        # i am failing the try when attempting to call parse(url)
        page = urllib.urlopen(url) # added this line
        doc = parse(page) # changed to pass page not url
    except _network_error_classes:
        raise RemoteDataError("Unable to parse expiry months from URL "
                              "{0!r}".format(url))
```

Hmm.  I don't know what the problem is there, I didn't write that code, works fine for me.  I have a couple fixes for the _get_expiry_months.  I'll add them.

Probably some proxy issue then.

## 

Sent from mobile

On Dec 11, 2013, at 15:36, dstephens99 notifications@github.com wrote:

Hmm. I don't know what the problem is there, I didn't write that code,
works fine for me. I have a couple fixes for the _get_expiry_months. I'll
add them.

—
Reply to this email directly or view it on
GitHubhttps://github.com/pydata/pandas/pull/5602#issuecomment-30366529
.

New to git, new to python, new to pandas. For that reason I'm only commenting here and if it is in the wrong place I apologize.  

There is an issue in data.py where if you request option data for CUR_MONTH and CUR_YEAR+1 you will only get CUR_MONTH/CUR_YEAR options.  This is due to an unnecessary check in _get_option_data() to see if the passed in month is CUR_MONTH and using the URL without additional date parameters if it is.  If you just simply pass the month and year parameters in the URL every time this problem goes away.

@mcneary70 

why don't you create a new issue for that, and show some code that can reprod the problem?

thanks

@dstephens99 if you can rebase and get this passing, and prob needs some more tests, we can consider for 0.14

Sure, I will do that.

@mcneary70 I fixed that issue in my last commit.

@dstephens99 That is awesome, thanks!

pls rebase this in current master

need to update the doc strings as well

I am ok with changing the API here where it makes sense
eg best for all of these routines to return dataframes always (possibly multi level so u can hold put/calls) - but should always do the same thing

Rebased - I'll work on this some more this week.

gr8

as I say popose a nice pandonic API for this

@dstephens99 update?

@dstephens99 this needs some API work, can you comment on that

I think you can clean up some of the logic using something like:

```
    df["rootExp"] = df.Symbol.str[0:-9]
    df["root"] = df.rootExp.str[0:-6]
    df["exp"] = pd.to_datetime(df.rootExp.str[-6:])
```

You can define `has_tick` (which would be better is named "is_nonstandard" (meaning the deliverable of the option basket is not 100 shares) to be: `df["root"]==symbol`

@dstephens99 can you show the API?

Sorry for the delay in responding, I've been really busy on a project at work.

What do you think about the API being a data frame if you request just calls or just puts and a dictionary of {'calls': call_df, 'puts' : put_df} if you request both?  Pythonic?

Never mind, I just saw your prior comment about always returning a dataframe.  I will do it that way.

FYI: The test that failed was unrelated to this pull request (worked on the Travis CI connected to my fork).
Error:
File "testing.pyx", line 140, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2387)
AssertionError: expected 0.00286 but got 0.00286

Working tests:
https://travis-ci.org/dstephens99/pandas/builds/23879925

Happy to make any changes you guys think are appropriate.

ignore the failure in test_sum for now

can u post a sample of the frame that you are returning now?
eg just post your doc string examples for the things u added/changed

Here's the IPython output of get_all_data()
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="qrichtext" content="1" /><style type="text/css">
</style></head><body style=" font-family:'Monospace'; font-size:10pt; font-weight:400; font-style:normal;">

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">2</span><span style=" color:#000080;">]:</span> from pandas.io.data import Options</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">3</span><span style=" color:#000080;">]:</span> import pandas as pd</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">4</span><span style=" color:#000080;">]:</span> pd.set_option('max_columns', 25)</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">   ...:</span> pd.set_option('display.width', 400)</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">   ...:</span> pd.set_option('display.max_rows', 100)</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">   ...:</span> </p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">5</span><span style=" color:#000080;">]:</span> aapl = Options('aapl', 'yahoo')</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">6</span><span style=" color:#000080;">]:</span> all_data = aapl.get_all_data()</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">7</span><span style=" color:#000080;">]:</span> all_data.head()</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#8b0000;">Out[</span><span style=" font-weight:600; color:#8b0000;">7</span><span style=" color:#8b0000;">]:</span> </p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">                                     Symbol    Last  Chg     Bid     Ask  Vol  Open Int is_nonstandard   Tag Underlying  Underlying_Price</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">Strike Expiry     Type                                                                                                                   </p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">280    2014-05-17 call  AAPL140517C00280000  235.19    0  290.90  293.00    4         4          False  None       AAPL            571.94</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">285    2014-05-17 call  AAPL140517C00285000  235.45    0  285.95  288.05   49        49          False  None       AAPL            571.94</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">290    2014-05-17 call  AAPL140517C00290000  236.30    0  280.95  283.05    5         5          False  None       AAPL            571.94</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">295    2014-05-17 call  AAPL140517C00295000  231.30    0  275.95  278.05    4        24          False  None       AAPL            571.94</p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">310    2014-05-17 call  AAPL140517C00310000  214.75    0  260.95  263.05    2        39          False  None       AAPL            571.94</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;">[5 rows x 11 columns]</p>

<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><br /></p>

<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px;"><span style=" color:#000080;">In [</span><span style=" font-weight:600; color:#000080;">8</span><span style=" color:#000080;">]:</span> </p></body></html>

I'd like to see an `infer_dtypes=True` keyword where you`.convert_objects()`` on the frame if True (the default). You may also need to replace some values to make them more useful (e.g. None -> NaN)

Sure I can add that.  It looks like the only column it changes is the tag column from object to float:

aapl.get_near_stock_price(call=True, put=True, month=5,year=2014).dtypes
Out[24]: 
Strike                     float64
Expiry              datetime64[ns]
Type                        object
Symbol                      object
Last                       float64
Chg                        float64
Bid                        float64
Ask                        float64
Vol                          int64
Open Int                     int64
is_nonstandard                bool
Tag                         object
Underlying                  object
Underlying_Price           float64
dtype: object

aapl.get_near_stock_price(call=True, put=True, month=5,year=2014).convert_objects(convert_numeric=True).dtypes
Out[25]: 
Strike                     float64
Expiry              datetime64[ns]
Type                        object
Symbol                      object
Last                       float64
Chg                        float64
Bid                        float64
Ask                        float64
Vol                          int64
Open Int                     int64
is_nonstandard                bool
Tag                        float64
Underlying                  object
Underlying_Price           float64
dtype: object

Yep.  Specifically BRK-B was the one that broke it for me.  The yahoo equity symbol is BRK-B, but its BRKB in the options symbol.  Ex: BRKB140517C00100000

Have you seen any sources of historical options prices and volumes? Unfortunately Yahoo only has most recent close / trading day.