Repository: pandas-dev/pandas

Issue:
yahoo test error connection reset
we really need to fix this...

https://travis-ci.org/cpcloud/pandas/jobs/8321921
https://travis-ci.org/cpcloud/pandas/jobs/8322345
https://travis-ci.org/jreback/pandas/jobs/8448538
https://travis-ci.org/pydata/pandas/jobs/8449118 (HTTP error here)



Discussion:
I thought the "@network" was supposed to catch these? are these not `IOError`?

no the error generated is `socket.error` which is basically a light wrapper around one of the C errno numbers

which apparently is not caught by `IOError`!
so `socket.error` should prob be added as a default list of things to catch in '@network'?

_so_ wish python 2 had [this](http://docs.python.org/3/library/exceptions.html?highlight=connectionerror#ConnectionResetError)

should we be catcing `OSError` then? 

here's why that might be problem

```
In [1]: open('asdfasdf', 'r')
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-1-efa01a311135> in <module>()
----> 1 open('asdfasdf', 'r')

IOError: [Errno 2] No such file or directory: 'asdfasdf'

In [2]: os.path.getsize('asd09g8')
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-0a206a03b7df> in <module>()
----> 1 os.path.getsize('asd09g8')

/home/phillip/.virtualenvs/pandas/lib/python2.7/genericpath.pyc in getsize(filename)
     47 def getsize(filename):
     48     """Return the size of a file, reported by os.stat()."""
---> 49     return os.stat(filename).st_size
     50
     51

OSError: [Errno 2] No such file or directory: 'asd09g8'
```

of course python3 has `FileNotFoundError` ARG

in python3 `socket.error` is an alias for `OSError` so i guess we should catch it then...that should work too for connection errors...pr coming in 100..99..98

These are network calls though, the point is to catch them (obviously if we can find a narrow exception would be good)

but if there happens to be a file opening error in a python2 test labelled networrk it won't be caught

so let's add the socket exceptions then (in addition to IOError)

i did

``` python
_network_error_classes = IOError, OSError

try:
    ConnectionError
except NameError:
    _network_error_classes += socket.error,
```

but probably better is

``` python
_network_error_classes = IOError,

try:
    _network_error_classes += ConnectionError,
except NameError:
    _network_error_classes += socket.error,
```

this should work in 2.x, 3.x since 3.2 follows py2 conventions regarding exceptions

are there other socket errors that would want to catch?

ssl possibly

I guess throw it in and we see more..then easy to add....

all exceptions in `socket` are subclasses of `IOError` so that's reassuring

```
In [32]: print x[0].map(lambda x: getattr(socket, x.split('.')[1]))[10:].map(lambda x: map(lambda y: y.__name__, x.mro())).values
[ ['error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['gaierror', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['herror', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']
 ['SSLError', 'error', 'IOError', 'EnvironmentError', 'StandardError', 'Exception', 'BaseException', 'object']]
```

here's how i got `x`

```
In [33]: psearch socket.*error*
socket.SO_ERROR
socket.SSL_ERROR_EOF
socket.SSL_ERROR_INVALID_ERROR_CODE
socket.SSL_ERROR_SSL
socket.SSL_ERROR_SYSCALL
socket.SSL_ERROR_WANT_CONNECT
socket.SSL_ERROR_WANT_READ
socket.SSL_ERROR_WANT_WRITE
socket.SSL_ERROR_WANT_X509_LOOKUP
socket.SSL_ERROR_ZERO_RETURN
socket.error
socket.gaierror
socket.herror
socket.sslerror

In [34]: # copy to clipboard via tmux

In [35]: x = read_clipboard(header=None)
```

missed one `socket.timeout` which is also a subclass of `IOError` which begs the question why aren't these being caught?

i'm not convinced that my PR is necessary, why would `IOError`s be caught but not `socket.error`?

let me see if i can somehow trigger this error across a vagrant machine

Should this be the responsibility of the function to catch these errors? Or
is it supposed to bubble up.

Also, if you hit this error with internet on and then don't with internet
off - then it could be an issue with the calling function rather than
anything internal. Post merge, I ran this sans - internet and only see
errors in two functions that return None or non-complete objects with
IOError.
On Jun 21, 2013 6:16 PM, "Phillip Cloud" notifications@github.com wrote:

> let me see if i can somehow trigger this error across a vagrant machine
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3982#issuecomment-19843883
> .

I thought the new function of the network decorator is to by default catch any network related error that is specified and then skip ?

it ran successfully with and without internet

So there's a separate socket error? When I tested, it was raising socket
error as IO... Is it platform dependent?

Also, if you do this, maybe we should create a function that checks whether
something is a network error or maybe context manager... Then could use it
elsewhere.
On Jun 21, 2013 8:25 PM, "Phillip Cloud" notifications@github.com wrote:

> it ran successfully with and without internet
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3982#issuecomment-19847467
> .

yes the `socket` module has a handful of exception classes, all subclasses of `IOError` but the one generated here wasn't being caught by the decorator

Probably one of the tests decorated by `with_connectivity_check`. Those try
to ping google (or another url) and if they succeed, they then raise the
error instead of catching it. Just swap out with `network`. If it's
actually not working that's more problematic. (I notice, for example, that
one test catches and _prints_ the socket errors but actually raises an
AttributeError.

On Fri, Jun 21, 2013 at 11:35 PM, Phillip Cloud notifications@github.comwrote:

> yes the socket module has a couple of exception classes, all subclasses
> of IOError but that wasn't being caught by the decorator
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/3982#issuecomment-19850454
> .

#3985 closes this, right?

@jreback I think it's #4002 that fixes this (adding in the `closing(urlopen(url))` syntax so that it doesn't leave unclosed `urlopen` results.

I see you moved #4002 to 0.12, this too? trying to close the remaining issues.....


Resolving patch:
diff --git a/pandas/io/data.py b/pandas/io/data.py
--- a/pandas/io/data.py
+++ b/pandas/io/data.py
@@ -8,9 +8,9 @@
 import numpy as np
 import datetime as dt
 import urllib
-import urllib2
 import time
-import warnings
+from contextlib import closing
+from urllib2 import urlopen
 
 from zipfile import ZipFile
 from pandas.util.py3compat import StringIO, BytesIO, bytes_to_str
@@ -109,10 +109,11 @@ def get_quote_yahoo(symbols):
 
     data = dict(zip(codes.keys(), [[] for i in range(len(codes))]))
 
-    urlStr = 'http://finance.yahoo.com/d/quotes.csv?s=%s&f=%s' % (
-        sym_list, request)
+    url_str = 'http://finance.yahoo.com/d/quotes.csv?s=%s&f=%s' % (sym_list,
+                                                                   request)
 
-    lines = urllib2.urlopen(urlStr).readlines()
+    with closing(urlopen(url_str)) as url:
+        lines = url.readlines()
 
     for line in lines:
         fields = line.decode('utf-8').strip().split(',')
@@ -151,29 +152,29 @@ def _get_hist_yahoo(sym=None, start=None, end=None, retry_count=3,
 
     yahoo_URL = 'http://ichart.yahoo.com/table.csv?'
 
-    url = yahoo_URL + 's=%s' % sym + \
-        '&a=%s' % (start.month - 1) + \
-        '&b=%s' % start.day + \
-        '&c=%s' % start.year + \
-        '&d=%s' % (end.month - 1) + \
-        '&e=%s' % end.day + \
-        '&f=%s' % end.year + \
-        '&g=d' + \
-        '&ignore=.csv'
-
-    for _ in range(retry_count):
-        resp = urllib2.urlopen(url)
-        if resp.code == 200:
-            lines = resp.read()
-            rs = read_csv(StringIO(bytes_to_str(lines)), index_col=0,
-                          parse_dates=True)[::-1]
-
-            # Yahoo! Finance sometimes does this awesome thing where they
-            # return 2 rows for the most recent business day
-            if len(rs) > 2 and rs.index[-1] == rs.index[-2]:  # pragma: no cover
-                rs = rs[:-1]
-
-            return rs
+    url = (yahoo_URL + 's=%s' % sym +
+           '&a=%s' % (start.month - 1) +
+           '&b=%s' % start.day +
+           '&c=%s' % start.year +
+           '&d=%s' % (end.month - 1) +
+           '&e=%s' % end.day +
+           '&f=%s' % end.year +
+           '&g=d' +
+           '&ignore=.csv')
+
+    for _ in xrange(retry_count):
+        with closing(urlopen(url)) as resp:
+            if resp.code == 200:
+                lines = resp.read()
+                rs = read_csv(StringIO(bytes_to_str(lines)), index_col=0,
+                              parse_dates=True)[::-1]
+
+                # Yahoo! Finance sometimes does this awesome thing where they
+                # return 2 rows for the most recent business day
+                if len(rs) > 2 and rs.index[-1] == rs.index[-2]:  # pragma: no cover
+                    rs = rs[:-1]
+
+                return rs
 
         time.sleep(pause)
 
@@ -198,17 +199,19 @@ def _get_hist_google(sym=None, start=None, end=None, retry_count=3,
     google_URL = 'http://www.google.com/finance/historical?'
 
     # www.google.com/finance/historical?q=GOOG&startdate=Jun+9%2C+2011&enddate=Jun+8%2C+2013&output=csv
-    url = google_URL + urllib.urlencode({"q": sym, \
-        "startdate": start.strftime('%b %d, %Y'), \
-        "enddate": end.strftime('%b %d, %Y'), "output": "csv" })
-    for _ in range(retry_count):
-        resp = urllib2.urlopen(url)
-        if resp.code == 200:
-            lines = resp.read()
-            rs = read_csv(StringIO(bytes_to_str(lines)), index_col=0,
-                          parse_dates=True)[::-1]
-
-            return rs
+    url = google_URL + urllib.urlencode({"q": sym,
+                                         "startdate": start.strftime('%b %d, '
+                                                                     '%Y'),
+                                         "enddate": end.strftime('%b %d, %Y'),
+                                         "output": "csv"})
+    for _ in xrange(retry_count):
+        with closing(urlopen(url)) as resp:
+            if resp.code == 200:
+                lines = resp.read()
+                rs = read_csv(StringIO(bytes_to_str(lines)), index_col=0,
+                              parse_dates=True)[::-1]
+
+                return rs
 
         time.sleep(pause)
 
@@ -280,19 +283,19 @@ def get_components_yahoo(idx_sym):
           '&e=.csv&h={2}'
 
     idx_mod = idx_sym.replace('^', '@%5E')
-    urlStr = url.format(idx_mod, stats, 1)
+    url_str = url.format(idx_mod, stats, 1)
 
     idx_df = DataFrame()
     mask = [True]
     comp_idx = 1
 
-    #LOOP across component index structure,
-    #break when no new components are found
-    while (True in mask):
-        urlStr = url.format(idx_mod, stats,  comp_idx)
-        lines = (urllib.urlopen(urlStr).read().decode('utf-8').strip().
-                 strip('"').split('"\r\n"'))
-
+    # LOOP across component index structure,
+    # break when no new components are found
+    while True in mask:
+        url_str = url.format(idx_mod, stats,  comp_idx)
+        with closing(urlopen(url_str)) as resp:
+            raw = resp.read()
+        lines = raw.decode('utf-8').strip().strip('"').split('"\r\n"')
         lines = [line.strip().split('","') for line in lines]
 
         temp_df = DataFrame(lines, columns=['ticker', 'name', 'exchange'])
@@ -468,11 +471,11 @@ def get_data_fred(name=None, start=dt.datetime(2010, 1, 1),
 
     fred_URL = "http://research.stlouisfed.org/fred2/series/"
 
-    url = fred_URL + '%s' % name + \
-        '/downloaddata/%s' % name + '.csv'
-    data = read_csv(urllib.urlopen(url), index_col=0, parse_dates=True,
-                    header=None, skiprows=1, names=["DATE", name],
-                    na_values='.')
+    url = fred_URL + '%s' % name + '/downloaddata/%s' % name + '.csv'
+    with closing(urlopen(url)) as resp:
+        data = read_csv(resp, index_col=0, parse_dates=True,
+                        header=None, skiprows=1, names=["DATE", name],
+                        na_values='.')
     try:
         return data.truncate(start, end)
     except KeyError:
@@ -489,9 +492,9 @@ def get_data_famafrench(name, start=None, end=None):
     # path of zip files
     zipFileURL = "http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/"
 
-    url = urllib.urlopen(zipFileURL + name + ".zip")
-    zipfile = ZipFile(StringIO(url.read()))
-    data = zipfile.open(name + ".txt").readlines()
+    with closing(urlopen(zipFileURL + name + ".zip")) as url:
+        with closing(ZipFile(StringIO(url.read()))) as zf:
+            data = zf.read(name + ".txt").splitlines()
 
     file_edges = np.where(np.array([len(d) for d in data]) == 2)[0]
 
@@ -638,7 +641,7 @@ def get_options_data(self, month=None, year=None, expiry=None):
             url = str('http://finance.yahoo.com/q/op?s=' + self.symbol +
                                                             '+Options')
 
-        parsed = parse(urllib2.urlopen(url))
+        parsed = parse(url)
         doc = parsed.getroot()
         tables = doc.findall('.//table')
         calls = tables[9]
@@ -709,7 +712,7 @@ def get_call_data(self, month=None, year=None, expiry=None):
             url = str('http://finance.yahoo.com/q/op?s=' + self.symbol +
                                                             '+Options')
 
-        parsed = parse(urllib2.urlopen(url))
+        parsed = parse(url)
         doc = parsed.getroot()
         tables = doc.findall('.//table')
         calls = tables[9]
@@ -777,7 +780,7 @@ def get_put_data(self, month=None, year=None, expiry=None):
             url = str('http://finance.yahoo.com/q/op?s=' + self.symbol +
                                                             '+Options')
 
-        parsed = parse(urllib2.urlopen(url))
+        parsed = parse(url)
         doc = parsed.getroot()
         tables = doc.findall('.//table')
         puts = tables[13]


Test patch: