Repository: pandas-dev/pandas

Issue:
BUG/API: data readers should return missing data as NaN rather than warn
I have pandas version:  pandas-0.13.1-py2.7-win32.egg

Routine pandas.io.data._dl_mult_symbols has an "except IOError" where the following is executed:
                    stocks[sym] = np.nan

The next thing that happens is:
try:
        return Panel(stocks).swapaxes('items', 'minor')

However, if any of the values in stocks equals nan, then "return Panel(stocks).swapaxes('items', 'minor')" throws an error that is not handled and terminates the script.

I discovered this by calling pandas.io.data.get_data_yahoo(['EQQQ.F'],'20140930','20140930',3,0.001,False,False,5)



Discussion:
in 0.14.1 and > this doesn't return any data (and returns a nice exception).
It works just fine for example with symbol of `['IBM']`
I do recall a similiar issue, but cannot reproduce.

Pls reopen / comment if this is still an issue.

Thanks, jreback for the very quick reply.
I suspect you may have misunderstood the issue. In any case, in v0.14.1 I still see the same problem.

[Note, this is my 1st time working with github or reporting an issue, so bear with me if there's any conventions I am unaware of.]

The problem is that in _stocks_, if there is even only 1 value equal to _nan_, then the _Entire_ set of results is discarded and an error is returned.
What would be better, is if the good data was returned, and only the _nan_ data was discarded.

For example, the below works. I indicate lines I changed compared to v0.14.1.

```
def _dl_mult_symbols(symbols, start, end, chunksize, retry_count, pause,
                     method):
    stocks = {}
    for sym_group in _in_chunks(symbols, chunksize):
        for sym in sym_group:
            try:
                stocks[sym] = method(sym, start, end, retry_count, pause)
            except IOError:
                warnings.warn('Failed to read symbol: {0!r}, omitting from '    # <---
                              'data set.'.format(sym), SymbolWarning)           # <---
                pass # <-- More a hack than a proper fix, 
                         # but I would rather leave this to experienced people than suggest amateur solutions.

    try:
        return Panel(stocks).swapaxes('items', 'minor')
    except AttributeError:
        # cannot construct a panel with just 1D nans indicating no data
        raise RemoteDataError("No data fetched using "
                              "{0!r}".format(method.__name__))

```

@gayodeji actually this reproduces correctly by passing `['IBM','EQQQ.F']` iow at least 1 valid symbol (these raise occurs if all symbols are invalid, but that's a separate issue).

Ok, thought we had an issue for this somewhere.

Going to fix the title. Your soln is not actually what we want here, rather returning a Panel with missing values for the non-returned data.

Pls feel free to work on this.


Resolving patch:
diff --git a/pandas/io/data.py b/pandas/io/data.py
--- a/pandas/io/data.py
+++ b/pandas/io/data.py
@@ -317,6 +317,7 @@ def get_components_yahoo(idx_sym):
 def _dl_mult_symbols(symbols, start, end, chunksize, retry_count, pause,
                      method):
     stocks = {}
+    failed = []
     for sym_group in _in_chunks(symbols, chunksize):
         for sym in sym_group:
             try:
@@ -324,9 +325,14 @@ def _dl_mult_symbols(symbols, start, end, chunksize, retry_count, pause,
             except IOError:
                 warnings.warn('Failed to read symbol: {0!r}, replacing with '
                               'NaN.'.format(sym), SymbolWarning)
-                stocks[sym] = np.nan
+                failed.append(sym)
 
     try:
+        if len(stocks) > 0 and len(failed) > 0:
+            df_na = stocks.values()[0].copy()
+            df_na[:] = np.nan
+            for sym in failed:
+                stocks[sym] = df_na
         return Panel(stocks).swapaxes('items', 'minor')
     except AttributeError:
         # cannot construct a panel with just 1D nans indicating no data


Test patch:
diff --git a/pandas/io/tests/test_data.py b/pandas/io/tests/test_data.py
--- a/pandas/io/tests/test_data.py
+++ b/pandas/io/tests/test_data.py
@@ -94,6 +94,12 @@ def test_get_multi1(self):
             else:
                 self.assertRaises(AttributeError, lambda: pan.Close)
 
+    @network
+    def test_get_multi_invalid(self):
+        sl = ['AAPL', 'AMZN', 'INVALID']
+        pan = web.get_data_google(sl, '2012')
+        self.assertIn('INVALID', pan.minor_axis)
+
     @network
     def test_get_multi2(self):
         with warnings.catch_warnings(record=True) as w: