Repository: quantopian/zipline

Issue:
Zipline is not to be used as an installed python module?
While trying to reproduce the example on my local machine:

```
http://nbviewer.ipython.org/3962843/
```

I installed zipline from the github repo with

```
sudo pip install git+https://github.com/quantopian/zipline.git
```

which causes loader.py dump_benchmark() to try to write benchmark.msgpack in 

```
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/data/benchmark.msgpack'
```

which fails of course with permission denied. It works as expected when run within a project but not installed in site libs. Can or should the msgpack files be stored in a configurable location?

Thank you for releasing this. :)



Discussion:
Thanks for taking the time to report. I tried to reproduce but couldn't. It is a little surprising because you execute the command with sudo.

Is there a more complete traceback perhaps?

Thanks for checking in! Yes there is. Disabling the locally installed module and running in either iPython notebook or via script when I get to:

```
my_algo = BuyApple() # Instantiate our algorithm
results_buy_apple = my_algo.run(data) # Backtest algorithm on dataframe.
```

I get

```
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-7102635a4d33> in <module>()
      1 my_algo = BuyApple() # Instantiate our algorithm
----> 2 results_buy_apple = my_algo.run(data) # Backtest algorithm on dataframe.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/algorithm.pyc in run(self, source, start, end)
    170             self.transforms.append(sf)
    171 
--> 172         environment = create_trading_environment(start=start, end=end)
    173 
    174         # create transforms and zipline

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.pyc in create_trading_environment(year, start, end)
     98 def create_trading_environment(year=2006, start=None, end=None):
     99     """Construct a complete environment with reasonable defaults"""
--> 100     benchmark_returns, treasury_curves = load_market_data()
    101 
    102     if start is None:

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/utils/factory.pyc in load_market_data()
     55 Fetching data from Yahoo Finance.
     56 """.strip()
---> 57         data.loader.dump_benchmarks()
     58         fp_bm = open(benchmark_data_path, "rb")
     59 

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/data/loader.pyc in dump_benchmarks()
     59     benchmark_path = os.path.join(os.path.dirname(__file__),
     60                                   "benchmark.msgpack")
---> 61     benchmark_fp = open(benchmark_path, "wb")
     62     benchmark_data = []
     63     for daily_return in get_benchmark_returns():

IOError: [Errno 13] Permission denied: '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/zipline/data/benchmark.msgpack'

data msgpacks aren't distribute with source.
Fetching data from Yahoo Finance.
```

So at that point it's trying to write benchmark.msgpack into that path which fails.

Ah, I see. One solution would then be to move the fetching into setup.py rather than during run-time.

Right, it looks we didn't consider the "installed into a root protected directory" properly, because of the use of virtualenv.

Since, these files should probably be updated over time (so that fresh data can be worked with), maybe we should install to a directory not in the library path? e.g. a `~/.zipline/data/` directory, which could then be made configurable later?

I see, yeah that'd work.

In general though I'm not a big fan of smaller tools installing things
outside of their assigned directory. What's wrong with:
1) Packaging the msgpack with the packaged version.
2) Checking if present and downloading to system dir in setup.py.

On Thu, Nov 8, 2012 at 1:32 PM, Eddie Hebert notifications@github.comwrote:

> Right, it looks we didn't consider the "installed into a root protected
> directory" properly, because of the use of virtualenv.
> 
> Since, these files should probably be updated over time (so that fresh
> data can be worked with), maybe we should install to a directory not in the
> library path? e.g. a ~/.zipline/data/ directory, which could then be made
> configurable later?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/quantopian/zipline/issues/11#issuecomment-10199194.

@twiecki 

With regard to 1) Trying not to directly distribute the Yahoo Finance data, rather putting the onus on the developers' machines, so as to comply with my understanding of Yahoo's ToS.

2) We could do it in the setup.py so that it installs along side... but if we start providing tools for easy downloading of other data sets or ability to point at user provided data sets, we'd probably want to have the user lazily load that data iff they want to use that data. Though that is a use case we don't have yet.

I'm thinking there is a distinction between where the library and where the data it runs over exist. But can be convinced otherwise.

1) Ah, you are right. Disregard.

2) The problem is that even for the simplest example we require benchmark
data (see #13). I think it should just work out of the box. Alternatively
we could just allow running of zipline without benchmark data (as I
mentioned in #13).

A mechanism to load data sets once they are needed would be really cool but
makes me think about http://en.wikipedia.org/wiki/YAGNI ;)

On Thu, Nov 8, 2012 at 2:12 PM, Eddie Hebert notifications@github.comwrote:

> @twiecki https://github.com/twiecki
> 
> With regard to 1) Trying not to directly distribute the Yahoo Finance
> data, rather putting the onus on the developers' machines, so as to comply
> with my understanding of Yahoo's ToS.
> 
> 2) We could do it in the setup.py so that it installs along side... but if
> we start providing tools for easy downloading of other data sets or ability
> to point at user provided data sets, we'd probably want to have the user
> lazily load that data iff they want to use that data. Though that is a use
> case we don't have yet.
> 
> I'm thinking there is a distinction between where the library and where
> the data it runs over exist. But can be convinced otherwise.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/quantopian/zipline/issues/11#issuecomment-10200660.

As a (brand spanking new) user I agree with point 2 here and YAGNI. It'd be neat but eventually the community would start coming with all kinds of unforeseen ways to pull in data especially since pandas/python readily facilitates this. The way we users use zipline will of course also go beyond what was originally planned. That's a given. :) Because of that, working "out of the box" may be challenging because we'll be reshaping the definition of that box even as you're making it. Which is what I'm doing which brought this problem to light. Heh.

So pandas made it easy to just pull my data in and I then assumed the system would Just Work™. I didn't know (previously) about the benchmark data requirement or the limits of that data.

For now I think I'll attempt to hack in my own benchmark data. Perhaps even a flat unchanging set of data or munging the existing data to allow my data to work with it and get an approximation. Then I can start testing strategies with either an approximated risk report for now or just a basic drawdown and sharpe report.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -42,6 +42,7 @@
           'Logbook',
           'blist',
           'pytz',
+          'requests',
           'numpy',
           'pandas'
       ],
diff --git a/zipline/data/loader.py b/zipline/data/loader.py
--- a/zipline/data/loader.py
+++ b/zipline/data/loader.py
@@ -15,17 +15,33 @@
 
 
 import os
-
-if __name__ == "__main__":
-    import sys
-    sys.path.append(os.path.abspath('.'))
-    print sys.path
+from os.path import expanduser
 
 import msgpack
 
 from treasuries import get_treasury_data
 from benchmarks import get_benchmark_returns
 
+# TODO: Make this path customizable.
+DATA_PATH = os.path.join(
+    expanduser("~"),
+    '.zipline',
+    'data'
+)
+
+
+def get_datafile(name, mode='r'):
+    """
+    Returns a handle to data file.
+
+    Creates containing directory, if needed.
+    """
+
+    if not os.path.exists(DATA_PATH):
+        os.makedirs(DATA_PATH)
+
+    return open(os.path.join(DATA_PATH, name), mode)
+
 
 def dump_treasury_curves():
     """
@@ -36,7 +52,6 @@ def dump_treasury_curves():
     tr_data = []
 
     for curve in get_treasury_data():
-        print curve
         date_as_tuple = curve['date'].timetuple()[0:6] + \
             (curve['date'].microsecond,)
         # Not ideal but massaging data into expected format
@@ -44,10 +59,8 @@ def dump_treasury_curves():
         tr = (date_as_tuple, curve)
         tr_data.append(tr)
 
-    tr_path = os.path.join(os.path.dirname(__file__),
-                           "treasury_curves.msgpack")
-    tr_fp = open(tr_path, "wb")
-    tr_fp.write(msgpack.dumps(tr_data))
+    with get_datafile('treasury_curves.msgpack', mode='wb') as tr_fp:
+        tr_fp.write(msgpack.dumps(tr_data))
 
 
 def dump_benchmarks():
@@ -56,16 +69,13 @@ def dump_benchmarks():
 
     Puts source treasury and data into zipline.
     """
-    benchmark_path = os.path.join(os.path.dirname(__file__),
-                                  "benchmark.msgpack")
-    benchmark_fp = open(benchmark_path, "wb")
     benchmark_data = []
     for daily_return in get_benchmark_returns():
-        print daily_return
         date_as_tuple = daily_return.date.timetuple()[0:6] + \
             (daily_return.date.microsecond,)
         # Not ideal but massaging data into expected format
         benchmark = (date_as_tuple, daily_return.returns)
         benchmark_data.append(benchmark)
 
-    benchmark_fp.write(msgpack.dumps(benchmark_data))
+    with get_datafile('benchmark.msgpack', mode='wb') as bmark_fp:
+        bmark_fp.write(msgpack.dumps(benchmark_data))
diff --git a/zipline/utils/factory.py b/zipline/utils/factory.py
--- a/zipline/utils/factory.py
+++ b/zipline/utils/factory.py
@@ -20,7 +20,6 @@
 import pytz
 import msgpack
 import random
-from os.path import join, abspath, dirname
 from operator import attrgetter
 from collections import OrderedDict
 
@@ -35,27 +34,23 @@
 from zipline.sources import SpecificEquityTrades, DataFrameSource
 from zipline.gens.utils import create_trade
 from zipline.finance.trading import TradingEnvironment
-
-from zipline import data
-
-
-# TODO
-def data_path():
-    data_path = dirname(abspath(data.__file__))
-    return data_path
+from zipline.data.loader import (
+    get_datafile,
+    dump_benchmarks,
+    dump_treasury_curves
+)
 
 
 def load_market_data():
-    benchmark_data_path = join(data_path(), "benchmark.msgpack")
     try:
-        fp_bm = open(benchmark_data_path, "rb")
+        fp_bm = get_datafile('benchmark.msgpack', "rb")
     except IOError:
         print """
 data msgpacks aren't distribute with source.
 Fetching data from Yahoo Finance.
 """.strip()
-        data.loader.dump_benchmarks()
-        fp_bm = open(benchmark_data_path, "rb")
+        dump_benchmarks()
+        fp_bm = get_datafile('benchmark.msgpack', "rb")
 
     bm_list = msgpack.loads(fp_bm.read())
     bm_returns = []
@@ -71,20 +66,20 @@ def load_market_data():
         daily_return = risk.DailyReturn(date=event_dt, returns=returns)
         bm_returns.append(daily_return)
 
+    fp_bm.close()
+
     bm_returns = sorted(bm_returns, key=attrgetter('date'))
 
-    treasury_data_path = join(data_path(), "treasury_curves.msgpack")
     try:
-        fp_bm = open(treasury_data_path, "rb")
+        fp_tr = get_datafile('treasury_curves.msgpack', "rb")
     except IOError:
         print """
 data msgpacks aren't distribute with source.
 Fetching data from data.treasury.gov
 """.strip()
-        data.loader.dump_treasury_curves()
-        fp_bm = open(treasury_data_path, "rb")
+        dump_treasury_curves()
+        fp_tr = get_datafile('treasury_curves.msgpack', "rb")
 
-    fp_tr = open(join(data_path(), "treasury_curves.msgpack"), "rb")
     tr_list = msgpack.loads(fp_tr.read())
     tr_curves = {}
     for packed_date, curve in tr_list:
@@ -92,6 +87,8 @@ def load_market_data():
         #tr_dt = tr_dt.replace(hour=0, minute=0, second=0, tzinfo=pytz.utc)
         tr_curves[tr_dt] = curve
 
+    fp_tr.close()
+
     return bm_returns, tr_curves
 
 


Test patch: