I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
The first thing we'll do with this function is compute this for each month and then average. This should give us a sense of the relationship over a long timeframe.
Can you write Python code for it?

number_of_baskets = 10
mean_basket_returns = np.zeros(number_of_baskets)
for m in range(1, 25):
    basket_returns = compute_basket_returns(cap_data_filtered, month_forward_returns, number_of_baskets, m)
    mean_basket_returns += basket_returns

mean_basket_returns /= 24

# Plot the returns of each basket
plt.bar(range(number_of_baskets), mean_basket_returns)
plt.ylabel('Returns')
plt.xlabel('Basket')
plt.legend(['Returns of Each Basket']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
We'll repeat the same analysis for PE Ratio.
Can you write Python code for it?

number_of_baskets = 10
mean_basket_returns = np.zeros(number_of_baskets)
for m in range(1, 25):
    basket_returns = compute_basket_returns(pe_data_filtered, month_forward_returns, number_of_baskets, m)
    mean_basket_returns += basket_returns

mean_basket_returns /= 24

# Plot the returns of each basket
plt.bar(range(number_of_baskets), mean_basket_returns)
plt.ylabel('Returns')
plt.xlabel('Basket')
plt.legend(['Returns of Each Basket']);

f, axarr = plt.subplots(3, 4)
for month in range(1, 13):
    basket_returns = compute_basket_returns(pe_data_filtered, month_forward_returns, 10, month)

    r = np.floor((month-1) / 4)
    c = (month-1) % 4
    axarr[r, c].bar(range(10), basket_returns)
    axarr[r, c].xaxis.set_visible(False) # Hide the axis lables so the plots aren't super messy
    axarr[r, c].set_title('Month ' + str(month))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Regression_Model_Instability/notebook.ipynb".
The first task is:
Biased noise
The particular sample we choose for the data affects the model generated, and unevenly distributed noise can lead to an inaccurate model. Below we're drawing from a normal distribution, but because we do not have very many data points, we get a significant downward bias. If we took more measurements, both of the regression coefficients would move toward zero.
Can you write Python code for it?

# Draw observations from normal distribution
np.random.seed(107) # Fix seed for random number generation
rand = np.random.randn(20)

# Conduct linear regression on the ordered list of observations
xs = np.arange(20)
a, b = linreg(xs, rand)
print 'Slope:', b, 'Intercept:', a

# Plot the raw data and the regression line
plt.scatter(xs, rand, alpha=0.7)
Y_hat = xs * b + a
plt.plot(xs, Y_hat, 'r', alpha=0.9);

import seaborn

seaborn.regplot(xs, rand)

# Draw more observations
rand2 = np.random.randn(100)

# Conduct linear regression on the ordered list of observations
xs2 = np.arange(100)
a2, b2 = linreg(xs2, rand2)
print 'Slope:', b2, 'Intercept:', a2

# Plot the raw data and the regression line
plt.scatter(xs2, rand2, alpha=0.7)
Y_hat2 = xs2 * b2 + a2
plt.plot(xs2, Y_hat2, 'r', alpha=0.9);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Regression_Model_Instability/notebook.ipynb".
The first task is:
Multicollinearity
Above we were only considering regressions of one dependent variable against one independent one. However, we can also have multiple independent variables. This leads to instability if the independent variables are highly correlated.
Imagine we are using two independent variables, $X_1$ and $X_2$, which are very highly correlated. Then the coefficients may shift drastically if we add a new observation that is slightly better explained by one of the two than by the other. In the extreme case, if $X_1 = X_2$, then the choice of coefficients will depend on the particular linear regression algorithm.
Below, we run a multiple linear regression in which the independent variables are highly correlated. If we take our sample period to be 2013-01-01 to 2015-01-01, then the coefficients are approximately .25 and .1. But if we extend the period to 2015-06-01, the coefficients become approximately .18 and .20, respectively.
Can you write Python code for it?

# Get pricing data for two benchmarks (stock indices) and a stock
start = '2013-01-01'
end = '2015-01-01'
b1 = get_pricing('SPY', fields='price', start_date=start, end_date=end)
b2 = get_pricing('MDY', fields='price', start_date=start, end_date=end)
asset = get_pricing('V', fields='price', start_date=start, end_date=end)

mlr = regression.linear_model.OLS(asset, sm.add_constant(np.column_stack((b1, b2)))).fit()
prediction = mlr.params[0] + mlr.params[1]*b1 + mlr.params[2]*b2
print 'Constant:', mlr.params[0], 'MLR beta to S&P 500:', mlr.params[1], ' MLR beta to MDY', mlr.params[2]

# Plot the asset pricing data and the regression model prediction, just for fun
asset.plot()
prediction.plot();
plt.ylabel('Price')
plt.legend(['Asset', 'Linear Regression Prediction']);

# Get pricing data for two benchmarks (stock indices) and a stock
start = '2013-01-01'
end = '2015-06-01'
b1 = get_pricing('SPY', fields='price', start_date=start, end_date=end)
b2 = get_pricing('MDY', fields='price', start_date=start, end_date=end)
asset = get_pricing('V', fields='price', start_date=start, end_date=end)

mlr = regression.linear_model.OLS(asset, sm.add_constant(np.column_stack((b1, b2)))).fit()
prediction = mlr.params[0] + mlr.params[1]*b1 + mlr.params[2]*b2
print 'Constant:', mlr.params[0], 'MLR beta to S&P 500:', mlr.params[1], ' MLR beta to MDY', mlr.params[2]

# Plot the asset pricing data and the regression model prediction, just for fun
asset.plot()
prediction.plot();
plt.ylabel('Price')
plt.legend(['Asset', 'Linear Regression Prediction']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Spearman_Rank_Correlation/notebook.ipynb".
The first task is:
Let's take a look at the distribution of measured correlation coefficients and compare the spearman with the regular metric.
Can you write Python code for it?

plt.hist(spearman_dist, bins=50, alpha=0.5)
plt.hist(correlation_dist, bins=50, alpha=0.5)
plt.legend(['Spearman Rank', 'Regular Correlation'])
plt.xlabel('Correlation Coefficient')
plt.ylabel('Frequency');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Spearman_Rank_Correlation/notebook.ipynb".
The first task is:
Sure enough, the relationship is not detected. It is important when using both regular and spearman correlation to check for lagged relationships by offsetting your data and testing for different offset values.
Built-In Function
We can also use the spearmanr function in the scipy.stats library:
Can you write Python code for it?

# Generate two random data sets
np.random.seed(161)
X = np.random.rand(10)
Y = np.random.rand(10)

r_s = stats.spearmanr(X, Y)
print 'Spearman Rank Coefficient: ', r_s[0]
print 'p-value: ', r_s[1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Spearman_Rank_Correlation/notebook.ipynb".
The first task is:
We can see that the Spearman rank correlation copes with the non-linear relationship much better at most levels of noise. Interestingly, at very high levels, it seems to do worse than regular correlation.
Delay in correlation
Of you might have the case that one process affects another, but after a time lag. Now let's see what happens if we add the delay.
Can you write Python code for it?

n = 100

X = np.random.rand(n)
Xrank = stats.rankdata(X, method='average')
# n-2 is the second to last element
Yrank = stats.rankdata([1,1] + list(X[:(n-2)]), method='average')

diffs = Xrank - Yrank # order doesn't matter since we'll be squaring these values
r_s = 1 - 6*sum(diffs*diffs)/(n*(n**2 - 1))
print r_s


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Spearman_Rank_Correlation/notebook.ipynb".
The first task is:
We now have ourselves an $r_S$, but how do we interpret it? It's positive, so we know that the variables are not anticorrelated. It's not very large, so we know they aren't perfectly positively correlated, but it's hard to say from a glance just how significant the correlation is. Luckily, spearmanr also computes the p-value for this coefficient and sample size for us. We can see that the p-value here is above 0.05; therefore, we cannot claim that $X$ and $Y$ are correlated.
Real World Example: Mutual Fund Expense Ratio
Now that we've seen how Spearman rank correlation works, we'll quickly go through the process again with some real data. For instance, we may wonder whether the expense ratio of a mutual fund is indicative of its three-year Sharpe ratio. That is, does spending more money on administration, management, etc. lower the risk or increase the returns? Quantopian does not currently support mutual funds, so we will pull the data from Yahoo Finance. Our p-value cutoff will be the usual default of 0.05.
Data Source
Thanks to Matthew Madurski for the data. To obtain the same data:

Download the csv from this link. https://gist.github.com/dursk/82eee65b7d1056b469ab
Upload it to the 'data' folder in your research account.
Can you write Python code for it?

mutual_fund_data = local_csv('mutual_fund_data.csv')
expense = mutual_fund_data['Annual Expense Ratio'].values
sharpe = mutual_fund_data['Three Year Sharpe Ratio'].values

plt.scatter(expense, sharpe)
plt.xlabel('Expense Ratio')
plt.ylabel('Sharpe Ratio')

r_S = stats.spearmanr(expense, sharpe)
print 'Spearman Rank Coefficient: ', r_S[0]
print 'p-value: ', r_S[1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Statistical_Moments/notebook.ipynb".
The first task is:
A distribution which is not symmetric is called <i>skewed</i>. For instance, a distribution can have many small positive and a few large negative values (negatively skewed) or vice versa (positively skewed), and still have a mean of 0. A symmetric distribution has skewness 0. Positively skewed unimodal (one mode) distributions have the property that mean > median > mode. Negatively skewed unimodal distributions are the reverse, with mean < median < mode. All three are equal for a symmetric unimodal distribution.
The explicit formula for skewness is:
$$ S_K = \frac{n}{(n-1)(n-2)} \frac{\sum_{i=1}^n (X_i - \mu)^3}{\sigma^3} $$
Where $n$ is the number of observations, $\mu$ is the arithmetic mean, and $\sigma$ is the standard deviation. The sign of this quantity describes the direction of the skew as described above. We can plot a positively skewed and a negatively skewed distribution to see what they look like. For unimodal distributions, a negative skew typically indicates that the tail is fatter on the left, while a positive skew indicates that the tail is fatter on the right.
Can you write Python code for it?

# Generate x-values for which we will plot the distribution
xs2 = np.linspace(stats.lognorm.ppf(0.01, .7, loc=-.1), stats.lognorm.ppf(0.99, .7, loc=-.1), 150)

# Negatively skewed distribution
lognormal = stats.lognorm.pdf(xs2, .7)
plt.plot(xs2, lognormal, label='Skew > 0')

# Positively skewed distribution
plt.plot(xs2, lognormal[::-1], label='Skew < 0')
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Statistical_Moments/notebook.ipynb".
The first task is:
Although skew is less obvious when graphing discrete data sets, we can still compute it. For example, below are the skew, mean, and median for S&P 500 returns 2012-2014. Note that the skew is negative, and so the mean is less than the median.
Can you write Python code for it?

start = '2012-01-01'
end = '2015-01-01'
pricing = get_pricing('SPY', fields='price', start_date=start, end_date=end)
returns = pricing.pct_change()[1:]

print 'Skew:', stats.skew(returns)
print 'Mean:', np.mean(returns)
print 'Median:', np.median(returns)

plt.hist(returns, 30);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Statistical_Moments/notebook.ipynb".
The first task is:
Great, if properly calibrated we should expect to be wrong $5\%$ of the time at a 0.05 significance level, and this is pretty close. This means that the test is working as we expect.
Can you write Python code for it?

_, pvalue, _, _ = jarque_bera(returns)

if pvalue > 0.05:
    print 'The returns are likely normal.'
else:
    print 'The returns are likely not normal.'


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Statistical_Moments/notebook.ipynb".
The first task is:
Kurtosis
Kurtosis attempts to measure the shape of the deviation from the mean. Generally, it describes how peaked a distribution is compared the the normal distribution, called mesokurtic. All normal distributions, regardless of mean and variance, have a kurtosis of 3. A leptokurtic distribution (kurtosis > 3) is highly peaked and has fat tails, while a platykurtic distribution (kurtosis < 3) is broad. Sometimes, however, kurtosis in excess of the normal distribution (kurtosis - 3) is used, and this is the default in scipy. A leptokurtic distribution has more frequent large jumps away from the mean than a normal distribution does while a platykurtic distribution has fewer.
Can you write Python code for it?

# Plot some example distributions
plt.plot(xs,stats.laplace.pdf(xs), label='Leptokurtic')
print 'Excess kurtosis of leptokurtic distribution:', (stats.laplace.stats(moments='k'))
plt.plot(xs, normal, label='Mesokurtic (normal)')
print 'Excess kurtosis of mesokurtic distribution:', (stats.norm.stats(moments='k'))
plt.plot(xs,stats.cosine.pdf(xs), label='Platykurtic')
print 'Excess kurtosis of platykurtic distribution:', (stats.cosine.stats(moments='k'))
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Statistical_Moments/notebook.ipynb".
The first task is:
Sometimes mean and variance are not enough to describe a distribution. When we calculate variance, we square the deviations around the mean. In the case of large deviations, we do not know whether they are likely to be positive or negative. This is where the skewness and symmetry of a distribution come in. A distribution is <i>symmetric</i> if the parts on either side of the mean are mirror images of each other. For example, the normal distribution is symmetric. The normal distribution with mean $\mu$ and standard deviation $\sigma$ is defined as
$$ f(x) = \frac{1}{\sigma \sqrt{2 \pi}} e^{-\frac{(x - \mu)^2}{2 \sigma^2}} $$
We can plot it to confirm that it is symmetric:
Can you write Python code for it?

# Plot a normal distribution with mean = 0 and standard deviation = 2
xs = np.linspace(-6,6, 300)
normal = stats.norm.pdf(xs)
plt.plot(xs, normal);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/The_Dangers_of_Overfitting/notebook.ipynb".
The first task is:
Clearly fitting to our sample data doesn't always give good results in the future. Just for fun, let's plot the length scores computed from the two different timeframes:
Can you write Python code for it?

plt.plot(length_scores)
plt.plot(length_scores2)
plt.xlabel('Window length')
plt.ylabel('Score')
plt.legend(['2011-2013', '2013-2015']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/The_Dangers_of_Overfitting/notebook.ipynb".
The first task is:
If we pick the length based on which seems best - say, on how well our model or algorithm performs - we are overfitting. Below we have a simple trading algorithm which bets on the stock price reverting to the rolling mean (for more details, check out the mean reversion notebook). We use the performance of this algorithm to score window lengths and find the best one. However, when we consider a different timeframe, this window length is far from optimal. This is because our original choice was overfitted to the sample data.
Can you write Python code for it?

# Trade using a simple mean-reversion strategy
def trade(stock, length):

    # If window length is 0, algorithm doesn't make sense, so exit
    if length == 0:
        return 0

    # Compute rolling mean and rolling standard deviation
    mu = pd.rolling_mean(stock, window=length)
    std = pd.rolling_std(stock, window=length)

    # Compute the z-scores for each day using the historical data up to that day
    zscores = (stock - mu)/std

    # Simulate trading
    # Start with no money and no positions
    money = 0
    count = 0
    for i in range(len(stock)):
        # Sell short if the z-score is > 1
        if zscores[i] > 1:
            money += stock[i]
            count -= 1
        # Buy long if the z-score is < 1
        elif zscores[i] < -1:
            money -= stock[i]
            count += 1
        # Clear positions if the z-score between -.5 and .5
        elif abs(zscores[i]) < 0.5:
            money += count*stock[i]
            count = 0
    return money

# Find the window length 0-254 that gives the highest returns using this strategy
length_scores = [trade(pricing, l) for l in range(255)]
best_length = np.argmax(length_scores)
print 'Best window length:', best_length

# Get pricing data for a different timeframe
start2 = '2013-01-01'
end2 = '2015-01-01'
pricing2 = get_pricing('MCD', fields='price', start_date=start2, end_date=end2)

# Find the returns during this period using what we think is the best window length
length_scores2 = [trade(pricing2, l) for l in range(255)]
print best_length, 'day window:', length_scores2[best_length]

# Find the best window length based on this dataset, and the returns using this window length
best_length2 = np.argmax(length_scores2)
print best_length2, 'day window:', length_scores2[best_length2]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/The_Dangers_of_Overfitting/notebook.ipynb".
The first task is:
If we wanted, we could scan our universe for variables that were correlated with the dependent variable, and construct an extremely overfitted model. However, in most cases the correlation will be spurious, and the relationship will not continue into the future.
Example: Rolling windows
One of the challenges in building a model that uses rolling parameter estimates, such as rolling mean or rolling beta, is choosing a window length. A longer window will take into account long-term trends and be less volatile, but it will also lag more when taking into account new observations. The choice of window length strongly affects the rolling parameter estimate and can change how we see and treat the data. Below we calculate the rolling averages of a stock price for different window lengths:
Can you write Python code for it?

# Load the pricing data for a stock
start = '2011-01-01'
end = '2013-01-01'
pricing = get_pricing('MCD', fields='price', start_date=start, end_date=end)

# Compute rolling averages for various window lengths
mu_30d = pd.rolling_mean(pricing, window=30)
mu_60d = pd.rolling_mean(pricing, window=60)
mu_100d = pd.rolling_mean(pricing, window=100)

# Plot asset pricing data with rolling means from the 100th day, when all the means become available
plt.plot(pricing[100:], label='Asset')
plt.plot(mu_30d[100:], label='30d MA')
plt.plot(mu_60d[100:], label='60d MA')
plt.plot(mu_100d[100:], label='100d MA')
plt.xlabel('Day')
plt.ylabel('Price')
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/The_Dangers_of_Overfitting/notebook.ipynb".
The first task is:
When working with real data, there is unlikely to ever be a situation where a ninth-degree polynomial is appropriate: our choice of function should reflect a belief about the underlying process, and real-world processes generally do not follow high-degree polynomial curves. This example is contrived, but it can be tempting to use a quadratic or cubic model just to decrease sample error.
Note: Model/Parameter Parsimony
Just as the most elegant physics models describe a tremendous amount of our world through a few equations, a good trading model should explain most of the data through a few rules. Any time you start to have a number of rules even close to the number of points in your data set, you can be sure you are overfitting. Since parameters can be thought of as rules as they equivalently constrain a model, the same is true of parameters. Fewer parameters is better, and it is better to explain 60% of the data with 2-3 paremeters than 90% with 10.
Beware of the perfect fit
Because there is almost always noise present in real data, a perfect fit is almost always indicative of overfitting. It is almost impossible to know the percentage noise/signal in a given data set while you are developing the model, but use your common sense. Are the predictions surprisingly good? Then you're probably overfitting.
Example: Regression parameters
How do we know which variables to include in a model? If we're afraid of omitting something important, we might try different ones and include all the variables we can find that improve the fit. Below we regress one asset that is in the same sector as the asset whose price we're trying to predict, and three other unrelated ones. In our initial timeframe, we are able to fit the model more closely to the data when using multiple variables than when using just one.
Can you write Python code for it?

# Load one year's worth of pricing data for five different assets
start = '2013-01-01'
end = '2014-01-01'
x1 = get_pricing('PEP', fields='price', start_date=start, end_date=end)
x2 = get_pricing('MCD', fields='price', start_date=start, end_date=end)
x3 = get_pricing('ATHN', fields='price', start_date=start, end_date=end)
x4 = get_pricing('DOW', fields='price', start_date=start, end_date=end)
y = get_pricing('PG', fields='price', start_date=start, end_date=end)

# Build a linear model using only x1 to explain y
slr = regression.linear_model.OLS(y, sm.add_constant(x1)).fit()
slr_prediction = slr.params[0] + slr.params[1]*x1

# Run multiple linear regression using x1, x2, x3, x4 to explain y
mlr = regression.linear_model.OLS(y, sm.add_constant(np.column_stack((x1,x2,x3,x4)))).fit()
mlr_prediction = mlr.params[0] + mlr.params[1]*x1 + mlr.params[2]*x2 + mlr.params[3]*x3 + mlr.params[4]*x4

# Compute adjusted R-squared for the two different models
print 'SLR R-squared:', slr.rsquared_adj
print 'SLR p-value:', slr.f_pvalue
print 'MLR R-squared:', mlr.rsquared_adj
print 'MLR p-value:', mlr.f_pvalue

# Plot y along with the two different predictions
y.plot()
slr_prediction.plot()
mlr_prediction.plot()
plt.ylabel('Price')
plt.xlabel('Date')
plt.legend(['PG', 'SLR', 'MLR']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Universe_Selection/notebook.ipynb".
The first task is:
From the above plots it is clear that there is a mild sector bias towards the financial services industry. Any big events that affect companies in this sector will have a large effect on this universe and any algorithm that uses it.
One option is to equal-weight the sectors, so that equities from each industry sector make up an identical proportion of the final universe. This, however, comes with its own disadvantages. In a sector-equal Lectures500, the universe would include some lower-revenue real estate equities at the expense of higher-revenue financial services equities.
Turnover
Another thing to consider when designing a universe is the rate at which the universe changes. Turnover is a way of measuring this rate of change. Turnover is defined as the number of equities to enter or exit the universe in a particular time window.
Let us imagine a universe with a turnover of 0. This universe would be completely unchanged by market movements. Moreover, stocks inappropriate for the universe would never be removed and stocks that should be included will never enter.
Conversely, imagine a universe that changes every one of its constituents every day. An algorithm built on this universe will be forced to sell its entire portfolio every day. This incurs transaction costs which erode returns.
When creating a universe, there is an inherent tradeoff between stagnation and sensitivity to the market.
Let's have a look at the turnover for the Lectures500!
Can you write Python code for it?

res = run_pipeline(Pipeline(columns={'Lectures500' : Lectures500}), '2015-01-01', '2016-01-01')
res = res.unstack().fillna(False)

def calculate_daily_turnover(unstacked):
    return (unstacked
            .diff()        # Get True/False showing where values changed from previous day.
            .iloc[1:]      # Drop first row, which is meaningless after diff().
            .astype(bool)  # diff() coerces from bool -> object :(.  Undo that.
            .groupby(axis=1, level=0)
            .sum())        # Group by universe and count number of True values in each row.

def plot_daily_turnover(unstacked):

    # Calculate locations where the inclusion state of an asset changed.
    turnover = calculate_daily_turnover(unstacked)

    # Write the data to an axis.
    ax = turnover.plot(figsize=(14, 8))

    # Add style to the axis.
    ax.grid(False)
    ax.set_title('Changes per Day')
    ax.set_ylabel('Number of Added or Removed Assets')

def print_daily_turnover_stats(unstacked):
    turnover = calculate_daily_turnover(unstacked)
    print(turnover.describe().loc[['mean', 'std', '25%', '50%', '75%', 'min', 'max']])

plot_daily_turnover(res)
print_daily_turnover_stats(res)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Universe_Selection/notebook.ipynb".
The first task is:
Looking at the metrics, we can see that the smoothed universe has a lower turnover than the original Lectures500. Since this is a good characteristic, we will add this logic to the universe.
NB: Smoothing can also be accomplished by downsampling.
Tradability Concerns
While controlling for sector exposure and turnover are important, perhaps the most important part of any universe selection methodology is ensuring that the securities included in the universe can be traded.
Recently we released two of our own universes, the Q500US and the Q1500US, built with the same basic methodology. These universes are very similar to the Lectures500 under the hood and are built with all the concerns listed here in mind. As part of this release, we developed a tradability filter. The filter will return True in the follow cases:

The stock must be the primary share class for its company.
The company issuing the stock must have known market
   capitalization.
The stock must not be a depository receipt.
The stock must not be traded over the counter (OTC).
The stock must not be for a limited partnership.
The stock must have a known previous-day close price.
The stock must have had nonzero volume on the previous trading
   day.

Let's add these tradability constraints to the Lectures500US.
Can you write Python code for it?

from quantopian.pipeline.filters import default_us_equity_universe_mask

Lectures500 = Lectures500 & default_us_equity_universe_mask()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Universe_Selection/notebook.ipynb".
The first task is:
Smoothing
A good way to reduce turnover is through smoothing functions. Smoothing is the process of taking noisy data and aggregating it in order to analyze its underlying trends. When applied to universe selection, a good smoothing function prevents equities at the universe boundary from entering and exiting frequently.
One example of a potential smoothing function is a filter that finds equities that have passed the Lectures500 criteria for 16 or more days out of the past 21 days. We will call this filter AtLeast16. This aggregation many days of data lends a certain degree of flexibility to the edges of our universe. If, for example, Equity XYZ is very close to the boundary for inclusion, in a given month, it may flit in and out of the Lectures500 day after day. However, with the AtLeast16 filter, Equity XYZ is allowed to enter and exit the daily universe a maximum of 5 times before it is excluded from the smoothed universe.
Let's apply a smoothing function to our universe and see its effect on turnover.
Can you write Python code for it?

from quantopian.pipeline.filters import AtLeastN

Lectures500 = AtLeastN(inputs=[Lectures500],
                       window_length=21,
                       N=16,)

res_smoothed = run_pipeline(Pipeline(columns={'Lectures500 Smoothed' : Lectures500}),
                            '2015-01-01',
                            '2016-01-01')
res_smoothed = res_smoothed.unstack().fillna(False)

plot_daily_turnover(res_smoothed)
print_daily_turnover_stats(res_smoothed)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Universe_Selection/notebook.ipynb".
The first task is:
This is a good start, but again, it is a very naive universe. Normally, high revenue is a characteristic of a healthy, thriving company, but there are many other things that play into the construction of a good universe. While this idea has a reasonable economic basis, more analysis has to be conducted to determine the efficacy of this universe. There may be more subtle things occurring independently of the revenue of its constituent companies.
For the rest of this notebook, we will design our own universe, profile it and check its performance. Let's create the Lectures500!
Lectures500
Sector Exposure
If I create a universe that only looks at equities in the technology sector, my algorithm will have an extreme sector bias. Companies in the same industry sector are affected by similar macroeconomic trends and therefore their performance tends to be correlated. In the case of particular strategies, we may find the benefits of working exclusively within a particular sector greater than the downside risks, but this is not suitable for creating a general-purpose, quality universe.
Let's have a look at the sector breakdown of the Lectures500.
Can you write Python code for it?

# Rename our universe to Lectures500
Lectures500 = universe

SECTOR_CODE_NAMES = {
    Sector.BASIC_MATERIALS: 'Basic Materials',
    Sector.CONSUMER_CYCLICAL: 'Consumer Cyclical',
    Sector.FINANCIAL_SERVICES: 'Financial Services',
    Sector.REAL_ESTATE: 'Real Estate',
    Sector.CONSUMER_DEFENSIVE: 'Consumer Defensive',
    Sector.HEALTHCARE: 'Healthcare',
    Sector.UTILITIES: 'Utilities',
    Sector.COMMUNICATION_SERVICES: 'Communication Services',
    Sector.ENERGY: 'Energy',
    Sector.INDUSTRIALS: 'Industrials',
    Sector.TECHNOLOGY: 'Technology',
    -1 : 'Misc'
}

def get_sector_codes(day, universe):
    pipe = Pipeline(columns={'Sector': Sector()}, screen=universe)
    # Drop the datetime level of the index, since we only have one day of data
    return run_pipeline(pipe, day, day).reset_index(level=0, drop=True)

def calculate_sector_counts(sectors):
    counts = (sectors.groupby('Sector').size())

    # Replace numeric sector codes with human-friendly names
    counts.index = counts.index.map(lambda code: SECTOR_CODE_NAMES[code])
    return counts

lectures500_sectors = get_sector_codes('2016-01-04', Lectures500)
lectures500_counts = calculate_sector_counts(lectures500_sectors)

def plot_sector_counts(sector_counts):

    bar = plt.subplot2grid((5,5), (0,0), rowspan=2, colspan=5)
    pie = plt.subplot2grid((5,5), (2,0), rowspan=3, colspan=5)

    # Bar chart
    sector_counts.plot(
        kind='bar',
        color='b',
        rot=30,
        ax=bar,
    )

    bar.set_title('Sector Exposure - Counts')

    # Pie chart
    sector_counts.plot(
        kind='pie',
        colormap='Set3',
        autopct='%.2f %%',
        fontsize=12,
        ax=pie,
    )
    pie.set_ylabel('')  # This overwrites default ylabel, which is None :(
    pie.set_title('Sector Exposure - Proportions')

    plt.tight_layout();

plot_sector_counts(lectures500_counts)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
CVaR is higher because it is capturing more information about the shape of the distribution, AKA the moments of the distribution. If the tails have more mass, this will capture that. In general it is considered to be a far superior metric compared with VaR and you should use it over VaR in most cases.
Let's visualize what it's capturing.
Can you write Python code for it?

lookback_days = 520
alpha = 0.95

# Multiply asset returns by weights to get one weighted portfolio return
portfolio_returns = returns.fillna(0.0).iloc[-lookback_days:].dot(weights)

portfolio_VaR = value_at_risk(value_invested, returns, weights, alpha=0.95)
# Need to express it as a return rather than absolute loss
portfolio_VaR_return = portfolio_VaR / value_invested

portfolio_CVaR = cvar(value_invested, returns, weights, alpha=0.95)
# Need to express it as a return rather than absolute loss
portfolio_CVaR_return = portfolio_CVaR / value_invested

# Plot only the observations > VaR on the main histogram so the plot comes out
# nicely and doesn't overlap.
plt.hist(portfolio_returns[portfolio_returns > portfolio_VaR_return], bins=20)
plt.hist(portfolio_returns[portfolio_returns < portfolio_VaR_return], bins=10)
plt.axvline(portfolio_VaR_return, color='red', linestyle='solid');
plt.axvline(portfolio_CVaR_return, color='red', linestyle='dashed');
plt.legend(['VaR for Specified Alpha as a Return',
            'CVaR for Specified Alpha as a Return',
            'Historical Returns Distribution',
            'Returns < VaR'])
plt.title('Historical VaR and CVaR');
plt.xlabel('Return');
plt.ylabel('Observation Frequency');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Conditional Value at Risk (CVaR)
CVaR is what many consider an improvement on VaR, as it takes into account the shape of the returns distribution. It is also known as Expected Shortfall (ES), as it is an expectation over all the different possible losses greater than VaR and their corresponding estimated likelihoods.
If you are not familiar with expectations, much content is available online. However we will provide a brief refresher.
Expected Value
Say you have a fair six sided die. Each number is equally likely. The notion of an expectation, written as $\mathrm{E}(X)$, is what should you expect to happen out of all the possible outcomes. To get this you multiply each event by the probability of that event and add that up, think of it as a probability weighted average. With a die we get
$$1/6 \times 1 + 1/6 \times 2 + 1/6 \times 3 + 1/6 \times 4 + 1/6 \times 5 + 1/6 \times 6 =  3.5$$
When the probabilities are unequal it gets more complicated, and when the outcomes are continuous we have to use integration in closed form equations. Here is the formula for CVaR.
$$CVaR_{\alpha}(x) \approx \frac{1}{(1-\alpha)} \int_{f(x,y) \geq VaR_{\alpha}(x)} f(x,y)p(y)dy dx$$
Can you write Python code for it?

def cvar(value_invested, returns, weights, alpha=0.95, lookback_days=520):
    # Call out to our existing function
    var = value_at_risk(value_invested, returns, weights, alpha, lookback_days=lookback_days)
    returns = returns.fillna(0.0)
    portfolio_returns = returns.iloc[-lookback_days:].dot(weights)

    # Get back to a return rather than an absolute loss
    var_pct_loss = var / value_invested

    return value_invested * np.nanmean(portfolio_returns[portfolio_returns < var_pct_loss])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Historical (Non-Parametric) Case
Historical VaR instead uses historical data to draw a discrete Probability Density Function, or histogram. Then finds the point at which only $100 \times (1-\alpha)\%$ of the points are below that return. It returns that return as the VaR return for coverage $\alpha$.
Can you write Python code for it?

lookback_days = 520
alpha = 0.95

# Multiply asset returns by weights to get one weighted portfolio return
portfolio_returns = returns.fillna(0.0).iloc[-lookback_days:].dot(weights)

portfolio_VaR = value_at_risk(value_invested, returns, weights, alpha=0.95)
# Need to express it as a return rather than absolute loss
portfolio_VaR_return = portfolio_VaR / value_invested

plt.hist(portfolio_returns, bins=20)
plt.axvline(portfolio_VaR_return, color='red', linestyle='solid');
plt.legend(['VaR for Specified Alpha as a Return', 'Historical Returns Distribution'])
plt.title('Historical VaR');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Interpreting this, we say that historically no more than $5\%$ of days resulted in losses more extreme than this, or that on each day your probability of losing this much is less than $5\%$. Keeping in mind that any forecast like this is just an estimate.
Normal vs. Non-Parametric Historical VaR
Normal Case
A special case of VaR is when you assume that the returns follow a given distribution rather than non-parametrically estiamting it historically. In this case a normal VaR would fit our data, because all our returns were simulated form a normal distribution. We can check this by using a normal distribution Cumulative Distribution Function (CDF), which sums the area under a normal curve to figure out how likely certain values are. We'll use an inverse CDF, or PPF, which for a given likelihood will tell us to which value that likelihood corresponds.
Specifically, the closed form formula for Normal VaR is
$$VaR_{\alpha}(x) = \mu - \sigma N^{-1}(\alpha)$$
Can you write Python code for it?

# Portfolio mean return is unchanged, but std has to be recomputed
# This is because independent variances sum, but std is sqrt of variance
portfolio_std = np.sqrt( np.power(sigma, 2) * num_assets ) / num_assets

# manually
(mu - portfolio_std * norm.ppf(0.95)) * value_invested


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Let's compute CVaR on our data and see how it compares with VaR.
Can you write Python code for it?

cvar(value_invested, returns, weights, lookback_days=500)

value_at_risk(value_invested, returns, weights, lookback_days=500)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Let's visualize this.
Can you write Python code for it?

lookback_days = 520
alpha = 0.95

# Multiply asset returns by weights to get one weighted portfolio return
portfolio_returns = returns.fillna(0.0).iloc[-lookback_days:].dot(weights)

portfolio_VaR = value_at_risk(value_invested, returns, weights, alpha=0.95)
# Need to express it as a return rather than absolute loss
portfolio_VaR_return = portfolio_VaR / value_invested

plt.hist(portfolio_returns, bins=20)
plt.axvline(portfolio_VaR_return, color='red', linestyle='solid');
plt.legend(['VaR for Specified Alpha as a Return', 'Historical Returns Distribution'])
plt.title('Historical VaR');
plt.xlabel('Return');
plt.ylabel('Observation Frequency');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Now let's compute the VaR for $\alpha = 0.95$. We'll write this as $VaR_{\alpha=0.95}$ from now on.
Can you write Python code for it?

value_at_risk(value_invested, returns, weights, alpha=0.95, lookback_days=520)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Seems close enough to within some random variance. Let's visualize the continuous normal case. Notice that the VaR is expressed as a return rather than an absolute loss. To get aboslute loss we just need to multiply by value invested.
Can you write Python code for it?

def value_at_risk_N(mu=0, sigma=1.0, alpha=0.95):
    return mu - sigma*norm.ppf(alpha)

x = np.linspace(-3*sigma,3*sigma,1000)
y = norm.pdf(x, loc=mu, scale=portfolio_std)
plt.plot(x,y);
plt.axvline(value_at_risk_N(mu = 0.01, sigma = portfolio_std, alpha=0.95), color='red', linestyle='solid');
plt.legend(['Return Distribution', 'VaR for Specified Alpha as a Return'])
plt.title('VaR in Closed Form for a Normal Distribution');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Simulated Data Example
Let's simulate some returns of 10 hypothetical assets.
NOTE
We use normal distributions to simulate the returns, in practice real returns will almost never follow normal distributions and usually have weird behavior including fat tails. We'll discuss this more later.
Can you write Python code for it?

# mu = 0.01, std = 0.10, 1000 bars, 10 assets
mu = 0.01
sigma = 0.10
bars = 1000
num_assets = 10

returns = np.random.normal(mu, sigma, (bars, num_assets))

# Fake asset names
names = ['Asset %s' %i  for i in range(num_assets)]

# Put in a pandas dataframe
returns = pd.DataFrame(returns, columns=names)

# Plot the last 50 bars
plt.plot(returns.head(50))
plt.xlabel('Time')
plt.ylabel('Return');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
Underlying Distributions Are Not Always Normal
In real financial data the underlying distributions are rarely normal. This is why we prefer historical VaR as opposed to an assumption of an underlying distribution. Historical VaR is also non-parametric, so we aren't at risk of overfitting distribution parameters to some data set.
Real Data Example
We'll show this on some real financial data.
Can you write Python code for it?

# OEX components as of 3/31/16
# http://www.cboe.com/products/indexcomponents.aspx?DIR=OPIndexComp&FILE=snp100.doc
oex = ['MMM','T','ABBV','ABT','ACN','ALL','GOOGL','GOOG','MO','AMZN','AXP','AIG','AMGN','AAPL','BAC',
       'BRK-B','BIIB','BLK','BA','BMY','CVS','COF','CAT','CELG','CVX','CSCO','C','KO','CL','CMCSA',
       'COP','CSOT','DHR','DOW','DUK','DD','EMC','EMR','EXC','XOM','FB','FDX','F','GD','GE','GM','GILD',
       'GS','HAL','HD','HON','INTC','IBM','JPM','JNJ','KMI','LLY','LMT','LOW','MA','MCD','MDT','MRK',
       'MET,','MSFT','MDZL','MON','MS','NKE','NEE','OXY','ORCL','PYPL','PEP','PFE','PM','PG','QCOM',
       'RTN','SLB','SPG','SO','SBUX','TGT','TXN','BK','PCLN','TWX','FOXA','FOX','USB','UNP','UPS','UTX',
       'UNH','VZ','V','WMT','WBA','DIS','WFC']
tickers = symbols(oex)
num_stocks = len(tickers)

start = time.time()
data = get_pricing(tickers, fields='close_price', start_date='2014-01-01', end_date='2016-04-04')
end = time.time()
print "Time: %0.2f seconds." % (end - start)

returns = data.pct_change()
returns = returns - returns.mean(skipna=True) # de-mean the returns

data.plot(legend=None);
returns.plot(legend=None);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/VaR_and_CVaR/notebook.ipynb".
The first task is:
We can see here that VaR does appear to converge within a 400-600 lookback window period. Therefore our 520 day parameter should be fine. In fact, 1000 may be better as it uses strictly more information, but more computationally intensive and prey to stationarity concerns.
It can be useful to do analyses like this when evaluating whether a VaR is meaningful. Another check we'll do is for stationarity of the portfolio returns over this time period.
Can you write Python code for it?

from statsmodels.tsa.stattools import adfuller

results = adfuller(portfolio_returns)
pvalue = results[1]

if pvalue < 0.05:
    print 'Process is likely stationary.'
else:
    print 'Process is likely non-stationary.'


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Variance/notebook.ipynb".
The first task is:
A related notion is target semivariance (and target semideviation), where we average the distance from a target of values which fall below that target:
$$ \frac{\sum_{X_i < B} (X_i - B)^2}{n_{<B}} $$
Can you write Python code for it?

B = 19
lows_B = [e for e in X if e <= B]
semivar_B = sum(map(lambda x: (x - B)**2,lows_B))/len(lows_B)

print 'Target semivariance of X:', semivar_B
print 'Target semideviation of X:', np.sqrt(semivar_B)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Variance/notebook.ipynb".
The first task is:
Mean Absolute Deviation (MAD)
The mean absolute deviation is the average of the distances of observations from the arithmetic mean. We use the absolute value of the deviation, so that 5 above the mean and 5 below the mean both contribute 5, because otherwise the deviations always sum to 0.
$$ MAD = \frac{\sum_{i=1}^n |X_i - \mu|}{n} $$
where $n$ is the number of observations and $\mu$ is their mean.
Can you write Python code for it?

abs_dispersion = [np.abs(mu - x) for x in X]
MAD = np.sum(abs_dispersion)/len(abs_dispersion)
print 'Mean absolute deviation of X:', MAD


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Variance/notebook.ipynb".
The first task is:
One way to interpret standard deviation is by referring to Chebyshev's inequality. This tells us that the proportion of samples within $k$ standard deviations (that is, within a distance of $k \cdot$ standard deviation) of the mean is at least $1 - 1/k^2$ for all $k>1$.
Let's check that this is true for our data set.
Can you write Python code for it?

k = 1.25
dist = k*np.std(X)
l = [x for x in X if abs(x - mu) <= dist]
print 'Observations within', k, 'stds of mean:', l
print 'Confirming that', float(len(l))/len(X), '>', 1 - 1/k**2


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Variance/notebook.ipynb".
The first task is:
Range
Range is simply the difference between the maximum and minimum values in a dataset. Not surprisingly, it is very sensitive to outliers. We'll use numpy's peak to peak (ptp) function for this.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Variance/notebook.ipynb".
The first task is:
The bound given by Chebyshev's inequality seems fairly loose in this case. This bound is rarely strict, but it is useful because it holds for all data sets and distributions.
Semivariance and semideviation
Although variance and standard deviation tell us how volatile a quantity is, they do not differentiate between deviations upward and deviations downward. Often, such as in the case of returns on an asset, we are more worried about deviations downward. This is addressed by semivariance and semideviation, which only count the observations that fall below the mean. Semivariance is defined as
$$ \frac{\sum_{X_i < \mu} (X_i - \mu)^2}{n_<} $$
where $n_<$ is the number of observations which are smaller than the mean. Semideviation is the square root of the semivariance.
Can you write Python code for it?

# Because there is no built-in semideviation, we'll compute it ourselves
lows = [e for e in X if e <= mu]

semivar = np.sum( (lows - mu) ** 2 ) / len(lows)

print 'Semivariance of X:', semivar
print 'Semideviation of X:', np.sqrt(semivar)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Violations_of_Regression_Models/notebook.ipynb".
The first task is:
# Multicollinearity
When using multiple independent variables, it is important to check for multicollinearity; that is, an approximate linear relation between the independent variables, such as
$$ X_2 \approx 5 X_1 - X_3 + 4.5 $$
With multicollinearity, it is difficult to identify the independent effect of each variable, since we can change around the coefficients according to the linear relation without changing the model. As with truly unnecessary variables, this will usually not hurt the accuracy of the model, but will cloud our analysis. In particular, the estimated coefficients will have large standard errors. The coefficients will also no longer represent the partial effect of each variable, since with multicollinearity we cannot change one variable while holding the others constant.
High correlation between independent variables is indicative of multicollinearity. However, it is not enough, since we would want to detect correlation between one of the variables and a linear combination of the other variables. If we have high R-squared but low t-statistics on the coefficients (the fit is good but the coefficients are not estimated precisely) we may suspect multicollinearity. To resolve the problem, we can drop one of the independent variables involved in the linear relation.
For instance, using two stock indices as our independent variables is likely to lead to multicollinearity. Below we can see that removing one of them improves the t-statistics without hurting R-squared.
Another important thing to determine here is which variable may be the casual one. If we hypothesize that the market influences both MDY and HPQ, then the market is the variable that we should use in our predictive model.
Can you write Python code for it?

# Load pricing data for asset and two market indices
start = '2014-01-01'
end = '2015-01-01'
b1 = get_pricing('SPY', fields='price', start_date=start, end_date=end)
b2 = get_pricing('MDY', fields='price', start_date=start, end_date=end)
a = get_pricing('HPQ', fields='price', start_date=start, end_date=end)

# Run multiple linear regression
mlr = regression.linear_model.OLS(a, sm.add_constant(np.column_stack((b1,b2)))).fit()

# Construct fit curve using dependent variables and estimated coefficients
mlr_prediction = mlr.params[0] + mlr.params[1]*b1 + mlr.params[2]*b2

# Print regression statistics
print 'R-squared:', mlr.rsquared_adj
print 't-statistics of coefficients:\n', mlr.tvalues

# Plot asset and model
a.plot()
mlr_prediction.plot()
plt.legend(['Asset', 'Model']);
plt.ylabel('Price')

# Perform linear regression
slr = regression.linear_model.OLS(a, sm.add_constant(b1)).fit()
slr_prediction = slr.params[0] + slr.params[1]*b1

# Print fit statistics
print 'R-squared:', slr.rsquared_adj
print 't-statistics of coefficients:\n', slr.tvalues

# Plot asset and model
a.plot()
slr_prediction.plot()
plt.ylabel('Price')
plt.legend(['Asset', 'Model']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Violations_of_Regression_Models/notebook.ipynb".
The first task is:
Serial correlation of errors
A common and serious problem is when errors are correlated across observations (known serial correlation or autocorrelation). This can occur, for instance, when some of the data points are related, or when we use time-series data with periodic fluctuations. If one of the independent variables depends on previous values of the dependent variable - such as when it is equal to the value of the dependent variable in the previous period - or if incorrect model specification leads to autocorrelation, then the coefficient estimates will be inconsistent and therefore invalid. Otherwise, the parameter estimates will be valid, but the fit statistics will be off. For instance, if the correlation is positive, we will have inflated F- and t-statistics, leading us to overestimate the significance of the model.
If the errors are homoskedastic, we can test for autocorrelation using the Durbin-Watson test, which is conveniently reported in the regression summary in statsmodels.
Can you write Python code for it?

# Load pricing data for an asset
start = '2014-01-01'
end = '2015-01-01'
y = get_pricing('DAL', fields='price', start_date=start, end_date=end)
x = np.arange(len(y))

# Regress pricing data against time
model = regression.linear_model.OLS(y, sm.add_constant(x)).fit()

# Construct the fit line
prediction = model.params[0] + model.params[1]*x

# Plot pricing data and regression line
plt.plot(x,y)
plt.plot(x, prediction, color='r')
plt.legend(['DAL Price', 'Regression Line'])
plt.xlabel('Time')
plt.ylabel('Price')

# Print summary of regression results
model.summary()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Violations_of_Regression_Models/notebook.ipynb".
The first task is:
Testing for Heteroskedasticity
You can test for heteroskedasticity using a few tests, we'll use the Breush Pagan test from the statsmodels library. We'll also test for normality, which in this case also picks up the weirdness in the second case. HOWEVER, it is possible to have normally distributed residuals which are also heteroskedastic, so both tests must be performed to be sure.
Can you write Python code for it?

residuals1 = y1-fit1
residuals2 = y2-fit2

xs_with_constant = sm.add_constant(xs)

_, jb_pvalue1, _, _ = statsmodels.stats.stattools.jarque_bera(residuals1)
_, jb_pvalue2, _, _ = statsmodels.stats.stattools.jarque_bera(residuals2)
print "p-value for residuals1 being normal", jb_pvalue1
print "p-value for residuals2 being normal", jb_pvalue2

_, pvalue1, _, _ = stats.diagnostic.het_breushpagan(residuals1, xs_with_constant)
_, pvalue2, _, _ = stats.diagnostic.het_breushpagan(residuals2, xs_with_constant)
print "p-value for residuals1 being heteroskedastic", pvalue1
print "p-value for residuals2 being heteroskedastic", pvalue2


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
Before we check how many significant results we got, let's run out some math to check how many we'd expect. The formula for the number of pairs given N series is
$$\frac{N(N-1)}{2}$$
There are no relationships in our data as it's all randomly generated. If our test is properly calibrated we should expect a false positive rate of 5% given our 5% cutoff. Therefore we should expect the following number of pairs that achieved significance based on pure random chance.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
Experiment - Running Many Tests
We'll start by defining a data frame.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
If we add some noise our coefficient will drop.
Can you write Python code for it?

X = pd.Series(np.random.normal(0, 1, 100))
Y = X + np.random.normal(0, 1, 100)

r_s = stats.spearmanr(Y, X)
print 'Spearman Rank Coefficient: ', r_s[0]
print 'p-value: ', r_s[1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
Now let's compare to how many we actually found.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
Now we'll populate it by adding N randomly generated timeseries of length T.
Can you write Python code for it?

N = 20
T = 100

for i in range(N):
    X = np.random.normal(0, 1, T)
    X = pd.Series(X)
    name = 'X%s' % i
    df[name] = X

df.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
Sensitivity / Specificity Tradeoff
As with any adjustment of p-value cutoff, we have a tradeoff. A lower cutoff decreases the rate of false positives, but also decreases the chance we find a real relationship (true positive). So you can't just decrease your cutoff to solve this problem.
https://en.wikipedia.org/wiki/Sensitivity_and_specificity
Reducing Multiple Comparisons Bias
You can't really eliminate multiple comparisons bias, but you can reduce how much it impacts you. To do so we have two options.
Option 1: Run fewer tests.
This is often the best option. Rather than just sweeping around hoping you hit an interesting signal, use your expert knowledge of the system to develop a great hypothesis and test that. This process of exploring the data, coming up with a hypothesis, then gathering more data and testing the hypothesis on the new data is considered the gold standard in statistical and scientific research. It's crucial that the data set on which you develop your hypothesis is not the one on which you test it. Because you found the effect while exploring, the test will likely pass and not really tell you anything. What you want to know is how consistent the effect is. Moving to new data and testing there will not only mean you only run one test, but will be an 'unbiased estimator' of whether your hypothesis is true. We discuss this a lot in other lectures.
Option 2: Adjustment Factors and Bon Ferroni Correction
WARNING: This section gets a little technical. Unless you're comfortable with significance levels, we recommend looking at the code examples first and maybe reading the linked articles before fully diving into the text.
If you must run many tests, try to correct your p-values. This means applying a correction factor to the cutoff you desire to obtain the one actually used when determining whether p-values are significant. The most conservative and common correction factor is Bon Ferroni.
Example: Bon Ferroni Correction
The concept behind Bon Ferroni is quite simple. It just says that if we run...
Can you write Python code for it?

num_experiments = 100

results = np.zeros((num_experiments,))

N = 20

T = 100

desired_level = 0.05

num_tests = N * (N - 1) / 2

new_cutoff = desired_level / num_tests

for i in range(num_experiments):
    # Run a single experiment
    result = do_experiment(20, 100, cutoff=new_cutoff)

    # Count how many pairs
    n = len(result)

    # Add to array
    results[i] = n

np.mean(results)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
We can see that with a lower cutoff we should expect to get fewer false positives. Let's check that with our above experiment.
Can you write Python code for it?

num_experiments = 100

results = np.zeros((num_experiments,))

for i in range(num_experiments):
    # Run a single experiment
    result = do_experiment(20, 100, cutoff=0.01)

    # Count how many pairs
    n = len(result)

    # Add to array
    results[i] = n

np.mean(results)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
We shouldn't expect the numbers to match too closely here on a consistent basis as we've only run one experiment. If we run many of these experiments we should see a convergence to what we'd expect.
Repeating the Experiment
Can you write Python code for it?

def do_experiment(N, T, cutoff=0.05):
    df = pd.DataFrame()

    # Make random data
    for i in range(N):
        X = np.random.normal(0, 1, T)
        X = pd.Series(X)
        name = 'X%s' % i
        df[name] = X

    significant_pairs = []

    # Look for relationships
    for i in range(N):
        for j in range(i+1, N):
            Xi = df.iloc[:, i]
            Xj = df.iloc[:, j]

            results = stats.spearmanr(Xi, Xj)

            pvalue = results[1]

            if pvalue < cutoff:
                significant_pairs.append((i, j))

    return significant_pairs

num_experiments = 100

results = np.zeros((num_experiments,))

for i in range(num_experiments):
    # Run a single experiment
    result = do_experiment(20, 100, cutoff=0.05)

    # Count how many pairs
    n = len(result)

    # Add to array
    results[i] = n


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
We'll now collect a bunch of pvalues. As in any case we'll want to collect quite a number of p-values to start getting a sense of how the underlying distribution looks. If we only collect few, it will be noisy like this:
Can you write Python code for it?

pvalues = get_pvalues_from_experiment(10, 100)
plt.hist(pvalues)
plt.ylabel('Frequency')
plt.title('Observed p-value');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/p-Hacking_and_Multiple_Comparisons_Bias/notebook.ipynb".
The first task is:
p-value Refresher
For more info on p-values see this lecture. What's important to remember is they're used to test a hypothesis given some data. Here we are testing the hypothesis that a relationship exists between two series given the series values.
IMPORTANT: p-values must be treated as binary.
A common mistake is that p-values are treated as more or less significant. This is bad practice as it allows for what's known as p-hacking and will result in more false positives than you expect. Effectively, you will be too likely to convince yourself that relationships exist in your data.
To treat p-values as binary, a cutoff must be set in advance. Then the p-value must be compared with the cutoff and treated as significant/not signficant. Here we'll show this.
The Cutoff is our Significance Level
We can refer to the cutoff as our significance level because a lower cutoff means that results which pass it are significant at a higher level of confidence. So if you have a cutoff of 0.05, then even on random data 5% of tests will pass based on chance. A cutoff of 0.01 reduces this to 1%, which is a more stringent test. We can therefore have more confidence in our results.
Can you write Python code for it?

# Setting a cutoff of 5% means that there is a 5% chance
# of us getting a significant p-value given no relationship
# in our data (false positive).
# NOTE: This is only true if the test's assumptions have been
# satisfied and the test is therefore properly calibrated.
# All tests have different assumptions.
cutoff = 0.05

X = pd.Series(np.random.normal(0, 1, 100))
Y = X + np.random.normal(0, 1, 100)

r_s = stats.spearmanr(Y, X)
print 'Spearman Rank Coefficient: ', r_s[0]
if r_s[1] < cutoff:
    print 'There is significant evidence of a relationship.'
else:
    print 'There is not significant evidence of a relationship.'


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/2_pipeline_lesson8/notebook.ipynb".
The first task is:
Using Sector is equivalent to morningstar.asset_classification.morningstar_sector_code.latest.
Building Filters from Classifiers
Classifiers can also be used to produce filters with methods like isnull, eq, and startswith. The full list of Classifier methods producing Filters can be found here. can be found here.
As an example, if we wanted a filter to select for securities trading on the New York Stock Exchange, we can use the eq method of our exchange classifier.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson3/notebook.ipynb".
The first task is:
Trading activity jumps from one contract to the next, and transitions happen just prior to the delivery of each contract.
As you might imagine, having to explicitly reference a series of transient contracts when trading or simulating futures can make it difficult to work with them. In the next lesson we introduce a solution to this problem in the form of Continuous Futures.

Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson3/notebook.ipynb".
The first task is:
When a list of Futures and a single field are specified, the return type is a pandas DataFrame indexed by date, with assets as columns:
Can you write Python code for it?

cl_fgh_16_volume = history(
    symbols=['CLF16', 'CLG16', 'CLH16'],
    fields='volume',
    frequency='daily',
    start_date='2015-10-21',
    end_date='2015-12-21'
)

cl_fgh_16_volume.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
Continuous Futures & history
We can use history to get pricing and volume data for a particular ContinuousFuture in the same way we do for Futures. Additionally, we can get the reference to its currently active Future contract by using the contract field.
Running the next cell will get pricing data for our CL continuous future and plot it:
Can you write Python code for it?

# Pricing data for CL `ContinuousFuture`.
cl_pricing = history(
    cl,
    fields='price',
    frequency='daily',
    start_date='2015-10-21',
    end_date='2016-06-01'
)

cl_pricing.plot()


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
Part of the job of our continuous future abstraction is to account for these discontinuities, as we will see next by plotting our CL continuous future price against the price history for individual active contracts.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
The price difference between contracts at a given time is not considered to be an increase in value in the future. Instead, it is associated with the carrying cost and the opportunity cost of holding the underlying commodity or asset prior to delivery. This concept is covered more in depth in the Introduction to Futures Contracts lecture from our Lecture Series.
Next, let's look at the price history for active contracts separately. We will notice that this difference in price creates discontinuities when a contract expires and the reference moves to the next contract:
Can you write Python code for it?

# Pricing and contract data for unadjusted CL `ContinuousFuture`.
# Adjustments are covered in the next section.
cl_unadjusted = continuous_future('CL', adjustment=None)

cl_history = history(
    cl_unadjusted,
    fields=['contract', 'price'],
    frequency='daily',
    start_date='2015-10-21',
    end_date='2016-06-01'
)

cl_active_contract_pricing = cl_history.pivot(index=cl_history.index, columns='contract')
cl_active_contract_pricing.plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
The volume for the CL ContinuousFuture is essentially the skyline of the individual contract volumes. As the volume moves from one contract to the next, the continuous future starts pointing to the next contract. Note that there are some points where the volume does not exactly match, most notably in the transition from CLK16 to CLM16 between April and May. This is because the rolls are currently computed daily, using only the previous day's volume to avoid lookahead bias.
Offset
The offset argument allows you to specify whether you want to maintain a reference to the front contract or to a back contract. Setting offset=0 (default) maintains a reference to the front contract, or the contract with the next soonest delivery. Setting offset=1 creates a continuous reference to the contract with the second closest date of delivery, etc.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
To better understand the need for continuous futures, let's use history to get pricing data for the chain of individual contracts we looked at in the previous lesson and plot it.
Can you write Python code for it?

cl_contracts = symbols(['CLF16', 'CLG16', 'CLH16', 'CLJ16', 'CLK16', 'CLM16'])

# Pricing data for our consecutive contracts from earlier.
cl_consecutive_contract_pricing = history(
    cl_contracts,
    fields='price',
    frequency='daily',
    start_date='2015-10-21',
    end_date='2016-06-01'
)

cl_consecutive_contract_pricing.plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson4/notebook.ipynb".
The first task is:
To create a continuous future, we just need to supply a root_symbol to the continuous_future function. The following cell creates a continuous future for Light Sweet Crude Oil.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/tutorials/4_futures_getting_started_lesson5/notebook.ipynb".
The first task is:
We can see in our plot that both prices follow a similar trend, so our strategy probably has some validity based on historical data.
As it turns out, this trend can be explained by a statistical property called Cointegration. Essentially, if two time series are cointegrated, there is an equilibrium relationship between them in the long run, even if they seem to diverge substantially from that equilibrium in the short term. Pairs Trading strategies take advantage of this trend by shorting one future in the pair and going long on the other as the spread increases or decreases, with the expectation that the spread will return to its state of equilibrium.
We can do a quick test for cointegration using the coint function from Python's statsmodels library. If the second value of coint's output (pvalue) is small, this means the input time series are cointegrated.
Can you write Python code for it?

import statsmodels.api as sm
import statsmodels.tsa as tsa

tsa.stattools.coint(cl_price, xb_price)[1]


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
Default Inputs
When writing a custom factor, we can set default inputs and window_length in our CustomFactor subclass. For example, let's define the TenDayMeanDifference custom factor to compute the mean difference between two data columns over a trailing window using numpy.nanmean. Let's set the default inputs to [USEquityPricing.close, USEquityPricing.open] and the default window_length to 10:
Can you write Python code for it?

class TenDayMeanDifference(CustomFactor):
    # Default inputs.
    inputs = [USEquityPricing.close, USEquityPricing.open]
    window_length = 10
    def compute(self, today, asset_ids, out, close, open):
        # Calculates the column-wise mean difference, ignoring NaNs
        out[:] = numpy.nanmean(close - open, axis=0)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
Further Example
Let's take another example where we build a momentum custom factor and use it to create a filter. We will then use that filter as a screen for our pipeline.
Let's start by defining a Momentum factor to be the division of the most recent close price by the close price from n days ago where n is the window_length.
Can you write Python code for it?

class Momentum(CustomFactor):
    # Default inputs
    inputs = [USEquityPricing.close]

    # Compute momentum
    def compute(self, today, assets, out, close):
        out[:] = close[-1] / close[0]


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
Next, let's add our momentum factors and our positive_momentum filter to make_pipeline. Let's also pass positive_momentum as a screen to our pipeline.
Can you write Python code for it?

def make_pipeline():

    ten_day_momentum = Momentum(window_length=10)
    twenty_day_momentum = Momentum(window_length=20)

    positive_momentum = ((ten_day_momentum > 1) & (twenty_day_momentum > 1))

    std_dev = StdDev(inputs=[USEquityPricing.close], window_length=5)

    return Pipeline(
        columns={
            'std_dev': std_dev,
            'ten_day_momentum': ten_day_momentum,
            'twenty_day_momentum': twenty_day_momentum
        },
        screen=positive_momentum
    )


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
Now, let's instantiate our Momentum factor (twice) to create a 10-day momentum factor and a 20-day momentum factor. Let's also create a positive_momentum filter returning True for securities with both a positive 10-day momentum and a positive 20-day momentum.
Can you write Python code for it?

ten_day_momentum = Momentum(window_length=10)
twenty_day_momentum = Momentum(window_length=20)

positive_momentum = ((ten_day_momentum > 1) & (twenty_day_momentum > 1))


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
Running this pipeline outputs the standard deviation and each of our momentum computations for securities with positive 10-day and 20-day momentum.
Can you write Python code for it?

result = run_pipeline(make_pipeline(), '2015-05-05', '2015-05-05')
result


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
The defaults can be manually overridden by specifying arguments in the constructor call.
Can you write Python code for it?

# Computes the 10-day mean difference between the daily high and low prices.
high_low_diff = TenDayMeanDifference(inputs=[USEquityPricing.high, USEquityPricing.low])


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_10.ipynb".
The first task is:
When this pipeline is run, StdDev.compute() will be called every day with data as follows:
- values: An M x N numpy array, where M is 20 (window_length), and N is ~8000 (the number of securities in our database on the day in question).
- out: An empty array of length N (~8000). In this example, the job of compute is to populate out with an array storing of 5-day close price standard deviations.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Built-in Base Universe
We have just defined our own base universe to select 'tradeable' securities with high dollar volume. However, Quantopian has two built-in filters that do something similar. The Q500US and the Q1500US are built-in pipeline filters that select a group of 500 or 1500 tradeable, liquid stocks each day. Constituents of these groups are chosen at the start of each calendar month by selecting the top 'tradeable' stocks by 200-day average dollar volume, capped at 30% of equities allocated to any single sector (more detail on the selection criteria of these filters can be found here).
To simplify our pipeline, let's replace what we've already written for our base_universe with the Q1500US built-in filter. First, we need to import it.
Can you write Python code for it?

from quantopian.pipeline.filters.morningstar import Q1500US


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Let's then combine shorts and longs to create a new filter that we can use as the screen of our pipeline:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Next, let's create filters for the top 25 and bottom 25 equities by percent_difference.
Can you write Python code for it?

# Create a filter to select securities to short.
shorts = percent_difference.top(25)

# Create a filter to select securities to long.
longs = percent_difference.bottom(25)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Note that when defining our filters, we used several Classifier methods that we haven't yet seen including notnull, startswith, endswith, and matches. Documentation on these methods is available here.
Next, let's create a filter for the top 30% of tradeable stocks by 20-day average dollar volume. We'll call this our base_universe.
Can you write Python code for it?

base_universe = AverageDollarVolume(window_length=20, mask=tradeable_stocks).percentile_between(70, 100)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Now that we have a filter base_universe that we can use to select a subset of securities, let's focus on creating factors for this subset. For this example, let's create a pipeline for a mean reversion strategy. In this strategy, we'll look at the 10-day and 30-day moving averages (close price). Let's plan to open equally weighted long positions in the 25 securities with the least (most negative) percent difference and equally weighted short positions in the 25 with the greatest percent difference. To do this, let's create two moving average factors using our base_universe filter as a mask. Then let's combine them into a factor computing the percent difference.
Can you write Python code for it?

# 10-day close price average.
mean_10 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=10, mask=base_universe)

# 30-day close price average.
mean_30 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=30, mask=base_universe)

percent_difference = (mean_10 - mean_30) / mean_30


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Now we can define our filters:
Can you write Python code for it?

# Filter for primary share equities. IsPrimaryShare is a built-in filter.
primary_share = IsPrimaryShare()

# Equities listed as common stock (as opposed to, say, preferred stock).
# 'ST00000001' indicates common stock.
common_stock = morningstar.share_class_reference.security_type.latest.eq('ST00000001')

# Non-depositary receipts. Recall that the ~ operator inverts filters,
# turning Trues into Falses and vice versa
not_depositary = ~morningstar.share_class_reference.is_depositary_receipt.latest

# Equities not trading over-the-counter.
not_otc = ~morningstar.share_class_reference.exchange_id.latest.startswith('OTC')

# Not when-issued equities.
not_wi = ~morningstar.share_class_reference.symbol.latest.endswith('.WI')

# Equities without LP in their name, .matches does a match using a regular
# expression
not_lp_name = ~morningstar.company_reference.standard_name.latest.matches('.* L[. ]?P.?$')

# Equities with a null value in the limited_partnership Morningstar
# fundamental field.
not_lp_balance_sheet = morningstar.balance_sheet.limited_partnership.latest.isnull()

# Equities whose most recent Morningstar market cap is not null have
# fundamental data and therefore are not ETFs.
have_market_cap = morningstar.valuation.market_cap.latest.notnull()

# Filter for stocks that pass all of our previous filters.
tradeable_stocks = (
    primary_share
    & common_stock
    & not_depositary
    & not_otc
    & not_wi
    & not_lp_name
    & not_lp_balance_sheet
    & have_market_cap
)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Running this pipeline will result in a DataFrame with 50 rows and 2 columns each day. Each day, the columns will contain boolean values that we can use to decide whether we want to open a long or short position in each security.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Since our earlier filters were used as masks as we built up to this final filter, when we use securities_to_trade as a screen, the output securities will meet the criteria outlined at the beginning of the lesson (primary shares, non-ETFs, etc.). They will also have high dollar volume.
Finally, let's instantiate our pipeline. Since we are planning on opening equally weighted long and short positions later, the only information that we actually need from our pipeline is which securities we want to trade (the pipeline index) and whether or not to open a long or a short position. Let's add our longs and shorts filters to our pipeline and set our screen to be securities_to_trade.
Can you write Python code for it?

def make_pipeline():

    # Base universe filter.
    base_universe = Q1500US()

    # 10-day close price average.
    mean_10 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=10, mask=base_universe)

    # 30-day close price average.
    mean_30 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=30, mask=base_universe)

    # Percent difference factor.
    percent_difference = (mean_10 - mean_30) / mean_30

    # Create a filter to select securities to short.
    shorts = percent_difference.top(25)

    # Create a filter to select securities to long.
    longs = percent_difference.bottom(25)

    # Filter for the securities that we want to trade.
    securities_to_trade = (shorts | longs)

    return Pipeline(
        columns={
            'longs': longs,
            'shorts': shorts
        },
        screen=securities_to_trade
    )


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_11.ipynb".
The first task is:
Then, let's set our base_universe to the Q1500US.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_3.ipynb".
The first task is:
And now, when we make and run our pipeline again, there are two columns in our output dataframe. One column has the 10-day mean close price of each security, and the other has the latest close price.
Can you write Python code for it?

result = run_pipeline(make_pipeline(), '2015-05-05', '2015-05-05')
result.head(5)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_3.ipynb".
The first task is:
Creating a Factor
Let's go back to our make_pipeline function from the previous lesson and instantiate a SimpleMovingAverage factor. To create a SimpleMovingAverage factor, we can call the SimpleMovingAverage constructor with two arguments: inputs, which must be a list of BoundColumn objects, and window_length, which must be an integer indicating how many days worth of data our moving average calculation should receive. (We'll discuss BoundColumn in more depth later; for now we just need to know that a BoundColumn is an object indicating what kind of data should be passed to our Factor.).
The following line creates a Factor for computing the 10-day mean close price of securities.
Can you write Python code for it?

mean_close_10 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=10)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_3.ipynb".
The first task is:
Note: factors can also be added to an existing Pipeline instance using the Pipeline.add method. Using add looks something like this:
    >>> my_pipe = Pipeline()
    >>> f1 = SomeFactor(...)
    >>> my_pipe.add(f1)
Latest
The most commonly used built-in Factor is Latest. The Latest factor gets the most recent value of a given data column. This factor is common enough that it is instantiated differently from other factors. The best way to get the latest value of a data column is by getting its .latest attribute. As an example, let's update make_pipeline to create a latest close price factor and add it to our pipeline:
Can you write Python code for it?

def make_pipeline():

    mean_close_10 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=10)
    latest_close = USEquityPricing.close.latest

    return Pipeline(
        columns={
            '10_day_mean_close': mean_close_10,
            'latest_close_price': latest_close
        }
    )


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_3.ipynb".
The first task is:
Now we have a column in our pipeline output with the 10-day average close price for all 8000+ securities (display truncated). Note that each row corresponds to the result of our computation for a given security on a given date stored. The DataFrame has a MultiIndex where the first level is a datetime representing the date of the computation and the second level is an Equity object corresponding to the security. For example, the first row (2015-05-05 00:00:00+00:00, Equity(2 [AA])) will contain the result of our mean_close_10 factor for AA on May 5th, 2015.
If we run our pipeline over more than one day, the output looks like this.
Can you write Python code for it?

result = run_pipeline(make_pipeline(), '2015-05-05', '2015-05-07')
result


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_7.ipynb".
The first task is:
Applying the mask to SimpleMovingAverage restricts the average close price factors to a computation over the ~2000 securities passing the high_dollar_volume filter, as opposed to ~8000 without a mask. When we combine mean_close_10 and mean_close_30 to form percent_difference, the computation is performed on the same ~2000 securities.
Masking Filters
Masks can be also be applied to methods that return filters like top, bottom, and percentile_between.
Masks are most useful when we want to apply a filter in the earlier steps of a combined computation. For example, suppose we want to get the 50 securities with the highest open price that are also in the top 10% of dollar volume. Suppose that we then want the 90th-100th percentile of these securities by close price. We can do this with the following:
Can you write Python code for it?

# Dollar volume factor
dollar_volume = AverageDollarVolume(window_length=30)

# High dollar volume filter
high_dollar_volume = dollar_volume.percentile_between(90,100)

# Top open price filter (high dollar volume securities)
top_open_price = USEquityPricing.open.latest.top(50, mask=high_dollar_volume)

# Top percentile close price filter (high dollar volume, top 50 open price)
high_close_price = USEquityPricing.close.latest.percentile_between(90, 100, mask=top_open_price)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_7.ipynb".
The first task is:
Masking
Sometimes we want to ignore certain assets when computing pipeline expresssions. There are two common cases where ignoring assets is useful:
1. We want to compute an expression that's computationally expensive, and we know we only care about results for certain assets. A common example of such an expensive expression is a Factor computing the coefficients of a regression (RollingLinearRegressionOfReturns).
2. We want to compute an expression that performs comparisons between assets, but we only want those comparisons to be performed against a subset of all assets. For example, we might want to use the Factor method top to compute the top 200 assets by earnings yield, ignoring assets that don't meet some liquidity constraint.
To support these two use-cases, all Factors and many Factor methods can accept a mask argument, which must be a Filter indicating which assets to consider when computing.
Masking Factors
Let's say we want our pipeline to output securities with a high or low percent difference but we also only want to consider securities with a dollar volume above $10,000,000. To do this, let's rearrange our make_pipeline function so that we first create the high_dollar_volume filter. We can then use this filter as a mask for moving average factors by passing high_dollar_volume as the mask argument to SimpleMovingAverage.
Can you write Python code for it?

# Dollar volume factor
dollar_volume = AverageDollarVolume(window_length=30)

# High dollar volume filter
high_dollar_volume = (dollar_volume > 10000000)

# Average close price factors
mean_close_10 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=10, mask=high_dollar_volume)
mean_close_30 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=30, mask=high_dollar_volume)

# Relative difference factor
percent_difference = (mean_close_10 - mean_close_30) / mean_close_30


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_7.ipynb".
The first task is:
Running this pipeline outputs 5 securities on May 5th, 2015.
Can you write Python code for it?

result = run_pipeline(make_pipeline(), '2015-05-05', '2015-05-05')
print 'Number of securities that passed the filter: %d' % len(result)


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_8.ipynb".
The first task is:
Classifiers
A classifier is a function from an asset and a moment in time to a categorical output such as a string or integer label:
F(asset, timestamp) -> category
An example of a classifier producing a string output is the exchange ID of a security. To create this classifier, we'll have to import morningstar.share_class_reference.exchange_id and use the latest attribute to instantiate our classifier:
Can you write Python code for it?

from quantopian.pipeline.data import morningstar

# Since the underlying data of morningstar.share_class_reference.exchange_id
# is of type string, .latest returns a Classifier
exchange = morningstar.share_class_reference.exchange_id.latest


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_8.ipynb".
The first task is:
Previously, we saw that the latest attribute produced an instance of a Factor. In this case, since the underlying data is of type string, latest produces a Classifier.
Similarly, a computation producing the latest Morningstar sector code of a security is a Classifier. In this case, the underlying type is an int, but the integer doesn't represent a numerical value (it's a category) so it produces a classifier. To get the latest sector code, we can use the built-in Sector classifier.
Can you write Python code for it?

from quantopian.pipeline.classifiers.morningstar import Sector
morningstar_sector = Sector()


I am working on the file "Notebooks/quantopian_research_public/tutorials/pipeline/pipeline_tutorial_lesson_8.ipynb".
The first task is:
This filter will return True for securities having 'NYS' as their most recent exchange_id.
Quantiles
Classifiers can also be produced from various Factor methods. The most general of these is the quantiles method which accepts a bin count as an argument. The quantiles method assigns a label from 0 to (bins - 1) to every non-NaN data point in the factor output and returns a Classifier with these labels. NaNs are labeled with -1. Aliases are available for quartiles (quantiles(4)), quintiles (quantiles(5)), and deciles (quantiles(10)). As an example, this is what a filter for the top decile of a factor might look like:
Can you write Python code for it?

dollar_volume_decile = AverageDollarVolume(window_length=10).deciles()
top_decile = (dollar_volume_decile.eq(9))


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
Enrichment
Enrichment values were calculated with g:Profiler (http://biit.cs.ut.ee/gprofiler/index.cgi)
Now we get the gProfiler output for each dataset, and see how the p-values change with trimming.
Can you write Python code for it?


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
Filtering parameters
Now we define 5 different datasets with different amounts of quality filter, ranged from least to most stringent
Dataset 1 - No filters
- All proteins included

Dataset 2 - Just entropy
- Entropy: < 1
- HGT: No filter
- NumDBsContributing: No Filter
- Bimodality: No Filter

Dataset 3 - Just HGT flags
- Entropy: No Filter
- HGT: Remove flagged proteins
- NumDBsContributing: No Filter
- Bimodality: No Filter

Dataset 4 - Just number of algorithms contributing to final age
- Entropy: No Filter
- HGT: No Filter
- NumDBsContributing: > 3
- Bimodality: No Filter

Dataset 5 - Just bimodality metric
- Entropy: No Filter
- HGT: No Filter
- NumDBsContributing: No Filter
- Bimodality: < 5

Dataset 6 - Filter on entropy, HGT flag, Num. algorithms, and bimodality
- Entropy: < 1
- HGT: Remove flagged proteins
- NumDBsContributing: > 3
- Bimodality: < 5
Can you write Python code for it?

d1_archs = [prot for prot in con[con["modeAge"] == "Euk_Archaea"].index]
d1_bacs = [prot for prot in con[con["modeAge"] == "Euk+Bac"].index]

d2_archs = [prot for prot in con[(con["modeAge"] == "Euk_Archaea")
                                 & (con["entropy"]<1)].index]
d2_bacs = [prot for prot in con[(con["modeAge"] == "Euk+Bac")
                                 & (con["entropy"]<1)].index]

d3_archs = [prot for prot in con[(con["modeAge"] == "Euk_Archaea") &
                                 (con["HGT_flag"] == False)].index]
d3_bacs = [prot for prot in con[(con["modeAge"] == "Euk+Bac") &
                                (con["HGT_flag"] == False)].index]

d4_archs = [prot for prot in con[(con["modeAge"] == "Euk_Archaea") &
                                 (con["NumDBsContributing"] > 3)].index]
d4_bacs = [prot for prot in con[(con["modeAge"] == "Euk+Bac") &
                                (con["NumDBsContributing"] > 3)].index]

d5_archs = [prot for prot in con[(con["modeAge"] == "Euk_Archaea") &
                                 (con["Bimodality"] < 5)].index]
d5_bacs = [prot for prot in con[(con["modeAge"] == "Euk+Bac") &
                                 (con["Bimodality"] < 5)].index]

d6_archs = [prot for prot in con[(con["modeAge"] == "Euk_Archaea") &
                                 (con["entropy"]<1) &
                                 (con["HGT_flag"] == False) &
                                 (con["Bimodality"] < 5) &
                                 (con["NumDBsContributing"] > 3)].index]
d6_bacs = [prot for prot in con[(con["modeAge"] == "Euk+Bac") &
                                (con["entropy"]<1) &
                                (con["HGT_flag"] == False) &
                                (con["Bimodality"] < 5) &
                                (con["NumDBsContributing"] > 3)].index]

## Write out the data files for submission to g:Profiler

archs = [d1_archs,d2_archs,d3_archs,d4_archs,d5_archs,d6_archs]
bacs = [d1_bacs,d2_bacs,d3_bacs,d4_bacs,d5_bacs,d6_bacs]

for index,prots in...


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
First we'll take a look at the data and the distributions of different quality measures
Can you write Python code for it?

con = pd.read_csv("../main_HUMAN.csv",index_col=0)
con.head()


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
Histogram of bimodality values
Can you write Python code for it?


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
Histogram of the number of algorithms contributing to each gene's age call
Can you write Python code for it?


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
P-value series
This is what the output looks like. Note that if the column "diff" is negative, it means that the heavy filtering on dataset 5 gave a stronger p-value.
Can you write Python code for it?

archs_pvalues_BP = pValue_series(dsets)
archs_pvalues_BP.head(n=10)

bacs_pvalues_BP = pValue_series(dsets,'bacs')

archs_pvalues_CC = pValue_series(dsets,'archs',"CC")
bacs_pvalues_CC = pValue_series(dsets,'bacs',"CC")

#Write out p-value series

archs_pvalues_BP.to_csv("archs_p-values_BP.csv")
bacs_pvalues_BP.to_csv("bacs_p-values_BP.csv")
archs_pvalues_CC.to_csv("archs_p-values_CC.csv")
bacs_pvalues_CC.to_csv("bacs_p-values_CC.csv")


I am working on the file "Notebooks/robustness_of_geneEnrichment.ipynb".
The first task is:
Store the datasets in a dictionary
Can you write Python code for it?

dsets = dict(zip(
    ["_".join(i.split("_")[:2]) for i in infiles],
    [pd.read_table(i) for i in infiles]))

dsets['d1_archs'].head()

for df in dsets:
    dsets[df].sort(["t type","p-value"],inplace=True)

## Write data files once they've been sorted

for df in dsets:
    dsets[df].to_csv(df+"_sorted.csv")


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Neutral evolution in a larger population of 200 individuals
Can you write Python code for it?


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Neutral evolution in a small population of 20 individuals
Can you write Python code for it?


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Neutral evolution in an even larger population of size 2000
Can you write Python code for it?


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Selection in a population of 200 individuals
Can you write Python code for it?

plotFreqs(simulateSelection (200, 50, 10))


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Selection in a population of 2000 individuals
Can you write Python code for it?

plotFreqs(simulateSelection (2000, 50, 10))


I am working on the file "Notebooks/simulateAllelicEvolution.ipynb".
The first task is:
Simulation of evolution under natural selection in poplations of different sizes
Same idea as above, except that the "True" allele now has a selective advantage of 5%.
Can you write Python code for it?

def generationSelection (genolist):
    newGenos = list()
    length = len(genolist)
    weights = list()
    for i in range(length):
        if genolist[i]:
            weights.append(1.05)
        else:
            weights.append(1.0)
    probas = [weights[i]/sum(weights) for i in range(len(weights))]
    newGenos = np.random.choice(genolist, length, p=probas)
    return(newGenos)

#############################
def simulateSelection (numIndividuals, numGenerations, numPopulations):
    # Two alleles: true and false
    init = [True]*int(numIndividuals/2) + [False] * int(numIndividuals/2)
    allFreqs = list()
    for pop in range(numPopulations):
        freqs = list()
        freqs.append(sum(init)/ numIndividuals)
        current = init
        for gen in range(numGenerations):
            new = generationSelection(current)
            freqs.append(sum(new) / numIndividuals)
            current = new
        allFreqs.append(freqs)
    return allFreqs


I am working on the file "Notebooks/switchiness_screen.ipynb".
The first task is:
Greedy filter based on effect size
Can you write Python code for it?

f2 = fp.ix[ti(rr.q < .0001)]
ff_u = filter_pathway_hits(fp.ix[ti(f2>0)].order()[::-1], gs2)
ff_p = filter_pathway_hits(fp.ix[ti(f2<0)].order(), gs2)
ff = ff_u.append(ff_p)

selected = rr.ix[ff.index].join(f2)
selected.sort('p')


I am working on the file "Notebooks/switchiness_screen.ipynb".
The first task is:
Greedy filter based on p-value
Can you write Python code for it?

f2 = fp.ix[ti(rr.q < .0001)]
ff_u = filter_pathway_hits(rr.ix[ti(f2>0)].p.order(), gs2)
ff_p = filter_pathway_hits(rr.ix[ti(f2<0)].p.order(), gs2)
ff = ff_u.append(ff_p)

selected = rr.ix[ff[ff < .00001].index].join(fp)
selected.sort('p')