I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Factor_Risk_Exposure/notebook.ipynb".
The first task is:
How did each factor do over 2014?
Can you write Python code for it?

SMB_CUM = np.cumprod(SMB+1)
HML_CUM = np.cumprod(HML+1)

plt.plot(SMB_CUM.index, SMB_CUM.values)
plt.plot(HML_CUM.index, HML_CUM.values)
plt.ylabel('Cumulative Return')
plt.legend(['SMB Portfolio Returns', 'HML Portfolio Returns']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Factor_Risk_Exposure/notebook.ipynb".
The first task is:
Now we'll look at FMCAR as it changes over time.
Can you write Python code for it?

# Remove the first 99, which are all NaN for each case

# Compute covariances
covariances = pd.rolling_cov(df[['F1', 'F2']], window=100)[99:]
# Compute active risk squared
active_risk_squared = pd.rolling_std(active, window = 100)[99:]**2
# Compute betas
betas = rolling_parameter_estimates[['F1', 'F2']]

# Set up empty dataframe
FMCAR = pd.DataFrame(index=betas.index, columns=betas.columns)

# For each factor
for factor in betas.columns:
    # For each bar in our data
    for t in betas.index:
        # Compute the sum of the betas and covariances
        s = np.sum(betas.loc[t] * covariances[t][factor])
        # Get the beta
        b = betas.loc[t][factor]
        # Get active risk squared
        AR = active_risk_squared.loc[t]
        # Put them all together to estimate FMCAR on that date
        FMCAR[factor][t] = b * s / AR


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Factor_Risk_Exposure/notebook.ipynb".
The first task is:
Problems with using this data
Whereas it may be interesting to know how a portfolio was exposed to certain factors historically, it is really only useful if we can make predictions about how it will be exposed to risk factors in the future. It's not always a safe assumption to say that future exposure will be the current exposure. As you saw the exposure varies quite a bit, so taking the average is dangerous. We could put confidence intervals around that average, but that would only work if the distribution of exposures were normal or well behaved. Let's check using our old buddy, the Jarque-Bera test.
Can you write Python code for it?

from statsmodels.stats.stattools import jarque_bera
_, pvalue1, _, _ = jarque_bera(FMCAR['F1'].dropna().values)
_, pvalue2, _, _ = jarque_bera(FMCAR['F2'].dropna().values)

print 'p-value F1_FMCAR is normally distributed', pvalue1
print 'p-value F2_FMCAR is normally distributed', pvalue2


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Factor_Risk_Exposure/notebook.ipynb".
The first task is:
The rest of the risk can be attributed to active specific risk, i.e. factors that we did not take into account or the asset's idiosyncratic risk.
However, as usual we will look at how the exposure to these factors changes over time. As we lose a tremendous amount of information by just looking at one data point. Let's look at what happens if we run a rolling regression over time.
Can you write Python code for it?

# Compute the rolling betas
model = pd.stats.ols.MovingOLS(y = df['R'], x=df[['F1', 'F2']],
                             window_type='rolling',
                             window=100)
rolling_parameter_estimates = model.beta
rolling_parameter_estimates.plot();
plt.title('Computed Betas');
plt.legend(['F1 Beta', 'F2 Beta', 'Intercept']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Factor_Risk_Exposure/notebook.ipynb".
The first task is:
Using the formula from the start of the notebook, we can compute the factors' marginal contributions to active risk squared:
Can you write Python code for it?

F1 = df['F1']
F2 = df['F2']
cov = np.cov(F1, F2)
ar_squared = (active.std())**2
fmcar1 = (b1*(b2*cov[0,1] + b1*cov[0,0]))/ar_squared
fmcar2 = (b2*(b1*cov[0,1] + b2*cov[1,1]))/ar_squared
print 'SMB Risk Contribution:', fmcar1
print 'HML Risk Contribution:', fmcar2


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Actually Running the Regression
Now that we have returns series representing our factors, we can compute the factor model for any return stream using a linear regression. Below, we compute the factor sensitivities for returns on a tech portfolio.
Can you write Python code for it?

# Get returns data for our portfolio
portfolio = get_pricing(['MSFT', 'AAPL', 'YHOO', 'FB', 'TSLA'],
                        fields='price', start_date=start_date, end_date=end_date).pct_change()[1:]
R = np.mean(portfolio, axis=1)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Finally, let's add another factor so you can see how the code changes.
Can you write Python code for it?

MKT = results['market_cap']['2011-1-3']
scipy.stats.mstats.winsorize(MKT, inplace=True, limits=0.01)
MKT_z = (MKT - np.mean(MKT)) / np.std(MKT)

constant = pd.TimeSeries(np.ones(len(R_day.index)), index=R_day.index)

df_day = pd.DataFrame({'R': R_day,
              'BTP_z': BTP_z,
              'MKT_z': MKT_z,
              'Constant': constant})
df_day = df_day.dropna()

# Perform linear regression to get the coefficients in the model
F1, F2 = regression.linear_model.OLS(df_day['R'], df_day[['BTP_z', 'MKT_z']]).fit().params
print F1, F2


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Great, we have all the data. Now we need to compute the returns of our portfolios over time. We have the daily returns for each equity, plus whether or not that equity was included in any given portfolio on any given day. We can combine that information in the following way to yield daily portfolio returns.
Step 1: Subset our results into only data belonging to our 'biggest' portfolio.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Now let's actually run it and check out our results.
Can you write Python code for it?

# This takes a few minutes.
results = run_pipeline(pipe, start_date, end_date)
results


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Now run through this computation for each portfolio and get our final results.
Can you write Python code for it?

R_biggest = results[results.biggest]['returns'].groupby(level=0).mean()
R_smallest = results[results.smallest]['returns'].groupby(level=0).mean()

R_highpb = results[results.highpb]['returns'].groupby(level=0).mean()
R_lowpb = results[results.lowpb]['returns'].groupby(level=0).mean()

SMB = R_smallest - R_biggest
HML = R_highpb - R_lowpb


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Now set up our data and estimate $F_j$ using linear regression.
Can you write Python code for it?

constant = pd.TimeSeries(np.ones(len(R_day.index)), index=R_day.index)

df_day = pd.DataFrame({'R': R_day,
              'BTP_z': BTP_z,
              'Constant': constant})
df_day = df_day.dropna()

# Perform linear regression to get the coefficients in the model
F1 = regression.linear_model.OLS(df_day['R'], df_day['BTP_z']).fit().params
print F1


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Now we initialize the pipeline.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Perform the regression. You'll notice that these are the sensitivities over an entire year. It can be valuable to look at the rolling sensitivities as well to determine how stable they are.
Can you write Python code for it?

# Perform linear regression to get the coefficients in the model
b1, b2, b3 = regression.linear_model.OLS(df['R'], df[['M', 'SMB', 'HML']]).fit().params

# Print the coefficients from the linear regression
print 'Historical Sensitivities of portfolio returns to factors:\nMarket: %f\nMarket cap: %f\nB/P: %f' %  (b1, b2, b3)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Problem: The Data is Weirdly Distributed
Notice how there are big outliers in the dataset that cause the z-scores to lose a lot of information. Basically the presence of some huge book to price datapoints causes the rest of the data to seem to occupy a relatively small area. We need to get around this issue using some data cleaning technique, here we're use winsorization.
Winsorization
Winzorization takes the top $n\%$ of a dataset and sets it all equal to the least extreme value in the top $n\%$. For example, if your dataset ranged from 0-10, plus a few crazy outliers, those outliers would be set to 0 or 10 depending on their direction. Here is an example.
Can you write Python code for it?

# Get some random data
X = np.random.normal(0, 1, 100)

# Put in some outliers
X[0] = 1000
X[1] = -1000

# Perform winsorization
print 'Before winsorization', np.min(X), np.max(X)
scipy.stats.mstats.winsorize(X, inplace=True, limits=0.01)
print 'After winsorization', np.min(X), np.max(X)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Set the date range for which we want data.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
Step 3: Group by day and take the mean. This is pretty deep into pandas logic, so if you don't understand this on first pass it is recommended to check out pandas' documentation on all the functions used. Especially groupby, which is very useful. Keep in mind that the index in our results is a MultiIndex rather than a regular Index, that can complicate things.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
The last data we need are the daily returns on the broad market.
Can you write Python code for it?

M = get_pricing('SPY', start_date='2011-1-1', end_date='2012-1-1', fields='price').pct_change()[1:]

plt.plot(M.index, M.values)
plt.ylabel('Daily Percent Return')
plt.legend(['Market Portfolio Returns']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
This function will allow us to run the pipeline.
Can you write Python code for it?

from quantopian.research import run_pipeline


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
We can visualize the dependency graph of our data computations here.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Fundamental_Factor_Models/notebook.ipynb".
The first task is:
We need the returns for that day as well.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Futures_Trading_Considerations/notebook.ipynb".
The first task is:
This is one of the most liquid futures contracts and we see this significant increase in volume traded during normal equity trading hours. These hours can be even more tight for less liquid commodities. For example, let's look at how Feeder Cattle trades during the same time period on the same day.
Can you write Python code for it?

contract = 'FCH17'
one_day_volume = get_pricing(contract, start_date='2017-02-01', end_date='2017-02-01', frequency='minute', fields='volume')

one_day_volume.tz_convert('EST').plot()
plt.title('Trading Volume for 3/01/2017 by Minute')
plt.xlabel('Minute')
plt.ylabel('Volume');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Hypothesis_Testing/notebook.ipynb".
The first task is:
Because the $p$-value is greater than our significance level, $\alpha = 0.05$, we fail to reject the null hypothesis.
After we make the statistical decision, we have to translate it into real life. Oftentimes this may be difficult to act upon directly, but the results can have other implications. In the case of our example, we have found that the daily returns of Microsoft in 2015 were not significantly different from $0$.
Hypothesis Testing on Means
A $z$-distribution, or a standard normal distribution, is an essential probability distribution in finance. We like it when things are normally distributed because it entails many useful properties. On top of this, many fundamental methods require an assumption of normality in order to proceed. However, in most cases a $z$-distribution will be inappropriate for our data. We rarely know the true parameter values (mean and variance) of our data and must rely upon approximations. In these cases, we should use the $t$-distribution, and approximation of the normal distribution. The $t$-distribution is more forgiving when it comes to small sample sizes and is meant to be used with sample mean and variance. It has fatter tails and a lower peak, giving more flexibility compared to a normal distribution.
Both the $t$ and $z$-distributions rely upon an underlying assumption of normality, which is typically the case when we are analyzing financial data. As such, in addition to testing individual means, it makes sense to use them to compare between two or more mean values. We can use a hypothesis test to determine whether the means of several data-sets are statistically different from one another. Here, we will use a $t$-distribution to demonstrate. We will compare the mean returns of the S&P500 and Apple stock with a hypothesis test to see if the differences are statistically significant or not.
Can you write Python code for it?

symbol_list = ["SPY", "AAPL"]
start = '2015-01-01'
end = '2016-01-01'
pricing_sample = get_pricing(symbol_list, start_date = start, end_date = end, fields='price')
pricing_sample.columns = map(lambda x: x.symbol, pricing_sample.columns)
returns_sample = pricing_sample.pct_change()[1:]
returns_sample.plot()
plt.ylabel('Returns');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Hypothesis_Testing/notebook.ipynb".
The first task is:
Here are the returns.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Hypothesis_Testing/notebook.ipynb".
The first task is:
In order to make the statistical decision for the test, we compare our test statistic to our critical value. Our test statistic as stated above is between the two critical values for a 95% two-tailed $z$-test so in this example we fail to reject our $H_0$, our hypothesis that MSFT returns are not $0$.
If we chose instead to determine the result of this hypothesis test with a $p$-value, we would calculate the $p$-value in the following way:
Can you write Python code for it?

from scipy.stats import t

p_val = 2 * (1 - t.cdf(test_statistic, n - 1))
print 'P-value is: ', p_val


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Hypothesis_Testing/notebook.ipynb".
The first task is:
Let's get some data.
Can you write Python code for it?

start = "2015-01-01"
end = "2016-01-01"
pricing_sample = get_pricing("MSFT", start_date = start, end_date = end, fields = 'price')

#transform it into returns
returns_sample = pricing_sample.pct_change()[1:]

# plot it
plt.plot(pricing_sample.index, pricing_sample.values)
plt.ylabel('Price');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Hypothesis_Testing/notebook.ipynb".
The first task is:
While these returns look to have the same mean, we still don't have enough evidence to say for sure. We use a hypothesis test to ground our suspicions in a statistical basis.
When comparing two means, our hypothesis tests are stated as the following:

$H_0: \mu_1 - \mu_2 = \theta_0, \ H_A: \mu_1 - \mu_2 \neq \theta_0$
$H_0: \mu_1 - \mu_2 \leq \theta_0, \ H_A: \mu_1 - \mu_2 > \theta_0$
$H_0: \mu_1 - \mu_2 \geq \theta_0, \ H_A: \mu_1 - \mu_2 < \theta_0$

Where $\mu_1, \mu_2$ are the respective means of SPY and AAPL and $\theta_0$ is the parameter we are testing. We will use the first hypothesis test to test the equality of the two returns. If we assume that the population variances are equal, our test statistic is calculated as:
$$ t = \frac{(\bar{X}_1 - \bar{X}_2) - (\mu_1 - \mu_2)}{(\frac{s_p^2}{n_1} + \frac{s_p^2}{n_2})^{1/2}} = \frac{\bar{X}_1 - \bar{X}_2}{(\frac{s_p^2}{n_1} + \frac{s_p^2}{n_2})^{1/2}}$$
With $s_p^2 = \frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2)}{n_1 + n_2 - 2}$ as the estimator of the common variance, known as the pooled variance, and $n_1 + n_2 - 2$ as the number of degrees of freedom ($n_1 - 1$ and $n_2 - 1$ for each dataset). A typical $t$-test on a mean assumes that all variances involved are equal with underlying normal distributions. If we are assuming that the variances are not equal, we have to calculate our test statistic differently. Our test statistic in this case is:
$$ t = \frac{(\bar{X}_1 - \bar{X}_2) - (\mu_1 - \mu_2)}{(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2})^{1/2}} = \frac{\bar{X}_1 - \bar{X}_2}{(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2})^{1/2}}$$
Where the number of degrees of freedom used to find the critical statistic is the modified degrees of freedom, the number of values that are free to vary, $df = \frac{(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2})^2}{\frac{(s_1^2/n_1)^2}{n_1} + \frac{(s_2^2/n_2)^2}{n_2}}$. This preserves the underlying normality of the data being tested while accounting for differing variances. Calculating the statistic this way removes a lot of problems that can...
Can you write Python code for it?

# Sample mean values
mu_spy, mu_aapl = returns_sample.mean()
s_spy, s_aapl = returns_sample.std()
n_spy = len(returns_sample['SPY'])
n_aapl = len(returns_sample['AAPL'])

test_statistic = ((mu_spy - mu_aapl) - 0)/((s_spy**2/n_spy) + (s_aapl**2/n_aapl))**0.5
df = ((s_spy**2/n_spy) + (s_aapl**2/n_aapl))**2/(((s_spy**2 / n_spy)**2 /n_spy)+((s_aapl**2 / n_aapl)**2/n_aapl))

print 't test statistic: ', test_statistic
print 'Degrees of freedom (modified): ', df


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
Example: mean and standard deviation
First, let's take a look at some samples from a normal distribution. We know that the mean of the distribution is 0 and the standard deviation is 1; but if we measure the parameters from our observations, we will get only approximately 0 and approximately 1. We can see how these estimates change as we take more and more samples:
Can you write Python code for it?

# Set a seed so we can play with the data without generating new random numbers every time
np.random.seed(123)

normal = np.random.randn(500)
print np.mean(normal[:10])
print np.mean(normal[:100])
print np.mean(normal[:250])
print np.mean(normal)

# Plot a stacked histogram of the data
plt.hist([normal[:10], normal[10:100], normal[100:250], normal], normed=1, histtype='bar', stacked=True);
plt.ylabel('Frequency')
plt.xlabel('Value');

print np.std(normal[:10])
print np.std(normal[:100])
print np.std(normal[:250])
print np.std(normal)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
In fact, the standard deviation, which we use to quantify variability, is itself variable. Below we plot the rolling standard deviation (for a 90-day window), and compute <i>its</i> mean and standard deviation.
Can you write Python code for it?

# Compute rolling standard deviation
std = pd.rolling_std(pricing, window=90)

# Plot rolling std
_, ax2 = plt.subplots()
ax2.plot(std)
ax2.set_xticklabels([pricing.index[i].date() for i in ticks[:-1]]) # Label x-axis with dates
plt.ylabel('Standard Deviation of Moving Average')
plt.xlabel('Date')

print 'Mean of rolling std:', np.mean(std)
print 'std of rolling std:', np.std(std)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
Notice that, although the probability of getting closer to 0 and 1 for the mean and standard deviation, respectively, increases with the number of samples, we do not always get better estimates by taking more data points. Whatever our expectation is, we can always get a different result, and our goal is often to compute the probability that the result is significantly different than expected.
With time series data, we usually care only about contiguous subsets of the data. The moving average (also called running or rolling) assigns the mean of the previous $n$ data points to each point in time. Below, we compute the 90-day moving average of a stock price and plot it to see how it changes. There is no result in the beginning because we first have to accumulate at least 90 days of data.
Example: Non-Normal Underlying Distribution
What happens if the underlying data isn't normal? A mean will be very deceptive. Because of this it's important to test for normality of your data. We'll use a Jarque-Bera test as an example.
Can you write Python code for it?

#Generate some data from a bi-modal distribution
def bimodal(n):
    X = np.zeros((n))
    for i in range(n):
        if np.random.binomial(1, 0.5) == 0:
            X[i] = np.random.normal(-5, 1)
        else:
            X[i] =  np.random.normal(5, 1)
    return X

X = bimodal(1000)

#Let's see how it looks
plt.hist(X, bins=50)
plt.ylabel('Frequency')
plt.xlabel('Value')
print 'mean:', np.mean(X)
print 'standard deviation:', np.std(X)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
This lets us see the instability/standard error of the mean, and helps anticipate future variability in the data. We can quantify this variability by computing the mean and standard deviation of the rolling mean.
Can you write Python code for it?

print 'Mean of rolling mean:', np.mean(mu)
print 'std of rolling mean:', np.std(mu)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
To see what this changing standard deviation means for our data set, let's plot the data again along with the Bollinger bands: the rolling mean, one rolling standard deviation (of the data) above the mean, and one standard deviation below.
Note that although standard deviations give us more information about the spread of the data, we cannot assign precise probabilities to our expectations for future observations without assuming a particular distribution for the underlying process.
Can you write Python code for it?

# Plot original data
_, ax3 = plt.subplots()
ax3.plot(pricing)
ax3.set_xticklabels([pricing.index[i].date() for i in ticks[:-1]]) # Label x-axis with dates

# Plot Bollinger bands
ax3.plot(mu)
ax3.plot(mu + std)
ax3.plot(mu - std);
plt.ylabel('Price')
plt.xlabel('Date')
plt.legend(['Price', 'Moving Average', 'Moving Average +1 Std', 'Moving Average -1 Std'])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Instability_of_Estimates/notebook.ipynb".
The first task is:
We'll test our data using the Jarque-Bera test to see if it's normal. A significant p-value indicates non-normality.
Can you write Python code for it?

from statsmodels.stats.stattools import jarque_bera

jarque_bera(X)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
And this is the short side payoff:
Can you write Python code for it?

plt.plot(S_T, short_payoff);
plt.axhline(0, color='black', alpha=0.3)
plt.axvline(0, color='black', alpha=0.3)
plt.xlim(0, 100)
plt.ylim(-100, 100)
plt.axvline(K, linestyle='dashed', color='r', label='K')
plt.ylabel('Payoff')
plt.xlabel('$S_T$')
plt.title('Payoff of a Short Forward Contract')
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
As one contract fades out of the spotlight, the contract for the next month fades in. It is common practice to roll over positions in contracts, closing the previous month's positions and opening up equivalent positions in the next set of contracts. Note that when you create a futures object, you can access the expiration_date attribute to see when the contract will stop trading.
Can you write Python code for it?

cl_january_contract = symbols('CLF16')
print cl_january_contract.expiration_date


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
Contango and Backwardation
Often in futures markets we expect the futures price to be above the spot price. In this case, we can infer that participants in the market are willing to pay a premium for avoiding storage costs and the like. We call the difference between the futures price and the spot price the basis. A higher futures price than spot price indicates a positive basis, a situation which we call contango. With our cost of carry model, a positive cost of carry indicates contango.
Can you write Python code for it?

# A toy example to show Contango
N = 100 # Days to expiry of futures contract
cost_of_carry = 0.01
spot_price = pd.Series(np.ones(N), name = "Spot Price")
futures_price = pd.Series(np.ones(N), name = "Futures Price")
spot_price[0] = 20
futures_price[0] = spot_price[0]*np.exp(cost_of_carry*N)
for n in range(1, N):
    spot_price[n] = spot_price[n-1]*(1 + np.random.normal(0, 0.05))
    futures_price[n] = spot_price[n]*np.exp(cost_of_carry*(N - n))

spot_price.plot()
futures_price.plot()
plt.legend()
plt.title('Contango')
plt.xlabel('Time')
plt.ylabel('Price');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
Derivatives
We have stated that a futures contract is a derivative, so let's be clear on what a derivative is. A derivative is a financial instrument whose value is dependent on the value of an underlying asset. This can be a complex relationship or it can be something very simple. Derivatives have been constructed for a variety of different purposes in order to make more and more intricate bets about the markets. They provide new ways to express your expectations of how the markets will move and are used to great effect in conjunction with more conventional investments. Large amounts of resources are devoted to the construction and pricing of exotic derivatives, though that is beyond the scope of this lecture.
A futures contract is a standardized version of one of the simplest possible derivatives, the forward contract. Let's look at how forward contracts behave to give us a little more background on futures.
Forward Contracts
A futures contract at its heart is based on a derivative called a forward contract. This is an agreement between two parties to pay a delivery price, $K$, for an asset at some predetermined point in the future. Forward contracts are basic over the counter (OTC) derivatives, typically used for hedging. They are used for neutralizing risk by locking the price for an asset, obligating both sides of the contract to follow through.
Entering into a long position on a forward contract entails agreeing to buy the underlying asset while entering into a short position entails agreeing to sell the underlying. Let's say that the price of the good is denotated by $S_i$, indexed with time, with $T$ being the maturity date of the forward contract. Then the payoff of a long position in a forward contract is:
$$ S_T - K $$
And the payoff of a short position in a forward contract is:
$$ K - S_T$$
Where $S_T$ is the value of the underlying at maturity and $K$ is the value agreed upon for the underlying at maturity. The specific value of $K$ is negotiated between the parties entering into a forward...
Can you write Python code for it?

# K is the delivery price agreed upon in the contract
K = 50
# Here we look at various different values that S_T can have
S_T = np.linspace(0, 100, 200)
# Calculate the long and short payoffs
long_payoff = S_T - K
short_payoff = K - S_T


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
For a long position on a forward contract, you benefit if the price at expiry is greater than the delivery price, while the opposite holds with a short position. However, even if you do not make a profit on your position there can be advantages. A forward contract locks in a price for a transaction, removing any uncertainty that you may have about a sale or purchase in the future. This is advantageous in cases where you know what you will need at some point in the future (or have a good idea of what you will need due to your models).
Hedging with a forward contract serves to help remove any sort of uncertainty about the price that you will pay (or be paid) for a good. If you are a producer, you can easily protect yourself against falling prices by using a short position in a forward contract with a delivery price that you find amenable. Similarly, if you are a buyer, you can easily protect yourself with a long position.
Say that you need a certain quantity of copper for circuit board production in May. You could wait until May to purchase the appropriate amount, but you will be at the mercy of the spot market, the market where assets are traded for immediate delivery. Depending on your risk model and the cost of housing large amounts of copper, it may be more reasonable to enter into a forward contract for delivery in May with a distributor today. This way you are more prepared to meet your production demands.
Forward Contract Issues
Of course, we seldom know exactly when we will need an asset. In addition, entering into a private agreement with another party exposes you to counterparty risk, the risk that one or the other party in a transaction will renege on a deal. How the contract is actually settled is also up to the two parties, whether it be with an exchange of assets or a simple cash reconciliation. Forward contracts leave the delivery date, delivery method, and quantity up for debate. Their OTC nature gives a large degree of customizability, but directly contributes to the aforementioned issues and a...
Can you write Python code for it?

contract = symbols('CNH17')
futures_position_value = get_pricing(contract, start_date = '2017-01-19', end_date = '2017-02-15', fields = 'price')
futures_position_value.name = futures_position_value.name.symbol
futures_position_value.plot()
plt.title('Corn Futures Price')
plt.xlabel('Date')
plt.ylabel('Price');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
Spot Prices and Futures Prices
An important feature of futures markets is that as a contract approaches its expiry, its futures price will converge to the spot price. To show this, we will examine how SPY and a S&P 500 E-Mini contract move against each other. SPY tracks the S&P 500 Index, which is the underlying for the S&P 500 E-Mini contract. If we plot ten times the price of the ETF (the value is scaled down from the actual index), then ideally the difference between them should go to 0 as we approach the expiry of the contract.
Can you write Python code for it?

assets = ['SPY', 'ESH16']
prices = get_pricing(assets, start_date = '2015-01-01', end_date = '2016-04-15', fields = 'price')
prices.columns = map(lambda x: x.symbol, prices.columns)
prices['ESH16'].plot()
(10*prices['SPY']).plot()
plt.legend()
plt.title('Price of a S&P 500 E-Mini Contract vs SPY')
plt.xlabel('Date')
plt.ylabel('Price');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
The expiration date for this crude oil contract is in December, but the delivery does not occur until January. This time lag between expiration and delivery varies for different underlyings. For example, the S&P 500 E-Mini contract, a financial future, has an expiration date in the same month as its delivery.
Can you write Python code for it?

es_march_contract = symbols('ESH17')
print es_march_contract.expiration_date


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
The plot shows some signifiant decreases in price over the chosen time period, which should be reflected by drops in the margin account.
Can you write Python code for it?

initial_margin = 990
maintenance_margin = 900
contract_count = 5


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
This indeed seems to corroborate the point that futures prices approach the spot at expiry. And this makes sense. If we are close to expiry, there should be little difference between the price of acquiring a commodity or asset now and the price at the expiry date.
Connection Between Spot and Futures Prices
There are several ways to theoretically model futures prices, just as there are many models to model equity prices. A very basic model of futures prices and spot prices  connects them through a parameter called the cost of carry. The cost of carry acts as a discount factor for futures prices, such that
$$ F(t, T) = S(t)\times (1 + c)^{T - t} $$
where $F(t, T)$ is the futures price at time $t$ for maturity $T$, $S(t)$ is the spot price at time $t$, and $c$ is the cost of carry (here assumed to be constant). With continuous compounding, this relationship becomes:
$$ F(t, T) = S(t)e^{c(T - t)} $$
This is a naive representation of the relationship in that it relies on a constant rate as well as a few other factors. Depending on the underlying asset, the cost of carry may be composed of several different things. For example, for a physical commodity, it may incorporate storage costs and the convenience yield for immediate access through the spot market, while for some financial commodities it may only encompass the risk free rate.
The cost of carry on futures can be thought of similarly to dividends on stocks. When considering futures prices of a single underlying through several different maturities, adjustments must be made to account for the cost of carry when switching to a new maturity.
The further out we are from expiry, the more the cost of carry impacts the price. Here is a plot of the prices of contracts on light sweet crude for January, February, March, and April 2017 delivery. The further out the contract is from expiry, the higher the price.
Can you write Python code for it?

contracts = symbols(['CLF17', 'CLG17', 'CLH17', 'CLJ17'])
prices = get_pricing(contracts, start_date='2016-11-01', end_date='2016-12-15', fields='price')
prices.columns = map(lambda x: x.symbol, prices.columns)

prices.plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Futures/notebook.ipynb".
The first task is:
This is the long side payoff:
Can you write Python code for it?

plt.plot(S_T, long_payoff)
plt.axhline(0, color='black', alpha=0.3)
plt.axvline(0, color='black', alpha=0.3)
plt.xlim(0, 100)
plt.ylim(-100, 100)
plt.axvline(K, linestyle='dashed', color='r', label='K')
plt.ylabel('Payoff')
plt.xlabel('$S_T$')
plt.title('Payoff of a Long Forward Contract')
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Accessing the index of an individual element will return only the element.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
And the first row of a 2-dimensional array like so:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Array functions
Functions built into NumPy can be easily called on arrays. Most functions are applied to an array element-wise (as scalar multiplication is). For example, if we call log() on an array, the logarithm will be taken of each element.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Arrays are indexed in much the same way as lists in Python. Elements of a list begin indexing from $0$ and end at $n - 1$, where $n$ is the length of the array.
Can you write Python code for it?

print returns[0], returns[len(returns) - 1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Calculating Expected Return
So we have a universe of stocks. Great! Now let's put them together in a portfolio and calculate its expected return and risk.
We will start off by generating $N$ random weights for each asset in our portfolio.
Can you write Python code for it?

weights = np.random.uniform(0, 1, N)
weights = weights/np.sum(weights)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Calling isnan() on an array will call the function on each value of the array, returning a value of True if the element is nan and False if the element is valid. Now, knowing whether your array contains nan values is all well and good, but how do we remove nans? Handily enough, NumPy arrays can be indexed by boolean values (True or False). If we use a boolean array to index an array, we will remove all values of the array that register as False under the condition. We use the isnan() function in create a boolean array, assigning a True value to everything that is not nan and a False to the nans and we use that to index the same array.
Can you write Python code for it?

ix = ~np.isnan(v) # the ~ indicates a logical not, inverting the bools
print v[ix] # We can also just write v = v[~np.isnan(v)]

print np.mean(v[ix])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
For further reading on the universal functions in NumPy, check out the documentation.
Return to the returns
Now let's modify our returns array with scalar values. If we add a scalar value to an array it will be added to every element of the array. If we multiply an array by a scalar value it will be multiplied against every element of the array. If we do both, both will happen!
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Here we generate the remaining $N - 1$ securities that we want in our universe by adding random noise to $R_1$. This ensures that our $N - 1$ other assets will be correlated with the base asset because they have some underlying information that is shared.
Let's plot what the mean return of each asset looks like:
Can you write Python code for it?

plt.bar(np.arange(len(mean_returns)), mean_returns)
plt.xlabel('Stock')
plt.ylabel('Returns')
plt.title('Returns for {0} Random Assets'.format(N));


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Notice that the above-defined matrices, $A$ and $B$, have different dimensions. $A$ is $3 \times 5$ and $B$ is $4 \times 3$. The general rule of what can and cannot be multiplied in which order is based on the dimensions of the matrices. Specifically, the number of columns in the matrix on the left must be equal to the number of rows in the matrix on the right. In super informal terms, let's say that we have an $m \times n$ matrix and a $p \times q$ matrix. If we multiply the first by the second on the right, we get the following:
$$ (m \times n) \cdot (p \times q) = (m \times q) $$
So the resultant product has the same number of rows as the left matrix and the same number of columns as the right matrix. This limitation of matrix multiplication with regards to dimensions is important to keep track of when writing code. To demonstrate this, we use the dot() function to multiply our matrices below:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
NumPy also has functions specifically built to operate on arrays. Let's take the mean and standard deviation of this group of returns.
Can you write Python code for it?

print "Mean: ", np.mean(returns), "Std Dev: ", np.std(returns)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Or the max() function will return the maximum element of an array.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
Portfolio Variance
Let's return to our portfolio example from before. We calculated the expected return of the portfolio, but how do we calculate the variance We start by trying to evaluate the portfolio as a sum of each individual asset, scaled by it's weight.
$$ VAR[P] = VAR[\omega_1 S_1 + \omega_2 S_2 + \cdots + \omega_N S_N] $$
Where $S_0, \cdots, S_N$ are the assets contained within our universe. If all of our assets were independent of each other, we could simply evaluate this as
$$ VAR[P] = VAR[\omega_1 S_1] + VAR[\omega_2 S_2] + \cdots + VAR[\omega_N S_N] = \omega_1^2\sigma_1^2 + \omega_2^2\sigma_2^2 + \cdots + \omega_N^2\sigma_N^2 $$
However, all of our assets depend on each other by their construction. They are all in some way related to our base asset and therefore each other. We thus have to calculate the variance of the portfolio by including the individual pairwise covariances of each asset. Our formula for the variance of the portfolio:
$$ VAR[P] = \sigma_P^2 = \sum_i \omega_i^2\sigma_i^2 + \sum_i\sum_{i\neq j} \omega_i\omega_j\sigma_i\sigma_j\rho_{i, j}, \ i, j \in \lbrace 1, 2, \cdots, N \rbrace $$
Where $\rho_{i,j}$ is the correlation between $S_i$ and $S_j$, $\rho_{i, j} = \frac{COV[S_i, S_j]}{\sigma_i\sigma_j}$. This seems exceedingly complicated, but we can easily handle all of this using NumPy arrays. First, we calculate the covariance matrix that relates all the individual stocks in our universe.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
The random module in NumPy is exceedingly useful. It contains methods for sampling from many different probability distributions, some of which are covered in the random variables lecture in the Quantopian lecture series. In this case we draw $N = 100$ random samples from a normal distribution with mean $1.01$ and standard deviation $0.03$. We treat these as the daily percentage returns of our asset and take the cumulative product of these samples to get the current price.
The way we have generated our universe, the the individual $R_i$ vectors are each 1-dimensional arrays and the returns and assets variables contain 2-dimensional arrays. Above, we set the initial row of both returns and assets to be the first $R_i$ vector and the cumulative asset price based on those returns, respectively.
We will now use this base asset to create a few other random assets that are correlated with it.
Can you write Python code for it?

# Generate assets that are correlated with R_1
for i in range(1, N):
    R_i = R_1 + np.random.normal(0.001, 0.02, 100)
    returns[i] = R_i # Set each row of returns equal to the new R_i array
    assets[i] = np.cumprod(R_i)

mean_returns = [(np.mean(R) - 1)*100 for R in returns]
return_volatilities = [np.std(R) for R in returns]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
The transpose looks like a mirror image of the same array.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
These results make sense in accordance with our rule. Multiplying a $3 \times 5$ matrix on the right by a $4 \times 3$ matrix results in an error while multiplying a $4 \times 3$ matrix on the right by a $3 \times 5$ matrix results in a $4 \times 5$ matrix.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
This array is not formatted particularly nicely, but a covariance matrix is a very important concept. The covariance matrix is of the form:
$$ \left[\begin{matrix}
VAR[S_1] & COV[S_1, S_2] & \cdots & COV[S_1, S_N] \
COV[S_2, S_1] & VAR[S_2] & \cdots & COV[S_2, S_N] \
\vdots & \vdots & \ddots & \vdots \
COV[S_N, S_1] & COV[S_N, S_2] & \cdots & VAR[S_N]
\end{matrix}\right] $$
So each diagonal entry is the variance of that asset at that index and each off-diagonal holds the covariance of two assets indexed by the column and row number. What is important is that once we have the covariance matrix we are able to do some very quick linear algebra to calculate the variance of the overall portfolio. We can represent the variance of the portfolio in array form as:
$$ \sigma_p^2 = \omega \ C \ \omega^\intercal$$
Where $C$ is the covariance matrix of all the assets and $\omega$ is the array containing the weights of each individual asset. The superscript $\intercal$ on the second $\omega$ listed above denotes the transpose of $\omega$. For a reference on the evaluation of the variance of a portfolio as a matrix equation, please see the Wikipedia article on modern portfolio theory.
The transpose of an array is what you get when you switch the rows and columns of an array. This has the effect of reflecting an array across what you might imagine as a diagonal. For example, take our array $A$ from before:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
To confirm this calculation, let's simply evaluate the volatility of the portfolio using only NumPy functions.
Can you write Python code for it?

# Confirming calculation
vol_p_alt = np.sqrt(np.var(np.dot(weights, returns), ddof=1))
print "Portfolio volatility: ", vol_p_alt


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
We can make an array by calling a function on the list:
Can you write Python code for it?

returns = np.array(stock_list)
print returns, type(returns)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_NumPy/notebook.ipynb".
The first task is:
We can take a slice of an array using a colon, just like in a list.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Correlation vs. Cointegration
Correlation and cointegration, while theoretically similar, are not the same. To demonstrate this, we'll show examples of series that are correlated, but not cointegrated, and vice versa. To start let's check the correlation of the series we just generated.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Examining the price ratio of a trading pair is a traditional way to handle pairs trading. Part of why this works as a signal is based in our assumptions of how stock prices move, specifically because stock prices are typically assumed to be log-normally distributed. What this implies is that by taking a ratio of the prices, we are taking a linear combination of the returns associated with them (since prices are just the exponentiated returns).
This can be a little irritating to deal with for our purposes as purchasing the precisely correct ratio of a trading pair may not be practical. We choose instead to move forward with simply calculating the spread between the cointegrated stocks using linear regression. This is a very simple way to handle the relationship, however, and is likely not feasible for non-toy examples. There are other potential methods for estimating the spread listed at the bottom of this lecture. If you want to get more into the theory of why having cointegrated stocks matters for pairs trading, again, please see the Integration, Cointegration, and Stationarity Lecture from the Quantopian Lecture Series.
So, back to our example. The absolute spread isn't very useful in statistical terms. It is more helpful to normalize our signal by treating it as a z-score.
WARNING
In practice this is usually done to try to give some scale to the data, but this assumes some underlying distribution. Usually normal. Under a normal distribution, we would know that approximately 84% of all spread values will be smaller. However, much financial data is not normally distributed, and one must be very careful not to assume normality, nor any specific distribution when generating statistics. It could be the case that the true distribution of spreads was very fat-tailed and prone to extreme values. This could mess up our model and result in large losses.
Can you write Python code for it?

def zscore(series):
    return (series - series.mean()) / np.std(series)

zscore(spread).plot()
plt.axhline(zscore(spread).mean(), color='black')
plt.axhline(1.0, color='red', linestyle='--')
plt.axhline(-1.0, color='green', linestyle='--')
plt.legend(['Spread z-score', 'Mean', '+1', '-1']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Example of how to get all the prices of all the stocks loaded using get_pricing() above in one pandas dataframe object
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Generating Two Fake Securities
We model X's daily returns by drawing from a normal distribution. Then we perform a cumulative sum to get the value of X on each day.
Can you write Python code for it?

X_returns = np.random.normal(0, 1, 100) # Generate the daily returns
# sum them and shift all the prices up into a reasonable range
X = pd.Series(np.cumsum(X_returns), name='X') + 50
X.plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Looking for Cointegrated Pairs of Alternative Energy Securities
We are looking through a set of solar company stocks to see if any of them are cointegrated. We'll start by defining the list of securities we want to look through. Then we'll get the pricing data for each security for the year of 2014.
Our approach here is somewhere in the middle of the spectrum that we mentioned before. We have formulated an economic hypothesis that there is some sort of link between a subset of securities within the energy sector and we want to test whether there are any cointegrated pairs. This incurs significantly less multiple comparisons bias than searching through hundreds of securities and slightly more than forming a hypothesis for an individual test.
NOTE: We include the market in our data. This is because the market drives the movement of so many securities that you often times might find two seemingingly cointegrated securities, but in reality they are not cointegrated and just both conintegrated with the market. This is known as a confounding variable and it is important to check for market involvement in any relationship you find.
get_pricing() is a Quantopian method that pulls in stock data, and loads it into a Python Pandas DataPanel object.  Available fields are 'price', 'open_price', 'high', 'low', 'volume'.  But for this example we will just use 'price' which is the daily closing price of the stock.
Can you write Python code for it?

symbol_list = ['ABGB', 'ASTI', 'CSUN', 'DQ', 'FSLR','SPY']
prices_df = get_pricing(symbol_list, fields=['price']
                               , start_date='2014-01-01', end_date='2015-01-01')['price']
prices_df.columns = map(lambda x: x.symbol, prices_df.columns)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Looks like 'ABGB' and 'FSLR' are cointegrated. Let's take a look at the prices to make sure there's nothing weird going on.
Can you write Python code for it?

S1 = prices_df['ABGB']
S2 = prices_df['FSLR']

score, pvalue, _ = coint(S1, S2)
pvalue


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Now we generate Y. Remember that Y is supposed to have a deep economic link to X, so the price of Y should vary pretty similarly. We model this by taking X, shifting it up and adding some random noise drawn from a normal distribution.
Can you write Python code for it?

some_noise = np.random.normal(0, 1, 100)
Y = X + 5 + some_noise
Y.name = 'Y'
pd.concat([X, Y], axis=1).plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Now we'll run our method on the list and see if any pairs are cointegrated.
Can you write Python code for it?

# Heatmap to show the p-values of the cointegration test between each pair of
# stocks. Only show the value in the upper-diagonal of the heatmap
scores, pvalues, pairs = find_cointegrated_pairs(prices_df)
import seaborn
seaborn.heatmap(pvalues, xticklabels=symbol_list, yticklabels=symbol_list, cmap='RdYlGn_r'
                , mask = (pvalues >= 0.05)
                )
print pairs


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Out of Sample Test
Now that we have constructed our spread appropriately and have an idea of how we will go about making trades, it is time to conduct some out of sample testing. Our whole model is based on the premise that these securities are cointegrated, but we built it on information from a certain time period. If we actually want to implement this model, we need to conduct an out of sample test to confirm that the principles of our model are still valid going forward.
Since we initially built the model on the 2014 - 2015 year, let's see if this cointegrated relationship holds for 2015 - 2016. Historical results do not guarantee future results so this is a sanity check to see if the work we have done holds strong.
Can you write Python code for it?

symbol_list = ['ABGB', 'FSLR']
prices_df = get_pricing(symbol_list, fields=['price']
                               , start_date='2015-01-01', end_date='2016-01-01')['price']
prices_df.columns = map(lambda x: x.symbol, prices_df.columns)

S1 = prices_df['ABGB']
S2 = prices_df['FSLR']

score, pvalue, _ = coint(S1, S2)
print 'p-value: ', pvalue


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Simple Strategy:

Go "Long" the spread whenever the z-score is below -1.0
Go "Short" the spread when the z-score is above 1.0
Exit positions when the z-score approaches zero

This is just the tip of the iceberg, and only a very simplistic example to illustrate the concepts.  In practice you would want to compute a more optimal weighting for how many shares to hold for S1 and S2.  Some additional resources on pair trading are listed at the end of this notebook
Trading using constantly updating statistics
In general taking a statistic over your whole sample size can be bad. For example, if the market is moving up, and both securities with it, then your average price over the last 3 years may not be representative of today. For this reason traders often use statistics that rely on rolling windows of the most recent data.
Moving Averages
A moving average is just an average over the last $n$ datapoints for each given time. It will be undefined for the first $n$ datapoints in our series. Shorter moving averages will be more jumpy and less reliable, but respond to new information quickly. Longer moving averages will be smoother, but take more time to incorporate new information.
We also need to use a rolling beta, a rolling estimate of how our spread should be calculated, in order to keep all of our parameters up to date.
Can you write Python code for it?

# Get the spread between the 2 stocks
# Calculate rolling beta coefficient
rolling_beta = pd.ols(y=S1, x=S2, window_type='rolling', window=30)
spread = S2 - rolling_beta.beta['x'] * S1
spread.name = 'spread'

# Get the 1 day moving average of the price spread
spread_mavg1 = pd.rolling_mean(spread, window=1)
spread_mavg1.name = 'spread 1d mavg'

# Get the 30 day moving average
spread_mavg30 = pd.rolling_mean(spread, window=30)
spread_mavg30.name = 'spread 30d mavg'

plt.plot(spread_mavg1.index, spread_mavg1.values)
plt.plot(spread_mavg30.index, spread_mavg30.values)

plt.legend(['1 Day Spread MAVG', '30 Day Spread MAVG'])

plt.ylabel('Spread');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
Testing for Cointegration
That's an intuitive definition, but how do we test for this statistically? There is a convenient cointegration test that lives in statsmodels.tsa.stattools. Let's say that our confidence level is $0.05$. We should see a p-value below our cutoff, as we've artifically created two series that are the textbook definition of cointegration.
Can you write Python code for it?

# compute the p-value of the cointegration test
# will inform us as to whether the spread between the 2 timeseries is stationary
# around its mean
score, pvalue, _ = coint(X,Y)
print pvalue


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pairs_Trading/notebook.ipynb".
The first task is:
That's very high, as we would expect. But how would two series that are correlated but not cointegrated look?
Correlation Without Cointegration
A simple example is two series that just diverge.
Can you write Python code for it?

X_returns = np.random.normal(1, 1, 100)
Y_returns = np.random.normal(2, 1, 100)

X_diverging = pd.Series(np.cumsum(X_returns), name='X')
Y_diverging = pd.Series(np.cumsum(Y_returns), name='Y')

pd.concat([X_diverging, Y_diverging], axis=1).plot();

print 'Correlation: ' + str(X_diverging.corr(Y_diverging))
score, pvalue, _ = coint(X_diverging,Y_diverging)
print 'Cointegration test p-value: ' + str(pvalue)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Accessing DataFrame elements
Again we see a lot of carryover from Series in how we access the elements of DataFrames. The key sticking point here is that everything has to take into account multiple dimensions now. The main way that this happens is through the access of the columns of a DataFrame, either individually or in groups. We can do this either by directly accessing the attributes or by using the methods we already are familiar with.
Can you write Python code for it?

symbol = ["CMG", "MCD", "SHAK", "WFM"]
start = "2012-01-01"
end = "2016-01-01"
prices = get_pricing(symbol, start_date=start, end_date=end, fields="price")
if isinstance(symbol, list):
    prices.columns = map(lambda x: x.symbol, prices.columns)
else:
    prices.name = symbol


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Accessing an individual column will return a Series, regardless of how we get it.
Can you write Python code for it?

print type(prices.CMG)
print type(prices.loc[:, 'CMG'])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Adding, Removing Columns, Combining DataFrames/Series
It is all well and good when you already have a DataFrame filled with data, but it is also important to be able to add to the data that you have.
We add a new column simply by assigning data to a column that does not already exist. Here we use the .loc[:, 'COL_NAME'] notation and store the output of get_pricing() (which returns a pandas Series if we only pass one security) there. This is the method that we would use to add a Series to an existing DataFrame.
Can you write Python code for it?

s_1 = get_pricing('TSLA', start_date=start, end_date=end, fields='price')
prices.loc[:, 'TSLA'] = s_1
prices.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Again, the describe() function will provide us with summary statistics of our data if we would rather have all of our typical statistics in a convenient visual instead of calculating them individually.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
And we can create linear combinations of Series themselves using the basic mathematical operators. pandas will group up matching indices and perform the calculations elementwise to produce a new Series.
Can you write Python code for it?

noisy_prices = prices + 5 * pd.Series(np.random.normal(0, 5, len(prices)), index=prices.index) + 20
noisy_prices.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
As mentioned above, we can combine Series into DataFrames. Concatatenating Series like this will match elements up based on their corresponding index. As the following Series do not have an index assigned, they each default to an integer index.
Can you write Python code for it?

s_1 = pd.Series([2, 4, 6, 8, 10], name='Evens')
s_2 = pd.Series([1, 3, 5, 7, 9], name="Odds")
numbers = pd.concat([s_1, s_2], axis=1)
print numbers


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
As well as the standard deviation:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
As with our toy example, we can specify a name for our time series, if only to clarify the name the get_pricing() provides us.
Can you write Python code for it?

print 'Old name: ', prices.name
prices.name = symbol
print 'New name: ', prices.name


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Boolean indexing
As with Series, sometimes we want to filter a DataFrame according to a set of criteria. We do this by indexing our DataFrame with boolean values.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
But again, the best choice in this case (since we are still using time series data, handling multiple time series at once) is still to simply drop the missing values.
Can you write Python code for it?

dropped_prices = prices.dropna()
dropped_prices.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
But again, this is a bad idea for the same reasons as the previous option. Both of these so-called solutions take into account future data that was not available at the time of the data points that we are trying to fill. In the case of using the mean or the median, these summary statistics are calculated by taking into account the entire time series. Backward filling is equivalent to saying that the price of a particular security today, right now, tomorrow's price. This also makes no sense. These two options are both examples of look-ahead bias, using data that would be unknown or unavailable at the desired time, and should be avoided.
Our next option is significantly more appealing. We could simply drop the missing data using the dropna() method. This is much better alternative than filling NaN values in with arbitrary numbers.
Can you write Python code for it?

dropped_prices = calendar_prices.dropna()
dropped_prices.head(10)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Each DataFrame has a few key attributes that we need to keep in mind. The first of these is the index attribute. We can easily include an index of Timestamp objects like we did with Series.
Can you write Python code for it?

frame_data = pd.DataFrame(dict_data, index=pd.date_range('2016-01-01', periods=5))
print frame_data


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Every Series has a name. We can give the series a name as a parameter or we can define it afterwards by directly accessing the name attribute. In this case, we have given our time series no name so the attribute should be empty.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Here we directly access the CMG column. Note that this style of access will only work if your column name has no spaces or unfriendly characters in it.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Here we use the pct_change() method to get a DataFrame of the multiplicative returns of the securities that we are looking at.
Can you write Python code for it?

mult_returns = prices.pct_change()[1:]
mult_returns.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
If there are no matching indices, however, we may get an empty Series in return.
Can you write Python code for it?

empty_series = prices + pd.Series(np.random.normal(0, 1, len(prices)))
empty_series.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
If we instead want to combine multiple DataFrames into one, we use the pandas.concat() method.
Can you write Python code for it?

df_1 = get_pricing(['SPY', 'VXX'], start_date=start, end_date=end, fields='price')
df_2 = get_pricing(['MSFT', 'AAPL', 'GOOG'], start_date=start, end_date=end, fields='price')
df_3 = pd.concat([df_1, df_2], axis=1)
df_3.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
If we so desire, we can group multiple conditions together using the logical operators &, |, and ~ (and, or, and not, respectively).
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
It is also just as easy to remove a column.
Can you write Python code for it?

prices = prices.drop('TSLA', axis=1)
prices.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Many NumPy functions will work on Series the same way that they work on 1-dimensional NumPy arrays.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Missing data (again)
Bringing real-life data into a DataFrame brings us the same problems that we had with it in a Series, only this time in more dimensions. We have access to the same methods as with Series, as demonstrated below.
Can you write Python code for it?

filled0_prices = prices.fillna(0)
filled0_prices.head(5)

bfilled_prices = prices.fillna(method='bfill')
bfilled_prices.head(5)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Notice how we pass a tuple into the loc[] method? This is a key difference between accessing a Series and accessing a DataFrame, grounded in the fact that a DataFrame has multiple dimensions. When you pass a 2-dimensional tuple into a DataFrame, the first element of the tuple is applied to the rows and the second is applied to the columns. So, to break it down, the above line of code tells the DataFrame to return every single row of the column with label 'CMG'. Lists of columns are also supported.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Notice that this DatetimeIndex has a collection of associated information. In particular it has an associated frequency (freq) and an associated timezone (tz). The frequency indicates whether the data is daily vs monthly vs some other period while the timezone indicates what locale this index is relative to. We can modify all of this extra information!
If we resample our Series, we can adjust the frequency of our data. We currently have daily data (excluding weekends) because get_pricing() pulls only data from market days. Let's up-sample from this daily data to monthly data using the resample() method.
Can you write Python code for it?

monthly_prices = prices.resample('M')
monthly_prices.head(10)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Now our time series is cleaned for the calendar year, with all of our NaN values properly handled. It is time to talk about how to actually do time series analysis with pandas data structures.
Time Series Analysis with pandas
Let's do some basic time series analysis on our original prices. Each pandas Series has a built-in plotting method.
Can you write Python code for it?

prices.plot();
# We still need to add the axis labels and title ourselves
plt.title(symbol + " Prices")
plt.ylabel("Price")
plt.xlabel("Date");


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Or even for a range of dates!
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Rather than looking at a time series itself, we may want to look at its first-order differences or percent change (in order to get additive or multiplicative returns, in our particular case). Both of these are built-in methods.
Can you write Python code for it?

add_returns = prices.diff()[1:]
mult_returns = prices.pct_change()[1:]

plt.title("Multiplicative returns of " + symbol)
plt.xlabel("Date")
plt.ylabel("Percent Returns")
mult_returns.plot();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Separate from the columns and index of a DataFrame, we can also directly access the values they contain by looking at the values attribute.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
The index of the Series is crucial for handling time series, which we will get into a little later.
Accessing Series Elements
Series are typically accessed using the iloc[] and loc[] methods. We use iloc[] to access elements by integer index and we use loc[] to access the index of the Series.
Can you write Python code for it?

print "First element of the series: ", s.iloc[0]
print "Last element of the series: ", s.iloc[len(s)-1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
The majority of these functions, however, are already implemented directly as Series and DataFrame methods.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
The same statistical functions from our interactions with Series resurface here with the addition of the axis parameter. By specifying the axis, we tell pandas to calculate the desired function along either the rows (axis=0) or the columns (axis=1). We can easily calculate the mean of each columns like so:
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
This is very convenient for getting only elements of a Series that fulfill specific criteria that we need. It gets even more convenient when we are handling DataFrames.
Indexing and Time Series
Since we use Series for handling time series, it's worth covering a little bit of how we handle the time component. For our purposes we use pandas Timestamp objects. Let's pull a full time series, complete with all the appropriate labels, by using our get_pricing() method. All data pulled with get_pricing() or using our Pipeline API will be in either Series or DataFrame format. We can modify this index however we like.
Can you write Python code for it?

symbol = "CMG"
start = "2012-01-01"
end = "2016-01-01"
prices = get_pricing(symbol, start_date=start, end_date=end, fields="price")


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
This name can be directly modified with no repercussions.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
This plays nicely with lists of columns, too.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
This returns a NumPy array.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
This returns a slice of the series that starts from the second to last element and ends at the third to last element (because the fourth to last is not included, taking steps of size $1$).
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Time Series Analysis with pandas
Using the built-in statistics methods for DataFrames, we can perform calculations on multiple time series at once! The code to perform calculations on DataFrames here is almost exactly the same as the methods used for Series above, so don't worry about re-learning everything.
The plot() method makes another appearance here, this time with a built-in legend that corresponds to the names of the columns that you are plotting.
Can you write Python code for it?

prices.plot()
plt.title("Collected Stock Prices")
plt.ylabel("Price")
plt.xlabel("Date");


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
Using iloc[] also works similarly, allowing you to access parts of the DataFrame by integer index.
Can you write Python code for it?

prices.iloc[0:2, 1]

# Access prices with integer index in
# [1, 3, 5, 7, 9, 11, 13, ..., 99]
# and in column 0 or 3
prices.iloc[[1, 3, 5] + range(7, 100, 2), [0, 3]].head(20)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We call the collected axis labels of a Series its index. An index can either passed to a Series as a parameter or added later, similarly to its name. In the absence of an index, a Series will simply contain an index composed of integers, starting at $0$, as in the case of our "Toy Series".
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can add multiple boolean conditions by using the logical operators &, |, and ~ (and, or, and not, respectively) again!
Can you write Python code for it?

prices.loc[(prices.MCD > prices.WFM) & ~prices.SHAK.isnull()].head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can also access a series by using the values of its index. Since we indexed s with a collection of dates (Timestamp objects) we can look at the value contained in s for a particular date.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can also adjust the timezone of a Series to adapt the time of real-world data. In our case, our time series is already localized to UTC, but let's say that we want to adjust the time to be 'US/Eastern'. In this case we use the tz_convert() method, since the time is already localized.
Can you write Python code for it?

eastern_prices = prices.tz_convert('US/Eastern')
eastern_prices.head(10)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can also simply access the DataFrame by index value using loc[], as with Series.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can also use loc[] to access an individual column like so.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can display the first few elements of our series by using the head() method and specifying the number of elements that we want. The analogous method for the last few elements is tail().
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can even specify how we want the calculation of the new period to be done. Here we create a custom_resampler() function that will return the first value of the period. In our specific case, this will return a Series where the monthly value is the first value of that month.
Can you write Python code for it?

def custom_resampler(array_like):
    """ Returns the first value of the period """
    return array_like[0]

first_of_month_prices = prices.resample('M', how=custom_resampler)
first_of_month_prices.head(10)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We can scale and add scalars to our DataFrame, as you might suspect after dealing with Series. This again works element-wise.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
We will use pandas.concat() again later to combine multiple DataFrames into one.
Each DataFrame also has a columns attribute. These can either be assigned when we call pandas.DataFrame or they can be modified directly like the index. Note that when we concatenated the two Series above, the column names were the names of those Series.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
With Series, we can just use the brackets ([]) to access elements, but this is not best practice. The brackets are ambiguous because they can be used to access Series (and DataFrames) using both index and integer values and the results will change based on context (especially with DataFrames).
Boolean Indexing
In addition to the above-mentioned access methods, you can filter Series using boolean arrays. Series are compatible with your standard comparators. Once compared with whatever condition you like, you get back yet another Series, this time filled with boolean values.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Pandas/notebook.ipynb".
The first task is:
You'll notice that we still have a couple of NaN values right at the beginning of our time series. This is because the first of January in 2012 was a Sunday and the second was a market holiday! Because these are the earliest data points and we don't have any information from before them, they cannot be forward-filled. We will take care of these NaN values in the next section, when we deal with missing data.
Missing Data
Whenever we deal with real data, there is a very real possibility of encountering missing values. Real data is riddled with holes and pandas provides us with ways to handle them. Sometimes resampling or reindexing can create NaN values. Fortunately, pandas provides us with ways to handle them. We have two primary means of coping with missing data. The first of these is filling in the missing data with  fillna(). For example, say that we want to fill in the missing days with the mean price of all days.
Can you write Python code for it?

meanfilled_prices = calendar_prices.fillna(calendar_prices.mean())
meanfilled_prices.head(10)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
A floating point number, or a float is a fancy name for a real number (again as in mathematics). To define a float, we need to either include a decimal point or specify that the value is a float.
Can you write Python code for it?

my_float = 1.0
print my_float, type(my_float)
my_float = float(1)
print my_float, type(my_float)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
A variable of type float will not round the number that you store in it, while a variable of type integer will. This makes floats more suitable for mathematical calculations where you want more than just integers.
Note that as we used the float() function to force an number to be considered a float, we can use the int() function to force a number to be considered an int.
Can you write Python code for it?

my_int = int(3.14159)
print my_int, type(my_int)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Adding a new key-value pair is as simple as defining it.
Can you write Python code for it?

my_dict["Historical Fiction"] = ["Pillars of the Earth"]
print my_dict["Historical Fiction"]

print my_dict


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
After defining a dictionary, we can access any individual value by indicating its key in brackets.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
And concatenate them the way that you would with strings!
Can you write Python code for it?

my_other_tuple = ('make', 'that', 50)
print my_tuple + my_other_tuple


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
And everything before a certain point
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
And so on, for the other collections and containers. None also evaluates as False. The number 1 is equivalent to True and the number 0 is equivalent to False as well, in a boolean context.
If-statements
We can create segments of code that only execute if a set of conditions is met. We use if-statements in conjunction with logical statements in order to create branches in our code.
An if block gets entered when the condition is considered to be True. If condition is evaluated as False, the if block will simply be skipped unless there is an else block to accompany it. Conditions are made using either logical operators or by using the truthiness of values in Python. An if-statement is defined with a colon and a block of indented text.
Can you write Python code for it?

# This is the basic format of an if statement. This is a vacuous example.
# The string "Condition" will always evaluated as True because it is a
# non-empty string. he purpose of this code is to show the formatting of
# an if-statement.
if "Condition":
    # This block of code will execute because the string is non-empty
    # Everything on these indented lines
    print True
else:
    # So if the condition that we examined with if is in fact False
    # This block of code will execute INSTEAD of the first block of code
    # Everything on these indented lines
    print False
# The else block here will never execute because "Condition" is a non-empty string.

i = 4
if i == 5:
    print 'The variable i has a value of 5'


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
And that one calculates the slope between two points!
Can you write Python code for it?

print "The slope-intercept form of the line between a and b, using point a, is: y - {0} = {2}(x - {1})".format(a[1], a[0], calculate_slope(a, b))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
As with other data types, == will check for whether the two things on either side of it have the same value. In this case, we compare whether the value of the strings are the same. Using > or < or any of the other comparators is not quite so intuitive, however, so we will stay from using comparators with strings in this lecture. Comparators will examine the lexicographical order of the strings, which might be a bit more in-depth than you might like.
Some built-in functions return a boolean value, so they can be used as conditions in an if-statement. User-defined functions can also be constructed so that they return a boolean value. This will be covered later with function definition!
The in keyword is generally used to check membership of a value within another value. We can check memebership in the context of an if-statement and use it to output a truth value.
Can you write Python code for it?

if 'a' in my_string or 'e' in my_string:
    print 'Those are my favorite vowels!'


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Because a set is unordered, we can't access individual elements using an index. We can, however, easily check for membership (to see if something is contained in a set) and take the unions and intersections of sets by using the built-in set functions.
Can you write Python code for it?

'cats' in animal_set # Here we check for membership using the `in` keyword.


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Booleans, or bools are binary variable types. A bool can only take on one of two values, these being True or False. There is much more to this idea of truth values when it comes to programming, which we cover later in the Logical Operators of this notebook.
Can you write Python code for it?

my_bool = True
print my_bool, type(my_bool)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Both are allowed so that we can include apostrophes or quotation marks in a string if we so choose.
Can you write Python code for it?

my_string = '"Jabberwocky", by Lewis Carroll'
print my_string
my_string = "'Twas brillig, and the slithy toves / Did gyre and gimble in the wabe;"
print my_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Built-in objects and classes often have special functions associated with them that are called methods. We access these methods by using a period ('.'). We will cover objects and their associated methods more in another lecture!
Using string methods we can count instances of a character or group of characters.
Can you write Python code for it?

print 'Count of the letter i in Supercalifragilisticexpialidocious: ', my_string.count('i')
print 'Count of "li" in the same word: ', my_string.count('li')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Collections
Lists
A list in Python is an ordered collection of objects that can contain any data type. We define a list using brackets ([]).
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Functions are defined with def, a function name, a list of parameters, and a colon. Everything indented below the colon will be included in the definition of the function.
We can have our functions do anything that you can do with a normal block of code. For example, our hello_world() function prints a string every time it is called. If we want to keep a value that a function calculates, we can define the function so that it will return the value we want. This is a very important feature of functions, as any variable defined purely within a function will not exist outside of it.
Can you write Python code for it?

def see_the_scope():
    in_function_string = "I'm stuck in here!"

see_the_scope()
print in_function_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Here we have selected the entire list (because 0:7 will yield elements 0 through 6) and we have selected a step size of 2. So this will spit out element 0 , element 2, element 4, and so on through the list element selected. We can skip indicated the beginning and end of our slice, only indicating the step, if we like.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Here we use in to check whether the variable my_string contains any particular letters. We will later use in to iterate through lists!
Loop Structures
Loop structures are one of the most important parts of programming. The for loop and the while loop provide a way to repeatedly run a block of code repeatedly. A while loop will iterate until a certain condition has been met. If at any point after an iteration that condition is no longer satisfied, the loop terminates. A for loop will iterate over a sequence of values and terminate when the sequence has ended. You can instead include conditions within the for loop to decide whether it should terminate early or you could simply let it run its course.
Can you write Python code for it?

i = 5
while i > 0: # We can write this as 'while i:' because 0 is False!
    i -= 1
    print 'I am looping! {0} more to go!'.format(i)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
If we just iterate over a dictionary without doing anything else, we will only get the keys. We can either use the keys to get the values, like so:
Can you write Python code for it?

for key in my_dict:
    print my_dict[key]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
If we want to, we can define a function so that it takes an arbitrary number of parameters. We tell Python that we want this by using an asterisk (*).
Can you write Python code for it?

def sum_values(*args):
    sum_val = 0
    for i in args:
        sum_val += i
    return sum_val

print sum_values(1, 2, 3)
print sum_values(10, 20, 30, 40, 50)
print sum_values(4, 2, 5, 1, 10, 249, 25, 24, 13, 6, 4)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
If you are not familiar with the the mod operator, it operates like a remainder function. If we type $15 \ \% \ 4$, it will return the remainder after dividing $15$ by $4$.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
In addition to accessing individual elements of a list, we can access groups of elements through slicing.
Can you write Python code for it?

my_list = ['friends', 'romans', 'countrymen', 'lend', 'me', 'your', 'ears']


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
In these examples, the %s indicates that Python should convert the values into strings. There are multiple conversion types that you can use to get more specific with the the formatting. See the string formatting documentation for additional examples and more complete details on use.
Logical Operators
Basic Logic
Logical operators deal with boolean values, as we briefly covered before. If you recall, a bool takes on one of two values, True or False (or $1$ or $0$). The basic logical statements that we can make are defined using the built-in comparators. These are == (equal), != (not equal), < (less than), > (greater than), <= (less than or equal to), and >= (greater than or equal to).
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Indexing in Python starts from $0$. If you have a list of length $n$, the first element of the list is at index $0$, the second element is at index $1$, and so on and so forth. The final element of the list will be at index $n-1$. Be careful! Trying to access a non-existent index will cause an error.
Can you write Python code for it?

print 'The first, second, and third list elements: ', my_list[0], my_list[1], my_list[2]
print 'Accessing outside the list bounds causes an error: ', my_list[3]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Lists can also contain multiple different data types at once!
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Logical statements can be as simple or complex as we like, depending on what we need to express. Evaluating the above logical statement step by step we see that we are evaluating (True and True) or (False and not False). This becomes True or (False and True), subsequently becoming True or False, ultimately being evaluated as True.
Truthiness
Data types in Python have a fun characteristic called truthiness. What this means is that most built-in types will evaluate as either True or False when a boolean value is needed (such as with an if-statement). As a general rule, containers like strings, tuples, dictionaries, lists, and sets, will return True if they contain anything at all and False if they contain nothing.
Can you write Python code for it?

# Similar to how float() and int() work, bool() forces a value to be considered a boolean!
print bool('')

print bool('I have character!')

print bool([])

print bool([1, 2, 3])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Make sure that your variables are floats if you want to have decimal points in your answer. If you perform math exclusively with integers, you get an integer. Including any float in the calculation will make the result a float.
Can you write Python code for it?

first_integer = 11
second_integer = 3
print first_integer / second_integer

first_number = 11.0
second_number = 3.0
print first_number / second_number