I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Make sure you read the comments within each code cell (if they are there). They will provide more real-time explanations of what is going on as you look at each line of code.
Variables
Variables provide names for values in programming. If you want to save a value for later or repeated use, you give the value a name, storing the contents in a variable. Variables in programming work in a fundamentally similar way to variables in algebra, but in Python they can take on various different data types.
The basic variable types that we will cover in this section are integers, floating point numbers, booleans, and strings.
An integer in programming is the same as in mathematics, a round number with no values after the decimal point. We use the built-in print function here to display the values of our variables as well as their types!
Can you write Python code for it?

my_integer = 50
print my_integer, type(my_integer)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Mathematical functions also work on variables!
Can you write Python code for it?

first_integer = 4
second_integer = 5
print first_integer * second_integer


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Note how any extra instances of the same item are removed in the final set. We can also create a set from a list, using the set() function.
Can you write Python code for it?

animal_list = ['cats', 'dogs', 'dogs', 'dogs', 'lizards', 'sponges', 'cows', 'bats', 'sponges']
animal_set = set(animal_list)
print animal_set # Removes all extra instances from the list


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Note that in this for loop we use the in keyword. Use of the in keyword is not limited to checking for membership as in the if-statement example. You can iterate over any collection with a for loop by using the in keyword.
In this next example, we will iterate over a set because we want to check for containment and add to a new set.
Can you write Python code for it?

my_list = {'cats', 'dogs', 'lizards', 'cows', 'bats', 'sponges', 'humans'} # Lists all the animals in the world
mammal_list = {'cats', 'dogs', 'cows', 'bats', 'humans'} # Lists all the mammals in the world
my_new_list = set()
for animal in my_list:
    if animal in mammal_list:
        # This adds any animal that is both in my_list and mammal_list to my_new_list
        my_new_list.add(animal)

print my_new_list


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Or we can use the iteritems() function to get both key and value at the same time.
Can you write Python code for it?

for key, value in my_dict.iteritems():
    print key, ':', value


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Pairing two sets together with & will calculate the intersection of both sets, returning a set that only contains what they have in common.
If you are interested in learning more about the built-in functions for sets, feel free to check out the documentation.
Dictionaries
Another essential data structure in Python is the dictionary. Dictionaries are defined with a combination of curly braces ({}) and colons (:). The braces define the beginning and end of a dictionary and the colons indicate key-value pairs. A dictionary is essentially a set of key-value pairs. The key of any entry must be an immutable data type. This makes both strings and tuples candidates. Keys can be both added and deleted.
In the following example, we have a dictionary composed of key-value pairs where the key is a genre of fiction (string) and the value is a list of books (list) within that genre. Since a collection is still considered a single entity, we can use one to collect multiple variables or values into one key-value pair.
Can you write Python code for it?

my_dict = {"High Fantasy": ["Wheel of Time", "Lord of the Rings"],
           "Sci-fi": ["Book of the New Sun", "Neuromancer", "Snow Crash"],
           "Weird Fiction": ["At the Mountains of Madness", "The House on the Borderland"]}


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Pairing two sets together with | combines the sets, removing any repetitions to make every set element unique.
Can you write Python code for it?

print animal_set & things_i_like # You can also write things_i_like & animal_set with no difference


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Python does not have native matrices, but with lists we can produce a working fascimile. Other packages, such as numpy, add matrices as a separate data type, but in base Python the best way to create a matrix is to use a list of lists.
We can also use built-in functions to generate lists. In particular we will look at range() (because we will be using it later!). Range can take several different inputs and will return a list.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Remember that we can group multiple conditions together by using the logical operators!
Can you write Python code for it?

i = 5
j = 12
if i < 10 and j > 11:
    print '{0} is less than 10 and {1} is greater than 11! How novel and interesting!'.format(i, j)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Slicing
We use the colon (:) to slice lists.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
String Formatting
Using the format() method we can add in variable values and generally format our strings.
Can you write Python code for it?

my_string = "{0} {1}".format('Marco', 'Polo')
print my_string

my_string = "{1} {0}".format('Marco', 'Polo')
print my_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
String Shenanigans
We already know that strings are generally used for text. We can used built-in operations to combine, split, and format strings easily, depending on our needs.
The + symbol indicates concatenation in string language. It will combine two strings into a longer string.
Can you write Python code for it?

first_string = '"Beware the Jabberwock, my son! /The jaws that bite, the claws that catch! /'
second_string = 'Beware the Jubjub bird, and shun /The frumious Bandersnatch!"/'
third_string = first_string + second_string
print third_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Strings are also indexed much in the same way that lists are.
Can you write Python code for it?

my_string = 'Supercalifragilisticexpialidocious'
print 'The first letter is: ', my_string[0] # Uppercase S
print 'The last letter is: ', my_string[-1] # lowercase s
print 'The second to last letter is: ', my_string[-2] # lowercase u
print 'The first five characters are: ', my_string[0:5] # Remember: slicing doesn't include the final element!
print 'Reverse it!: ', my_string[::-1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The % symbol basically cues Python to create a placeholder. Whatever character follows the % (in the string) indicates what sort of type the value put into the placeholder will have. This character is called a conversion type. Once the string has been closed, we need another % that will be followed by the values to insert. In the case of one value, you can just put it there. If you are inserting more than one value, they must be enclosed in a tuple.
Can you write Python code for it?

print 'There are %s cats in my %s' % (13, 'apartment')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The continue statement will tell the loop to immediately end this iteration and continue onto the next iteration of the loop.
Can you write Python code for it?

i = 0
while i < 5:
    i += 1
    if i == 3:
        continue
    print i


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The int() function will also truncate any digits that a number may have after the decimal point!
Strings allow you to include text as a variable to operate on. They are defined using either single quotes ('') or double quotes ("").
Can you write Python code for it?

my_string = 'This is a string with single quotes'
print my_string
my_string = "This is a string with double quotes"
print my_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The iteritems() function creates a tuple of each key-value pair and the for loop stores unpacks that tuple into key, value on each separate execution of the loop!
Functions
A function is a reusable block of code that you can call repeatedly to make calculations, output data, or really do anything that you want. This is one of the key aspects of using a programming language. To add to the built-in functions in Python, you can define your own!
Can you write Python code for it?

def hello_world():
    """ Prints Hello, world! """
    print 'Hello, world!'

hello_world()

for i in range(5):
    hello_world()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The math library adds a long list of new mathematical functions to Python. Feel free to check out the documentation for the full list and details. It concludes some mathematical constants
Can you write Python code for it?

print 'Pi: ', math.pi
print "Euler's Constant: ", math.e


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The scope of a variable is the part of a block of code where that variable is tied to a particular value. Functions in Python have an enclosed scope, making it so that variables defined within them can only be accessed directly within them. If we pass those values to a return statement we can get them out of the function. This makes it so that the function call returns values so that you can store them in variables that have a greater scope.
In this case specifically,including a return statement allows us to keep the string value that we define in the function.
Can you write Python code for it?

def free_the_scope():
    in_function_string = "Anything you can do I can do better!"
    return in_function_string
my_string = free_the_scope()
print my_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
The time to use *args as a parameter for your function is when you do not know how many values may be passed to it, as in the case of our sum function. The asterisk in this case is the syntax that tells Python that you are going to pass an arbitrary number of parameters into your function. These parameters are stored in the form of a tuple.
Can you write Python code for it?

def test_args(*args):
    print type(args)

test_args(1, 2, 3, 4, 5, 6)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
There are also some methods that are unique to strings. The function upper() will convert all characters in a string to uppercase, while lower() will convert all characters in a string to lowercase!
Can you write Python code for it?

my_string = "I can't hear you"
print my_string.upper()
my_string = "I said HELLO"
print my_string.lower()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
There are two statements that are very helpful in dealing with both for and while loops. These are break and continue. If break is encountered at any point while a loop is executing, the loop will immediately end.
Can you write Python code for it?

i = 10
while True:
    if i == 14:
        break
    i += 1 # This is shorthand for i = i + 1. It increments i with each iteration.
    print i

for i in range(5):
    if i == 2:
        break
    print i


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
These comparators also work in conjunction with variables.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
This loop skips printing the number $3$ because of the continue statement that executes when we enter the if-statement. The code never sees the command to print the number $3$ because it has already moved to the next iteration. The break and continue statements are further tools to help you control the flow of your loops and, as a result, your code.
The variable that we use to iterate over a loop will retain its value when the loop exits. Similarly, any variables defined within the context of the loop will continue to exist outside of it.
Can you write Python code for it?

for i in range(5):
    loop_string = 'I transcend the loop!'
    print 'I am eternal! I am {0} and I exist everywhere!'.format(i)

print 'I persist! My value is {0}'.format(i)
print loop_string


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Tuples
A tuple is a data type similar to a list in that it can hold different kinds of data types. The key difference here is that a tuple is immutable. We define a tuple by separating the elements we want to include by commas. It is conventional to surround a tuple with parentheses.
Can you write Python code for it?

my_tuple = 'I', 'have', 30, 'cats'
print my_tuple

my_tuple = ('I', 'have', 30, 'cats')
print my_tuple


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Unpacking assigns each value of the tuple in order to each variable on the left hand side of the equals sign. Some functions, including user-defined functions, may return tuples, so we can use this to directly unpack them and access the values that we want.
Sets
A set is a collection of unordered, unique elements. It works almost exactly as you would expect a normal set of things in mathematics to work and is defined using braces ({}).
Can you write Python code for it?

things_i_like = {'dogs', 7, 'the number 4', 4, 4, 4, 42, 'lizards', 'man I just LOVE the number 4'}
print things_i_like, type(things_i_like)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
Using : we can select a group of elements in the list starting from the first element indicated and going up to  (but not including) the last element indicated.
We can also select everything after a certain point
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
We can 'pack' values together, creating a tuple (as above), or we can 'unpack' values from a tuple, taking them out.
Can you write Python code for it?

str_1, str_2, int_1 = my_other_tuple
print str_1, str_2, int_1


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
We can also change the value associated with a given key
Can you write Python code for it?

my_dict["Sci-fi"] = "I can't read"
print my_dict["Sci-fi"]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
We can also iterate over a dictionary!
Can you write Python code for it?

my_dict = {'firstname' : 'Inigo', 'lastname' : 'Montoya', 'nemesis' : 'Rugen'}

for key in my_dict:
    print key


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
We can see the number of elements in a list by calling the len() function.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
We can string these comparators together to make more complex logical statements using the logical operators or, and, and not.
Can you write Python code for it?

statement_1 = 10 > 2
statement_2 = 4 <= 6
print "Statement 1 truth value: {0}".format(statement_1)
print "Statement 2 truth value: {0}".format(statement_2)
print "Statement 1 and Statement 2: {0}".format(statement_1 and statement_2)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
With a negative step size we can even reverse the list!
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Python/notebook.ipynb".
The first task is:
You can also add a third component to slicing. Instead of simply indicating the first and final parts of your slice, you can specify the step size that you want to take. So instead of taking every single element, you can take every other element.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Getting Documentation Help
Placing a question mark after a function and executing that line of code will give you the documentation IPython has for that function. It's often best to do this in a new cell, as you avoid re-executing other code and running into bugs.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Getting Real Pricing Data
Randomly sampled data can be great for testing ideas, but let's get some real data. We can use get_pricing to do that. You can use the ? syntax as discussed above to get more information on get_pricing's arguments.
Can you write Python code for it?

data = get_pricing('MSFT', start_date='2012-1-1', end_date='2015-6-1')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Importing Libraries
The vast majority of the time, you'll want to use functions from pre-built libraries. You can't import every library on Quantopian due to security issues, but you can import most of the common scientific ones. Here I import numpy and pandas, the two most common and useful libraries in quant finance. I recommend copying this import statement to every new notebook.
Notice that you can rename libraries to whatever you want after importing. The as statement allows this. Here we use np and pd as aliases for numpy and pandas. This is a very common aliasing and will be found in most code snippets around the web. The point behind this is to allow you to type fewer characters when you are frequently accessing these libraries.
Can you write Python code for it?

import numpy as np
import pandas as pd

# This is a plotting library for pretty pictures.
import matplotlib.pyplot as plt


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Knowing When a Cell is Running
While a cell is running, a [*] will dsiplay on the left. When a cell has yet to be executed, [ ] will display. When it has been run, a number will display indicating the order in which it was run during the execution of the notebook [5]. Try on this cell and note it happening.
Can you write Python code for it?

#Take some time to run something
c = 0
for i in range(10000000):
    c = c + i
c


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Our data is now a dataframe. You can see the datetime index and the colums with different pricing data.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Plotting
We can use the plotting library we imported as follows.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Remember that only the result from the last line is printed.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Sometimes there is no result to be printed, as is the case with assignment.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Squelching Line Output
You might have noticed the annoying line of the form [<matplotlib.lines.Line2D at 0x7f72fdbc1710>] before the plots. This is because the .plot function actually produces output. Sometimes we wish not to display output, we can accomplish this with the semi-colon as follows.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
Tab Autocomplete
Pressing tab will give you a list of IPython's best guesses for what you might want to type next. This is incredibly valuable and will save you a lot of time. If there is only one possible option for what you could type next, IPython will fill that in for you. Try pressing tab very frequently, it will seldom fill in anything you don't want, as if there is ambiguity a list will be shown. This is a great way to see what functions are available in a library.
Try placing your cursor after the . and pressing tab.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
We can get statistics again on real data.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Research/notebook.ipynb".
The first task is:
We can plot the returns distribution as a histogram.
Can you write Python code for it?

plt.hist(R, bins=20)
plt.xlabel('Return')
plt.ylabel('Frequency')
plt.legend(['MSFT Returns']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Volume_Slippage_and_Liquidity/notebook.ipynb".
The first task is:
Broad Liquidity Considerations in Equities

Liquidity will vary by market cap. Higher cap stocks experience more absolute trading volume and are more liquid. Micro-cap stocks are
Some stocks are hard to borrow/short, and may experience one sided liquidity. Longs are easy to purchase, but shorts are hard to obtain. Quantopian does not currently provide hard to borrow data.

Liquidity in Futures

Futures have much higher liquidity per instrument compared to equities.
Liquidity varies by type/class of future. Certain types of futures will be far less illiquid. We'll show an example here.
Liquidity has structure depending on expiration date. The volume of trading of a future contract will increase over time up to the expiration date. Far in advance there will be little volume as most people will be trading more current contracts. This volume effect must be considered when trying to trade futures. We'll show an example of this here.
Can you write Python code for it?

crude_future_volume = get_pricing('CLM16', start_date=start_date, end_date=end_date, fields='volume')

spy_future_volume.plot()
crude_future_volume.plot()
plt.legend();

palladium_future_volume = get_pricing('PAM16', start_date=start_date, end_date=end_date, fields='volume')

crude_future_volume.plot()
palladium_future_volume.plot()
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Volume_Slippage_and_Liquidity/notebook.ipynb".
The first task is:
Let's look at how this volume is balanced out over a single day.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Volume_Slippage_and_Liquidity/notebook.ipynb".
The first task is:
Notice that just plotting the volume gives us lots of gaps. This is because we are plotting data with time on the x-axis, and equities are only traded during part of the day. The rest of the day we have no data for volume. There is a small amount of overnight trading that occurs for equities, but it is often only useful as a data source as the liquidity is too low to trade. Quantopian does not currently have data on overnight trading, just for regular market hours.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Volume_Slippage_and_Liquidity/notebook.ipynb".
The first task is:
Often in real datasets you will be given averaged or 'last traded' data rather than individual trades. With averaged data, some average is taken over a bar (time period). With last traded data, only the final observation from that bar is reported. It is important to know if the data is averaged, volume averaged, or simply last traded. All of these will need to be treated differently.
For pricing data, Quantopian currently (as of April 2017) provides the last traded prices at a minute resolution. The volume is the sum of all volume in that bar. While we do not offer minutely volume-weighted price, daily volume-weighted price can be approximated from the minute bars.
Let's look at some volume data.
Can you write Python code for it?

data = get_pricing('SPY', start_date='2015-6-1', end_date='2015-6-20', frequency='minute')

data.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Introduction_to_Volume_Slippage_and_Liquidity/notebook.ipynb".
The first task is:
This is pretty noisy, but with pandas we can easily take an average across all the days in our sample to get a sense of how volume fluctuates on average. Here is the average volume for a given minute over the whole time period.
Can you write Python code for it?

avg_minute_volume = data.groupby([data.index.hour, data.index.minute]).mean()['volume']

avg_minute_volume.plot()
plt.ylabel('Volume (USD)');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Leverage/notebook.ipynb".
The first task is:
Here is the leverage ratio of this algorithm plotted over time. Notice how it jumps around quite frequently. The ratio is below $1$ when it is not using all of its base capital and it spikes above $1$ whenever it makes a trade on margin. The algorithm associated with this leverage ratio is a long-short equity algorithm based on a combination of fundamental factors. For an overview of how a long-short equity strategy works, please see the lectures page.
A key feature of this sort of strategy is that it can trade hundreds, sometimes even thousands of equities at once. As such, we run the risk of incurring some fairly large rebalancing costs, depending on how frequently we rebalance. This algorithm specifically rebalances on a monthly basis. As we can see on the above graph of the leverage ratio, a lot of the largest changes occur aroud the start of each month.
To see how the rebalancing structure and maximum leverage can affect the leverage ratio of the algorithm when it is executed, go into the the template and modify these parameters. Changing the type of algorithm will also drastically affect how it uses leverage. Feel free to experiment.
Things to try:
1. Change the timing of the rebalancing between daily, monthly, and weekly.
2. Modify the amount of leverage that the portfolio is allowed to take on
3. Restrict the universe that the algorithm trades within by applying more filters to your trading universe.
4. Instead of making all portfolio weights equally-weighted, use a portfolio optimization scheme like Markowitz or Mean Absolute Deviation portfolio optimization.
Here are the associated returns of this algorithm. The volatility of these returns is important to take into account when examining the leverage of a strategy.
Can you write Python code for it?

plt.plot(daily_returns);
plt.xlabel('Date');
plt.ylabel('Daily Returns');
plt.legend();

print "Mean return: ", daily_returns.mean()
print "Volatility: ", daily_returns.std()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Leverage/notebook.ipynb".
The first task is:
This is what portfolio returns look like without leverage. Let's add some debt, leveraging the portfolio, and see how the returns change.
Can you write Python code for it?

debt = 100000

r_lvg = (capital_base + debt) * r_p
r_lvg_pct = r_lvg / capital_base
# Returns are calculated over the initial capital base
# Think of the debt as an asset purchased and added to the portfolio
lvg_ratio = (debt + capital_base) / capital_base
print 'Portfolio returns with leverage: {0}'.format(r_lvg)
print 'Percentage returns with {1}x leverage: {0}'.format(r_lvg_pct, lvg_ratio)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Leverage/notebook.ipynb".
The first task is:
Using leverage can be dangerous when you are dealing with a more volatile strategy. Because you are trading with borrowed money, we are on the hook to return it. We have to make sure that the broker gets his money back before we get our profit. If we end up in a position where we get a margin call, we have to pony up more funds if we want to hold our positions. Monitoring your strategy's volatility and ensuring you are only taking on palatable amounts of debt are key aspects of determining the quality of your trading strategy.
Risk-Adjusted Returns
Comparing returns of different investment opportunities without taking risk into account is meaningless. Some return streams may be higher than others, but this may be due to the risks taken on rather than any merit in the strategy itself. Taking on higher risk should in theory lead to higher returns, but then how do we judge the quality of these returns for the amount of risk we have to handle? This is where risk-adjusted returns and methods of risk-adjustment come into play. If we adjust several return streams for risk then we can consider them on equal footing, independent of the risk. This allows us to effectively compare and determine which return streams are the best for a given desired risk profile.`
One of the most prominent risk-adjusted measures is the Sharpe Ratio, defined as follows.
$$ \text{Sharpe Ratio} = \frac{r_p - r_f}{\sigma_p} $$
The Sharpe Ratio essentially normalizes the returns of a portfolio, giving us a metric that we can use as a measure of quality relative to other revenue streams. A higher Sharpe Ratio indicates that you are getting more return relative to the risk that your strategy is taking on.
Compare Strategies by Sharpe Ratio and then Lever as Needed
In general you want to compare the Sharpe Ratio of two strategies you may be interested in. Pick the strategy with the better Sharpe Ratio and then use leverage to multiply the returns up to where you want them. Assuming constraints like capital capacity don't kick in, you can add more...
Can you write Python code for it?

# Note these are all expected returns. You need to validate that
# your strategy will continue to produce these returns and volatility using other methods.
# Just measuring historically is a very poor predictor of future performance.
# We'll for now assume you've already validated that you can expect to keep seeing numbers similar to these.

strat_A_ann_return = 0.22
strat_A_ann_vol = 0.15

strat_B_ann_return = 0.05
strat_B_ann_vol = 0.02

# We'll assume a risk free rate of 0.02
risk_free_rate = 0.02

print 'Strategy A Sharpe: %s' % ((strat_A_ann_return - risk_free_rate) / strat_A_ann_vol)
print 'Strategy B Sharpe: %s' % ((strat_B_ann_return - risk_free_rate) / strat_B_ann_vol)

# Add in leverage to B

leverage = 3
# Expressed in returns

print 'Strategy B Sharpe: %s' % (
    (strat_B_ann_return * leverage - risk_free_rate * leverage) / (strat_B_ann_vol * leverage)
)
print 'Strategy B Levered Annual Returns: %s' % (
    (strat_B_ann_return * leverage - risk_free_rate * leverage)
)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Constructing a portfolio of uncorrelated assets
Another reason that correlation is useful in finance is that uncorrelated assets produce the best portfolios. The intuition for this is that if the assets are uncorrelated, a drawdown in one will not correspond with a drawdown in another. This leads to a very stable return stream when many uncorrelated assets are combined.
Limitations
Significance
It's hard to rigorously determine whether or not a correlation is significant, especially when, as here, the variables are not normally distributed. Their correlation coefficient is close to 1, so it's pretty safe to say that the two stock prices are correlated over the time period we use, but is this indicative of future correlation? If we examine the correlation of each of them with the S&P 500, we see that it is also quite high. So, AAPL and LRCX are slightly more correlated with each other than with the average stock.
One fundamental problem is that it is easy to datamine correlations by picking the right time period. To avoid this, one should compute the correlation of two quantities over many historical time periods and examine the distibution of the correlation coefficient. More details on why single point estimates are bad will be covered in future notebooks.
As an example, remember that the correlation of AAPL and LRCX from 2013-1-1 to 2015-1-1 was 0.95. Let's take the rolling 60 day correlation between the two to see how that varies.
Can you write Python code for it?

rolling_correlation = pd.rolling_corr(a1, a2, 60)
plt.plot(rolling_correlation)
plt.xlabel('Day')
plt.ylabel('60-day Rolling Correlation')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Finally, let's see what an inverse relationship looks like.
Can you write Python code for it?

X = np.random.rand(50)
Y = -X + np.random.normal(0, .1, 50)

plt.scatter(X,Y)
plt.xlabel('X Value')
plt.ylabel('Y Value')

print 'Correlation: ' + str(np.corrcoef(X, Y)[0, 1])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Let's dial down the relationship by introducing more noise.
Can you write Python code for it?

X = np.random.rand(50)
Y = X + np.random.normal(0, .2, 50)

plt.scatter(X,Y)
plt.xlabel('X Value')
plt.ylabel('Y Value')

print 'Correlation: ' + str(np.corrcoef(X, Y)[0, 1])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Non-Linear Relationships
The correlation coefficient can be useful for examining the strength of the relationship between two variables. However, it's important to remember that two variables may be associated in different, predictable ways which this analysis would not pick up. For instance, one variable might precisely follow the behavior of a second, but with a delay. There are techniques for dealing with this lagged correlation. Alternatively, a variable may be related to the rate of change of another. Neither of these relationships are linear, but can be very useful if detected.
Additionally, the correlation coefficient can be very sensitive to outliers. This means that including or excluding even a couple of data points can alter your result, and it is not always clear whether these points contain information or are simply noise.
As an example, let's make the noise distribution poisson rather than normal and see what happens.
Can you write Python code for it?

X = np.random.rand(100)
Y = X + np.random.poisson(size=100)

plt.scatter(X, Y)

np.corrcoef(X, Y)[0, 1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Now let's see what two correlated sets of data look like.
Can you write Python code for it?

X = np.random.rand(50)
Y = X + np.random.normal(0, 0.1, 50)

plt.scatter(X,Y)
plt.xlabel('X Value')
plt.ylabel('Y Value')

print 'Correlation: ' + str(np.corrcoef(X, Y)[0, 1])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
So now what? What does this mean? Correlation uses information about the variance of X and Y to normalize this metric. Once we've normalized the metric to the -1 to 1 scale, we can make meaningful statements and compare correlations.
To see how this is done consider the formula.
$$\frac{Cov(X, Y)}{std(X)std(Y)}$$
$$= \frac{Cov(X, Y)}{\sqrt{var(X)}\sqrt{var(Y)}}$$
$$= \frac{Cov(X, Y)}{\sqrt{Cov(X, X)}\sqrt{Cov(Y, Y)}}$$
To demonstrate this let's compare the correlation and covariance of two series.
Can you write Python code for it?

X = np.random.rand(50)
Y = 2 * X + 4

print 'Covariance of X and Y: \n' + str(np.cov(X, Y))
print 'Correlation of X and Y: \n' + str(np.corrcoef(X, Y))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
We see a little bit of rounding error, but they are clearly the same value.
How is this useful in finance?
Determining related assets
Once we've established that two series are probably related, we can use that in an effort to predict future values of the series. For example. let's loook at the price of Apple and a semiconductor equipment manufacturer, Lam Research Corporation.
Can you write Python code for it?

# Pull the pricing data for our two stocks and S&P 500
start = '2013-01-01'
end = '2015-01-01'
bench = get_pricing('SPY', fields='price', start_date=start, end_date=end)
a1 = get_pricing('LRCX', fields='price', start_date=start, end_date=end)
a2 = get_pricing('AAPL', fields='price', start_date=start, end_date=end)

plt.scatter(a1,a2)
plt.xlabel('LRCX')
plt.ylabel('AAPL')
plt.title('Stock prices from ' + start + ' to ' + end)
print "Correlation coefficients"
print "LRCX and AAPL: ", np.corrcoef(a1,a2)[0,1]
print "LRCX and SPY: ", np.corrcoef(a1,bench)[0,1]
print "AAPL and SPY: ", np.corrcoef(bench,a2)[0,1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Correlation_Analysis/notebook.ipynb".
The first task is:
Why do both np.cov and np.corrcoef return matricies?
The covariance matrix is an important concept in statistics. Often people will refer to the covariance of two variables $X$ and $Y$, but in reality that is just one entry in the covariance matrix of $X$ and $Y$. For each input variable we have one row and one column. The diagonal is just the variance of that variable, or $Cov(X, X)$, entries off the diagonal are covariances between different variables. The matrix is symmetric across the diagonal. Let's check that this is true.
Can you write Python code for it?

cov_matrix = np.cov(X, Y)

# We need to manually set the degrees of freedom on X to 1, as numpy defaults to 0 for variance
# This is usually fine, but will result in a slight mismatch as np.cov defaults to 1
error = cov_matrix[0, 0] - X.var(ddof=1)

print 'error: ' + str(error)

X = np.random.rand(50)
Y = np.random.rand(50)

plt.scatter(X,Y)
plt.xlabel('X Value')
plt.ylabel('Y Value')

# taking the relevant value from the matrix returned by np.cov
print 'Correlation: ' + str(np.cov(X,Y)[0,1]/(np.std(X)*np.std(Y)))
# Let's also use the builtin correlation function
print 'Built-in Correlation: ' + str(np.corrcoef(X, Y)[0, 1])


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Regression/notebook.ipynb".
The first task is:
First we'll define a function that performs linear regression and plots the results.
Can you write Python code for it?

def linreg(X,Y):
    # Running the linear regression
    X = sm.add_constant(X)
    model = regression.linear_model.OLS(Y, X).fit()
    a = model.params[0]
    b = model.params[1]
    X = X[:, 1]

    # Return summary of the regression and plot results
    X2 = np.linspace(X.min(), X.max(), 100)
    Y_hat = X2 * b + a
    plt.scatter(X, Y, alpha=0.3) # Plot the raw data
    plt.plot(X2, Y_hat, 'r', alpha=0.9);  # Add the regression line, colored in red
    plt.xlabel('X Value')
    plt.ylabel('Y Value')
    return model.summary()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Linear_Regression/notebook.ipynb".
The first task is:
In a situation like the above, the line of best fit does indeed model the dependent variable Y quite well (with a high $R^2$ value).
Evaluating and reporting results
The regression model relies on several assumptions:
* The independent variable is not random.
* The variance of the error term is constant across observations. This is important for evaluating the goodness of the fit.
* The errors are not autocorrelated. The Durbin-Watson statistic detects this; if it is close to 2, there is no autocorrelation.
* The errors are normally distributed. If this does not hold, we cannot use some of the statistics, such as the F-test.
If we confirm that the necessary assumptions of the regression model are satisfied, we can safely use the statistics reported to analyze the fit. For example, the $R^2$ value tells us the fraction of the total variation of $Y$ that is explained by the model.
When making a prediction based on the model, it's useful to report not only a single value but a confidence interval. The linear regression reports 95% confidence intervals for the regression parameters, and we can visualize what this means using the seaborn library, which plots the regression line and highlights the 95% (by default) confidence interval for the regression line:
Can you write Python code for it?

import seaborn

start = '2014-01-01'
end = '2015-01-01'
asset = get_pricing('TSLA', fields='price', start_date=start, end_date=end)
benchmark = get_pricing('SPY', fields='price', start_date=start, end_date=end)

# We have to take the percent changes to get to returns
# Get rid of the first (0th) element because it is NAN
r_a = asset.pct_change()[1:]
r_b = benchmark.pct_change()[1:]

seaborn.regplot(r_b.values, r_a.values);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Long-Short_Equity/notebook.ipynb".
The first task is:
Market Neutrality is Built-In
The nice thing about making money based on the spread of the ranking is that it is unaffected by what the market does.
Can you write Python code for it?

# We'll generate a random factor
current_factor_values = np.random.normal(0, 1, 10000)
equity_names = ['Equity ' + str(x) for x in range(10000)]
# Put it into a dataframe
factor_data = pd.Series(current_factor_values, index = equity_names)
factor_data = pd.DataFrame(factor_data, columns=['Factor Value'])

# Now let's say our future returns are dependent on our factor values
future_returns = -10 + current_factor_values + np.random.normal(0, 1, 10000)

returns_data = pd.Series(future_returns, index=equity_names)
returns_data = pd.DataFrame(returns_data, columns=['Returns'])
# Put both the factor values and returns into one dataframe
data = returns_data.join(factor_data)

# Rank the equities
ranked_data = data.sort('Factor Value')

# Compute the returns of each basket
# Baskets of size 500, so we create an empty array of shape (10000/500
number_of_baskets = 10000/500
basket_returns = np.zeros(number_of_baskets)

for i in range(number_of_baskets):
    start = i * 500
    end = i * 500 + 500
    basket_returns[i] = ranked_data[start:end]['Returns'].mean()

basket_returns[number_of_baskets-1] - basket_returns[0]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Long-Short_Equity/notebook.ipynb".
The first task is:
Now that we have factor values and returns, we can see what would happen if we ranked our equities based on factor values, and then entered the long and short positions.
Can you write Python code for it?

# Rank the equities
ranked_data = data.sort('Factor Value')

# Compute the returns of each basket
# Baskets of size 500, so we create an empty array of shape (10000/500)
number_of_baskets = 10000/500
basket_returns = np.zeros(number_of_baskets)

for i in range(number_of_baskets):
    start = i * 500
    end = i * 500 + 500
    basket_returns[i] = ranked_data[start:end]['Returns'].mean()

# Plot the returns of each basket
plt.bar(range(number_of_baskets), basket_returns)
plt.ylabel('Returns')
plt.xlabel('Basket')
plt.legend(['Returns of Each Basket']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
Exponential Distribution
Let's do the same thing, but for the exponential distribution. We'll start by sampling some data.
Can you write Python code for it?

TRUE_LAMBDA = 5
X = np.random.exponential(TRUE_LAMBDA, 1000)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
MLE for Asset Returns
Now we'll fetch some real returns and try to fit a normal distribution to them using MLE.
Can you write Python code for it?

prices = get_pricing('TSLA', fields='price', start_date='2014-01-01', end_date='2015-01-01')
# This will give us the number of dollars returned each day
absolute_returns = np.diff(prices)
# This will give us the percentage return over the last day's value
# the [:-1] notation gives us all but the last item in the array
# We do this because there are no returns on the final price in the array.
returns = absolute_returns/prices[:-1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
Now let's plot the distribution PDF along with the data to see how well it fits. We can do that by accessing the pdf provided in scipy.stats.norm.pdf.
Can you write Python code for it?

pdf = scipy.stats.norm.pdf
# We would like to plot our data along an x-axis ranging from 0-80 with 80 intervals
# (increments of 1)
x = np.linspace(0, 80, 80)
plt.hist(X, bins=x, normed='true')
plt.plot(pdf(x, loc=mu, scale=std))
plt.xlabel('Value')
plt.ylabel('Observed Frequency')
plt.legend(['Fitted Distribution PDF', 'Observed Data', ]);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
Now let's try our functions out on our sample data and see how they compare to the built-in np.mean and np.std
Can you write Python code for it?

print "Mean Estimation"
print normal_mu_MLE(X)
print np.mean(X)
print "Standard Deviation Estimation"
print normal_sigma_MLE(X)
print np.std(X)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
Of course, this fit is meaningless unless we've tested that they obey a normal distribution first. We can test this using the Jarque-Bera normality test. The Jarque-Bera test will reject the hypothesis of a normal distribution if the p-value is under a c
Can you write Python code for it?

from statsmodels.stats.stattools import jarque_bera
jarque_bera(returns)

jarque_bera(np.random.normal(0, 1, 100))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Maximum_Likelihood_Estimation/notebook.ipynb".
The first task is:
numpy defines the exponential distribution as
$$\frac{1}{\lambda}e^{-\frac{x}{\lambda}}$$
So we need to invert the MLE from the lecture notes. There it is
$$\hat\lambda = \frac{T}{\sum_{t=1}^{T} x_t}$$
Here it's just the reciprocal, so
$$\hat\lambda = \frac{\sum_{t=1}^{T} x_t}{T}$$
Can you write Python code for it?

def exp_lamda_MLE(X):
    T = len(X)
    s = sum(X)
    return s/T

print "lambda estimate: " + str(exp_lamda_MLE(X))

# The scipy version of the exponential distribution has a location parameter
# that can skew the distribution. We ignore this by fixing the location
# parameter to 0 with floc=0
_, l = scipy.stats.expon.fit(X, floc=0)

pdf = scipy.stats.expon.pdf
x = range(0, 80)
plt.hist(X, bins=x, normed='true')
plt.plot(pdf(x, scale=l))
plt.xlabel('Value')
plt.ylabel('Observed Frequency')
plt.legend(['Fitted Distribution PDF', 'Observed Data', ]);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Mean_Reversion_on_Futures/notebook.ipynb".
The first task is:
A coefficient of around $-0.02$ on the lagged futures return implies that if the oil price increased by 1% yesterday, the pure-play refiner is expected to go down by $2$ bp today. This would require more testing to formulate a functioning model, but it indicates that there might be some signal in drawing out the underreaction or overreaction of equity prices to changes in futures prices.
This conjecture could be total data mining, but perhaps when the connection between the futures and stock is exceedingly obvious, like oil stocks and oil exploration companies or gold stocks and gold miners, the market overreacts to fundamental information, but when the relationship is more subtle, the market underreacts.
Also, there may be other lead/lag effects over longer time scales than one-day, but as always, this could also lead to more data mining.
Can you write Python code for it?

data['futures_lag_diff'].plot(alpha=0.50, legend=True)
data['stock_excess'].plot(alpha=0.50, legend=True);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Mean_Reversion_on_Futures/notebook.ipynb".
The first task is:
And when we form a linear regression of the excess returns of XOM on the lagged futures returns, the coefficient is significant and negative. This and the above correlations indicate that there might be a slight overreaction to the shift in oil prices.
Can you write Python code for it?

result = sm.OLS(data['stock_excess'], sm.add_constant(data['futures_lag_diff'])).fit()
result.summary2()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Mean_Reversion_on_Futures/notebook.ipynb".
The first task is:
However, from looking at the p-value for our test, we conclude that soybean meal and soybean meal and soybean oil are not cointegrated.
Can you write Python code for it?

print 'p-value: ', coint(sm_price, bo_price)[1]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Mean_Reversion_on_Futures/notebook.ipynb".
The first task is:
We have a high positive contemporaneous correlation, but a slightly negative lagged correlation.
Can you write Python code for it?

#Compute contemporaneous correlation
contemp_corr = data['stock_excess'].shift(1).corr(data['futures_lag_diff'])
#Compute correlation of excess stock returns with lagged futures returns
lagged_corr = data['stock_excess'].corr(data['futures_lag_diff'])
print 'Contemporaneous correlation: ', contemp_corr
print 'Lagged correlation         : ', lagged_corr


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Mean_Reversion_on_Futures/notebook.ipynb".
The first task is:
We still have this compelling economic link, though. Both soybean oil and soybean meal have a root product in soybeans themselves. Let's see if we can suss out any signal by creating a spread between soybean prices and the refined products together, by implementing the crush spread.
Can you write Python code for it?

sm_future = continuous_future('SM', offset=1, roll='calendar', adjustment='mul')
sm_price = history(sm_future, fields='price', start_date='2014-01-01', end_date='2017-01-01')

bo_future = continuous_future('BO', offset=1, roll='calendar', adjustment='mul')
bo_price = history(bo_future, fields='price', start_date='2014-01-01', end_date='2017-01-01')

sy_future = continuous_future('SY', offset=0, roll='calendar', adjustment='mul')
sy_price = history(sy_future, fields='price', start_date='2014-01-01', end_date='2017-01-01')

crush = sy_price - (sm_price + bo_price)
crush.plot()
plt.ylabel('Crush Spread');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Means/notebook.ipynb".
The first task is:
For data that can take on many different values, such as returns data, there may not be any values that appear more than once. In this case we can bin values, like we do when constructing a histogram, and then find the mode of the data set where each value is replaced with the name of its bin. That is, we find which bin elements fall into most often.
Can you write Python code for it?

# Get return data for an asset and compute the mode of the data set
start = '2014-01-01'
end = '2015-01-01'
pricing = get_pricing('SPY', fields='price', start_date=start, end_date=end)
returns = pricing.pct_change()[1:]
print 'Mode of returns:', mode(returns)

# Since all of the returns are distinct, we use a frequency distribution to get an alternative mode.
# np.histogram returns the frequency distribution over the bins as well as the endpoints of the bins
hist, bins = np.histogram(returns, 20) # Break data up into 20 bins
maxfreq = max(hist)
# Find all of the bins that are hit with frequency maxfreq, then print the intervals corresponding to them
print 'Mode of bins:', [(bins[i], bins[i+1]) for i, j in enumerate(hist) if j == maxfreq]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Means/notebook.ipynb".
The first task is:
Harmonic mean
The harmonic mean is less commonly used than the other types of means. It is defined as
$$ H = \frac{n}{\sum_{i=1}^n \frac{1}{X_i}} $$
As with the geometric mean, we can rewrite the harmonic mean to look like an arithmetic mean. The reciprocal of the harmonic mean is the arithmetic mean of the reciprocals of the observations:
$$ \frac{1}{H} = \frac{\sum_{i=1}^n \frac{1}{X_i}}{n} $$
The harmonic mean for nonnegative numbers $X_i$ is always at most the geometric mean (which is at most the arithmetic mean), and they are equal only when all of the observations are equal.
Can you write Python code for it?

print 'Harmonic mean of x1:', stats.hmean(x1)
print 'Harmonic mean of x2:', stats.hmean(x2)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Means/notebook.ipynb".
The first task is:
The geometric mean is defined so that if the rate of return over the whole time period were constant and equal to $R_G$, the final price of the security would be the same as in the case of returns $R_1, \ldots, R_T$.
Can you write Python code for it?

T = len(returns)
init_price = pricing[0]
final_price = pricing[T]
print 'Initial price:', init_price
print 'Final price:', final_price
print 'Final price as computed with R_G:', init_price*(1 + R_G)**T


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Means/notebook.ipynb".
The first task is:
We can also define a <i>weighted</i> arithmetic mean, which is useful for explicitly specifying the number of times each observation should be counted. For instance, in computing the average value of a portfolio, it is more convenient to say that 70% of your stocks are of type X rather than making a list of every share you hold.
The weighted arithmetic mean is defined as
$$\sum_{i=1}^n w_i X_i $$
where $\sum_{i=1}^n w_i = 1$. In the usual arithmetic mean, we have $w_i = 1/n$ for all $i$.
Median
The median of a set of data is the number which appears in the middle of the list when it is sorted in increasing or decreasing order. When we have an odd number $n$ of data points, this is simply the value in position $(n+1)/2$. When we have an even number of data points, the list splits in half and there is no item in the middle; so we define the median as the average of the values in positions $n/2$ and $(n+2)/2$.
The median is less affected by extreme values in the data than the arithmetic mean. It tells us the value that splits the data set in half, but not how much smaller or larger the other values are.
Can you write Python code for it?

print 'Median of x1:', np.median(x1)
print 'Median of x2:', np.median(x2)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Means/notebook.ipynb".
The first task is:
What if we want to compute the geometric mean when we have negative observations? This problem is easy to solve in the case of asset returns, where our values are always at least $-1$. We can add 1 to a return $R_t$ to get $1 + R_t$, which is the ratio of the price of the asset for two consecutive periods (as opposed to the percent change between the prices, $R_t$). This quantity will always be nonnegative. So we can compute the geometric mean return,
$$ R_G = \sqrt[T]{(1 + R_1)\ldots (1 + R_T)} - 1$$
Can you write Python code for it?

# Add 1 to every value in the returns array and then compute R_G
ratios = returns + np.ones(len(returns))
R_G = stats.gmean(ratios) - 1
print 'Geometric mean of returns:', R_G


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Multiple_Linear_Regression/notebook.ipynb".
The first task is:
Multiple linear regression generalizes linear regression, allowing the dependent variable to be a linear function of multiple independent variables. As before, we assume that the variable $Y$ is a linear function of $X_1,\ldots, X_k$:
$$ Y_i = \beta_0 + \beta_1 X_{1i} + \ldots + \beta_k X_{ki} + \epsilon_i $$
Often in finance the form will be written as follows, but it is just the variable name that changes and otherwise the model is identical.
$$ Y_i = \alpha + \beta_1 X_{1i} + \ldots + \beta_k X_{ki} + \epsilon_i $$
For observations $i = 1,2,\ldots, n$. In order to find the plane (or hyperplane) of best fit, we will use the method of ordinary least-squares (OLS), which seeks to minimize the squared error between predictions and observations, $\sum_{i=1}^n \epsilon_i^2$. The square makes positive and negative errors equally bad, and magnifies large errors. It also makes the closed form math behind linear regression nice, but we won't go into that now. For an example of squared error, see the following.
Let's say Y is our actual data, and Y_hat is the predictions made by linear regression.
Can you write Python code for it?

Y = np.array([1, 3.5, 4, 8, 12])
Y_hat = np.array([1, 3, 5, 7, 9])

print 'Error ' + str(Y_hat - Y)

# Compute squared error
SE = (Y_hat - Y) ** 2

print 'Squared Error ' + str(SE)
print 'Sum Squared Error ' + str(np.sum(SE))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Multiple_Linear_Regression/notebook.ipynb".
The first task is:
Once we have used this method to determine the coefficients of the regression, we will be able to use new observed values of $X$ to predict values of $Y$.
Each coefficient $\beta_j$ tells us how much $Y_i$ will change if we change $X_j$ by one while holding all of the other dependent variables constant. This lets us separate out the contributions of different effects. This is assuming the linear model is the correct one.
We start by artificially constructing a $Y$, $X_1$, and $X_2$ in which we know the precise relationship.
Can you write Python code for it?

# Construct a simple linear curve of 1, 2, 3, ...
X1 = np.arange(100)

# Make a parabola and add X1 to it, this is X2
X2 = np.array([i ** 2 for i in range(100)]) + X1

# This is our real Y, constructed using a linear combination of X1 and X2
Y = X1 + X2

plt.plot(X1, label='X1')
plt.plot(X2, label='X2')
plt.plot(Y, label='Y')
plt.legend();


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Multiple_Linear_Regression/notebook.ipynb".
The first task is:
The same care must be taken with these results as with partial derivatives. The formula for $Y$ is ostensibly
$$X_1 + X_2 = X_1 + X^2 + X_1 = 2 X_1 + X^2$$
Or $2X_1$ plus a parabola.
However, the coefficient of $X_1$ is 1. That is because $Y$ changes by 1 if we change $X_1$ by 1 <i>while holding $X_2$ constant</i>. Multiple linear regression separates out contributions from different variables.
Similarly, running a linear regression on two securities might give a high $\beta$. However, if we bring in a third security (like SPY, which tracks the S&P 500) as an independent variable, we may find that the correlation between the first two securities is almost entirely due to them both being correlated with the S&P 500. This is useful because the S&P 500 may then be a more reliable predictor of both securities than they were of each other. This method allows us to better gauge the significance between the two securities and the problem is known as confounding.
Can you write Python code for it?

# Load pricing data for two arbitrarily-chosen assets and SPY
start = '2014-01-01'
end = '2015-01-01'
asset1 = get_pricing('DTV', fields='price', start_date=start, end_date=end)
asset2 = get_pricing('FISV', fields='price', start_date=start, end_date=end)
benchmark = get_pricing('SPY', fields='price', start_date=start, end_date=end)

# First, run a linear regression on the two assets
slr = regression.linear_model.OLS(asset1, sm.add_constant(asset2)).fit()
print 'SLR beta of asset2:', slr.params[1]

# Run multiple linear regression using asset2 and SPY as independent variables
mlr = regression.linear_model.OLS(asset1, sm.add_constant(np.column_stack((asset2, benchmark)))).fit()

prediction = mlr.params[0] + mlr.params[1]*asset2 + mlr.params[2]*benchmark
prediction.name = 'Prediction'

print 'MLR beta of asset2:', mlr.params[1], '\nMLR beta of S&P 500:', mlr.params[2]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Multiple_Linear_Regression/notebook.ipynb".
The first task is:
We can use the same function from statsmodels as we did for a single linear regression lecture.
Can you write Python code for it?

# Use column_stack to combine independent variables, then add a column of ones so we can fit an intercept
X = sm.add_constant( np.column_stack( (X1, X2) ) )

# Run the model
results = regression.linear_model.OLS(Y, X).fit()

print 'Beta_0:', results.params[0]
print 'Beta_1:', results.params[1]
print 'Beta_2:', results.params[2]


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Plotting_Data/notebook.ipynb".
The first task is:
Indexing into the data with data['AAPL'] will yield an error because the type of the columns are equity objects and not simple strings. Let's change that using this little piece of Python code. Don't worry about understanding it right now, unless you do, in which case congratulations.
Can you write Python code for it?

data.columns = [e.symbol for e in data.columns]
data.head()


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Plotting_Data/notebook.ipynb".
The first task is:
Line graph
A line graph can be used when we want to track the development of the y value as the x value changes. For instance, when we are plotting the price of a stock, showing it as a line graph instead of just plotting the data points makes it easier to follow the price over time. This necessarily involves "connecting the dots" between the data points, which can mask out changes that happened between the time we took measurements.
Can you write Python code for it?

plt.plot(data['MSFT'])
plt.plot(data['AAPL'])
plt.ylabel('Price')
plt.legend(['MSFT', 'AAPL']);

# Remove the first element because percent change from nothing to something is NaN
R = data['MSFT'].pct_change()[1:]

plt.plot(R)
plt.ylabel('Return')
plt.title('MSFT Returns');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Plotting_Data/notebook.ipynb".
The first task is:
Returns Histogram
In finance rarely will we look at the distribution of prices. The reason for this is that prices are non-stationary and move around a lot. For more info on non-stationarity please see this lecture. Instead we will use daily returns. Let's try that now.
Can you write Python code for it?

# Remove the first element because percent change from nothing to something is NaN
R = data['MSFT'].pct_change()[1:]

# Plot a histogram using 20 bins
plt.hist(R, bins=20)
plt.xlabel('Return')
plt.ylabel('Number of Days Observed')
plt.title('Frequency Distribution of MSFT Returns, 2014');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Plotting_Data/notebook.ipynb".
The first task is:
Scatter plot
A scatter plot is useful for visualizing the relationship between two data sets. We use two data sets which have some sort of correspondence, such as the date on which the measurement was taken. Each point represents two corresponding values from the two data sets. However, we don't plot the date that the measurements were taken on.
Can you write Python code for it?

plt.scatter(data['MSFT'], data['AAPL'])
plt.xlabel('MSFT')
plt.ylabel('AAPL')
plt.title('Daily Prices in 2014');

R_msft = data['MSFT'].pct_change()[1:]
R_aapl = data['AAPL'].pct_change()[1:]

plt.scatter(R_msft, R_aapl)
plt.xlabel('MSFT')
plt.ylabel('AAPL')
plt.title('Daily Returns in 2014');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Plotting_Data/notebook.ipynb".
The first task is:
The graph above shows, for example, that the daily returns of MSFT were above 0.03 on fewer than 5 days in 2014. Note that we are completely discarding the dates corresponding to these returns.
IMPORTANT: Note also that this does not imply that future returns will have the same distribution.
Cumulative Histogram (Discrete Estimated CDF)
An alternative way to display the data would be using a cumulative distribution function, in which the height of a bar represents the number of observations that lie in that bin or in one of the previous ones. This graph is always nondecreasing since you cannot have a negative number of observations. The choice of graph depends on the information you are interested in.
Can you write Python code for it?

# Remove the first element because percent change from nothing to something is NaN
R = data['MSFT'].pct_change()[1:]

# Plot a histogram using 20 bins
plt.hist(R, bins=20, cumulative=True)
plt.xlabel('Return')
plt.ylabel('Number of Days Observed')
plt.title('Cumulative Distribution of MSFT Returns, 2014');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
And now we can easily perform the same calculations that we did before to determine overall portfolio value and risk.
Can you write Python code for it?

P = np.dot(w, S)
var_p = np.dot(np.dot(w, cov), w.T)
sigma_p = np.sqrt(var_p)
print "Portfolio Value: ", P
print "Portfolio Volatility: ", sigma_p


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Here you can see the portfolio accompanied by all the assets, the assets being drawn much softer. The important thing to note is that the portfolio undergoes all the same shocks as the assets, because when one asset is up or down, all the others are likely to be so as well. This is the problem with correlated assets. Let's take a look at the volatility of the assets and the volatility of the portfolio.
The mean volatility of our assets is the same as the portfolio volatility. We haven't gained anything by making more bets. You can think of correlated bets as identical to the original bet. If the outcome of the second bet is correlated with the first, then really you've just made the same bet twice and you haven't reduced your volatility.
Case 3: Investing in Many Uncorrelated Assets
In this case we independently generate a bunch of assets an construct a portfolio that combines all of them.
Can you write Python code for it?

N = 10

assets = np.zeros((N, 100))
returns = np.zeros((N, 100))

for i in range(N):
    R_i = np.random.normal(1.01, 0.03, 100)
    returns[i] = R_i
    assets[i] = np.cumprod(R_i)

    plt.plot(assets[i], alpha=0.1)

R_P = np.mean(returns, axis=0)
P = np.mean(assets, axis=0)
plt.plot(P)
plt.xlabel('Time')
plt.ylabel('Price');

print 'Asset Volatilities'
print [np.std(R) for R in returns]
print 'Mean Asset Volatility'
print np.mean([np.std(R) for R in returns])
print 'Portfolio Volatility'
print np.std(R_P)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
In this case we're much safer, as the expected edge is 100 games.
NOTE: There is a subtlety that it's not always valid to use a standard deviation, as the underlying distribution of data in this case is not normal. We use it here because standard deviation is the metric of volatility used in finance, and it still reflects how much 'spread' exists in the data. Be careful not to abuse standard deviation in practice by assuming the underlying data is normal.
Expanding to Portfolio Theory
The same exact principle exists in portfolio theory. If you think you have an edge over the market in picking stocks that will go up or down, you should try to make as many independent bets as possible. This can be accomplished by investing in as many uncorrelated assets as possible. Let's take a look at an example.
Remember that in finance, volatility is measured by the standard deviation of a time series, and the amount of future risk of a portfolio is estimated by past portfolio volatility.
Case 1: Investing in Few Assets
Let's simulate some assets by sampling from a normal distribution.
NOTE: In practice real financial asset returns rarely are normally distributed, so this is not a great assumption. However it's okay here to get our point across because we are just concerned with correlation and level of volaility.
Can you write Python code for it?

R_1 = np.random.normal(1.01, 0.03, 100)
A_1 = np.cumprod(R_1)
P = A_1
plt.plot(P)
plt.xlabel('Time')
plt.ylabel('Price');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
In this case, we're totally exposed to the volatility of that asset, as our portfolio is entirely that asset.
Case 2: Investing in Many Correlated Assets
In this case we expand our asset pool, but there is still a large amount of pairwise correlation between the returns. We simulate this by simulating assets 2 through N as asset 1 plus some noise.
Can you write Python code for it?

N = 10

returns = np.zeros((N, 100))
assets = np.zeros((N, 100))

R_1 = np.random.normal(1.01, 0.03, 100)
returns[0] = R_1
assets[0] = np.cumprod(R_1)
plt.plot(assets[0], alpha=0.1)

for i in range(1, N):
    R_i = R_1 + np.random.normal(0.001, 0.01, 100)
    returns[i] = R_i
    assets[i] = np.cumprod(R_i)

    plt.plot(assets[i], alpha=0.1)

R_P = np.mean(returns, axis=0)
P = np.mean(assets, axis=0)
plt.plot(P)
plt.xlabel('Time')
plt.ylabel('Price');

print 'Asset Volatilities'
print [np.std(R) for R in returns]
print 'Mean Asset Volatility'
print np.mean([np.std(R) for R in returns])
print 'Portfolio Volatility'
print np.std(R_P)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Intuition
Let's say you learned to card count at Blackjack, whereas most casinos will throw you out if you are caught, it will give you a 1% edge over the house. If you walked into the casino with \$10,000 to bet, it would clearly be insane to place all the money on one game. Whereas you have a 51% chance of winning that game, the house still has a 49% chance. The expected outcome is for you to win the game, but the variance is increadibly high.
Let's say you placed your money on 100 different tables. This is known as making independent bets, because the outcome of one table doesn't affect any of the others. Your variance will be reduced as you make more and more bets. You would still expect to win 51% of the tables, but the chance of losing money is greatly reduced. Let's see this in action.
Simulating Blackjack Games
Each game will be won with a 51% probability. We can simulate this using a binomial distribution, which is parameterized with the number of trials we perform (games), and the chance of each trial succeeding.
First we'll simulate 1000 different universes in which you walk into the casino and play one game.
Can you write Python code for it?

universes = 1000

results = np.zeros((universes, 1))
for i in range(universes):
    results[i] = np.random.binomial(n = 1, p=0.51)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Now let's assume that the correlations between all of these assets are $0$, that they are all pairwise independent. We will assume that everything else remains the same as before. Using our calcualtions for the portfolio's risk we get:
Can you write Python code for it?

covariance_matrix = np.array([[0.05**2, 0, 0, 0],
                             [0, 0.07**2, 0, 0],
                             [0, 0, 0.11**2, 0],
                             [0, 0, 0, 0.09**2]])
var_p = np.dot(np.dot(weights, covariance_matrix), weights.T).flatten()
sigma_p = np.sqrt(var_p)
print "The overall risk of the portfolio is: ", sigma_p


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Now let's check the mean and standard deviation of the results. We see that because there are so many 0s and so many 1s, and nothing in between, the standard deviation is very high. This is saying that you should expect to win half a game, with the potential outcomes being approximately evenly distributed between a loss and a win. Because you played so few games, you have given no time for your edge to work.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Now let's simulate 1000 universes in which you walk into the casino and play 100 games.
Can you write Python code for it?

universes = 1000

results = np.zeros((universes, 1))
for i in range(universes):
    results[i] = np.random.binomial(n = 100, p=0.51)

np.mean(results), np.std(results)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Position_Concentration_Risk/notebook.ipynb".
The first task is:
Now we see that the average result is much closer to 51 games won, with a smaller standard deviation. We see here that you're likely still not safe, as your expected edge is only one game, whereas the standard deviation is many games. This would indicate that you can reasonably expect to lose more games than you win. Finally let's try 10,000 games.
Can you write Python code for it?

universes = 1000

results = np.zeros((universes, 1))
for i in range(universes):
    results[i] = np.random.binomial(n = 10000, p=0.51)

np.mean(results), np.std(results)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Again, as in all cases of sampling, the more samples that you take, the more consistent your resulting distribution looks:
Can you write Python code for it?

plt.hist(StockProbabilities.draw(10000), bins = [0, 1, 2, 3, 4, 5, 6], align = 'left')
plt.xlabel('Value')
plt.ylabel('Occurences');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
As before in the discrete uniform case, the continuous uniform distribution PDF is constant for all values the variable can take on. The only difference here is that we cannot take the probability for any individual point. The CDF, which we get from integrating the PDF is:
$$ F(x) = \begin{cases} 0 & \text{for $x \leq a$} \ \frac{x - a}{b - a} & \text{for $a < x < b$} \ 1 & \text{for $x \geq b$}\end{cases}$$
And is plotted on the same interval as the PDF as:
Can you write Python code for it?

y = [(i - a)/(b - a) for i in x]
plt.plot(x, y)
plt.xlabel('Value')
plt.ylabel('Probability');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
By changing the mean and standard deviation of the normal distribution, we can change the depth and width of the bell curve. With a larger standard deviation, the values of the distribution are less concentrated around the mean.
Rather than using normal distribution to model stock prices, we use it to model returns. Stock prices cannot go below $0$ while the normal distribution can take on all values on the real line, making it better suited to returns. Given the mean and variance of a normal distribution, we can make the following statements:

Around $68\%$ of all observations fall within one standard deviations around the mean ($\mu \pm \sigma$)
Around $95\%$ of all observations fall within two standard deviations around the mean ($\mu \pm 2\sigma$)
Around $99\%$ of all observations fall within three standard deviations aroud the mean ($\mu \pm 3\sigma$)

These values are important for understanding confidence intervals as they relate to the normal distribution. When considering the mean and variance of a sample distribution, we like to look at different confidence intervals around the mean.
Using the central limit theorem, we can standardize different random variables so that they become normal random variables. A very common tool in statistics is a standard normal probability table, used for looking up the values of the standard normal CDF for given values of $x$. By changing random variables into a standard normal we can simply check these tables for probability values. We standardize a random variable $X$ by subtracting the mean and dividing by the variance, resulting in the standard normal random variable $Z$.
$$
Z = \frac{X - \mu}{\sigma}
$$
Let's look at the case where $X$ ~ $B(n, p)$ is a binomial random variable. In the case of a binomial random variable, the mean is $\mu = np$ and the variance is $\sigma^2 = np(1 - p)$.
Can you write Python code for it?

n = 50
p = 0.25
X = BinomialRandomVariable(n, p)
X_samples = X.draw(10000)
Z_samples = (X_samples - n * p) / np.sqrt(n * p * (1 - p))

plt.hist(X_samples, bins = range(0, n + 2), align = 'left')
plt.xlabel('Value')
plt.ylabel('Probability');

plt.hist(Z_samples, bins=20)
plt.xlabel('Value')
plt.ylabel('Probability');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Changing the value of $p$ from $0.50$ to $0.25$ clearly makes our distribution asymmetric. We can extend this idea of stock price moving with a binomial random variable into a framework that we call the Binomial Model of Stock Price Movement. This is used as one of the foundations for option pricing. In the Binomial Model, it is assumed that for any given time period a stock price can move up or down by a value determined by the up or down probabilities. This turns the stock price into the function of a binomial random variable, the magnitude of upward or downward movement, and the initial stock price. We can vary these parameters in order to approximate different stock price distributions.
Continuous Random Variables
Continuous random variables differ from discrete random variables in that continuous ones can take infinitely many outcomes. They cannot be counted or described as a list. As such, it means very little when we assign individual probabilities to outcomes. Because there are infinitely many outcomes, the probability of hitting any individual outcome has a probability of 0.
We can resolve this issue by instead taking probabilities across ranges of outcomes. This is managed by using calculus, though in order to use our sampling techniques here we do not actually have to use any. With a continuous random variable $P(X = 0)$ is meaningless. Instead we would look for something more like $P(-1 < X < 1)$. For continous random variables, rather than using a PMF, we define a probability density function (PDF), $f_X(x)$, such that we can say:
$$P(a < X < b) = \int_a^b f_X(x)dx$$
Similar to our requirement for discrete distributions that all probabilities add to $1$, here we require that:

$f_X(x) \geq 0$ for all values of $X$
$P(-\infty < X < \infty) = \int_{-\infty}^{\infty} f_X(x) dx = 1$

It is worth noting that because the probability at an individual point with a continuous distribution is $0$, the probabilities at the endpoints of a range are $0$. Hence, $P(a \leq X \leq b) = P(a < X \leq b) = P(a \leq...
Can you write Python code for it?

class ContinuousRandomVariable:
    def __init__(self, a = 0, b = 1):
        self.variableType = ""
        self.low = a
        self.high = b
        return
    def draw(self, numberOfSamples):
        samples = np.random.uniform(self.low, self.high, numberOfSamples)
        return samples


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Discrete Random Variables
A discrete random variable is one with a countable number of outcomes. Each of these outcomes has a separate probability associated with it. Consider a coin flip or a die roll, some of the most basic uniformly distributed random variables. For the coin flip, there are two possible outcomes, either heads or tails, each with a $1/2$ probability of occurring. Discrete random variables do not always have equal weights for all outcomes. The basic unit of a discrete random variable is its probability mass function (PMF), another name for the probability function $p(x)$. The PMF, or probability function, gives a probability, a mass, to each point in the domain of the probability distribution. A probability function has two main properties:

$0 \leq p(x) \leq 1$ because all probabilities are in the interval $[0, 1]$
The sum of all probabilities $p(x)$ over all values of X is equal to $1$. The total weights for all values of the random variable must add to $1$.

Here we will consider some examples of the most prevalent discrete probability distributions.
Can you write Python code for it?

class DiscreteRandomVariable:
    def __init__(self, a=0, b=1):
        self.variableType = ""
        self.low = a
        self.high = b
        return
    def draw(self, numberOfSamples):
        samples = np.random.random_integers(self.low, self.high, numberOfSamples)
        return samples


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Each time we roll the die, we have an equal chance of getting each face. In the short run this looks uneven, but if we take many samples it is apparent that each face is occurring the same percentage of rolls.
Can you write Python code for it?

plt.hist(DieRolls.draw(10000), bins = [1,2,3,4,5,6,7], align = 'mid')
plt.xlabel('Value')
plt.ylabel('Occurences')
plt.legend(['Die Rolls']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Note how the returns of our portfolio, $W$, are also normally distributed
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Now let's see how a theoretical normal curve fits against the actual values.
Can you write Python code for it?

x = np.linspace(-(sample_mean + 4 * sample_std_dev), (sample_mean + 4 * sample_std_dev), len(returns))
sample_distribution = ((1/(sample_std_dev * 2 * np.pi)) *
                       np.exp(-(x - sample_mean)*(x - sample_mean) / (2 * sample_std_dev * sample_std_dev)))
plt.hist(returns.price, bins = 20, normed = True);
plt.plot(x, sample_distribution)
plt.xlabel('Value')
plt.ylabel('Occurrences');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Say that we changed our parameters so that $p = 0.25$. This makes it so that $P(X = 0) = 0.23730$, skewing our distribution much more towards lower values. We can see this easily in the following graph:
Can you write Python code for it?

StockProbabilities = BinomialRandomVariable(5, 0.25)
plt.hist(StockProbabilities.draw(10000), bins = [0, 1, 2, 3, 4, 5, 6], align = 'left')
plt.xlabel('Value')
plt.ylabel('Occurences');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Say that we have some other stock, $Z$, and that we have a portfolio of $Y$ and $Z$, called $W$.
Can you write Python code for it?

Z_initial = 50
Z_returns = X.draw(100)
Z = pd.Series(np.cumsum(Z_returns), name = 'Z') + Z_initial
Z.plot()
plt.xlabel('Time')
plt.ylabel('Value');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
So with a die roll, we can easily see illustrated that the $p(x) = 1/6$ for all values of the random variable $X$. Let's look at the possibilities for all values of both the probability function and the cumulative distribution function:
Value: $X = x$ | PMF: $p(x) = P(X = x)$ | CDF: $F(x) = P(X \leq x)$ |
--- | --- | --- |
1 | $1/6$ | $1/6$
2 | $1/6$ | $1/3$
3 | $1/6$ | $1/2$
4 | $1/6$ | $2/3$
5 | $1/6$ | $5/6$
6 | $1/6$ | $1$
Using this table we can easily see that the probability function satisfies the necessary conditions. Each value of the probability function is in the interval $[0,1]$, satisfying the first condition. The second condition is satisfied because all values of $p(x)$ sum to $1$, as evidenced in the cumulative distribution function. The demonstrates two properties of the cumulative distribution function:

The CDF is between $0$ and $1$ for all $x$. This parallels the value of the probability distribution function.
The CDF is nondecreasing in $x$. This means that as $x$ increases, the CDF either increases or remains constant.

When attempting to sample other probability distributions, we can use compositions of the uniform distribution with certain functions in order to get the appropriate samples. However, this method can be tremendously inefficient. As such, we will instead use the built-in NumPy functions for each distribution to simplify matters.
Binomial Distribution
A binomial distribution is used to describe successes and failures. This can be very useful in an investment context as many of our choices tend to be binary like this. When we take a single success/failure trial, we call it a Bernoulli trial. With the Bernoulli random variable, we have two possible outcomes:
$$p(1) = P(Y = 1) = p \ \ \ \ \ \ \
p(0) = P(Y = 0) = 1-p$$
We consider $Y$ taking on a value of $1$ to be a success, so the probability of a success occurring in a single trial is $p$.
A binomial distribution takes a set of $n$ Bernoulli trials. As such, we can have somewhere between $0$ and $n$ successes. Each trial...
Can you write Python code for it?

class BinomialRandomVariable(DiscreteRandomVariable):
    def __init__(self, numberOfTrials = 10, probabilityOfSuccess = 0.5):
        self.variableType = "Binomial"
        self.numberOfTrials = numberOfTrials
        self.probabilityOfSuccess = probabilityOfSuccess
        return
    def draw(self, numberOfSamples):
        samples = np.random.binomial(self.numberOfTrials, self.probabilityOfSuccess, numberOfSamples)
        return samples


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Take the example of a stock price moving up or down, each with probability $p = 0.5$. We can consider a move up, or $U$, to be a success and a move down, or $D$ to be a failure. With this, we can analyze the probability of each event using a binomial random variable. We will also consider an $n$-value of $5$ for $5$ observations of the stock price over time. The following table shows the probability of each event:
Number of Up moves, $x$ | Ways of reaching $x$ Up moves $\binom{n}{x}$ | Independent Trials with $p = 0.50$ | $p(x)$ Value | CDF: $F(x) = P(X \leq x)$ |
--- | --- | --- | --- | --- |
$0$ | $1$ | $0.50^0 (1 - 0.50)^5 = 0.03125$ | $0.03125$ | $0.03125$
$1$ | $5$ | $0.50^1 (1 - 0.50)^4 = 0.03125$ | $0.15635$ | $0.18750$
$2$ | $10$ | $0.50^2 (1 - 0.50)^3 = 0.03125$ | $0.31250$ | $0.50000$
$3$ | $10$ | $0.50^3 (1 - 0.50)^2 = 0.03125$ | $0.31250$ | $0.81250$
$4$ | $5$ | $0.50^4 (1 - 0.50)^1 = 0.03125$ | $0.15635$ | $0.96875$
$5$ | $1$ | $0.50^5 (1 - 0.50)^0 = 0.03125$ | $0.03125$ | $1.00000$
Here we see that in the particular case where $p = 0.50$, the binomial distribution is symmetric. Because we have an equal probability for both an upward and a downward move, the only differentiating factor between probabilities ends up being the combination aspect of the probability function, which is itself symmetric. If we were to slightly modify the value of $p$ we would end up with an asymmetric distribution.
Now we will draw some samples for the parameters above, where $X$ ~ $B(5, 0.50)$:
Can you write Python code for it?

StockProbabilities = BinomialRandomVariable(5, 0.50)
plt.hist(StockProbabilities.draw(50), bins = [0, 1, 2, 3, 4, 5, 6], align = 'left')
plt.xlabel('Value')
plt.ylabel('Occurences')
plt.legend(['Die Rolls']);


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
The low p-value of the JB test leads us to reject the null hypothesis that the returns are normally distributed. This is due to the high kurtosis (normal distributions have a kurtosis of $3$).
We will proceed from here assuming that the returns are normally distributed so that we can go through the steps of fitting a distribution. Next we calculate the sample mean and standard deviation of the series.
Can you write Python code for it?

# Take the sample mean and standard deviation of the returns
sample_mean = np.mean(returns.price)
sample_std_dev = np.std(returns.price)


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
Uniform Distribution
The uniform distribution can also be defined within the framework of a continous random variable. We take $a$ and $b$ to be constant, where $b$ is the highest possible value and $a$ is the lowest possible value that the outcome can obtain. Then the PDF of a uniform random variable is:
$$f(x) = \begin{cases}\frac{1}{b - a} & \text{for $a < x < b$} \ 0 & \text{otherwise}\end{cases}$$
Since this function is defined on a continuous interval, the PDF covers all values between $a$ and $b$. Here we have a plot of the PDF (feel free to vary the values of $a$ and $b$):
Can you write Python code for it?

a = 0.0
b = 8.0
x = np.linspace(a, b, 100)
y = [1/(b-a) for i in x]
plt.plot(x, y)
plt.xlabel('Value')
plt.ylabel('Probability');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Random_Variables/notebook.ipynb".
The first task is:
We construct $W$ by taking a weighted average of $Y$ and $Z$ based on their quantity.
Can you write Python code for it?

Y_quantity = 20
Z_quantity = 50
Y_weight = Y_quantity/(Y_quantity + Z_quantity)
Z_weight = 1 - Y_weight

W_initial = Y_weight * Y_initial + Z_weight * Z_initial
W_returns = Y_weight * Y_returns + Z_weight * Z_returns
W = pd.Series(np.cumsum(W_returns), name = 'Portfolio') + W_initial
W.plot()
plt.xlabel('Time')
plt.ylabel('Value');

pd.concat([Y, Z, W], axis = 1).plot()
plt.xlabel('Time')
plt.ylabel('Value');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
Because we're dealing with ranking systems, at several points we're going to want to rank our data. Let's check how our data looks when ranked to get a sense for this.
Can you write Python code for it?


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
Let's take a look at the data to get a quick sense of what we have. This may take a while.
Can you write Python code for it?

from quantopian.research import run_pipeline

start_date = '2013-01-01'
end_date = '2015-02-01'

data = run_pipeline(pipe, start_date, end_date)

# remove NaN values
data = data.dropna()

# show data
data


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
Now, we need to take each of these individual factors, clean them to remove NaN values and aggregate them for each month.
Can you write Python code for it?

cap_data = data['Market Cap'].transpose().unstack() # extract series of data
cap_data = cap_data.T.dropna().T # remove NaN values
cap_data = cap_data.resample('M', how='last') # use last instance in month to aggregate

pe_data = data['PE Ratio'].transpose().unstack()
pe_data = pe_data.T.dropna().T
pe_data = pe_data.resample('M', how='last')

month_data = data['Monthly Returns'].transpose().unstack()
month_data = month_data.T.dropna().T
month_data = month_data.resample('M', how='last')


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
Sometimes Factors are Just Other Factors
Often times a new factor will be discovered that seems to induce spread, but it turns out that it is just a new and potentially more complicated way to compute a well known factor. Consider for instance the case in which you have poured tons of resources into developing a new factor, it looks great, but how do you know it's not just another factor in disguise?
To check for this, there are many analyses that can be done.
Correlation Analysis
One of the most intuitive ways is to check what the correlation of the factors is over time. We'll plot that here.
Can you write Python code for it?

scores = np.zeros(24)
pvalues = np.zeros(24)
for i in range(24):
    score, pvalue = stats.spearmanr(cap_data_filtered.iloc[i], pe_data_filtered.iloc[i])
    pvalues[i] = pvalue
    scores[i] = score

plt.bar(range(1,25),scores)
plt.hlines(np.mean(scores), 1, 25, colors='r', linestyles='dashed')
plt.xlabel('Month')
plt.xlim((1, 25))
plt.legend(['Mean Correlation over All Months', 'Monthly Rank Correlation'])
plt.ylabel('Rank correlation between Market Cap and PE Ratio');


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
Spread Consistency
Of course, that's just the average relationship. To get a sense of how consistent this is, and whether or not we would want to trade on it, we should look at it over time. Here we'll look at the monthly spreads for the first year. We can see a lot of variation, and further analysis should be done to determine whether Market Cap is tradeable.
Can you write Python code for it?

f, axarr = plt.subplots(3, 4)
for month in range(1, 13):
    basket_returns = compute_basket_returns(cap_data_filtered, month_forward_returns, 10, month)

    r = np.floor((month-1) / 4)
    c = (month-1) % 4
    axarr[r, c].bar(range(number_of_baskets), basket_returns)
    axarr[r, c].xaxis.set_visible(False) # Hide the axis lables so the plots aren't super messy
    axarr[r, c].set_title('Month ' + str(month))


I am working on the file "Notebooks/quantopian_research_public/notebooks/lectures/Ranking_Universes_by_Factors/notebook.ipynb".
The first task is:
The correlation of PE Ratio and 30-day returns seems to be near 0 on average. It's important to note that this monthly and between 2012 and 2015. Different factors are predictive on differen timeframes and frequencies, so the fact that PE Ratio doesn't appear predictive here is not necessary throwing it out as a useful factor. Beyond it's usefulness in predicting returns, it can be used for risk exposure analysis as discussed in the Factor Risk Exposure Lecture.
Basket Returns
The next step is to compute the returns of baskets taken out of our ranking. If we rank all equities and then split them into $n$ groups, what would the mean return be of each group? We can answer this question in the following way. The first step is to create a function that will give us the mean return in each basket in a given the month and a ranking factor.
Can you write Python code for it?

def compute_basket_returns(factor_data, forward_returns, number_of_baskets, month):

    data = pd.concat([factor_data.iloc[month-1],forward_returns.iloc[month-1]], axis=1)
    # Rank the equities on the factor values
    data.columns = ['Factor Value', 'Month Forward Returns']
    data.sort('Factor Value', inplace=True)

    # How many equities per basket
    equities_per_basket = np.floor(len(data.index) / number_of_baskets)

    basket_returns = np.zeros(number_of_baskets)

    # Compute the returns of each basket
    for i in range(number_of_baskets):
        start = i * equities_per_basket
        if i == number_of_baskets - 1:
            # Handle having a few extra in the last basket when our number of equities doesn't divide well
            end = len(data.index) - 1
        else:
            end = i * equities_per_basket + equities_per_basket
        # Actually compute the mean returns for each basket
        basket_returns[i] = data.iloc[start:end]['Month Forward Returns'].mean()

    return basket_returns