I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
The last line of an error message is usually the most informative.
We will look at error messages in detail later.

Python is case-sensitive.

Python thinks that upper- and lower-case letters are different,
    so Name and name are different variables.
Again,
    there are conventions around using upper-case letters at the start of variable names
    so we will use lower-case letters for now.

Use meaningful variable names.

Python doesn't care what you call variables as long as they obey the rules
    (alphanumeric characters and the underscore).
Can you write Python code for it?

flabadab = 42
ewr_422_yY = 'Ahmed'
print(ewr_422_yY, 'is', flabadab, 'years old')


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Types control what operations can be done on values.

A value's type determines what the program can do to it.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Use an item’s index to fetch it from a list.

Each value in a list is stored in a particular location.
Locations are numbered from 0 rather than 1.
Use the location’s index in square brackets to access the value it contains.
Can you write Python code for it?

print('the first item is:', country_list[0])
print('the fourth item is:', country_list[3])


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Use del to remove items from a list entirely.

del list_name[index] removes an item from a list and shortens the list.
Not a function or a method, but a statement in the language.
Can you write Python code for it?

print("original list was:", country_list)
del country_list[3]
print("the list is now:", country_list)


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Use len to find out how many values are in a list.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Use meaningful variable names to help other people understand what the program does.
The most important "other person" is your future self.

Variables can be used in calculations.

We can use variables in calculations just as if they were values.
Remember, we assigned 42 to age a few lines ago.
Can you write Python code for it?

age = age + 3
print('Age in three years:', age)


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
Variable names:
cannot start with a digit
cannot contain spaces, quotation marks, or other punctuation
may contain an underscore (typically used to separate words in long variable names)

Underscores at the start like __alistairs_real_age have a special meaning
    so we won't do that until we understand the convention.

Use print to display values.

Python has a built-in function called print that prints things as text.
Call the function (i.e., tell Python to run it) by using its name.
Provide values to the function (e.g., things to print) in parentheses.
Can you write Python code for it?

print(first_name, 'is', age, 'years old')


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
What do you think should happen when you take upper of an int?  What about a string representation of an int?
Challenge 3: Write your name

Make two string variables, one with your first name and one with your last name.
Concatenate both strings to form your full name and assign it to a variable.
Assign a new variable that has your full name in all upper case.
Slice that string to get your first name again.

*Challenge 4: Exploring string methods *
In our next meeting we will be looking at Twitter activity from the March4Trump event in Berkeley on March 4, 2017. Below was the most retweeted tweet during the March4Trump event. Use this tweet to explore the methods above.
Can you write Python code for it?

tweet = 'RT @JasonBelich: #March4Trump #berkeley elderly man pepper sprayed by #antifa https://t.co/5z3O6UZuhL'


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
You can see some of the logic for this when we consider implicit indices.
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
You can splice strings using  [ : ]

if you want a range (or "slice") of a sequence, you get everything before the second index:
Can you write Python code for it?


I am working on the file "01-Intro-to-Computational-Tools-Python.ipynb".
The first task is:
print automatically puts a single space between items to separate them.
And wraps around to a new line at the end.

Variables persist between cells.

Variables defined in one cell exist in all following cells.
Notebook cells are just a way to organize a program:
    as far as Python is concerned,
    all of the source code is one long set of instructions.

Variables must be created before they are used.

If a variable doesn't exist yet, or if the name has been mis-spelled,
    Python reports an error.
Can you write Python code for it?


I am working on the file "01-IntroToPython/.ipynb_checkpoints/00-PythonBasics-checkpoint.ipynb".
The first task is:

List Comprehension
List comprehensions are a fairly advanced programming technique that we will spend more time talking about tomorrow. For now, you can think of them as list filters. Often, we don't need every value in a list, just a few that fulfill certain criteria.
Can you write Python code for it?

# 'list1' had contained three words, two of which were in title case.
# We can automatically return those words using a list comprehension

[word for word in list1 if word.istitle()]

# Or we can include all the words in the list but just take their first letters

[word[0] for word in list1]
for word in list1:
    print(word[0])

## EX. Using the list of words you produced by splitting 'new_string', create
##     a new list that contains only the words whose last letter is "y"
y_list = [word for word in new_string_list if word.endswith('y')]
print(y_list)
## EX. Create a new list that contains the first letter of each word.
first_letter = [word[0] for word in new_string_list]
print(first_letter)
## EX. Create a new list that contains only words longer than two letters.
long_words = [word for word in new_string_list if len(word)>2]
print(long_words)


I am working on the file "01-IntroToPython/.ipynb_checkpoints/00-PythonBasics-checkpoint.ipynb".
The first task is:

Lists
The numbers and strings we have just looked at are the two basic data types that we will focus our attention on in this workshop. (In a few days, we will look at a third data type, boolean, which consists of just True/False values.) When we are working with just a few numbers or strings, it is easy to keep track of them, but as we collect more we will want a system to organize them.
One such organizational system is a list. This contains values (regardless of type) in order, and we can perform operations on it very similarly to the way we did with numbers.
A list in which each element is a string
The creators of Python recognize that human language has many important yet idiosyncratic features, so they have tried to make it easy for us to identify and manipulate them. For example, in the demonstration at the very beginning of the workshop, we referred to the idea of the suffix: the final letters of a word tell us something about its grammatical role and potentially the author's argument.
We can analyze or manipulate certain features of a string using its methods. These are basically internal functions that every string automatically possesses. Note that even though the method may transform the string at hand, they don't change it permanently!
Can you write Python code for it?

# Let's assign a couple lists to variables

list1 = ['Call', 'me', 'Ishmael']
list2 = ['In', 'the', 'beginning']

## Q. Predict what will happen when we perform the following operations

print(list1+list2)
print(list1*5)

# As with a string, we can find out the length of a list

len(list1)

# Sometimes we just want a single value from the list at a time

print(list1[0])
print(list1[1])
print(list1[2])

# Or maybe we want the first few

print(list1[0:2])
print(list1[:2])

# Of course, lists can contain numbers or even a mix of numbers and strings

list3 = [7,8,9]
list4 = [7,'ate',9]

# And python is smart with numbers, so we can add them easily!

sum(list3)

## EX. Concatenate 'list1' and 'list2' into a single list.
##     Retrieve the third element from the combined list.
##     Retrieve the fourth through sixth elements from the combined list.
new_list = list1+list2
new_list[3:]


I am working on the file "01-IntroToPython/.ipynb_checkpoints/00-PythonBasics-checkpoint.ipynb".
The first task is:

Variable assignment
Assigning variables is something that we do all the time in programming. These aren't quite like the variables from high school algebra, where x represents an unknown to solve for. Instead these are like notes to ourselves that we want to save some value(s) for later use.
Note that the equals sign is directional, like an arrow, telling the computer to give a certain value to a certain label.
Can you write Python code for it?

# 'a' is being given the value 2; 'b' is given 5

a = 2
b = 5

# Let's perform an operation on the variables

a+b

# Variables can have many different kinds of names

this_number = 2
b/this_number


I am working on the file "01-IntroToPython/.ipynb_checkpoints/01-WorkingWithTexts_ExerciseSolutions-checkpoint.ipynb".
The first task is:
First, create variables that split the strings into lists of words. I'll print out the first 10 words to make sure everything looks good.
Can you write Python code for it?

austen_words = austen_string.split()
alcott_words = alcott_string.split()
print(austen_words[:10])
print(alcott_words[:10])


I am working on the file "01-IntroToPython/.ipynb_checkpoints/01-WorkingWithTexts_ExerciseSolutions-checkpoint.ipynb".
The first task is:
To count the number of long words I create two new variables and use list comprehension to keep only words longer than 7 characters. I then divide the length of these lists by the total number of words, to get proportion.
Can you write Python code for it?

##     How many words longer than 7 characters are in each novel? (don't worry about punctuation for now)
##     What proportion of the total words are the long words in each novel

austen_long = [word for word in austen_words if len(word)>7]
print("Number of long words in Pride and Prejudice:")
print(len(austen_long))

alcott_long = [word for word in alcott_words if len(word)>7]
print("Number of long words in A Garland for Girls:")
print(len(alcott_long))

print("Proportion of words that are long in Pride and Prejudice:")
print(len(austen_long)/len(austen_words))
print("Proportion of words that are long in A Garland for Girls:")
print(len(alcott_long)/len(alcott_words))

dir()
locals()


I am working on the file "01-IntroToPython/.ipynb_checkpoints/01-WorkingWithTexts_ExerciseSolutions-checkpoint.ipynb".
The first task is:
To count the number of words that are in title case we can use list comprehension.
Can you write Python code for it?

#How many words in each novel appear in title case?
print("Number of words in title case in Pride and Prejudice:")
print(len([word for word in austen_words if word.istitle()]))
print("Number of words in title case in A Garland for Girls:")
print(len([word for word in alcott_words if word.istitle()]))


I am working on the file "01-IntroToPython/.ipynb_checkpoints/01-WorkingWithTexts_ExerciseSolutions-checkpoint.ipynb".
The first task is:
To get the average length of the words in each novel we first transform each word into its length, using the len function and list comprehension. I'll print out the first 10 word length to make sure we did it right. We can then sum the list and divide by the total number of words.
Can you write Python code for it?

austen_word_length = [len(word) for word in austen_words]
print(austen_word_length[:10])
alcott_word_length = [len(word) for word in alcott_words]
print(alcott_word_length[:10])
print("Average word length in Pride and Prejudice:")
print(sum(austen_word_length)/len(austen_word_length))
print("Average word length in A Garland for Girls:")
print(sum(alcott_word_length)/len(alcott_word_length))


I am working on the file "01-IntroToPython/.ipynb_checkpoints/02-WorkingWithNumbers-checkpoint.ipynb".
The first task is:

Summarzing Real Data
Now let's read in a dataset containing real world data. The data is a .csv file you should have saved in the 'data' folder. These data are from the National Survey of Family Growth. It's from Cycle 6, the years 2002-2003. The survey contains information on pregnancy and births in the U.S.
First read in the data:
Can you write Python code for it?

#we'll assign it to the variable name df again, to keep things consistent
df = pandas.read_csv("../data/nsfg_data1.csv.bz2", compression='bz2', index_col = 0)
#view our data. Note: 'NaN' indicates the value is missing
df

#Typically the first thing we want to do is describe our data.
df.describe()


I am working on the file "01-IntroToPython/.ipynb_checkpoints/02-WorkingWithNumbers-checkpoint.ipynb".
The first task is:

DataFrame Subsetting
Can you write Python code for it?

# Slice out a column

df['income']

# Evaluate whether each element in the column is equal to 100

df['income']==100

# We can also use evaluation to subset the table. This time we'll use the greater than evaluator.
#df['income']>200
df[df['income']>200]

## EX. Slice 'df' to contain only rows in which 'education' equals 'BA' (remeber how to specify a string!)
df[df['education']=='BA']


I am working on the file "01-IntroToPython/.ipynb_checkpoints/02-WorkingWithNumbers-checkpoint.ipynb".
The first task is:
We can't do it! Why not?
Can you write Python code for it?

#We can do it if we specify a column

df['income']+10

# Of course our dataframe hasn't changed

df

# What if we just want to add the values in the column?

df['income'].sum()

# We can also perform operations among columns
# Pandas knows to match up individual entries in each column

df['income_age'] = df['income']/df['age']
df

# EX: .sum() adds the values in a column. .mean() calculates the mean value in a column.
#     Find the mean income and the mean age for the dataframe df()


I am working on the file "01-IntroToPython/00-PythonBasics.ipynb".
The first task is:

List Comprehension
List comprehensions are a fairly advanced programming technique that we will spend more time talking about tomorrow. For now, you can think of them as list filters. Often, we don't need every value in a list, just a few that fulfill certain criteria.
Can you write Python code for it?

# 'list1' had contained three words, two of which were in title case.
# We can automatically return those words using a list comprehension

[word for word in list1 if word.istitle()]

# Or we can include all the words in the list but just take their first letters

[word[0] for word in list1]

## EX.1  Using the list of words you produced by splitting 'new_string', create
##     a new list that contains only the words whose last letter is "y"

##EX.2 Create a new list that contains the first letter of each word.

##EX.3 Create a new list that contains only words longer than two letters.


I am working on the file "01-IntroToPython/00-PythonBasics.ipynb".
The first task is:

Lists
The numbers and strings we have just looked at are the two basic data types that we will focus our attention on in this workshop. (In a few days, we will look at a third data type, boolean, which consists of just True/False values.) When we are working with just a few numbers or strings, it is easy to keep track of them, but as we collect more we will want a system to organize them.
One such organizational system is a list. This contains values (regardless of type) in order, and we can perform operations on it very similarly to the way we did with numbers.
A list in which each element is a string
The creators of Python recognize that human language has many important yet idiosyncratic features, so they have tried to make it easy for us to identify and manipulate them. For example, in the demonstration at the very beginning of the workshop, we referred to the idea of the suffix: the final letters of a word tell us something about its grammatical role and potentially the author's argument.
We can analyze or manipulate certain features of a string using its methods. These are basically internal functions that every string automatically possesses. Note that even though the method may transform the string at hand, they don't change it permanently!
Can you write Python code for it?

# Let's assign a couple lists to variables

list1 = ['Call', 'me', 'Ishmael']
list2 = ['In', 'the', 'beginning']

## Q. Predict what will happen when we perform the following operations

print(list1+list2)
print(list1*5)

# As with a string, we can find out the length of a list

len(list1)

# Sometimes we just want a single value from the list at a time

print(list1[0])
print(list1[1])
print(list1[2])

# Or maybe we want the first few

print(list1[0:2])
print(list1[:2])

# Of course, lists can contain numbers or even a mix of numbers and strings

list3 = [7,8,9]
list4 = [7,'ate',9]

# And python is smart with numbers, so we can add them easily!

sum(list3)

## EX. Concatenate 'list1' and 'list2' into a single list.
##     Retrieve the third element from the combined list.
##     Retrieve the fourth through sixth elements from the combined list.


I am working on the file "01-IntroToPython/00-PythonBasics.ipynb".
The first task is:

A couple of useful tricks

String Methods
The creators of Python recognize that human language has many important yet idiosyncratic features, so they have tried to make it easy for us to identify and manipulate them. For example, in the demonstration at the very beginning of the workshop, we referred to the idea of the suffix: the final letters of a word tell us something about its grammatical role and potentially the author's argument.
We can analyze or manipulate certain features of a string using its methods. These are basically internal functions that every string automatically possesses. Note that even though the method may transform the string at hand, they don't change it permanently!
Can you write Python code for it?

# Let's assign a variable to perform methods upon

greeting = "Hello, World!"

# We saw the 'endswith' method at the very beginning
# Note the type of output that gets printed

greeting.startswith('H'), greeting.endswith('d')

# We can check whether the string is a letter or a number

this_string = 'f'

this_string.isalpha()

# When there are multiple characters, it checks whether *all*
# of the characters belong to that category

greeting.isalpha(), greeting.isdigit()

# Similarly, we can check whether the string is lower or upper case

greeting.islower(), greeting.isupper(), greeting.istitle()

# Sometimes we want not just to check, but to change the string

greeting.lower(), greeting.upper()

# The case of the string hasn't changed!

greeting

# But if we want to permanently make it lower case we re-assign it

greeting = greeting.lower()

greeting

# Oh hey. And strings are kind of like lists, so we can slice them similarly

greeting[:3]

# Strings may be like lists of characters, but as humans we often treat them as
# lists of words. We tell the computer to can perform that conversion.

greeting.split()

## EX. Return the second through eighth characters in 'greeting'

## EX. Split the string below into a list of words and assign this to a new variable
## Note: A slash at the end of a line allows a string to continue unbroken onto the next

new_string = "It seems very strange that one must turn back, \
and be transported to the very beginnings of history, \
in order to arrive at an understanding of humanity as it is at present."


I am working on the file "01-IntroToPython/00-PythonBasics.ipynb".
The first task is:

Variable assignment
Assigning variables is something that we do all the time in programming. These aren't quite like the variables from high school algebra, where x represents an unknown to solve for. Instead these are like notes to ourselves that we want to save some value(s) for later use.
Note that the equals sign is directional, like an arrow, telling the computer to give a certain value to a certain label.
Can you write Python code for it?


I am working on the file "01-IntroToPython/00-PythonBasics_ExerciseSolutions.ipynb".
The first task is:
(2) EX. Create a new list that contains the first letter of each word.
We can again use list comprehension, combined with string splicing, to produce a new list that contain only the first letter of each word. Remember in Python counting starts at 0.
Can you write Python code for it?

word_list_firstletter = [word[0] for word in new_string_list]
#print our new list
word_list_firstletter


I am working on the file "01-IntroToPython/01-WorkingWithTexts_ExerciseSolutions.ipynb".
The first task is:
First, create variables that split the strings into lists of words. I'll print out the first 10 words to make sure everything looks good.
Can you write Python code for it?


I am working on the file "01-IntroToPython/01-WorkingWithTexts_ExerciseSolutions.ipynb".
The first task is:
To count the number of long words I create two new variables and use list comprehension to keep only words longer than 7 characters. I then divide the length of these lists by the total number of words, to get proportion.
Can you write Python code for it?


I am working on the file "01-IntroToPython/01-WorkingWithTexts_ExerciseSolutions.ipynb".
The first task is:
To count the number of words, simply take the length of each list and print the result. You can add print statements to make the output look prett.
Can you write Python code for it?

#How many words are in each novel?
print("Number of words in Pride and Prejudice:")
print(len(austen_words))
print("Number of words in A Garland for Girls")
print(len(alcott_words))


I am working on the file "01-IntroToPython/01-WorkingWithTexts_ExerciseSolutions.ipynb".
The first task is:
To get the average length of the words in each novel we first transform each word into its length, using the len function and list comprehension. I'll print out the first 10 word length to make sure we did it right. We can then sum the list and divide by the total number of words.
Can you write Python code for it?


I am working on the file "01-IntroToPython/02-WorkingWithNumbers.ipynb".
The first task is:

Arithmetic!
Can you write Python code for it?

# Our dataframe

df

# Pandas will produce a few descriptive statistics for each row, but only columns that are numbers

df.describe()

# Multiply entries of the dataframe by 10

df*10

# Add 10 to each entry

df+10


I am working on the file "01-IntroToPython/02-WorkingWithNumbers.ipynb".
The first task is:

DataFrame Subsetting
Can you write Python code for it?


I am working on the file "01-IntroToPython/02-WorkingWithNumbers.ipynb".
The first task is:
This describes the columns that contain numbers as values. To summarize the columns that contain strings, we can use the .value_counts() function. This counts the number of rows for each unique value in the specified column.
Can you write Python code for it?


I am working on the file "01-IntroToPython/02-WorkingWithNumbers.ipynb".
The first task is:
We can visualize the distribution of the two number columns by creating a histogram.
Can you write Python code for it?

# The main visualization, or plotting, library in Python is matplotlib
import matplotlib.pyplot as plt
df.hist()
plt.show()


I am working on the file "01-IntroToPython/02-WorkingWithNumbers.ipynb".
The first task is:
We have to first introduce a new datatype, lists of lists, or lists with sub-lists. We will use this to create a toy example of a dataset in pandas. Our toy dataset will include a sample of three people, and their weekly income and education level.
Can you write Python code for it?

# Create a list of three sub-lists, each with two entries.

square_list = [[100,'high school'],[400,'BA'],[700,'MA']]

# We can slice lists of lists like we do lists
# Question: What do you expect the output to be?

square_list[2]

# Now the interesting stuff. Create a dataframe from that list.
# The output here should look familiar, like a spreadsheet

pandas.DataFrame(square_list)

# We can create a list for our column labels to add to our dataframe

column_names = ['income', 'education']
row_names = ['Laura', 'Hovig', 'Ashley']

# A-ha!

pandas.DataFrame(square_list, columns = column_names, index=row_names)

# Assign this to a variable

df = pandas.DataFrame(square_list, columns = column_names, index=row_names)
df

# Call up a column of the dataframe

df['income']
df.income

# Make that column into a list

list(df['income'])

# Get the indices for the entries in the column

df['income'].index

# Call up a row from the indices
df.iloc[1]

# Call up a couple of rows, using a list of indices

df.loc[['Laura','Hovig']]

# Get a specific entry by calling both row and column

df.loc['Laura']['income']

# Temporarily re-order the dataframe by values in the 'income' column

df.sort_values('income', ascending=False)

# Create a new column

df['age'] = [22,31,35]

# Inspect

df

## EX. Call up the entries (400 and BA) from the middle of the dataframe 'df' individually

## CHALLENGE: Call up both entries at the same time


I am working on the file "01-Iterators.ipynb".
The first task is:
Don't actually call the __next__ and __iter__ methods though
I showed these methods to display how any object in python can implement the interface. For actual use, Python has builtin functions next and iter that you should use.
Can you write Python code for it?

my_iter = iter(my_list)
print("Type:", type(my_iter))

print(next(my_iter))
print(next(my_iter))
print(next(my_iter))


I am working on the file "01-Iterators.ipynb".
The first task is:
StopIteration
While a for loop will handle the StopIteration exception and know to terminate the loop, if we reach the end of the iterator outside of a for loop, we can see the exception as it's raised.
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Acesando o sistema operacional
Nota: os comandos abaixo funcionam em Linux e Macs, mas não em Windows.
Can you write Python code for it?

!pwd

files = !ls
print(u"Conteúdo do diretório:")
print(files)

!echo $files

!echo {files[0].upper()}


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Ajuda com tab
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Funções mágicas
As magic functions são um conjunto de comandos chamados através do um ou dois sinas de %.  Alguns ainda implementam opções extras através de argumento --, parêntesis ou vírgulas. As duas razões por trás das funções magic são:

Fornecer um namespace ortogonal para controlar o próprio IPython e expor funcionalidades do sistema.

Para fornecer um mode de comandos que requer menos digitação.
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Histórico do input
Can you write Python code for it?

In[11]

print(u'último: {}'.format(_i))
print(u'penúltimo: {}'.format(_ii))
print(u'antepenúltimo: {}'.format(_iii))

%history


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
IPython reference card:
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Interactiviade: input, output, history
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Magics podem ativar outra sintaxe que não necessariamente é python:
Can you write Python code for it?

%%bash
echo "My shell is:" $SHELL
echo "User:" $USER


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Notebook
Interface

Dashboard
Notebook e células
Menu
Toolbar

Típos de células

Código
Markdown
Raw text
Heading

$\LaTeX$ equations
$$
\frac{D\mathbf{v}}{Dt} = -\frac{1}{\rho}\nabla p - 2\Omega \times \mathbf{v} + \mathbf{g} + \mathbf{R_r}
$$
Atalhos de teclado

Ctrl-Enter para rodar
Shift-Enter para rodar e ir para a próxima
Os outros atalhos começam com: Ctrl-m ?
Examplo Ctrl-m h

Trabalhando com Notebooks

IPython Notebooks são arquivos .ipynb
Tudo dentro do arquivo são dados JSONs
O servidor dos Notebooks abre o que está no diretório

Para começar digite:
ipython notebook
Can you write Python code for it?

from IPython.nbformat import current

with open('01-Jupyter-Basics.ipynb') as f:
    nb = current.read(f, 'json')

nb.worksheets[0].cells[0]


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Os últimos 3 podem ser acessados rapidamente com:
Can you write Python code for it?

print(u'último: {}'.format(_))
print(u'penúltimo: {}'.format(__))
print(u'antepenúltimo: {}'.format(___))


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Output
Você pode suprimir a saída e o cache usando ; no final do comando:
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Pedir ajuda
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
Tab completion
É uma forma conveniente, especialmente para atributos, de explorar
a estrutura dos objetos que você está lidando.
Can you write Python code for it?


I am working on the file "01-Jupyter-Basics.ipynb".
The first task is:
magics de linha podem ser usadas em loops:
Can you write Python code for it?

for i in range(5):
    size = i*100
    print('size: {}'.format(size))
    %timeit range(size)


I am working on the file "01-LinearBeam.ipynb".
The first task is:
And elements as lists of nodes
Can you write Python code for it?

elements = np.array([[i,i+1] for i in range(n_elements)])
elements


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Deformations
Within the polynomial approximation above, the deformations are given by
Can you write Python code for it?

eps = u.diff(s)
kappa = v.diff(s,2)
[eps,kappa]


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Elastic energy and stiffness matrix
We can write the extensional energy as
$$
\mathcal E_{\mathrm{ext}} =\int_0^L\dfrac{ES}{2} e(s)^2\,ds = \dfrac{1}{2}K^e_{\mathrm{ext}} U\cdot U
$$
where
Can you write Python code for it?

ES = sp.Symbol('ES')
Ke_ext = ES*sp.integrate(B[0,:].transpose()*B[0,:],(s,0,L))
Ke_ext


I am working on the file "01-LinearBeam.ipynb".
The first task is:
For example, the global indeces of the dof of the element $e=2$ are (python indeces start from $0$)
Can you write Python code for it?

def dof_map_e(e):
    return [dof_map(e, i_local) for  i_local in range(6)]
dof_map_e(2)


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Imposing boundary conditions (alternative method) and solve
To impose th bcs, we solve a reduced system, where we eliminate the rows and columns corresponding to blocked dof. We call K_red and F_red the new matrix
Can you write Python code for it?

K_red = np.delete(K, blocked_dof, axis = (0))
K_red = np.delete(K_red, blocked_dof, axis = (1))
F_red = np.delete(F, blocked_dof, axis = (0))
sp.Matrix(K_red),sp.Matrix(F_red)


I am working on the file "01-LinearBeam.ipynb".
The first task is:
It is useful also to define the following vector, saying if a giving dof is blocked or not
Can you write Python code for it?

bc_flags = 0*F.astype("bool")
bc_flags[blocked_dof] = 1
bc_flags


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Let us now define the vector collecting the displacement and rotation at lefth and right ends of the beam
Can you write Python code for it?


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Numerical numpy array version
For the following developments, it is convenient to work with numerical quantities. We start using the  python package for numerical linear algebra, numpy
Can you write Python code for it?

import numpy as np
Ken = np.array(Ke.subs({EI:1., ES:100., L:1.}))
Fen = np.array(Fe.subs({L:1., p:1.}))
Ken,Fen


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Or you can have fun and make it prettier
Can you write Python code for it?

with plt.xkcd():
    fig, ax = plt.subplots()
    ax.plot(xnodes,ynodes,'o-',lw=2, color='black', ms=10)
    for i in range(n_nodes):
        ax.text(xnodes[i], -.02, np.array([i,i+1,i+2]))
        ax.text(xnodes[i], .01, i, color = 'b')
    for e in range(n_elements):
        ax.text(xnodes[e]+(xnodes[e+1]-xnodes[e])/2., .02, str(e), bbox=dict(facecolor='red'))
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    ax.set_title("Elements, nodes, and dofs")
    ax.set_xlim(-0.2, 1.2)
    ax.set_ylim(-0.05, 0.05)


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Or, if we want to print it out nicely
Can you write Python code for it?


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Similarly for the bending energy
$$
\mathcal E_{\mathrm{bend}} = \int_0^L\dfrac{EI}{2} v''(s)^2 = \dfrac{1}{2}K^e_{\mathrm{ben}} U\cdot U
$$
where
Can you write Python code for it?

EI = sp.Symbol('EI')
Ke_bend = EI*sp.integrate(B[1,:].transpose()*B[1,:],(s,0,L))
Ke_bend


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Solve the system
We have finally a linear system to solve of the type
$$
K\,U = F
$$
However some of the components of $U$ are prescribed by the boundary conditions. This should be accounted for.
Boundary conditions
In our case of a clamped 4-element beam, the first three and the last three component of $U$ are blocked.
Can you write Python code for it?

blocked_dof = np.array([0, 1, 2, F.size-3, F.size-2, F.size-1])
blocked_dof


I am working on the file "01-LinearBeam.ipynb".
The first task is:
The polynomial approximation allows us to express the displacement in any point in the of the end displacement $U$. To do it we have to resolve a linear system. We do it by using the capability of sympy.
Can you write Python code for it?

L = sp.Symbol('L')
solu = sp.solve([u.subs({s:0})-u0, u.subs({s:L})-u1], (c0,c1))
solv = sp.solve([v.subs({s:0})-v0, q.subs({s:0})-q0, v.subs({s:L})-v1, q.subs({s:L})-q1], (c2,c3,c4,c5))
u = u.subs(solu)
v = v.subs(solv)
sp.Matrix([u, v])


I am working on the file "01-LinearBeam.ipynb".
The first task is:
The unit tangent venctor to the current configuration is given by
\begin{equation}
\underline{t}(s) = \dfrac{\underline{x}'(s)}{\Vert{\underline{x}'(s)}\Vert} = \cos\theta(s)\,\underline{e}_1 + \sin\theta(s) \,\underline{e}_2
\end{equation}
where in the last equality we introduced the angle $\theta(s)$ between the tangent $\underline{t}(s)$ and $\underline{e}_1$.
We define the following deformations measures for the beam:
\begin{equation}
\begin{cases}
\epsilon(s) = \Vert{\underline{x}'(s)}\Vert - 1 = \sqrt{(1+u'(s))^2+v'(s)^2}-1&(extension)\
\kappa(s) = \theta'(s) & (curvature)
\end{cases}
\end{equation}
When $\epsilon(s)=0$ the beam is inextensible and $s$ is a arc-length also in the current configurations. Otherwise the arclength in the current configuration is given by $\tilde s$, such that $\mathrm d \tilde s ^2= \mathrm d x^2+ \mathrm d y^2 = \Vert{\underline{x}'(s)}\Vert^2 ds^2 $.
The definions of the extension and the tangent give the following expressions for the  derivatives of the displacement field:
$$
\begin{cases}
u'(s)&=(1+\epsilon(s))\cos\theta(s) - 1\
v'(s)&=(1+\epsilon(s))\sin\theta(s)
\end{cases}
$$
Moderation rotations, small extension approximation
Let us assume that $\theta$ and $e$ are small.
Then by a Taylor expansion
$$
\begin{cases}
u'(s)&=(1+\epsilon(s))\cos\theta(s) - 1\simeq \epsilon(s)-\dfrac{\theta(s)^2}{2}\
v'(s)&=(1+\epsilon(s))\sin\theta(s) \simeq \theta(s)+o(\theta^2)
\end{cases}
$$
Linearized kinematics
Retaing only the linear terms in the Taylor expansion above one get the fully linearized kinematics for a planar Euler-Bernoulli beam:
$$
\begin{cases}
\epsilon(s) = u'(s)\
\theta(s) = v'(s)\quad\Rightarrow\quad \kappa(s)=v''(s)
\end{cases}
$$
Variational Formulation - linearized model
We present below the linear model, assuming the fully linearized kinematics above. This model is valid for small loadings. Let us consider the case of beam of length $L$, bending stiffness $EI$ and axial stiffness $ES$, which submitted to a distributed force per unit length $\underline f(s)=...
Can you write Python code for it?

# I use sympy http://docs.sympy.org/
import sympy as sp
from sympy.interactive import printing
printing.init_printing()
s = sp.Symbol('s')
c0, c1, c2, c3, c4, c5 = sp.symbols('c0, c1, c2, c3, c4, c5')
u = c0 + c1*s
v = c2 + c3*s + c4*s**2 + c5*s**3
sp.Matrix([u, v])


I am working on the file "01-LinearBeam.ipynb".
The first task is:
Then we can do it better with the current configuration
Can you write Python code for it?

x = lambda s : xR(s) + .1*s
y = lambda s : yR(s) + .3*s**2

with plt.xkcd():
    fig, ax = plt.subplots()
    ax.plot(x(s),y(s),"r",color='b', ls='-', lw=3)
    ax.plot(xR(s),yR(s),"r",color='b', ls='--', lw=3)
    ax.quiver(xR(.7),yR(.7),x(.7)-xR(.7),y(.7)-yR(.7),color='r',scale_units='xy', angles='xy', scale=1)
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
    ax.set_title("Reference and deformed configuration")
    ax.set_xlim(-0.2, 1.2)
    ax.set_ylim(-0.05, .35)
    ax.text(xR(.7)-.2,yR(.7)-.03,r'$({x}_R(s),y_R(s))$')
    ax.text(x(.7)-.3,y(.7)+.01,r'$({x}(s),y(s))$')


I am working on the file "01-LinearBeam.ipynb".
The first task is:
This is a simple visualization of the solution
Can you write Python code for it?

def plot_deformed(Usol):
    x_c = nodes[:,0]+[Usol[3*e] for e in range(n_nodes)]
    y_c = nodes[:,1]+[Usol[3*e+1] for e in range(n_nodes)]
    return plt.plot(x_c, y_c, '-o')
plot_deformed(Usol)


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We can also represent it graphically
Can you write Python code for it?


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We can represent it using matplotlib
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(xR(s),yR(s))


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We can represent the basis functions using matplotlib
Can you write Python code for it?

sn = np.linspace(0.,1.,20)
for i in range(2):
    Sn = np.array([S[0,:].subs({s:si, L:1}) for si in sn])
    plt.plot(sn,Sn)
plt.xlabel('s')
plt.ylabel('shape functions')

for i in range(6):
    Sn = np.array([S[1,:].subs({s:si, L:1}) for si in sn])
    plt.plot(sn,Sn)
    plt.xlabel('s')
    plt.ylabel('shape functions')


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We can then define the following shape functions $S_{ij}$ that gives the displacement field in the beam as a function of the displacement and rotations at the ends:
$$
\begin{bmatrix}
u(s)\
v(s)
\end{bmatrix}
=
S(s)U
$$
We can calculate the matrix S  by using the derivative to extract the coefficients with respect to the components of $U$
Can you write Python code for it?

S = sp.Matrix([[u.diff(Ui) for Ui in U],[v.diff(Ui) for Ui in U]])
S


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We denote by $q(s)=v'(s)$ the rotation field. To calculate it we can use the symbolic differntiation in sympy
Can you write Python code for it?


I am working on the file "01-LinearBeam.ipynb".
The first task is:
We show below how to plot the element and the nodes in python using matplotlib (http://matplotlib.org)
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.plot(xnodes,ynodes,'o-')


I am working on the file "01-Phonemes-to-words.ipynb".
The first task is:
Create an instance of the Wrapper
Can you write Python code for it?

import seq2seq_wrapper

import importlib
importlib.reload(seq2seq_wrapper)

model = seq2seq_wrapper.Seq2Seq(xseq_len=xseq_len,
                               yseq_len=yseq_len,
                               xvocab_size=xvocab_size,
                               yvocab_size=yvocab_size,
                               ckpt_path='ckpt/cmudict/',
                               emb_dim=emb_dim,
                               num_layers=3
                               )


I am working on the file "01-Phonemes-to-words.ipynb".
The first task is:
Create data generators
Read data_utils.py for more information
Can you write Python code for it?

val_batch_gen = data_utils.rand_batch_gen(validX, validY, 16)
train_batch_gen = data_utils.rand_batch_gen(trainX, trainY, 128)


I am working on the file "01-Phonemes-to-words.ipynb".
The first task is:
Let us decode and see the words
Can you write Python code for it?

for oi in output:
    print(data_utils.decode(sequence=oi, lookup=data_ctl['idx2alpha'],
                           separator=''))


I am working on the file "01-Phonemes-to-words.ipynb".
The first task is:
Predict
Can you write Python code for it?

output = model.predict(sess, val_batch_gen.__next__()[0])
print(output.shape)

output


I am working on the file "01-Phonemes-to-words.ipynb".
The first task is:
Restore last saved session from disk
Can you write Python code for it?


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
*Se tentarmos estimar a performance do modelo no mesmo conjunto de dados que foi usado para treiná-lo, obteremos uma estimativa enviesada já que o modelo foi treinado para minimizar sua taxa de erro exatamente nos exemplos presentes no conjunto de treinamento. Para estimar corretamente o quão bem o modelo se comportará na prática, ele precisa se testado em um conjunto de dados com o qual ele nunca teve contato.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

lr = # Treine um modelo LinearRegression aqui usando o conjunto de treinamento

lr.predict(6)

# Calcule os preços previstos pelo modelo treinado
predicted_prices = lr.predict(x)

# Crie um scatterplot dessas duas propriedades usando plt.scatter()
plt.scatter(rooms, prices)
# Crie um line plot exibindo os valores previstos em vermelho
plt.plot(rooms, predicted_prices, 'r')
# Crie labels para os eixos X e Y
plt.xlabel('Number of rooms')
plt.ylabel('House price')
# Exiba o gráfico
plt.show()


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
A chamada values.reshape(-1, 1) é necessária nesse caso porque o scikit-learn espera que os preditores estejam na forma de uma matriz - isto é, em um formato de array bidimensional. Como estamos usando apenas um preditor, o pandas acaba representando isso como um array unidimensional, então precisamos "reformatá-lo" em uma "matriz de uma coluna só". Esse passo não é necessário se estivermos usando mais de um preditor para treinar um modelo do scikit-learn, como será visto no próximo exemplo.
Agora que temos o dataset isolado em preditores e saídas, eles precisam ser divididos em dois conjuntos diferentes: um conjunto de treinamento e um conjunto de teste.
Esse passo é necessário caso você precise estimar o quão bem seu modelo treinado se comportará quando for usado para prever preços de novas casas: é necessário usar o conjunto de treinamento para treinar o modelo e então calcular a sua taxa de erros no conjunto de teste.
Can you write Python code for it?

# Use a função train_test_split() do scikit-learn para dividir os dados em dois conjuntos.
# http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html
from sklearn.model_selection import train_test_split

RANDOM_STATE = 4321
xtr, xts, ytr, yts = # Chame a função train_test_split aqui


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
Neste exercício, usaremos o dataset [Boston Housinh]((http://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html) para prever preços de casas a partir de características delas e de sua vizinhança.
Can you write Python code for it?

# Carregue o arquivo 'datasets/boston.csv' usando o pandas


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
O método describe() apenas mostra estatísticas de features numéricas. Se uma feature contém strings, por exemplo, ele não será capaz de mostrar informações sobre ela.
Visualização de dados
Após ler os dados em um DataFrame do pandas e ter obtido uma visão geral do dataset, podemos criar gráficos para visualizar o "formato" desses dados.
Usaremos a biblitoeca Matplotlib para criar esses gráficos.
Exemplo
Suponha que lhe seja dada a seguinte informação sobre quatro datasets:
Can you write Python code for it?

datasets = pd.read_csv('./datasets/anscombe.csv')

for i in range(1, 5):
    dataset = datasets[datasets.Source == 1]
    print('Dataset {} (X, Y) mean: {}'.format(i, (dataset.x.mean(), dataset.y.mean())))

print('\n')
for i in range(1, 5):
    dataset = datasets[datasets.Source == 1]
    print('Dataset {} (X, Y) std deviation: {}'.format(i, (dataset.x.std(), dataset.y.std())))

print('\n')
for i in range(1, 5):
    dataset = datasets[datasets.Source == 1]
    print('Dataset {} correlation between X and Y: {}'.format(i, dataset.x.corr(dataset.y)))


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
O método head() imprime as primeiras cinco linhas por padrão. Ele pode receber opcionalmente um argumento que especifique quantas linhas devem ser exibidas, como boston.head(n=10).
Can you write Python code for it?

# Use o método info() para exibir algumas informações sobre o dataset


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
O método info() exibe vários detalhes sobre o dataset, como a sua quantidade de linhas, quais  features estão presentes, qual é o tipo de cada feature e se existem valores em branco.
Can you write Python code for it?

# Use o método describe() apra exibir algumas estatísticas do dataset


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
Pandas permite a leitura de nossos dados a partir de diferentes formatos. Veja esse link para uma lista de formatos suportados e as respectivas funções usadas para lê-los.
O tipo usado pelo pandas para representar essa tabela com nosso dataset carregado é chamada de DataFrame.
Can you write Python code for it?

# Use o método head() para exibir as primeiras cinco linhas do dataset


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
Podemos agora usar a função mean_squared_error do Scikit-Learn para calcular o erro total médio do modelo nos dados do conjunto de teste.
Can you write Python code for it?

# Use o conjunto de testes para avaliar a performace do modelo
from sklearn.metrics import mean_squared_error

# Calcule o mean_squared_error do modelo aqui


I am working on the file "01-Predicao de precos de casas.ipynb".
The first task is:
Todos eles possuem aproximadamente a mesma média, desvio-padrão e correlação. Quão parecidos esses datasets devem ser?

Esse conjunto de datasets são conhecidos como o Quarteto de Anscombe e eles costumam ser usados para ilustrar como confiar apenas em estatísticas como uma forma de caracterizar conjuntos de dados podem induzir a conclusões incorretas.
Can you write Python code for it?

# Na primeira vez que o matplotlib é importado, pode ser exibido algum tipo
# de alerta relacionado às fontes do sistema dependendo da sua configuração
import matplotlib.pyplot as plt
# Essa linha permite que os gráficos gerados apareçam diretamente no notebook
# ao invés de serem abertos em uma janela ou arquivo separado.
%matplotlib inline

# Extraia os preços das casas e a quantidade média de cômodos em duas variáveis separadas
prices =
rooms =

# Crie um scatterplot dessas duas features usando plt.scatter()

# Especifique labels para os eixos X e Y

# Exiba o gráfico

# Extraia os preços das casas e o índice de poluição da vizinhança em duas variáveis separadas
prices =
nox =

# Crie um scatterplot dessas duas features usando plt.scatter()

# Especifique labels para os eixos X e Y

# Exiba o gráfico


I am working on the file "01-Regression/CV.ipynb".
The first task is:
In the first iteration, the accuracy is 58%, in the second iteration the accuracy is 65% and so on.
Generally a 10-folds is a good compromise between speed and accuracy.
A model with polynomial features
Now we try to improve our simple model's accuracy by adding polynomial terms to our feature, i.e. transforming it into its power.
Can you write Python code for it?

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=2) # quadratic function

  # we first transform our data
x_train2 = poly.fit_transform(x_train)
x_test2  = poly.fit_transform(x_test)

x_train2[0:5]


I am working on the file "01-Regression/CV.ipynb".
The first task is:
It looks better, so we can try with a cubic function:
Can you write Python code for it?

poly = PolynomialFeatures(degree=3) # cubic function
x_train3 = poly.fit_transform(x_train) # again, transform the data
x_test3  = poly.fit_transform(x_test)

model3 = linear_model.LinearRegression()
model3.fit(x_train3, y_train) # fit a new model

model3.score(x_test3, y_test)


I am working on the file "01-Regression/CV.ipynb".
The first task is:
Note: I use .iloc to index the Pandas dataframe to avoid that some data become NaN after the split and get this error:
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Can you write Python code for it?

scores = []
for k, (train_index, test_index) in enumerate(kf):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    Y_train, Y_test = Y.iloc[train_index], Y.iloc[test_index]
    model.fit(X_train, Y_train)
    scoreK = model.score(X_train, Y_train)
    scores.append(scoreK)

    print('Fold: {:2d}, Acc: {:.3f}'.format(k+1, scoreK))

print('\nCV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores),
                                                np.std(scores)))


I am working on the file "01-Regression/CV.ipynb".
The first task is:
Note: not printing each fold here because LOO is computationally more expensive.
Can you write Python code for it?

scores = []
for k, (train_index, test_index) in enumerate(LOO):

    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    Y_train, Y_test = Y.iloc[train_index], Y.iloc[test_index]

    model.fit(X_train, Y_train)
    scoreK = model.score(X_train, Y_train)
    scores.append(scoreK)

print('\nCV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores),
                                                np.std(scores)))


I am working on the file "01-Regression/CV.ipynb".
The first task is:
Now the accuracy of 0.606 is a much more reliable measure than the one from the test-set approach, which could change depending on which test set has been randomly selected.
A generic cross-validation score evaluating function.
The sklearn helper cross_val_score can quickly estimate the score of a model, using one CV approach, therefore substituting the above code with a more compact form:
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

# We are passing the entirety of X and y, not X_train or y_train
scores = cross_val_score(model, X, Y, cv=KFold(n_splits=5, shuffle=True,
                                              random_state=1))
print ("Cross-validated scores:", scores)
print('CV accuracy: {:.3f}  +/- {:.3f} '.format (np.mean(scores), np.std(scores)))


I am working on the file "01-Regression/CV.ipynb".
The first task is:
There are NaNs. We remove those observations before proceeding.
Can you write Python code for it?

autoData[autoData.isnull().any(axis=1)]

autoData.dropna(axis=0, inplace=True) # Drop any row that has NaNs within it

autoData[autoData.isnull().any(axis=1)]

  # horsepower feature is our input X
X = autoData[['horsepower']]
X.shape

X.describe()

m = X.shape[0]  # number of examples
m

  # mpg is our output Y, what we want to predict
Y = autoData.mpg.copy() # copy “y” column values out
len(Y)


I am working on the file "01-Regression/CV.ipynb".
The first task is:
These are the model's coefficients:
Can you write Python code for it?


I am working on the file "01-Regression/CV.ipynb".
The first task is:
These results show that a model which predicts mpg using a quadratic function of horsepower performs better than a model that involves only a linear function of horsepower, and there is little evidence in favour of a model that uses a cubic function of horsepower.
Using the CV approach to tune parameters (model selection)
We can repeat this procedure for increasingly complex polynomial fits.
This is called model selection.
To automate the process, we use a for()  loop, which iteratively fits polynomial regressions for polynomials of order i = 1 to i = 15, computes the associated cross-validation score and stores it.
Can you write Python code for it?

# search for an optimal value of poly n for LR

  # range of n we want to try
n_range = range(1, 15)
  # empty list to store scores
n_scores = []

model_n = linear_model.LinearRegression()

  # 1. we will loop through values of n
for n in n_range:
      # 2. transform features with n degree
    poly = PolynomialFeatures(degree=n)
    x_val_n = poly.fit_transform(X)

      # 3. obtain cross_val_score for model_n
    scores = cross_val_score(model_n, x_val_n, Y, cv=KFold(n_splits=5,
                                                           shuffle=True,
                                                           random_state=1))
    # 4. append mean of scores to n_scores list
    n_scores.append(scores.mean())

    #print ("Cross-validated scores:", scores)

print('CV accuracy: {:.3f} +/- {:.3f} '.format(np.mean(n_scores), np.std(n_scores)))

# in essence, this is basically running the k-fold cross-validation method n times :
print('Length of list', len(n_scores))
print('Max of list', max(n_scores))

# plot how accuracy changes as we vary the polynomial degree

plt.plot(n_range, n_scores)
plt.xlabel('Degree for horsepower')
plt.ylabel('Cross-validated accuracy')


I am working on the file "01-Regression/CV.ipynb".
The first task is:
We are interested in the mpg and the horsepower features.
Can you write Python code for it?

autoData.head(5)

autoData.info()

autoData.isnull().values.any()


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Almost unrecognisable.
Will the linear regression model be able to reconstruct the audio?
Can you write Python code for it?

y_test = test[test_samples:] # remaining audio part is the label


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Can you hear it? Now it's only the first syllable, "ze" ...
But we can even delete more and leave only the first quarter!
Can you write Python code for it?

Provided_Portion = 0.25 # let's delete three quarters of the audio!

test_samples = int(Provided_Portion * n_audio_samples)
X_test = test[0:test_samples] # first ones

wavfile.write('../outputs/ChoppedTestClip.wav', sample_rate, X_test)
IPython.display.Audio("../outputs/ChoppedTestClip.wav")


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Duplicate the same process for X_train, y_train.
Can you write Python code for it?

X_train = train[:, 0:test_samples] # first ones: data
y_train = train[:, test_samples:]  # remaining ones: label


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Embedding the audio file.
Note that this is not working directly in GitHub (I think all JavaScript is stripped out), fork it or download it to play the audio
Can you write Python code for it?

from IPython.display import Audio
Audio("../outputs/OriginalTestClip.wav")


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
Obviously, if you look only at Rsquared it seems that it was a totally useless result.
But let's listen to the generated audio.
First, take the first Provided_Portion portion of the test clip, the part you fed into your linear regression model. Then, stitch that
together with the abomination the predictor model generated for you,
and then save the completed audio clip:
Can you write Python code for it?

completed_clip = np.hstack((X_test, y_test_prediction))
wavfile.write('../outputs/ExtrapolatedClip.wav', sample_rate, completed_clip[0])

IPython.display.Audio("../outputs/ExtrapolatedClip.wav")


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
There are 500 recordings, 50 of each digit.
Each .wav file is actually just a bunch of numeric samples, "sampled"
from the analog signal. Sampling is a type of discretization. When we mention 'samples', we mean observations. When we mention 'audio samples', we mean the actually "features" of the audio file.
The goal of this notebook is to use multi-target, linear regression to generate by extrapolation, the missing portion of the test audio file.
Each one audio_sample features will be the output of an equation,
which is a function of the provided portion of the audio_samples:
missing_samples = f(provided_samples)
prepare the data
Convert zero into a DataFrame and set the dtype to np.int16, since the input audio files are 16 bits per sample. This is important otherwise the produced audio samples will be encoded as 64 bits per sample and will be too short.
Can you write Python code for it?

import numpy as np
import pandas as pd

zeroDF = pd.DataFrame(zero, dtype=np.int16)

zeroDF.info()


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
carve out the labels Y
The data will have two parts: X and y (the true labels).
X is going to be the first portion of the audio file, which we will be providing the computer as input (the "chopped" audio).
y, the "label", is going to be the remaining portion of the audio file.   In this way the computer will use linear regression to derive the missing portion of the sound file based off of the training data it has received!
ProvidedPortion is how much of the audio file will be provided, in percent. The remaining percent of the file will be generated via linear extrapolation.
Can you write Python code for it?

Provided_Portion = 0.5 # let's delete half of the audio

test_samples = int(Provided_Portion * n_audio_samples)
X_test = test[0:test_samples] # first ones

IPython.display.Audio(data=X_test, rate= sample_rate)


I am working on the file "01-Regression/LRaudio.ipynb".
The first task is:
split the data into training and testing sets
There are 50 takes of each clip. You want to pull out just one of them, randomly, and that one will NOT be used in the training of the model. In other words, the one file we'll be testing / scoring on will be an unseen sample, independent to the rest of the training set.
Can you write Python code for it?

from sklearn.utils.validation import check_random_state

rng   = check_random_state(7)
random_idx = rng.randint(zero.shape[0])

test  = zero[random_idx] # the test sample
train = np.delete(zero, [random_idx], axis=0)

print(train.shape)
print(test.shape)


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
Is there synergy among the advertising media?
Adding radio to the model leads to a substantial improvement in R2. This implies that a model that uses TV and radio expenditures to predict sales is substantially better than one that uses only TV advertising.
In our previous analysis of the Advertising data, we concluded that both TV and radio seem to be associated with sales. The linear models that formed the basis for this conclusion assumed that the effect on sales of increasing one advertising medium is independent of the amount spent on the other media.
For example, the linear model states that the average effect on sales of a one-unit increase in TV is always β1, regardless of the amount spent on radio.
However, this simple model may be incorrect. Suppose that spending money on radio advertising actually increases the effectiveness of TV advertising, so that the slope term for TV should increase as radio increases. In this situation, given a fixed budget of $100K spending half on radio and half on TV may increase sales more than allocating the entire amount to either TV or to radio.
In marketing, this is known as a synergy effect. The figure above suggests that such an effect may be present in the advertising data. Notice that when levels of either TV or radio are low, then the true sales are lower than predicted by the linear model. But when advertising is split between the two media, then the model tends to underestimate sales.
Can you write Python code for it?

modelSynergy = sm.ols('Sales ~ TV + Radio + TV*Radio', ad).fit()
modelSynergy.summary().tables[1]


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
Let's plot the actual values as red points and the model predictions as a cyan plane:
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
fig.suptitle('Regression: Sales ~ Radio + TV Advertising')
ax = Axes3D(fig)

ax.set_xlabel('Radio')
ax.set_ylabel('TV')
ax.set_zlabel('Sales')
ax.scatter(ad.Radio, ad.TV, ad.Sales, c='red')

ax.plot_surface(x,y,z, color='cyan', alpha=0.3)


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
M3 is slightly better than M2 (but remember that R2 always increases when adding new variables) so we call the approach completed and decide that the M2 model with TV and Radio is the good compromise. Adding the newspaper could possibly overfits on new test data.
Next year no budget for newspaper advertising and that amount will be used for TV and Radio instead.
Can you write Python code for it?


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
Now we need the Total Sum of Squares (TSS): the total variance in the response Y, and can be  thought of as the amount of variability inherent in the response before the regression is performed.
The distance from any point in a collection of data, to the mean of the data, is the deviation.
Can you write Python code for it?

y_mean = np.mean(ad.Sales) # mean of sales

TSS = np.sum((ad.Sales - y_mean)**2)
TSS


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
Plotting the model
The M2 model has two variables therefore can be plotted as a plane in a 3D chart.
Can you write Python code for it?


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
RSE is 1.68 units while the mean value for the response is 14.02, indicating a percentage error of roughly 12%.
Second, the R2 statistic records the percentage of variability in the response that is explained by the predictors.
The predictors explain almost 90% of the variance in sales.
Summary
statsmodels has a handy function that provides the above metrics in one single table:
Can you write Python code for it?


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
The M2 model can be described by this equation:
Sales = 0.19 * Radio + 0.05 * TV + 2.9  which I can write as:
 0.19x + 0.05y - z + 2.9 = 0
Its normal is (0.19, 0.05, -1)
and a point on the plane is (-2.9/0.19,0,0) = (-15.26,0,0)
Can you write Python code for it?

normal = np.array([0.19,0.05,-1])
point  = np.array([-15.26,0,0])
# a plane is a*x + b*y +c*z + d = 0
# [a,b,c] is the normal. Thus, we have to calculate
# d and we're set
d = -np.sum(point*normal) # dot product
# create x,y
x, y = np.meshgrid(range(50), range(300))
# calculate corresponding z
z = (-normal[0]*x - normal[1]*y - d)*1./normal[2]


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
The R2 for this model is 96.8 %, compared to only 89.7% for the model M2 that predicts sales using TV and radio without an interaction term. This means that (96.8 − 89.7)/(100 − 89.7) = 69% of the variability in sales that remains after fitting the additive model has been explained by the interaction term.
A linear model that uses radio, TV, and an interaction between the two to predict sales takes the form:
 sales = β0 + β1 × TV + β2 × radio + β3 × (radio×TV) + ε
Can you write Python code for it?


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
The lowest RSS and the highest R2 are for the TV medium.
Now we have a best model M1 which contains TV advertising.
We then add to this M1 model the variable that results
in the lowest RSS for the new two-variable model.
This approach is continued until some stopping rule is satisfied.
Can you write Python code for it?

modelTVRadio = sm.ols('Sales ~ TV + Radio', ad).fit()
modelTVRadio.summary().tables[1]

evaluateModel(modelTVRadio)

modelTVPaper = sm.ols('Sales ~ TV + Newspaper', ad).fit()
modelTVPaper.summary().tables[1]

evaluateModel(modelTVPaper)


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
The model containing only TV as a predictor had an RSS=2103 and an R2 of 0.61
Can you write Python code for it?

modelRadio = sm.ols('Sales ~ Radio', ad).fit()
modelRadio.summary().tables[1]

evaluateModel(modelRadio)

modelPaper = sm.ols('Sales ~ Newspaper', ad).fit()
modelPaper.summary().tables[1]

evaluateModel(modelPaper)


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
Unfortunately, there are a total of 2^p models that contain subsets of p variables.
For three predictors, it would still be manageable, only 8 models to fit and evaluate but as p increases, the number of models grows exponentially.
Instead, we can use other approaches. The three classical ways are the forward selection (start with no features and add one after the other until a threshold is reached); the backward selection (start with all features and remove one by one) and the mixed selection (a combination of the two).
We try here the forward selection.
Forward selection
We start with a null model (no features), we then fit three (p=3) simple linear regressions and add to the null model the variable that results in the lowest RSS.
Can you write Python code for it?

modelTV = sm.ols('Sales ~ TV', ad).fit()
modelTV.summary().tables[1]

evaluateModel(modelTV)


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
We interpret these results as follows: for a given amount of TV and newspaper advertising, spending an additional 1000 dollars on radio advertising leads to an increase in sales by approximately 189 units.
In contrast, the coefficient for newspaper represents the average effect (negligible) of increasing newspaper spending by 1000 dollars while holding TV and radio fixed.
Is at least one of the features useful in predicting Sales?
We use a hypothesis test to answer this question.
The most common hypothesis test involves testing the null hypothesis of:
H0: There is no relationship between the media and sales versus the alternative hypothesis
Ha: There is some relationship between the media and sales.
Mathematically, this corresponds to testing
H0: β1 = β2 = β3 = β4 = 0
versus
Ha: at least one βi is non-zero.
This hypothesis test is performed by computing the F-statistic
The F-statistic
We need first of all the Residual Sum of Squares (RSS), i.e. the sum of all squared errors (differences between actual sales and predictions from the regression line). Remember this is the number that the regression is trying to minimise.
Can you write Python code for it?

y_pred = modelAll.predict(ad)

import numpy as np
RSS = np.sum((y_pred - ad.Sales)**2)
RSS


I am working on the file "01-Regression/LRfeatures.ipynb".
The first task is:
When there is no relationship between the response and predictors, one would expect the F-statistic to take on a value close to 1.
On the other hand, if Ha is true, then we expect F to be greater than 1.
In this case, F is far larger than 1:  at least one of the three advertising media must be related to sales.
How strong is the relationship?
Once we have rejected the null hypothesis in favor of the alternative hypothesis, it is natural to want to quantify the extent to which the model fits the data.
The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the R2 statistic (the square of the correlation of the response and the variable, when close to 1 means high correlation).
Can you write Python code for it?

RSE = np.sqrt((1/(n-2))*RSS);
RSE

np.mean(ad.Sales)

R2 = 1 - RSS/TSS;
R2


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
85 SIN$ (per defect or excess) is the biggest difference done by the model.
Don't confuse errors and residuals.
The error is the deviation of the observed value from the (unobservable) true value of a quantity of interest (for example, a population mean), and the residual is the difference between the observed value and the estimated value of the quantity of interest (for example, a sample mean).
We can learn many things from the residuals.
One is that their distribution and properties can give us an indication about the model fit.
Residuals should not show any pattern
The residuals and their plot can highlight a poor model fit.
Here we plot the residuals versus the fitted values:
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(simpleModel.fittedvalues, residuals, 'o') # as round marks

   # pretty-up the plot
ax.plot ((0, 1200), (0,0)) # draw also a line at y=0
fig.suptitle("Residuals versus fitted prices")
ax.set_ylabel('Residuals [SIN $]')
ax.set_xlabel('Price [SIN $]')
ax.grid(True)


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
As you can see, the slope is the same as the previous model, only the intercept shifted.
This is always valid when you shift your X values.
Thus $500.1 is the expected price for the average sized diamond of the initial dataset (=0.2042 carats).
This is an intercept making much more sense.
You can shift the X input by a certain value but you can also re-scale them.
This can be useful when one unit is quite large and we would prefer a finer unit.
For example, in our case, one carat is worth of almost 4K SIN$ and could make sense to talk about tenth of carats (= 1/10).
Can you write Python code for it?

Xtenth = diamondData.carats *10 # rescale the X

Xtenth = sm.add_constant(Xtenth)
tenthModel = sm.OLS(diamondData.price, Xtenth).fit() # again fit the model
tenthModel.params


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Confidence
How accurately can we predict the diamond prices?
For any given weight in carats, what is our prediction for the price, and what is the accuracy of this prediction?
In statistics, a sequence of random variables is independent and identically distributed (IID) if each random variable has the same probability distribution as the others and all are mutually independent.
Inference for regression
In the case of regression with IID sampling assumptions and normal distributed residuals, the statistics for our estimated beta coefficients:
- will follow a finite sample T-distributions and be normally distributed
- can be used to test null hypotesis
- can be used to create a confidence interval
In probability and statistics, the t-distribution is any member of a family of continuous probability distributions that arises when estimating the mean of a normally distributed population in situations where the sample size is small and population standard deviation is unknown.
Whereas a normal distribution describes a full population, t-distributions describe samples drawn from a full population.
The t-distribution becomes closer to the normal (Gaussian) distribution as its degrees of freedom (df) increases.

The t-distribution arises in a variety of statistical estimation problems where the goal is to estimate an unknown parameter, such as a mean value, in a setting where the data are observed with additive errors.
If the population standard deviation of these errors is unknown and has to be estimated from the data, the t-distribution is often used to account for the extra uncertainty that results from this estimation.
Confidence intervals and hypothesis tests are two statistical procedures in which the quantiles of the sampling distribution of a particular statistic (e.g. the standard score) are required.
Confidence levels are expressed as a percentage (for example, a 95% confidence level). It means that should you take a sample over and over again, 95 percent of the time your results will match the...
Can you write Python code for it?

  # prepare the data
y = diamondData.price
x = diamondData.carats
n = len(y)

  # calculate beta1
beta1 = (np.corrcoef (y,x) * np.std(y) / np.std(x))[0][1]
beta1

  # calculate beta0
beta0 = np.mean(y) - beta1 * np.mean(x)
beta0


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Confidence intervals
A 95% confidence interval is defined as a range of values such that with 95% probability, the range will contain the true unknown value of the parameter.
Recall of quantiles and percentiles of a distribution
If you were the 95th percentile on an exam, it means that 95% of people scored worse than you and 5% scored better.
These are sample quantiles.
Now for a population: the i-th quantile of a distribution with distribution function F is simply the point x_i so that :
F(x_i) = i
A percentile is simply a quantile with i expressed as a percent.
The 95th percentile of a distribution is the point where the probability that a random variable drawn from the population is less than 95%.
Approximately 68%, 95% and 99% of the normal density lies respectively within 1,2 and 3 standard deviations from the mean.
Estimating the Standard Errors
Now we need to calculate the standard errors.
Can you write Python code for it?

ssx = sum((x - np.mean(x))**2)

  # calculate standard error for beta0
seBeta0 = (1 / n + np.mean(x) ** 2 / ssx) ** 0.5 * sigma
seBeta0

  # calculate standard error for beta1
seBeta1 = sigma / np.sqrt(ssx)
seBeta1


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Conveniently, residuals are also stored in the results attribute resid:
Can you write Python code for it?

residuals = simpleModel.resid

max(abs(residuals))


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Estimating residual variation
The residual variation measures how well the regression line fits the data points.
It is the variation in the dependent variable (Price) that is not explained by the regression model and is represented by the residuals. We want the residual variation to be as small as possible.
Each residual is distributed normally with mean 0 and variance = sigma_squared.
We have previously seen that the ML Estimate of variance, sigma_squared, is sum(residuals squared) divided by n   and we called it the Mean Squared Error (MSE).
Most people use (n-2) instead of n so that the estimator is unbiased (the -2 is accounting for the degrees of freedom for intercept and slope).
The square root of the estimate, sigma, is called the Root Mean Squared Error (RMSE).
We want both MSE and RMSE to be as small as possible.
In our diamonds example the estimated residual variation (unbiased RMSE) is :
Can you write Python code for it?

n = len(y)
MSE = sum(residuals**2) / (n-2)
RMSE = np.sqrt(MSE)
RMSE


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
I expect to pay around 485 SIN $.
Or I can use the predict() function available in the statsmodel package:
Can you write Python code for it?

newDiamond = [1, 0.2] # remember to add always the intercept!

simpleModel.predict(newDiamond)


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
If the p-value is less than the significance level (0.05 in our case) then the model explains the variation in the response.
T confidence intervals
For small samples we can use the t-distribution to calculate the confidence intervals.
The t-distribution has been invented by William Gosset in 1908 and is indexed by a degrees of freedom (df): it gets more like a standard normal as df gets larger.
ppf() is the Percent Point Function from SciPy.stats, that has as input the quantile (for a 2-sided 95% probability) and the degrees of freedom.
Can you write Python code for it?

alpha=0.05 # confidence interval for two-sided hypothesis
qt = 1 - (alpha/2)  # =0.975 for a 2-sided 95% probability
t_value = t.ppf(qt, df=degreesOfFreedom)
t_value


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Instead of X as input for the model, we take X centered around the mean, i.e. we shift X of a value equal to the sample mean:
Can you write Python code for it?

XmeanCentered = diamondData.carats - diamondData.carats.mean()

XmeanCentered = sm.add_constant(XmeanCentered) # this append a column of ones
meanCenteredModel = sm.OLS(diamondData.price, XmeanCentered).fit() # fit a new model
meanCenteredModel.params


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Let's summarise the values calculated until now:
Can you write Python code for it?

print ("##            Estimate     Std. Error    t-value        p-value")
print ("Intercept: ", beta0, seBeta0, tBeta0, pBeta0)
print ("Carats:     ", beta1, seBeta1, tBeta1, pBeta1)


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Now we can calculate the intervals for beta0 and beta1:
Can you write Python code for it?

limits=[-1,1]
[beta0 + i*t_value*seBeta0 for i in limits]

[beta1 + i*t_value*seBeta1 for i in limits]


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Of course, it is also available from the model results:
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
RMSE can be used to calculate the standardized residuals too.
They equal the value of a residual divided by an estimate of its standard deviation (so, RMSE).
Large standardized residuals are an indication of an outlier.
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Result: for 0.16, 0.27, and 0.34 carats, we predict the prices to be 335.74, 745.05, 1005.52 (SIN) dollars
Model fit
How strong is the relationship?
We know that there is a relationship between diamonds carats and prices, we would like to know the strength of this relationship. In other words, given a certain diamond weight, can we predict the price with a high level of accuracy? This would be a strong relationship. Or is a prediction of prices based on weight  only slightly better than a random guess? This would be a weak relationship.
Residuals
As we have seen previously, the residuals are the difference between the observed (y) and the predicted outcome (y_hat):
Can you write Python code for it?

y = diamondData.price

y_hat = simpleModel.fittedvalues

max (abs (y - y_hat))


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Summarizing the variation: R-squared
The total variation is the residual variation (variation after removing predictors) plus the systematic variation (variation explained by regression model).
R-squared is the percentage of variability explained by the regression model:
R-squared = explained / total variation = 1 - residual / total variation
R-squared is always between 0 and 1 (0% and 100%):
- 0% indicates that the model explains none of the variability of the response data around its mean.
- 100% indicates that the model explains all the variability of the response data around its mean.
In general, the higher the R-squared, the better the model fits your data.
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
Summary of statistic values
The statsmodel package offers an overview of the model values, similar to what we calculated above:
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
The intercept (beta0) is -259.6 and the slope (beta1) is 3721
Therefore our simple (one variable only) model looks like:
Diamond Price = -259.6 + 3721 * Diamond Weight
We can plot the obtained regression line together with the input data X.
We can do it by drawing a line using the beta parameters just calculated or also plotting the fitted values:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.scatter(diamondData.carats, diamondData.price)
   # draw linear regression line
x = [0.1,0.4]
y = [-259.6 + 3721 * i for i in x]
ax.plot(x, y)
   # alternatively, plot the fitted values
#y_hat = simpleModel.fittedvalues
#ax.plot(diamondData.carats, y_hat)
   # pretty-up the plot
fig.suptitle("Relation between diamonds' price and weight")
ax.set_ylabel('Price [SIN $]')
ax.set_xlabel('Weight [carat]')
ax.grid(True)


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
The intercept is the same as in the original model, only the slope coefficient is divided by 10.
This is always valid when you re-scale the X values.
We expect a 372.102 (SIN) dollar change in price for every 1/10th of a carat increase in mass of diamond.
Predicting the price of a diamond
Once we have a model of the relation, we can use it for predictions.
The statsmodel package has a method predict() associated to each model, that takes a new set of input and will output the predicted values, according to the model.
Let's say that I want to buy a 0.2 carats diamond. How much should I expect it to cost?
I can use the beta parameters estimated by the model and just putting them into the linear regression formula:
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
The mean of the residuals is expected to be zero.
This comes directly from the fact that OLS minimises the sum of square residuals
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
The residuals should be distributed uniformly without showing any pattern and having a constant variance.
If we see from the residuals vs. fitted plot that the variance of the residuals increases as the fitted values increase (takes a form of a horizontal cone) this is the sign of heteroscedasticity.
Homoscedasticity describes a situation in which the error term (that is, the “noise” or random disturbance in the relationship between the independent variables and the dependent variable) is the same across all values of the independent variables.
Heteroscedasticity (the violation of homoscedasticity) is present when the size of the error term differs across values of an independent variable.
Examining the scatterplot of the residuals against the predicted values of the dependent variable would show the classic cone-shaped pattern of heteroscedasticity.

Other patterns could be:
- curvilinear  (indicate is non-linear / missing higher-order term)
- a single point is far away from zero (probably an outlier)
- a single point is far away from the others in the x-direction (probably an influential point)
Residuals should be normally distributed
The sum of the residuals is expected to be zero (when there is an intercept).
This follows directly from the normal equation, i.e. the equation that the OLS estimator solves.
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
The standard error of the parameter measures the precision of the estimate of the parameter.
The smaller the standard error, the more precise the estimate.
Hypothesis testing
Hypothesis testing is concerned with making decisions using data.
A null hypothesis is specified that represents the status quo, usually labeled H0.
The null hypothesis is assumed true and statistical evidence is required to reject it in favour of an alternative hypothesis.

Consider testing H0: mu = mu0
If we take the set of all possible values for which you fail to reject H0, this set is an alfa% confidence interval for mu, alfa depending on the set.
Getting the T-values
Testing for null hypotesis H0: estimated beta0 and beta1 are equal to real coefficients
Dividing the parameter by its standard error calculates a t-value:
Can you write Python code for it?

tBeta0 = beta0 / seBeta0
tBeta0

tBeta1 = beta1 / seBeta1
tBeta1


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
This is one of the assumptions for regression analysis: residuals should have a normal (or Gaussian) distribution.
Can you write Python code for it?


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
We are quite close to a perfect model.
You can use a fitted line plot to graphically illustrate different R-squared values.
The more variation that is explained by the model, the closer the data points fall to the line. Theoretically, if a model could explain 100% of the variation, the fitted values would always equal the observed values and all of the data points would fall on the  line.
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(simpleModel.fittedvalues, diamondData.price, 'o') # as round marks
   # identity line
plt.plot(simpleModel.fittedvalues, simpleModel.fittedvalues, '--')
   # pretty-up the plot
fig.suptitle("Relation between estimated and actual diamonds' prices")
ax.set_ylabel('Estimated Price [SIN $]')
ax.set_xlabel('Actual Price [SIN $]')
ax.grid(True)


I am working on the file "01-Regression/LRinference.ipynb".
The first task is:
You can see all the values available using dir():
Can you write Python code for it?


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
As you can see, in this example the baseline never predicts the correct value, only a couple of times goes close but – worst of all – does not follow the trend at all: if the temperature is lower or higher than average, it doesn’t matter.
Let’s see now the more sophisticated model that Ashenfelter used: the Linear Regression Model and see if is better than our baseline.
The linear regression is one (a simple one) of the supervised learning algorithms part of machine learning.
Supervised learning means that examples with the right answers (labeled data) are used to build (or train) the model which then extrapolates and predicts outcomes based on different inputs.
Unsupervised learning would be without any previously labeled data.
In our case we have labeled data: the wine price of the past years. The idea is to “train” our model on these examples such as it will be able to predict future prices.
The Regression Model
The basic regression model is the single variable regression model, where the independent variable – x – is only one:
$$ y = \beta_{0} + \beta_{1}\cdot x$$
The goal of linear regression is to create a predictive line through the data.
Beta0 and Beta1 are specially selected coefficients.
Beta0 = intercept of the line
Beta1 = slope of the line
Let's see an example, using as independent variable x the average temperature (AGST) which was the one most correlated with the wine price:
y = wine price
x = average temperature.
This means the price will increase / decrease linearly based on increase / decrease of temperature.
Let's see how the relation would look like with y = 0.5 * x - 1.25 (beta coefficients arbitrarily picked up, just to follow the trend more or less):
Can you write Python code for it?

import numpy as np
x = np.linspace(15, 18, 10)  # 100 evenly-spaced values from 0 to 50
y = 0.5 * x - 1.25  # a simple model

plt.scatter(wine.AGST, wine.Price)
plt.hlines(baseline, 15, 18, color = 'red', label="Baseline")
plt.plot(x,y, label="Simple linear regression")
plt.legend(loc="upper left")
plt.title("Wine price vs. Average Growing Season Temperature")
plt.xlabel("Temperature [Celsius degrees]")
plt.ylabel("Log of Price")
plt.grid()
plt.show()


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
Here wine.Price is a vector (specifically, it is a pandas Series which is based on numpy arrays) which can be subtracted to a scalar (our baseline), squared and produces a new vector, which can be the input of the numpy function sum() and this just outputs the sum of its elements.
As baseline we choose simply the average value of the actual wine prices, as shown above, which has the advantage of being a value independent on any other variable or factor.
As we can see, the SSE metric for this baseline is around 10.15 (the sum of all squared errors).
Let's compare it to the SSE for our simple regression model:
Can you write Python code for it?

        # Get SSE for our simple linear regression (LR) model
predictedPrices = 0.5 * wine.AGST - 1.25
SSE_simpleLR = np.sum((wine.Price - predictedPrices)**2)
SSE_simpleLR


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
How can I use the formula to predict a price?
Let’s say this year the average temperature was 18 degrees, then here is the prediction:
Can you write Python code for it?


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
R2 is by definition a number between 0 and 1 (assuming the SST is not zero):
0 when the SSE from a model is equal to the SST (means there is no improvement) and 1 when SSE is null, means it is a perfect predictive model.
Higher R2, better the model.
R2 is unit-less and universally interpretable but should be used only to compare models inside the same problems. Hard problems may have an R2 close to zero even if the model is good ...
Closed form
We have seen that the regression line best fitting is the one that minimises the sum of these squared losses. But we still need to find it.
One way is to use its closed form.
An equation is said to be a closed-form solution if it solves a given problem in terms of functions and mathematical operations from a given generally accepted set.
The general linear model - as we have seen - is:
$$ \bar{y} = \sum_{i}^{N} (x_{i} \cdot \beta_{i}) $$
The idea is to minimise the costs over all data points, i.e. we can choose as cost function the least squares and to minimise the SSE:
$$  min (\sum_{i}^{N} ((y_{i} - (\beta_{0} + \beta_{1} \cdot x_{i})^{2})) $$
which is a function of two variables : beta0 and beta1.
This is a convex function which has exactly one and only one minimum, that can be found putting the derivative to zero.
Note: it is possible to solve the closed form also for more than two beta variables by using the gradient.
In mathematics, the gradient is a generalisation of the usual concept of derivative to functions of several variables. Given a differentiable function of several variables, its gradient is the vector whose components are the n partial derivatives.
For linear regression, the gradient (usually expressed with the greek symbol nabla) would be:
$$ \nabla g(\beta) = [\frac{\partial g}{\partial \beta_{i}}] $$
where these are n - dimensional vectors; the vector of all beta coefficients is calculated via partial derivatives ( like a derivate with respect to beta_i, treating all other variables as constant).
End of note.
Taking the derivative from
$$...
Can you write Python code for it?

X = wine.AGST   # the known variable
Y = wine.Price  # the variable to predict

numerator = sum(X*Y) - (1/n)*((sum(X)*sum(Y)))
numerator

denominator = sum(X**2) - (1/n)*((sum(X)*sum(X)))
denominator

slope = numerator / denominator
slope  # this is beta1 coefficient

intercept = Y.mean() - slope*(X.mean())
intercept  # this is beta0 coefficient


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
The first data point is then represented by this formula including the residual:
$$ y_{1} = -1.25 + 0.5 \cdot x_{1} + 0.18665$$
Plot the residuals
Let's plot the residuals related to this simple regression line, i.e. the vertical lines separating the actual point from the predicted point.
Can you write Python code for it?

y_bottom = 6
y_top = 9
y_range = y_top - y_bottom

plt.ylim(y_bottom, y_top)
plt.scatter(wine.AGST, wine.Price) # the actual points
plt.plot(x,y, label="Simple linear regression") # the regression line

    # now let's plot the residuals
for i in range(0, len(wine)):
            # first we need the coordinates of the actual point
    x_point = wine.AGST[i]
    y_point = wine.Price[i]
            # then we need the say how long is the vertical line
            # the vertical line must be between 0 and 1
    y1 = (y_point - y_bottom) / y_range  # scale
    y2 = ((0.5 * x_point - 1.25) - y_bottom) / y_range  # scale
            # now we can plot the vertical RED residual line
    plt.axvline(x_point, ymin=y1, ymax=y2, color="red")

plt.title("Residuals of simple model y=0.5x-1.25")
plt.xlabel("Temperature [Celsius degrees]")
plt.ylabel("Log of Price")
plt.grid()


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
The formula seems the same but in this case the vector wine.AGST is multiplied to a scalar and then subtracted to a different scalar [10], then wine.Price is subtracted this time to another vector, the wine.AGST and squared and numpy takes care of this operation [11]. Very practical!
The Sum of Squared Errors for our simple LR model is much smaller than the baseline, which tells us that this model can make smaller errors when predicting wine prices, therefore - according to our chosen metric of fitness - this model is better than the baseline.
Better metrics: RMSE and R-squared
The MSE is the Mean Squared Error and is basically just the average SSE:
Can you write Python code for it?

n = len(wine.Price) # number of examples
MSE_baseline = SSE_baseline / n; MSE_baseline

MSE_simpleLR = SSE_simpleLR / n; MSE_simpleLR


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
The residuals are the vertical red lines
Loss metrics
One way - the most common - to find the best model (i.e., the best choice of coefficients) is to look for the line which has the smallest error terms.
This means that is the line with the smallest residuals in total (i.e., the shortest vertical lines).
The idea is to measure the "closeness" between the actual price and its prediction (the regression line) based on the fact that is better to be off by one than by ten ...
Are there other way to find the best model?
As a matter of fact, yes. For example, I could be interested to find a model that prefer to underestimate a prediction instead of overestimating, because maybe is the latter is a much more expensive error. In this case I would need to weight the error with vertical lines "over" the regression line more than the error "under" it.
In the case of looking for the smallest error, the "loss" function or metric could be to take the absolute loss:
$$ | y_{i} - \bar{y_{i}} | $$
for each data point (y_i = actual value and y_bar = predicted value)  and then sum them all together. We would then look for the line minimising this value.
Even better - with nicer mathematical properties - is the squared loss:
$$ (y_{i} - \bar{y_{i}})^{2}$$
This specific metric - and its goodness of fit - is named SSE: the Sum of Squared Errors between actual values and predictions, which is very easy to calculate using Python numpy and vector operations:
Can you write Python code for it?

SSE_baseline = np.sum((wine.Price - baseline)**2); SSE_baseline


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
The simple model y=0.5*x-1.25 would predict for a Temperature = 17.1167 a price of :
Can you write Python code for it?

predictedPrice = 0.5 * 17.1167 - 1.25; print (predictedPrice)


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
Therefore the final linear regression model is :
winePrice = -3.4178 + 0.6351 * x
where our x known variable is the average temperature.
Can you write Python code for it?

y2 = 0.6351 * x - 3.4178 # solution from closed form
plt.scatter(wine.AGST, wine.Price)
plt.hlines(baseline, 15, 18, color = 'red', label="Baseline")
plt.plot(x,y, label="Simple linear regression")
plt.plot(x,y2, label = "Optimal linear regression")
plt.legend(loc="upper left")
plt.title("Wine price vs. Average Temperature")
plt.xlabel("Temperature [Celsius degrees]")
plt.ylabel("Log of Price")
plt.grid()
plt.show()


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
that gives an SSE of 5.73, an RMSE of 0.479 and an R2 of 0.435
Interpreting the coefficients
Beta0 is the intercept of the regression line, not very meaningful.
Beta1 is the slope. It is the predicted change in the output per unit change in input.
Be careful because the magnitude depends on units of both features and observations, so always indicate the unit.
In our case, it means that every Celsius degree additional in the average temperature causes an increase in the log of price of 0.6351
Can you write Python code for it?

y2 = 0.6351 * x - 3.4178
plt.plot(x,y2) # regression line

    # draw dashed lines
x1_point = 17.0
x2_point = 16.0
y1_point = 0.6351 * x1_point - 3.4178
y2_point = 0.6351 * x2_point - 3.4178

plt.plot((x1_point, x1_point), (y1_point, y2_point), 'g--')
plt.plot((x2_point, x1_point), (y2_point, y2_point), 'g--')

    # add notes next to the dashed lines
plt.annotate("Price difference for 1 degree difference", xy=(17.05,7))
plt.annotate("1 degree ", xy=(16.4, 6.6))

plt.title("Interpreting the slope")
plt.xlabel("Temperature [Celsius degrees]")
plt.ylabel("Log of Price")
plt.grid()
plt.show()


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
which tells us that the average squared error for each data point is around 0.41 for the baseline and a better 0.24 for our LR model.
MSE is a slightly better metric than SSE because does not depend on the number of data points N.
SSE and MSE can be hard to interpret: the former depends on N (the size of the dataset) and the latter unit is hard to understand (a squared value), therefore is normal to use the Root-Mean-Square Error (RMSE) which is just the root of MSE:
Can you write Python code for it?

RMSE_baseline = MSE_baseline ** 0.5; RMSE_baseline

RMSE_simpleLR = MSE_simpleLR ** 0.5; RMSE_simpleLR


I am working on the file "01-Regression/LinearRegression.ipynb".
The first task is:
which tells us the average error is around 0.64 for the baseline and a better 0.49 for our LR model.
Another useful metric is the R-squared which compares an SSE from a given model with the one from a baseline:
$$ R^{2} = 1 - \frac{SSE}{SST}$$
(SST being the baseline SSE)
Can you write Python code for it?

R2 = 1 - SSE_simpleLR / SSE_baseline; R2