I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
The satisfaction level increases with the salary, as expected. It seems though to be a more sensible difference between low and medium than medium and high salaries, but again we would need to test to see if the difference is significant.
Can you write Python code for it?

# Given an employees salary level (low, medium, or high), calculate the probability that
# they worked more than two standard deviations of the average monthly hours across all groups.
# In other words, compute P(hours > 2*sigma|salary) = P(salary|hours > 2*\sigma)*P(hours > 2*sigma)/P(salary)

# Creating a dataset for each salary level
df_low = df[df['salary'] == 'low']
df_medium = df[df['salary'] == 'medium']
df_high = df[df['salary'] == 'high']

# And one for employees who have worked more than two std above
sigma_hours = df.average_montly_hours.mean() + 2*df.average_montly_hours.std()
df_above = df[df['average_montly_hours'] > sigma_hours]
P_hours = len(df_above) / n_employees

for d, level in zip([df_low, df_medium, df_high], ['low', 'medium', 'high']):
    P_salary = len(d) / n_employees
    P_salary_hours = len(df_above[df_above['salary'] == level]) / len(df_above)
    P = (P_salary_hours * P_hours) / P_salary

    P_hours_salary = len(d[d['average_montly_hours'] > sigma_hours]) / len(d)
    print('{} salary level probability: {:.5f}, {:.5f}'.format(level, P, P_hours_salary))


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
The variance in the satisfaction levels is larger for employees who left, so the satisfaction level for this employees is more spread out around the mean. This may indicate that the employees leaving the company have a level of satisfaction more variable than those who stay.
Can you write Python code for it?

# Compute the mean satisfaction level for each salary category. Comment on your results.
df.groupby('salary').satisfaction_level.mean()


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
There seems to be a difference but before we draw any conclusion we would need to perform a hypothesis test:
Can you write Python code for it?

# Compute the variance and standard deviation of hours worked.
print(df.average_montly_hours.var())
print(df.average_montly_hours.std())

# Compare the variance between the satisfaction levels of employees who left versus those who stayed.
# Which is larger? What does this mean?
print(df_left.satisfaction_level.var())
print(df_stayed.satisfaction_level.var())


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
We can't reject the null hypothesis, let's plot the data:
Can you write Python code for it?

fig = plt.figure(figsize=(15, 10))
ax = plt.axes()
x1 = np.linspace(df_left.last_evaluation.min(), df_left.last_evaluation.max(), 1000)
ax.plot(x1, stats.norm.pdf(x1, df_left.last_evaluation.mean(), df_left.last_evaluation.std() / np.sqrt(len(df_left))), label='left')
x2 = np.linspace(df_stayed.last_evaluation.min(), df_stayed.last_evaluation.max(), 1000)
ax.plot(x2, stats.norm.pdf(x2, df_stayed.last_evaluation.mean(), df_stayed.last_evaluation.std() / np.sqrt(len(df_stayed))), label='stayed')
ax.legend();


I am working on the file "Statistical Inference/HR-Exercise.ipynb".
The first task is:
What can you say about your results in part 6?
There seems to be a clear trend and difference between the probability that an employee who worked more time is in a given salary level: the employee is more probable to be found in low and medium salary levels than high. This difference, though it seems quite marked, should be verified through a test.
Can you write Python code for it?

# Repeat parts 6 and 7 for P(left|salary) = P(salary|left)*P(left)/P(salary)

P_left = len(df_left) / n_employees

for d, level in zip([df_low, df_medium, df_high], ['low', 'medium', 'high']):
    P_salary = len(d) / n_employees
    P_salary_left = len(df_left[df_left['salary'] == level]) / len(df_left)
    P = (P_salary_left * P_left) / P_salary

    P_left_salary = len(d[d['left'] == 1]) / len(d)
    print('{} salary level probability: {:.5f}, {:.5f}'.format(level, P, P_left_salary))


I am working on the file "Statistical Learning.ipynb".
The first task is:
$MSE_{test}$ should bottom out around the "true" function. $MSE_{test}$ should never drop below the "true" amount of error/residuals. Goal is to minimize $MSE_{test}$.
Bias/Variance trade-off

It can be shown that for $y = f(x) + \epsilon$,

$E[ y_0 - \hat{f}(x_0)]^2 = Var(\hat{f}(x_0) + [bias(\hat{f}(x_0))]^2 + Var(\epsilon)$

$E[y_0 - \hat{f}(x_0)]^2$ -- Expected test set MSE
$Var(\hat{f}(x_0)$ -- Measure of how much the $\hat{f}$ function would change if I got new data. If model is well-fit, this should be small.
$Bias(\hat{f}) = E[f(x_0) - \hat{f}(x_0)]$ -- How much am I going to be wrong because my $\hat{f}$ is too restrictive. Want a model that is flexible enough that this bias is small.

$y_0$ is training data
Classification
Assessing accuracy

$\hat{y}$ will be categorical (as is $y$)
Measure will be % of cases mis-classified

Training error rate: $ER = \frac{1}{n}\sum{I(y_i \neq \hat{y}_i)}$
$I(u) =$ 1 if TRUE, 0 if FALSE$
Can you write Python code for it?


I am working on the file "Statistical Learning.ipynb".
The first task is:
Can fit this perfectly with a cubic model. But assuming that this is correct.
What happens when we get a new data point: $(x_0, y_0)$
for non-parametric methods we need some way to penalize "wiggliness"
wiggliness df: cumulative change in the second derivative, $f''$.
Pros & cons:
* Parametric:
  * Pros:
    * More interpretable
    * Requires fewer data
  * Cons:
    * More rigid
    * More assumptions to make
* Non-parametric
  * Pros:
    * More flexible
    * Fewer assumptions
  * Cons:
    * Need more data
    * Harder to interpret
Supervised vs. unsupervised algorithms

in the supervised algorithm we have response variable, $y$
unsupervised case, no response variable
the response variable, $y$, supervises our selection of important covariates, $X$

Examples:
* Regression -- supervised
* NMDS/PCA -- unsupervised
* Diabetes risk -- supervised
Can you write Python code for it?

plt.figure(figsize=(10, 5))
plt.subplot(121)
plt.title('Supervised')
plt.scatter([.0, .2, .1, .3], [.2, .1, .3, .4], c='red', label='nondiabetic')
plt.scatter([.6, .8, .9, .7], [.55, .74, .5, .8], c='blue', label='diabetic')
plt.ylabel('Weekly sugar intake')
plt.xlabel('BMI')
plt.legend(loc=2)

plt.subplot(122)
plt.title('Unsupervised')
plt.scatter([.6, .8, .9, .7]+[.0, .2, .1, .3], [.55, .74, .5, .8]+[.2, .1, .3, .4], c='black', label='diabetic')
plt.ylabel('Weekly sugar intake')
plt.xlabel('BMI')

plt.tight_layout()


I am working on the file "Statistical Learning.ipynb".
The first task is:
Goal: to develop an estimate of $f$, $\hat{f}$
Two reasons to estimate $f$ with $\hat{f}$:
$\hat{f}$ -> making inference; want to know how covariates X affects y.
Can you write Python code for it?

models = ['Subset selection lasso', 'least squares', 'generalized additive model trees',
          'bagging, boosting', 'support vector machines']
pos = [(0, 1), (0.2, 0.8), (0.4, 0.6), (0.6, 0.1), (0.7, 0.3)]
xlabels = ['Restrictive', 'Flexible']
ylabels = ['Low', 'High']

plt.figure(figsize=(10, 7))
for m, p in zip(models, pos):
    plt.text(p[0]+ 0.02, p[1]-0.05, m, size=16)

plt.xticks([0.07, 0.95], xlabels, size=16)
plt.yticks([0, 1], ylabels, size=16)
plt.ylabel('Interpretability', size=20)
plt.xlabel('Flexibility', size=20)

plt.show()


I am working on the file "Statistical Learning.ipynb".
The first task is:
How do we estimate $\hat{f}$?
Parametric vs non-parametric methods
Parametric methods
* Assume some form for the relationship between X and y. For example:
$y = \beta_0 + \beta_1x + \epsilon$
$y = X\beta + \epsilon$
$logit(y) = X\beta + \epsilon$
* And fit data by tweaking a few $p << n$ beta terms (much few parameters than the number of observations).
Non-parametric methods
* Assume no form for $f$,
* or the form has $p \simeq n$
Can you write Python code for it?


I am working on the file "Statistical Learning.ipynb".
The first task is:
Model A is better because the $Ave(y-\hat{y})^2$ (Mean Squared Error) is smaller.
Consider the model where we have n parameters (e.g. n-degree polynomial). It can go through every data point: no MSE!
If the model is too flexible (and we overfit the data), then we tend to do a bad job at predicting a new data point that was not used in tuning the model.
Test data & training data
Take our data and split into two groups:
1. Training data: data used to tune the model(s) of interest
2. Test data: data used to assess the accuracy of each model (typically use MSE)
In general, $MSE_{training} \leq MSE_{test}$
Want to look at the impact of model complexity on both $MSE_{training}$ and $MSE_{test}$.
Can you write Python code for it?

plt.figure(figsize=(7, 5))

x = np.linspace(1, 10, 99)
plt.plot(x, 1./x**0.5 - 0.1, label='$MSE_training$', lw=3)

plt.plot(np.linspace(1, 10, 7), [0.9, 0.6, 0.5, 0.45, 0.55, 0.7, 0.9], label='$MSE_{test}$', lw=3)
plt.ylabel('$MSE$')
plt.xlabel('flexibility')

plt.legend()
plt.show()


I am working on the file "Statistical Learning.ipynb".
The first task is:
Statistical Learning
Different from machine learning, in that not merely interested in how well a model fits: also interested in how to interpret/derive meaning.

How to relate my covariates $X = { x_1, x_2, x_3, ... x_p }$ to the response $y$.
Our model of the data is $y = f(x) + \epsilon$
$f(x)$ is not necessarily linear,
Error terms need not be normal.
Goal: to develop an estimate of $f$, $\hat{f}$
Two reasons to estimate $f$ with $\hat{f}$:
Make predictions (not necessarily informed by mechanisms, relationships among covariates),
Want $\hat{y}$ to be close to $y$; $\hat{y} = \hat{f}(x)$
Minimize Mean Squared Error:

$E(y-\hat{y})^2 = E[f(x) + \epsilon - \hat{f}(x)]^2$
$E(y-\hat{y})^2 = [f(x) - \hat{f}(x)]^2 + Var(\epsilon)$
Can you write Python code for it?

import numpy as np
from scipy.stats import uniform

f = lambda x: np.log(x)
x = np.linspace(0.1, 5.1, 100)
y = f(x)
Eps = uniform.rvs(-1., 2., size=(100,))

plt.plot(x, y, label='$f(x)$', lw=3)
plt.scatter(x, y + Eps, label='y')
plt.xlabel('x')
plt.legend(loc='best')
plt.show()


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Because the median (14.4542) is between -17.46132647619951 and 81.86974241334872, we can conclude that it is in fact with one standard deviation of the mean.
Question 5
How much more expensive was the 90th percentile ticket than the 5th percentile ticket? Are they the same class?
Can you write Python code for it?

p90, p5 = np.percentile(titanic_data.Fare, [90,5])
diff = p90 - p5
print("The 5th percentile: " + str(p5))
print("The 90th percentile: " + str(p90))
print('The difference between the 90th and 5th percentile in ticket cost is ${}'.format(diff))

class_90 = titanic_data[titanic_data.Fare == p90].Pclass
print("Tickets at the 90th percentile are associated with passengers in class: " + str(class_90.values[1]))

class_5 = titanic_data[titanic_data.Fare == p5].Pclass
print("Tickets at the 90th percentile are associated with passengers in class: " + str(class_5.values[1]))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Cherbourg is the port with the highest price average.
Question 7
Which port has passengers from the most similar passenger class?
Can you write Python code for it?

s_class_ave = titanic_data[titanic_data.Embarked=='S'].Pclass
import scipy.stats as sp
s_class_ave = sp.mode(s_class_ave)
print("Passengers from Southampton are mostly from class " + str(s_class_ave[0][0]) + " with a count of " + str(s_class_ave[1][0]))

q_class_ave = titanic_data[titanic_data.Embarked=='Q'].Pclass
q_class_ave = sp.mode(q_class_ave)
print("Passengers from Queenstown are mostly from class " + str(q_class_ave[0][0]) + " with a count of " + str(q_class_ave[1][0]))

c_class_ave = titanic_data[titanic_data.Embarked=='C'].Pclass
c_class_ave = sp.mode(c_class_ave)
print("Passengers from Cherbourg are mostly from class " + str(c_class_ave[0][0]) + " with a count of " + str(c_class_ave[1][0]))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
How big is the data?
Can you write Python code for it?

print("There are " + str(len(titanic_data)) + " rows in out dataset. \n")
print("The column names are " + str(list(titanic_data.columns.values)))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
More often survivors had 0 sibliings or spouses onboards.
Question 4
Within what range of standard deviations from the mean (0-1, 1-2, 2-3) is the median ticket price? Above or below the mean?
Can you write Python code for it?

fare_mean = np.mean(titanic_data.Fare)
print("The average fare prices is: " + str(fare_mean))

fare_median = np.median(titanic_data.Fare)
print("The median fare prices is: " + str(fare_median))

std_all = np.std(titanic_data.Fare)
print("The standard deviation of the fare prices is: " + str(std_all))

max = (fare_mean + std_all)
min = (fare_mean - std_all)
print("Anything within the " + str(min) + " and " + str(max) + " is within one standard deviation of the mean.")


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
NO first-class male survivors paid less the the median.
Question 9
How much older/younger was the average surviving passenger with family members than the average non-surviving passenger without them?
Can you write Python code for it?

surv_fam = np.mean(survivors[(survivors.SibSp > 0)|(survivors.Parch > 0)].Age)
print("Average age of surviving passengers with any sort of family members: " + str(surv_fam) + "\n")

dead_fam = np.mean(titanic_data[(titanic_data.Survived == 0) & (titanic_data.SibSp == 0) & (titanic_data.Parch==0)].Age)
print("Average age of dead passengers without any sort of family members: " + str(dead_fam) + "\n")

print("Age difference between surviving passengers w/ family members vs dead passengers w/o family members: " + str(surv_fam - dead_fam))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Question 2
What's the average age of..
- any Titanic passenger
- a survivor
- a non-surviving first-class passenger
- male survivors older than 30 from anywhere but Queenstown
Can you write Python code for it?

Ages = np.mean(titanic_data.Age)
print('Average age of all passengers with data is {}'.format(Ages))

survivors = titanic_data[titanic_data.Survived == 1]

srv_ages = np.mean(survivors.Age)
print('Average survivor age is {}'.format(srv_ages))

first_dead = titanic_data[(titanic_data.Survived == 0) & (titanic_data.Pclass ==1)]
first_dead_ages = np.mean(first_dead.Age)
print('Average first class passenger who died age is {}'.format(first_dead_ages))

live_males = titanic_data[(titanic_data.Survived==1) & (titanic_data.Age>30) & (titanic_data.Sex=='male') & (titanic_data.Embarked!= 'Q')]
live_males_ave = np.mean(live_males.Age)
print('Average age of male not dead over 30 not from Queenstown is {}'.format(live_males_ave))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Question 3
What's the most common..
- passenger class
- port of Embarkation
- number of siblings or spouses aboard for surviors
Can you write Python code for it?


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Question 6
Which port has the heightest average ticket price paid by passengers?
Can you write Python code for it?

s_average = np.mean(titanic_data.Fare[titanic_data.Embarked=='S'])
print("South Hampton passengers paid an average of: $" + str(s_average))

q_average = np.mean(titanic_data.Fare[titanic_data.Embarked=='Q'])
print("Queenstown passengers paid an average of: $" + str(q_average))

c_average = np.mean(titanic_data.Fare[titanic_data.Embarked=='C'])
print("Cherbourg passengers paid an average of: $" + str(c_average))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
Southampton has the most passangers from the same class which is third at a count of 353.
Question 8
How many male survivors in first class paid lower then the overall median ticket price?
Can you write Python code for it?

male_over_median = titanic_data[(titanic_data.Fare < fare_median) & (titanic_data.Pclass == 1) & (titanic_data.Survived == 1) & (titanic_data.Sex == "male")]
print("First-class male surviors that paid less then the median: " + str(male_over_median.values))


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
The most common passengers are in the third class.
Can you write Python code for it?


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
The most common port to embark from is Southamptom.
Can you write Python code for it?


I am working on the file "Statistical Titanic - Step 1.ipynb".
The first task is:
The surviving passegners with family were 6.888 years younger then the dead without.   See above for exact number.
Question 10
Display the relationship between survival rate and the quantile of the ticket price for 20 integer quantiles.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
quant_list = []
surv_percents = []
for i in range(20):
    i_per, iplus_per = np.percentile(titanic_data.Fare, [i*5, (i+1)*5])
    total = np.sum((titanic_data.Fare > i_per) & (titanic_data.Fare <= iplus_per))
    cut = (titanic_data.Fare > i_per) & (titanic_data.Fare <= iplus_per)
    surv = cut & (titanic_data.Survived==1)
    surv_percents.append(np.sum(surv)/total)
    quant_list.append(i_per)

plt.figure(figsize=(10, 5))
plt.plot(quant_list, surv_percents)
plt.xlabel("Ticket Price")
plt.ylabel("Percent of Survival")
plt.show()


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Filling all NaN ages with the mean of all the ages and confirming with .info() method. We later compensate for this with a functions that remove the mean.  This will be pointed out with a 'COMPENSATION:' and define the action as it arises.
Can you write Python code for it?

titanic_data.Age = titanic_data.Age.fillna(np.mean(titanic_data.Age))

titanic_data.info()


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Our second choice of question is to find the probablity that a survivor was from Cherbourg.
Can you write Python code for it?

c_port = survivors[survivors.Embarked == 'C']
prob_c = (len(c_port) / len(survivors))
print("There is a " + str(prob_c) + " percent probability that a survivor was from Cherbourg.")


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Question 1
Calculate the probability of survival during Titanic crash.
Can you write Python code for it?

survivors = titanic_data[titanic_data.Survived == 1]
survivor_prob = (len(survivors) / len(titanic_data))
print("There is a " + str(survivor_prob) + " percent chance of survival.")


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Question 2
CHOOSE TWO OF...

A passenger was male
A passenger was female and had atleast 1 SibSp on board
A survivor was from Cherbourg

Our first choice of question is to see the probability that a passenger was male.
Can you write Python code for it?

male_passenger = titanic_data[titanic_data.Sex == 'male']
prob_male = (len(male_passenger) / len(titanic_data))
print("There is a " + str(prob_male) + " percent probability that a passenger was male.")


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Question 4
Find the probability that a passenger was less than 10 years old.
Can you write Python code for it?

less_then_ten = []
for i in all_ages:
    if i < 10:
        less_then_ten.append(i)

prob_less_then_ten = (len(less_then_ten) / len(all_ages))
print("There is a " + str(round(prob_less_then_ten, 3)) + " probabililty that a passenger was less then 10 year old.")


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
Question 7
Is there a statistically significant difference between the fares paid by passengers between Queentown and the passengers from Cherbourg?
Can you write Python code for it?

from scipy.stats import ttest_ind
fare_from_q = titanic_data[titanic_data.Embarked == 'Q']
fare_from_c = titanic_data[titanic_data.Embarked == 'C']
t_stat, p_value = ttest_ind(fare_from_q.Fare, fare_from_c.Fare)

print("Results:\n\tt-statistic: %.5f\n\tp-value: %g" % (t_stat, p_value))


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
There is a 0.0613 probability that exactly 42 passenger survive out of 100.  See above 'Out' for a more precise probability.
Question 5
What is the probability that at least 42 of those 100 passenger survive?
Can you write Python code for it?


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
There is a 0.198 probability that at least 42 of those 100 passenger survive.  See above 'Out' for a more precise probability.
Question 6
Is there a statistically significant difference between the age of male and female survivors?
COMPENSATION:  Within our male and female survival groups we also had a match to throw out any age that matches the mean to the third the decimal to remove the skew.  Post skew compensation increased the p value, indicating a correct compensation.
Can you write Python code for it?

from scipy.stats import ttest_ind

survivors_male = survivors[(survivors.Sex == 'male') & (round(survivors.Age,3) != round(age_mean, 3)) ]
survivors_female = survivors[(survivors.Sex == 'female') & (round(survivors.Age, 3) != round(age_mean, 3))]
t_stat, p_value = ttest_ind(survivors_male.Age, survivors_female.Age)

print("Results:\n\tt-statistic: %.5f\n\tp-value: %.5f" % (t_stat, p_value))


I am working on the file "Statistical Titanic - Step 2.ipynb".
The first task is:
There is statistical difference in the fares paid between the passengers at Queentown and Cherboug. This is indicated by the p-value that is less than 0.01.
Can you write Python code for it?

plt.figure(figsize=(10, 4))
opacity = 0.5

plt.hist(fare_from_q.Fare, bins=np.arange(0, 90, 5), alpha=opacity, label="Queenstown")
plt.hist(fare_from_c.Fare, bins=np.arange(0, 90, 5), alpha=opacity, label="Cherbourg")
plt.legend()
plt.title("Fare Distribution from Queenstown to Cherbourg")
plt.xlabel("Fare Price")
plt.ylabel("Number of Passengers")
plt.show()


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
Based off the high U-stat and the very low p-value we can reject the null hypothesis that there is no difference in fare payed between the survivors and the dead.
Task: Find The Best Linear Model For The Relationship Between Fare and Age.
The Gloal is to find the best slope to have the model
Fare = slope * Age
Fit to the data
Looking again at the original data.
Can you write Python code for it?


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
Based off this graph it is clear that these two distributions are best compared using a Mann-Whitney U-test
Can you write Python code for it?

from scipy.stats import mannwhitneyu
u, p = mannwhitneyu(surivors.Fare, dead.Fare)
print("Results:\n\tU-statistic: %.5f\n\tp-value: %g" % (u, p * 2))


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
Data
Bring In Titanic Passenger Data
Can you write Python code for it?

td = pandas.read_csv('titanic_train.csv')
td.info()


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
The data we care about for this hypothesis(Survived, Fare) has no NaN values so no need to modify.
Setup a Survivor Data Set and a Non-Survivor Data Set.
Can you write Python code for it?

surivors = td[td.Survived==1]
dead = td[td.Survived==0]


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
The inital range of (-20,20,1) was narrowed down to (0,2,100) based off chi-squared being closer to 1.
Can you write Python code for it?

slopes = np.linspace(0,2,100)
chi_results = []
for s in slopes:
    model_fare = linear(valid_age,s)
    chi_results.append(chi_sq(valid_fare, model_fare, valid_fare.std(), dof=1))
chi_results = np.array(chi_results)
print("Best Chi_Squared: {}".format(chi_results[chi_results.argmin()]))
print("Best Slope: {}".format(slopes[chi_results.argmin()]))


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
Visual This Data With a Scatter Plot focusing on the highest density area.
Can you write Python code for it?

plt.figure(figsize=(7,4))
plt.scatter(valid_age, valid_fare)
plt.xlim(0,80)
plt.ylim(0,150)
plt.title('Comparision of Age and Fare')
plt.xlabel('Age')
plt.ylabel('Fare')
plt.show()


I am working on the file "Statistical Titanic - Step 3.ipynb".
The first task is:
Visualize the linear model over the data.
Can you write Python code for it?

plt.figure(figsize=(7,4))
plt.scatter(td.Age,td.Fare)
plt.xlim(0,80)
plt.ylim(0,150)
plt.plot(td.Age,linear(td.Age,slopes[chi_results.argmin()]))
plt.show()


I am working on the file "Statistical_Analysis/Collaboration-Unapp_vs_Approved_KStest.ipynb".
The first task is:
2. Approved Loans
Can you write Python code for it?

df_app_2015 = pd.read_csv('Data/approved_loan_2015_clean.csv', low_memory=False).loc[:, 'loan_amnt':]

#df_app_2015 = pd.read_csv('Data/LoanStats3d_securev1.csv.zip', compression='zip',header=1, low_memory=False)

df_app_2015.head(3)

## Pick approved and fully paid loans
df_app_2015_paid = df_app_2015[df_app_2015['Target']==1]

## Pick apprived and default loans
df_app_2015_default= df_app_2015[df_app_2015['Target']==0]

## pick columns and drop nan
df_app_15 = df_app_2015[['loan_amnt', 'fico_range_low', 'dti']].dropna()

df_app_15_paid = df_app_2015_paid[['loan_amnt', 'fico_range_low', 'dti']].dropna()
df_app_15_default = df_app_2015_default[['loan_amnt', 'fico_range_low', 'dti']].dropna()

df_app_15.shape

df_app_15_paid.shape

df_app_15_default.shape

df_app_15.head(3)

### using pre-set "Outlier Boundry" for loan amount to exclude the potential outliers

outlier_bound = 33600.0

df_app_test = df_app_15[df_app_15['loan_amnt'] <= outlier_bound]  # approved loans

df_app_paid_test = df_app_15_paid[df_app_15_paid['loan_amnt'] <= outlier_bound]  # approved and fully paid loans
df_app_default_test = df_app_15_default[df_app_15_default['loan_amnt'] <= outlier_bound]  # approved loans and default loans

len(df_app_test)


I am working on the file "Statistical_Analysis/Collaboration-Unapp_vs_Approved_KStest.ipynb".
The first task is:
Credit Score: Top 25% of Uapproved v.s. Bottom 25% of Approved loans
Can you write Python code for it?

## Top25% scores lower bound -- whole clean data
df_unapp_test.describe().Risk_Score

## Top25% scores lower bound -- sample data
df_unapp_test_s.describe().Risk_Score[6]

## Bottom 25% scores upper bound -- whole clean data
df_app_test.describe().fico_range_low

## Bottom 25% scores upper bound -- sample data
df_app_test_s.describe().fico_range_low[4]

### TEST with Samples using differen random states
random_states = np.random.randint(1, 100, 5)

for rs in random_states:

    # Sampling
    df_unapp_test_s = df_unapp_test.sample(n=10000, random_state=rs)
    df_app_test_s = df_app_test.sample(n=10000, random_state=rs)

    ### Select applications based on top 25% and bottom 25% credit scores boundries
    top25_bound_unapp = df_unapp_test_s.describe().Risk_Score[6]  ## Top25% scores lower bound -- sample data
    bot25_bound_app = df_app_test_s.describe().fico_range_low[4] ## Bottom 25% scores upper bound -- sample data

    df_unapp_top25= df_unapp_test_s[df_unapp_test_s['Risk_Score'] >= top25_bound_unapp]  # unapproved loans
    df_app_bot25 = df_app_test_s[df_app_test_s['fico_range_low'] <= bot25_bound_app]  # approved loans

    ## KS two sample test to compare unapproved and approved loan samples
    print '\nrandom_state = {}'.format(rs)
    print 'KS result on loan amounts:\n', stats.ks_2samp(sorted(df_unapp_top25['Amount Requested']), \
                                                         sorted(df_app_bot25['loan_amnt']))

    ## KS two sample test to compare unapproved and approved loans -- samples
    print 'KS result on credit scores:\n', stats.ks_2samp(sorted(df_unapp_top25['Risk_Score']), \
                                                          sorted(df_app_bot25['fico_range_low']))

    ## KS two sample test to compare unapproved and approved loans -- samples
    print 'KS result on dti:\n', stats.ks_2samp(sorted(df_unapp_top25['Debt-To-Income Ratio']), \
                                                sorted(df_app_bot25['dti']))


I am working on the file "Statistical_Analysis/Collaboration-Unapp_vs_Approved_KStest.ipynb".
The first task is:
Upproved vs Approved & Fully Paid Loans
25% top and 25% bottom
Can you write Python code for it?

## Credit Score
plot_feature_dist(df_unapp_test, 'Risk_Score', df_app_paid_test, 'fico_range_low',\
                  ylabel='Credit Score', heading='Unapproved vs Fully Paid Loans-Credit Score')

## Loan Amounts
plot_feature_dist(df_unapp_test, 'Amount Requested', df_app_paid_test, 'loan_amnt',\
                  ylabel='Loan Amount', heading='Unapproved vs Fully Paid Loans-Loan Amount')

## Debt-To-Income Ratio
plot_feature_dist(df_unapp_test, 'Debt-To-Income Ratio', df_app_paid_test, 'dti',\
                  ylabel='Debt-To-Income Ratio', heading='Unapproved vs Fully Paid Loans-Debt-To-Income Ratio')


I am working on the file "Statistical_Physics/Percolation.ipynb".
The first task is:
L'idée est de les rendre aléatoire pour mieux visualiser, d'où le code suivant :
Can you write Python code for it?

b = arange(lw.max() + 1)
shuffle(b)
shuffledLw = b[lw]
imshow(shuffledLw, origin='lower', interpolation='nearest')
colorbar()
title("Labeled clusters")
show()


I am working on the file "Statistical_Physics/Percolation.ipynb".
The first task is:
Pour finir, Voici ce qui se passe pour différentes valeurs de $p$. Ainsi nous mettrons en évidence le phénomène de percolation.
Can you write Python code for it?

for i,p in enumerate([0.2,0.4,0.5,0.6,0.8]) :

 fig = plt.figure()

 r = rand(L,L)
 z = r < p

 lw, num = measurements.label(z)
 area = measurements.sum(z, lw, index=arange(lw.max() + 1))
 areaImg = area[lw]
 im = imshow(areaImg, origin='lower', interpolation='nearest')

 fig.colorbar(im)

 title("Clusters by area p = "+str(p))

 sliced = measurements.find_objects(areaImg == areaImg.max())
 if(len(sliced) > 0):
    sliceX = sliced[0][1]
    sliceY = sliced[0][0]
    plotxlim=im.axes.get_xlim()
    plotylim=im.axes.get_ylim()
    plot([sliceX.start, sliceX.start, sliceX.stop, sliceX.stop, sliceX.start], \
                      [sliceY.start, sliceY.stop, sliceY.stop, sliceY.start, sliceY.start], \
                      color="red")
    xlim(plotxlim)
    ylim(plotylim)

 show()

from pylab import *
from scipy.ndimage import measurements
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Set up the figures
fig1,ax1=plt.subplots(5,3,figsize=(10, 6))
ax1 = ax1.ravel()

fig2,ax2=plt.subplots(5,3,figsize=(10, 6))
ax2 = ax2.ravel()

#Set up variables
p=np.arange(0., 1., 0.05)
areamax=np.zeros(len(p))
lwmax=np.zeros(len(p))

for ind,L in enumerate([50, 100, 200, 500, 1000]):

  for ind2,marker in enumerate(['ko', 'bo', 'ro']):

    # Randomise the percolation network
    r = rand(L,L)

    for i,p2 in enumerate(p):
      # Determine the connectivity for probability p
      z = r<p2

      lw, num = measurements.label(z)

      # Plot the clusters for given p

      area = measurements.sum(z, lw, index=arange(lw.max() + 1))
      areaImg = area[lw]
      lwmax[i] = lw.max()
      areamax[i] = area.max()
      # Plot the number of clusters

    ax2[3*ind + ind2 ].plot(p, areamax, marker)
    ax1[3*ind + ind2 ].plot(p, lwmax, marker)

      # Plot the size of the largest cluster
      #plt.figure(2)
      #plt.plot(p, area.max(), marker)
      #plt.draw()
fig1.tight_layout()
st = fig1.suptitle('Number of cluster',...


I am working on the file "Statistics 4.ipynb".
The first task is:
Below gives us our model versus the actual data.
Can you write Python code for it?

fig = plt.figure(figsize=(12, 4))
fig.subplots_adjust(wspace=0)
ax = plt.subplot(121)
ax.scatter(all_stats.HR, all_stats.salary, edgecolor="None",
            s=5, c='k', alpha=0.2)
ax.set_yscale("log")
ax.set_xlabel("Home Runs", fontsize=12); ax.set_ylabel("Salary ($)", fontsize=12)
ax.set_xlim(-50, 800); ax.minorticks_on()

ax = plt.subplot(122)
ax.scatter(Xvals[:, 1], regressor.predict(Xvals), edgecolor="None",
            s=2, c='r')
ax.set_xlabel("Home Runs", fontsize=12)
ax.set_ylim(1E4, 1E9)
ax.set_yscale("log"); ax.set_yticklabels([])
ax.set_xlim(-50, 800); ax.minorticks_on()

plt.show()


I am working on the file "Statistics 4.ipynb".
The first task is:
Multivariate Linear Regression Model

Using no more than 4 characteristics, create a multivariate Linear Regression model predicting player salary.
Report the R2R2 score for your multivariate model. Aim for R2>0.5R2>0.5.
Make a scatter plot of your data side-by-side with a scatter plot of your model predictions.

We have included the categories games, home runs, hit by pitcher, and stint for our four mulitvariant categories.
Can you write Python code for it?

N_folds = 10
kfolds = cv.KFold(len(all_stats), n_folds=N_folds)

regressor = linear_model.LinearRegression()
valid_data = ["HR", 'HBP', 'G', 'stint']
Xvals = np.array(all_stats[valid_data])
yvals = np.array(all_stats.salary)

coeffs, intercepts = [], []

for train_index, test_index in kfolds:
    X_train, X_test = Xvals[train_index], Xvals[test_index]
    y_train, y_test = yvals[train_index], yvals[test_index]
    regressor.fit(X_train, y_train)
    coeffs.append(regressor.coef_)
    intercepts.append(regressor.intercept_)

coeffs = np.array(coeffs).mean(axis=0) #averages each column
intercept = np.array(intercepts).mean(axis=0)

regressor.coef_ = coeffs
regressor.intercept_ = intercept


I am working on the file "Statistics 4.ipynb".
The first task is:
Our r^2 value is .376 as seen below.
Can you write Python code for it?


I am working on the file "Statistics.ipynb".
The first task is:
Calculate percentages for plotting purposes
Can you write Python code for it?

normalised_bias_ratings.loc[:,'HC_Baseline_pcnt'] = normalised_bias_ratings.HC_Baseline/normalised_bias_ratings.HC_Baseline.sum()*100
normalised_bias_ratings.loc[:,'HC_ImageBox_pcnt'] = normalised_bias_ratings.HC_ImageBox/normalised_bias_ratings.HC_ImageBox.sum()*100
normalised_bias_ratings.loc[:,'DT_Baseline_pcnt'] = normalised_bias_ratings.DT_Baseline/normalised_bias_ratings.DT_Baseline.sum()*100
normalised_bias_ratings.loc[:,'DT_ImageBox_pcnt'] = normalised_bias_ratings.DT_ImageBox/normalised_bias_ratings.DT_ImageBox.sum()*100

normalised_bias_ratings

normalised_bias_ratings.columns

HC_percentages = normalised_bias_ratings[['HC_Baseline_pcnt', 'HC_ImageBox_pcnt']]
DT_percentages = normalised_bias_ratings[['DT_Baseline_pcnt', 'DT_ImageBox_pcnt']]


I am working on the file "Statistics.ipynb".
The first task is:
Find the marginal distribution:
* $f_{X,Y}(X=x,Y=y) = \frac{1}{12}(2x + y), 0 \leq x \leq 2, 0 \leq y \leq 2$
* $f_X(X=x) = \int_0^2 \frac{1}{12}(2x + y) dy$
* $ = \frac{1}{12}[2xy + \frac{1}{2}y^2 + d]_0^2$
* $ = \frac{1}{12}[4x + 2 + d - d]$
* $ = \frac{4x + 2}{12}$
* $ = \frac{2x + 1}{6}$
Inversion sampling example:
* $F_X(X=x) = \int_0^x \dfrac{2x+1}{6}dx$
* $= \frac{1}{6}[x^2 + x + d]_0^x$
* $= \frac{x(x + 1)}{6}$
* $u = \frac{x^2 + x}{6}$
* $0 = x^2 + x - 6u$
* $x = \frac{-1 \pm \sqrt{1 + 4 \times 6u}}{2}$
Can you write Python code for it?

u = np.random.uniform(size=100000)
x = (-1 + (1 + 24*u)**.5) / 2
df = pd.DataFrame({'x':x})
ggplot(aes(x='x'), data=df) + geom_histogram()


I am working on the file "Statistics.ipynb".
The first task is:
Inversion sampling example
First find the normalizing constant:
$$
\begin{align}
    f_X(X=x) &&= cx^2, 0 \leq x \leq 2 \
    1 &&= c\int_0^2 x^2 dx \
    &&= c[\frac{1}{3}x^3 + d]_0^2 \
    &&= c[\frac{8}{3} + d - d] \
    &&= c[\frac{8}{3}] \
    f_X(X=x) &&= \frac{3}{8}x^2, 0 \leq x \leq 2
\end{align}
$$
Next find the cumulative distribution function:
* $F_X(X=x) = \int_0^x \frac{3}{8}x^2dx$
* $=\frac{3}{8}[\frac{1}{3}x^3 + d]_0^x$
* $=\frac{3}{8}[\frac{1}{3}x^3 + d - d]$
* $=\frac{1}{8}x^3$
We can randomly generate values from a standard uniform distribution and set equal to the CDF. Solve for $x$. Plug the randomly generated values into the equation and plot the histogram or density of $x$ to get the shape of the distribution:
* $u = \frac{1}{8}x^3$
* $x^3 = 8u$
* $x = 2u^{\frac{1}{3}}$
Can you write Python code for it?

%%latex

\begin{align*}
    f_X(X=x) &= cx^2, 0 \leq x \leq 2 \\
    1 &= c\int_0^2 x^2 dx \\
    &= c[\frac{1}{3}x^3 + d]_0^2 \\
    &= c[\frac{8}{3} + d - d] \\
    &= c[\frac{8}{3}] \\
    f_X(X=x) &= \frac{3}{8}x^2, 0 \leq x \leq 2
\end{align*}

u = np.random.uniform(size=100000)
x = 2 * u**.3333

df = pd.DataFrame({'x':x})
print df.describe()
ggplot(aes(x='x'), data=df) + geom_histogram()


I am working on the file "Statistics.ipynb".
The first task is:
Joint Distribution
Find the normalizing constant:
* $f_{X,Y}(X=x,Y=y) = c(2x + y), 0 \leq x \leq 2, 0 \leq y \leq 2$
* $1 = \int_0^2 \int_0^2 c(2x + y) dy dx$
* $  = c\int_0^2 [2xy + \frac{1}{2}y^2 + d]0^2 dx$
* $  = c\int_0^2 [4x + \frac{1}{2}4 + d - d] dx$
* $  = c\int_0^2 (4x + 2) dx$
* $  = c[2x^2 + 2x + d]_0^2$
* $  = c[2(4) + 2(2) + d - d]$
* $  = 12c$
* $c = \frac{1}{12}$
* $f{X,Y}(X=x,Y=y) = \frac{1}{12}(2x + y), 0 \leq x \leq 2, 0 \leq y \leq 2$
Can you write Python code for it?

x = np.random.uniform(size=10000)
y = np.random.uniform(size=10000)


I am working on the file "Statistics.ipynb".
The first task is:
Remove Unknown / unreliable row
Can you write Python code for it?


I am working on the file "Statistics.ipynb".
The first task is:
Statistical analysis on Allsides bias rating:
No sources from the images boxes were rated in the Allsides bias rating dataset. Therefore comparisons between bias of baseline sources versus image box sources could not be performed.
Statistical analysis on Facebook Study bias rating:
Hillary Clinton Image Box images versus Baseline images source bias according to Facebook bias ratings:
Can you write Python code for it?

print("Baseline skew: ", stats.skew(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3]))
print("Image Box skew: ", stats.skew(HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3]))


I am working on the file "Statistics.ipynb".
The first task is:
Test Donald Trump Image Box images against Basline images:
Can you write Python code for it?

stats.chisquare(f_exp=normalised_bias_ratings.DT_Baseline,
               f_obs=normalised_bias_ratings.DT_ImageBox)

DT_percentages.plot.bar()


I am working on the file "Statistics.ipynb".
The first task is:
The Kolmogorov-Smirnov analyis shows that the distribution of political representation across image sources is different between the baseline images and those found in the image box.

Statistical analysis on Allsides + Facebook + MondoTimes + my bias ratings:
Convert strings to integers:
Can you write Python code for it?

def convert_to_ints(col):
    if col == 'Left':
        return -1
    elif col == 'Center':
        return 0
    elif col == 'Right':
        return 1
    else:
        return np.nan

HC_imagebox['final_rating_ints'] = HC_imagebox.final_rating.apply(convert_to_ints)
DT_imagebox['final_rating_ints'] = DT_imagebox.final_rating.apply(convert_to_ints)
HC_baseline['final_rating_ints'] = HC_baseline.final_rating.apply(convert_to_ints)
DT_baseline['final_rating_ints'] = DT_baseline.final_rating.apply(convert_to_ints)

HC_imagebox.final_rating_ints.value_counts()

DT_imagebox.final_rating_ints.value_counts()


I am working on the file "Statistics.ipynb".
The first task is:
from the stats page "For normally distributed data, the skewness should be about 0. A skewness value > 0 means that there is more weight in the left tail of the distribution. The function skewtest can be used to determine if the skewness value is close enough to 0, statistically speaking."
Can you write Python code for it?

print("Baseline skew: ", stats.skewtest(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3]))
print("Image Box skew: ", stats.skewtest(HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3]))

stats.ks_2samp(HC_baseline.facebookbias_rating[HC_baseline.facebookbias_rating<3],
                               HC_imagebox.facebookbias_rating[HC_imagebox.facebookbias_rating<3])

HC_imagebox.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='blue')

HC_baseline.facebookbias_rating.plot.hist(alpha=0.5, bins=20, range=(-1,1), color='green')


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
8.3235 is a value without a lot of context. Think about what the units of that are: it's essentially 2022.61 dollars squared ($\$^2$). Do you every pay for things in dollars squared? Or if we use two variables with totally different units, like salary (dollars) and years, we get units of dollar-years. This is really difficult to get a sense of scale or closeness of relationship.
Correlation
Instead, statisticians will often use the correlation (or Pearson Correlation Coefficient to be formal) to report a number that everyone can understand. Correlation is a number between -1 and +1. If it's 1, there's a perfect positive relationship and vice versa. A correlation of 0 means there's absolutely no relationship. But keep in mind, just because there's a correlation of 0, doesn't mean that there's no linear correlation between values.
Take this helpful image from Wikipedia:

The correlation coefficient is almost always denoted by the variable $r$. The formula for a sample is:
$$ r = \frac{\sum\nolimits_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y}))}{\sqrt{\sum\nolimits_{i=1}^{n}(x_i-\bar{x})^2\sum\nolimits_{i=1}^{n}(y_i-\bar{y})^2}} $$
Before you freak out, we've seen all of these pieces before. This is just the covariance divided by the product of the standard deviations. Perhaps a simpler equation?
$$ \rho_{X,Y} = \frac{cov(X,Y)}{\sigma_X\sigma_Y} $$
Now in code:
Can you write Python code for it?

def correlation(x, y):
    std_x = x.std() # again, just use Panda's method
    std_y = y.std()
    if std_x > 0 and std_y > 0:
        return covariance(x,y) / (std_x*std_y)
    else:
        return 0

correlation(tips["total_bill"], tips["tip"])


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
Basic statistics
Now we want to examine our dataset. In particular, we want to see the mean, standard devaition, range, and the distribution. Refer to the Descriptive Statistics notebook for more details on how we do this manually. For now, we'll use the Series class from Pandas to do all of it for us:
Can you write Python code for it?


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
From the above, we can see that the distributions are roughly the same. It's not quite normal because of the right skew (that is, the tail is long and to the right). But even though these look similar, it does not say anything definitive about the relationship. For example, each tip could be completely independently matched up with the total bill, like in the extreme case of the \$10.00 max tip getting matched to the \$3.07 minimum bill (hey, it could happen!). For a better sense of this relationship, let's introduce the
Scatter Plot
Another familiar graph, it will plot the total_bill against the tip. In this case, total_bill is the explanatory variable and the tip is the result variable. This means we are looking at the total_bill value to explain the tip value.
Let's take a look at the scatter plot (or lmplot as seaborn calls it) and all will become clear:
Can you write Python code for it?

sns.lmplot(x="total_bill", y="tip", data=tips, fit_reg=False)


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
Take a couple minutes to examine those statistics. What do they tell you?
Which mean?
Think back to the previous lesson on Descriptive Statistics. We went over 3 different types of means. If we wanted to get the mean for the tip percent, which should we use?
You might see the percent sign and jump straight for the geometric mean. Afterall, that's how we averaged stock returns. But remember that these are not returns but proportions -- in other words, a tip percent has no dependence on any previous value.
You may also want to jump straight for the arithmetic mean by creating all of tip percent and averaging the value (in other words, $\frac{0.15 + 0.18 + 0.21}{3}=0.18$. This is a reasonable assumption, but is it the best mean?
Well the answer depends on what data you are trying to describe! The arithmetic mean will give you the average tip percent, which answers the question "what tip percent can I expect any given customer to give?"
But the harmonic mean answers a slightly different question: for every dollar I sell, how much tip can I expect to receive. (Remember, the harmonic mean is all about proportions). The difference is subtle, but important. Imagine if everyone that has small bills tips 20%, but everyone with large bills tips 10%. The man for my tip will be around 15%. But because those with large bills contribute so much more to my total bill amount, their tips will drag down the proportion and make it closer to 10%.
So the harmonic mean may be better for predicting how many tips you expect given a total revenue, whereas the arithmetic mean will be better at predicting what tip percent my next customer will make.
For our example, let's take a look at these two values:
Can you write Python code for it?

def arithmetic_mean(tip, bills):
    n = tip.count()
    return sum(tip/bills)/n

arithmetic_mean(tips["tip"], tips["total_bill"])

def harmonic_mean(tip, bills):
    return sum(tip)/sum(bills)

harmonic_mean(tips["tip"], tips["total_bill"])


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
That looks pretty good, but of course mathematicians aren't excited by "looks pretty good." They need numbers to back that claim up. We'll get in to finding the best line in the next notebook on linear regression, but for now be settled that it there's a clear positive relationship (meaning $y$ goes up as $x$ goes up) between the variables.
Refere back tou the section on "Which mean?" See how the higher bills tend to be below the blue line? This again is more confirmation that the larger bills tend to tip less. Of course in a small dataset, you can't call this a significant trend, but it does corrobrate the discrepency in our earlier analysis.
Covariance and Correlation
Covariance
Let's now talk about how strong this positive relationship is. In other words, how sure can we be that if $x$ goes up $y$ will follow? Statisticians look at the way these values vary together with a term called covariance. When two variables tend to move together, their covariance is positive. If they move opposite, it's negative. If they are completely independent, it's 0.
The covariance for $N$ samples with random variables $X$ and $Y$ is given by the formula
$$ Q = \frac{1}{N-1} \sum_{i=1}^{N} (x_{i}-\bar{x})(y_i-\bar{y}) $$
In Python:
Can you write Python code for it?

# A useful helper function will be to define the dot product between two vectors
# dot product is defined as <v1, v2> * <u1, u2> = v1*u1 + v2*u2
from operator import mul
def dot(v, u):
    return sum(map(mul, v,u))

# Another useful helper function will be to take a list and output another list
# that contains the differences of the means
def diff_mean(values):
    mean = values.mean() # use Panda's arithmetic mean function
    deltas = []
    for v in values:
        deltas.append(v-mean)
    return deltas

def covariance(x, y):
    n = len(x)
    return dot(diff_mean(x), diff_mean(y)) / (n-1)

covariance(tips["total_bill"], tips["tip"])


I am working on the file "Statistics/Bivariate Data.ipynb".
The first task is:
The type of our tips dataset is a pandas DataFrame. Think of a DataFrame as Python's version of an excel table. Again, don't worry about it too much. There's plent of resources out there on Pandas if you want to learn more.
There are 7 variables in this dataset, but I'm going to scrub the table so we have the two pieces we want to examine: total_bill and tip.
Can you write Python code for it?

tips.drop(labels=["sex", "smoker", "day", "time", "size"], axis=1, inplace=True)


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
2. Geometric mean
The geometric mean is a similar idea but instead uses the product. It says if I multiply each value in our list together, what one value could I use instead to get the same result?
The geometric mean is very useful in things like growth or returns (e.g. stocks) because adding returns doesn't give us the ability to get returns over a longer length of time. In other words, if I have a stock growing at 5% per year, what will be the total returns after 5 years?
If you said 25%, you are wrong. It would be $1.05^5 - 1  \approx 27.63\%$
Mathematically, our geometric is:
$$ GM(x) = \sqrt[n]{x_1 \times x_2 \times \cdots \times x_n }$$
Can you write Python code for it?

returns = [1.05, 1.06, .98, 1.08]

def product(vals):
    '''
    This is a function that will multiply every item in the list
    together reducing it to a single number. The Pythonic way to
    do this would be to use the 'reduce' function like so:
    > reduce(lambda x, y: x * y, vals)
    We are explicit here for clairty.
    '''
    prod = 1
    for x in vals:
        prod = prod * x
    return prod

def geometric_mean(vals):
    geo_mean = product(vals) ** (1/len(vals)) # raising to 1/n is the same as nth root
    return geo_mean

geom = geometric_mean(returns)
geom


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
A classic uniform distribution: centered about 500, the median goes right down the middle, and the whiskers are evenly spaced. Now let's take a look at the skewed list:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
A few clues that this distribution is skewed:
* The mean is significantly different from the median
* The percentiles cluster around 25. A uniform distribution we'd expect 25, 50, and 75 for our percentiles.
* The max i smuch higher than the mean, median, or even 75th percentile.
Let's take a look at a simple plot to describe all of these statistics to us:
The Boxplot
Also sometimes the Box-and-Whisker plot, this is a great way to visualize a lot of the information our print_statistics function displayed. In particular, we can see in one graph
* Median
* 75th percentile (called the third quartile)
* 25th percentile (called the first quartile)
* The reach ($\pm1.5*IQR$), which shows outliers
It does not show the mean, but it can be intuited by looking at the plot. Let's take a look at plots for values and skewed:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
A much more reasonable \$7200! Now let's see what happens when Moneybags goes home:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Above we used the Counter class in the standard library. This class is a subclass of the dict that holds a dictionary of keys to their counts. We can build our own version of it like so:
Can you write Python code for it?

# Here we use the defaultdict that will initialize our first value if it doesn't yet exist
from collections import defaultdict
def make_counter(values):
    counts = defaultdict(int)
    for v in values:
        counts[v] += 1
    return counts

counts = make_counter([1, 2, 2, 3, 5, 6, 6, 6])
counts


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Below we'll use Seaborn to plot and visualize some of our data. Don't worry about this too much. Visualization, while important, is not the focus of this notebook.
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
But that's enough about modes. Check out wikipedia if you want more because there's no point spending more time on them then they're worth.
Hang in there, because we're getting close. Still to come is Percentiles, Boxplots, and Histograms. Three very import things.
Let's get to it.
Percentiles
A percentile is familiar to anyone who has taken the SAT. It answers the question: what percentage of students are dumber than I am? Well the College Board would love to tell you: congratulations, you're in the 92nd percentile!
Let's take a look at our old friend Mr. values with 10,000 numbers from 1-1000. Since this list is uniformly distributed, meaning every value is as likely to occur as any other, we expect that 25% of the numbers to be below 250, 50% to be below 500, and 75% to be below 750. Let's verify:
Can you write Python code for it?

def percentile(vals, elem):
    '''Returns the percent of numbers
    below the index.
    '''
    count = 0
    sorted_val = sorted(values)
    for val in sorted_val:
        if val > elem:
            return count/len(values)
        count += 1

for num in [250, 500, 750]:
    print("Percentile for {}: {}%".format(num, percentile(values, num)*100))


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
From this we see our average value is 502.1696. Let's double check that with our intuitive definition using the sum:
Can you write Python code for it?

avg_sum = len(values) * arithmetic_mean(values) #10,000 * 502.1696
print("{} =? {}".format(sum(values), avg_sum))


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Get used to this image because you will see it everywhere in statistics, data science, and life. Even though there are many many distributions out there (and even more variations on each of those), most people will be happy to apply the "bell curve" to almost anything. Chance are, though, they're right.
Variability
Let's talk about one more descriptive statistic that describes to us how much the values vary. In other words, if we're looking at test scores, did everyone do about the same? Or were there some big winners and losers?
Mathemeticians call this the variability. There are three major measures of variability: range, inter-quartile range (IQR), and variation/standard devaition.
1. Range
Range is a very simple measure: how far apart could my values possibly be? In our generated datasets above, the answer was pretty obvious. We generated a random number from 0 to 1000, so the range was $1000-0 = 1000$. The values of x could never go outside of this range.
But what about our gaussian values? The tandard normal distribution has asymptotic end points instead of absolute end points, so it's not so clean. We can see from the graph above that it doesn't look like there are any values above and below 4, so we'd expect a range of something around 8:
Can you write Python code for it?

print("Max: {}\nMin: {}\nRange: {}".format(
    max(gaussValues),
    min(gaussValues),
    max(gaussValues) - min(gaussValues)))


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
How do you interpret this? The max value is 100, so is a variance of 934.8 a lot or a little? Also, let's say the skewed distribution was a measure of price in dollars. Therefore, the units of variance would be 934.8 dollars squared. Doesn't make a whole lot of sense.
For this reason, most people will take the square root of the variance to give them a value called the standard deviation: $\sigma = \sqrt{\sigma^2}$.
Can you write Python code for it?

def stddev(vals):
    return sqrt(variance(vals))
stddev(skewed)


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Just like we'd expect. Now if the data set is not so nice and uniform, we expect these values to be quite different. Let's write a function to give us an element at a particular percentile:
Can you write Python code for it?

from math import ceil
def pct_value(vals, pct):
    sorted_vals = sorted(vals)
    n = len(vals)
    return sorted_vals[ceil(n*pct)]

for pct in [.25, .5, .75]:
    print("Element at percentile {}%: {}".format(pct*100, pct_value(values, pct)))


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Looks pretty different? Instead of being centered around 50, it looks like the box is centered around 40. The median is at 27 and much of the box is to the right of it. This shows us that the distribution is skewed to the right.
There's another important way to visualize a distribution and that is
Histogram
Ah, the moment we've all been waiting for. I keep teaching you ways to describe a dataset, but sometimes a picture is worth a thousand words. That picture is the Histogram.
A histogram is a bar chart in which the values of the dataset are plotted horizontally on the X axis and the frequencies (i.e. how many times that value was seen) are plotted on the Y axis. If you remember our functions to make a counter, a histogram is essentially a chart of those counts.
Think for a minute on what the histogram for our uniform dataset of 10,000 randomly generated numbers would look like? Pretty boring right?
Can you write Python code for it?

# Seaborn gives an easy way to plot histograms. Plotting from scratch is beyond the
# scope of the programming we will do
sns.distplot(values, kde=False, bins=100)


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Notice how the element at the 50th percentile is also our median! Now we have a second definition of the median.
Let's take a look now at a highly skewed set. It will range from 0-100 but we'll cluster it around 10
Can you write Python code for it?

skewed = []
for i in range(1,100):
    skewed += [i]*random.randint(0,int(4+i//abs(10.1-i)))

def print_statistics(vals, calc_mode=True):
    print("Count: {}".format(len(vals)))
    print("Mean: {:.2f}".format(arithmetic_mean(vals)))
    print("Median: {}".format(median(vals)))
    if calc_mode: print("Mode: {}".format(mode(Counter(vals))))
    print("Max: {}".format(max(vals)))
    print("Min: {}".format(min(vals)))
    print("Range: {}".format(max(vals)-min(vals)))
    for pct in [.25, .5, .75]:
        print("{:.0f}th Percentile: {}".format(pct*100, pct_value(vals, pct)))
    print("IQR: {}".format(pct_value(vals, 0.75) - pct_value(vals, 0.25)))

print_statistics(skewed)


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Now look at what happens with the arithmetic mean:
Can you write Python code for it?

arm = arithmetic_mean(returns)
arm

arm**4


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Remember this part because it will show up very soon when we talk about histograms, the chef's knife of a data scientist's data exploration kitchen.
But first, there's one more descriptive statistic that we should cover. And that is
The Mode
The mode is simply the most common element. If there are multiple elements with the same count, then there are two modes. If all elements have the same count, there are no modes. If the distribution is continuous (meaning it can take uncountably infinite values, which we will discuss in the Distributions chapter), then we use ranges of values to determine the mode. Honestly, I don't really find the mode too useful. A good time to use it is if there's a lot of categorical data (meaning values like "blue", "red", "green" instead of numerical data like 1,2,3). You might want to know what color car your dealership has the most of.
Let's take a look at that example now. I've built a set of cars with up to 20 cars of any of four colors.
Can you write Python code for it?

car_colors = ["red"] * random.randint(1,20) + \
    ["green"] * random.randint(1,20) + \
    ["blue"] * random.randint(1,20) + \
    ["black"] * random.randint(1,20)
car_colors

#Using our familiar counter
color_count = Counter(car_colors)
color_count

# We can see the mode above is 'blue' because we have 18. Let's verify:
def mode(counter):
    # store a list of name:count tuples in case multiple modes
    modes = [('',0)]
    for k,v in counter.items():
        highest_count = modes[0][1]
        if v > highest_count:
            modes = [(k,v)]
        elif v == highest_count:
            modes.append((k,v))

    return modes
mode(color_count)

# If we have multiple modes?
mode(Counter(['blue']*3 + ['green']*3 + ['black']*2))


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Seaborn's helpful sns.distplot() method simply turns a dataset into a histogram for us. The bins parameter allows us to make bins where instead of the frequency count of each variable, we plot the frequency count of a range of variables. This is very useful when we have continuous distribution (i.e. one that can take an infinite number of values of the range), as plotting every individual value is unfeasible and would make for an ugly graph.
Let's take a look at our skewed data:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
The median drops down to \$60,000 (which is the average of \$48,000 and \$72,000).
Let's take a look at our original values list of 10,000 numbers.
Can you write Python code for it?

median(values)

# Recall our values list is even, meaning 506.0 was both item 5000 and 5001
len(values)

# Lopping off the end returns the same value
median(values[:-1])

# Why? There are 9 506s in the list
from collections import Counter
c = Counter(values)
c[506]


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
The variance four our generated Gaussian numbers is roughly 1, which is another condition of a standard normal distribution (mean is 0, variance is 1). So no surprise. But the variance is a bit of a tricky number to intuit because it is the average of the squared differences between the mean. Take our skewed for example:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
This graph is pretty cluttered. That makes sense because it's 10,000 values between 1 and 1,000. That tells us there should be an average of 10 entries for each value. I'll leave counting this average as an exercise for the reader.
Let's make a sparse number line with just 200 values between 1 and 1000. There should be a lot more white space.
Can you write Python code for it?

sparse_values = [random.randrange(1,1001) for _ in range (200)]
sns.stripplot(x=sparse_values, jitter=True)


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Useful Python functions
Many statistics require knowing the length or sum of your data. Let's chug through some useful built-in functions.
Random
We'll use the random module a lot to generate random numbers to make fake datasets. The plain vanilla random generator will pull from a uniform distribution. There are also options to pull from other datasets. As we add more tools to our data science toolbox, we'll find that NumPy's random number generators to be more full-featured and play really nicely with Pandas, another key data science library. For now, we're going to avoid the overhead of learning another library and just use Python's standard library.
Can you write Python code for it?

random.seed(42)
values = [random.randrange(1,1001,1) for _ in range(10000)]
values[0:10]


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
Using our product function above, we can easily multiply all the values together to get what your return after 4 years is:
Can you write Python code for it?


I am working on the file "Statistics/Descriptive Statistics.ipynb".
The first task is:
or roughly $17.8\%$. Using our geometric mean should give us the same result:
Can you write Python code for it?


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
1.2 Hypergeometric distribution
$$ P(X=k) = \frac {C_M^k C_{N-M}^{n-k}} {C_N^n} $$
then $ X \sim$ Hypergeometric distribution with parameters ${N,M,n}$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

ngood, nbad, nsamp = 90, 10, 50

plt.hist(np.random.hypergeometric(ngood, nbad, nsamp, 5000))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
1.4 Poisson distribution $P(\lambda)$
$$ P(X=k) = e^{-\lambda} \frac{\lambda^k}{k!}, k = 0,1,2,... $$
then $ X \sim P(\lambda)$, $\lambda>0$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.poisson(5, 10000))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
2.2 Exponential distribution $E(\lambda)$
$$ f(x)=\left{\begin{array}{ll}
\lambda e^{-\lambda x}, & x>0 \
0, & otherwise
\end{array}\right.$$
then $X \sim E(\lambda)$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.exponential(scale=1.0, size=1000))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
2.3 Normal distribution $N(\mu,\sigma^2)$
$$ f(x) = \frac {1} {\sqrt{2\pi\sigma}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$
then $X \sim N(\mu,\sigma^2)$.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.normal(size=4000))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
3.2 t distribution $t(n)$
Given $X \sim N(0,1)$, $Y \sim \chi^2(n)$,
$$ T \hat{=} \frac{X}{\sqrt{\frac{Y}{n}}} \sim t(n)$$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.standard_t(2,50))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
Contious distribution

Homogeneous distribution $R(a,b)$
Exponential distribution $E(\lambda)$
Normal distribution $N(\mu,\sigma^2)$

2.1 Homogeneous distribution $R(a,b)$
$$ f(x)=\left{\begin{array}{ll}
c, & a<x<b \
0, & otherwise
\end{array}\right.$$
then $X \sim R(a,b)$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.random_sample(1000))
plt.show()


I am working on the file "Statistics/Distribution.ipynb".
The first task is:
Sample(small) distribution

$\chi^2$ distribution $\chi^2(n)$
t distribution $t(n)$
F distribution $F(n)$

3.1 $\chi^2$ distribution $\chi^2(n)$
Given $X_i \sim N(0,1)$,
$$ Y = \sum_{i=1}^{N} X_i^2 \sim \chi^2(n) $$
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(np.random.chisquare(3,1000))
plt.show()


I am working on the file "Statistics/Hypothesis Testing.ipynb".
The first task is:
Even though we flipped the coin 1,000 times, we still didn't get perfect $p$-values of 0.5 each. So we want to test our null hypothesis by saying "let us assume that the probability of heads and tails are both 0.5. What are the odds that we then get the results we did? And is it so unlikely that we have to reject the hypothesis that the two coins have the same probability?"
The normal distribution
Remember our brief discussion of the normal distribution in descriptive statistics? We haven't gone deep into distributions yet, but here are some spoilers that you can feel free to take at face value:

The number of heads in an experiment of $n$ flips of a coin with probability $p$ will be normally distributed with mean $\mu = pn$ and $\sigma = \sqrt{p(1-p)*n}$
Can you write Python code for it?

import math
def normal_approx_to_binomial(n, p):
    mu = p * n
    sigma = math.sqrt(p * (1 - p)  * n)
    return mu, sigma

normal_approx_to_binomial(1000, 0.5)


I am working on the file "Statistics/Hypothesis Testing.ipynb".
The first task is:
Let's play around with this for a bit so you get a sense. Let's use our coin flip example were $\mu=500$ and $\sigma=15.81$ and see the probability that the number of heads is greater than various values.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Above the result in either case is a DataFrame, but notice that the first result includes the end of the slice; the second doesn’t.
Limits of PMFs
PMFs work well if the number of values is small. But as the number of values increases, the probability associated with each value gets smaller and the effect of random noise increases.
For example, if we are interested in the distribution of age.
The parts of this figure are hard to interpret. There are many spikes and valleys, and some apparent differences between the distributions. It is hard to tell which of these features are meaningful. Also, it is hard to see overall patterns; for example, which distribution do you think has the higher mean?
These problems can be mitigated by binning the data; that is, dividing the range of values into non-overlapping intervals and counting the number of values in each bin. Binning can be useful, but it is tricky to get the size of the bins right. If they are big enough to smooth out noise, they might also smooth out useful information.
An alternative that avoids these problems is the cumulative distribution function (CDF), which is the subject of this chapter. But before I can explain CDFs, I have to explain percentiles.
Cumulative Distribution Function
The difference between “percentile” and “percentile rank” can be confusing, and people do not always use the terms precisely. To summarize, PercentileRank takes a value and computes its percentile rank in a set of values; Percentile takes a percentile rank and computes the corresponding value.
Can you write Python code for it?

def PercentileRank(scores, your_score):
    count = 0
    for score in scores:
        if score <= your_score:
            count += 1
    percentile_rank = 100.0 * count / len(scores)
    return percentile_rank

social_network.dropna(inplace=True)

salary = list(social_network["EstimatedSalary"])

my_sal = 100000

PercentileRank(salary, my_sal)

def Percentile(scores, percentile_rank):
    scores.sort()
    for score in scores:
        if PercentileRank(scores, score) >= percentile_rank:
            return score

Percentile(salary, 50)

def Percentile2(scores, percentile_rank):
    scores.sort()
    index = percentile_rank * (len(scores)-1) // 100
    return scores[index]

Percentile2(salary, 50)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
After the shift, the first lag values are nan, so I use a slice to remove them before computing Corr.
If we apply SerialCorr to the raw price data with lag 1, we find serial correlation 0.48 for the high quality category, 0.16 for medium and 0.10 for low. In any time series with a long-term trend, we expect to see strong serial correlations; for example, if prices are falling, we expect to see values above the mean in the first half of the series and values below the mean in the second half.
It is more interesting to see if the correlation persists if you subtract away the trend. For example, we can compute the residual of the EWMA and then compute its serial correlation:
Can you write Python code for it?

ewma = reindexed.ppg.ewm(30).mean()
resid = reindexed.ppg - ewma
corr = SerialCorr(resid, 1)
print(corr)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
By default Cov computes deviations from the sample means, or you can provide known means. If xs and ys are Python sequences, np.asarray converts them to NumPy arrays. If they are already NumPy arrays, np.asarray does nothing.
This implementation of covariance is meant to be simple for purposes of explanation. NumPy and pandas also provide implementations of covariance, but both of them apply a correction for small sample sizes that we have not covered yet, and np.cov returns a covariance matrix, which is more than we need for now.
Pearson Correlation
Covariance is useful in some computations, but it is seldom reported as a summary statistic because it is hard to interpret. Among other problems, its units are the product of the units of X and Y .
One solution to this problem is to divide the deviations by the standard deviation, which yields standard scores, and
compute the product of standard scores:

                                    p i = ( x i − x ̄ )*( y i − y ̄ )/ SX*SY

Where SX and SY are the standard deviations of X and Y . The mean of these products is
ρ = 1/n SUMMATION􏱅 pi

Or we can rewrite ρ by factoring out SX and SY :
ρ= Cov(X,Y)/SX*SY

This value is called Pearson’s correlation after Karl Pearson, an influential early statistician. It is easy to compute and easy to interpret. Because standard scores are dimensionless, so is ρ.
Can you write Python code for it?

def Corr(xs, ys):
    xs = np.asarray(xs)
    ys = np.asarray(ys)
    meanx, varx = np.mean(xs), np.var(xs)
    meany, vary = np.mean(ys), np.var(ys)
    corr = Cov(xs, ys, meanx, meany) / math.sqrt(varx * vary)
    return corr


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Central limit theorem
If we add values drawn from normal distributions, the distribution of the sum is normal. Most other distributions don’t have this property; if we add values drawn from other distributions, the sum does not generally have an analytic distribution.
But if we add up n values from almost any distribution, the distribution of the sum converges to normal as n increases.
More specifically, if the distribution of the values has mean and standard deviation μ and σ, the distribution of the sum is approximately N(nμ,nσ2).
This result is the Central Limit Theorem (CLT). It is one of the most useful tools for statistical analysis, but it comes with caveats:
* The values have to be drawn independently. If they are correlated, the CLT doesn’t apply (although this is seldom a problem in practice).
* The values have to come from the same distribution (although this requirement can be relaxed).
* The values have to be drawn from a distribution with finite mean and variance. So most Pareto distributions are out.
* The rate of convergence depends on the skewness of the distribution. Sums from an exponential distribution converge for small n. Sums from a lognormal distribution require larger sizes.
The Central Limit Theorem explains the prevalence of normal distributions in the natural world. Many characteristics of living things are affected by genetic and environmental factors whose effect is additive. The character- istics we measure are the sum of a large number of small effects, so their distribution tends to be normal.
Correlation test
we used a permutation test for the correlation between Age and Estimated Salary, and found that it is statistically significant, with p-value less than 0.001.
Now we can do the same thing analytically. The method is based on this mathematical result: given two variables that are normally distributed and uncorrelated, if we generate a sample with size n, compute Pearson’s corre- lation, r, and then compute the transformed correlation
                                    t...
Can you write Python code for it?

def StudentCdf(n):
    ts = np.linspace(-3, 3, 101)
    ps = scipy.stats.t.cdf(ts, df=n-2)
    rs = ts / np.sqrt(n - 2 + ts**2)
    return thinkstats2.Cdf(rs, ps)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Dataframe Indexing
Can you write Python code for it?

### Create Dataframe from array

array = np.random.randn(4, 2)
df = pd.DataFrame(array)

df

columns = ['A', 'B']
df = pd.DataFrame(array, columns=columns)

index = ['a', 'b', 'c', 'd']
df = pd.DataFrame(array, columns=columns, index=index)
df


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
DiffMeansOneSided inherits MakeModel and RunModel from above testing technique; the only difference is that TestStatistic does not take the absolute value of the difference. This kind of test is called one-sided because it only counts one side of the distribution of differences. The previous test, using both sides, is two-sided.
For this version of the test, the p-value is half of previous. In general the p-value for a one-sided test is about half the p-value for a two-sided test, depending on the shape of the distribution.
The one-sided hypothesis, that purchasing customer is old, is more specific than the two-sided hypothesis, so the p-value is smaller.
We can use the same framework to test for a difference in standard deviation. So we might hypothesize that the standard deviation is higher. Here’s how we can test that:
DiffStdPermute
Can you write Python code for it?

def TestStatistic(data):
    group1, group2 = data
    test_stat = group1.std() - group2.std()
    return test_stat

def MakeModel(data):
    group1, group2 = data
    n, m = len(group1), len(group2)
    pool = np.hstack((group1, group2))
    #print(pool.shape)
    return pool, n

def RunModel(pool, n):
    np.random.shuffle(pool)
    data = pool[:n], pool[n:]
    return data

def sample_generator(data):
    pool, n = MakeModel(data)
    return RunModel(pool, n)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Effect Size
An effect size is a summary statistic intended to describe (wait for it) the size of an effect. For example, to describe the difference between two groups, one obvious choice is the difference in the means.
https://en.wikipedia.org/wiki/Effect_size
Can you write Python code for it?

purchased_customer.Age.mean() - no_purchase.Age.mean()

purchased_customer["EstimatedSalary"].mean() - no_purchase["EstimatedSalary"].mean()

def CohenEffectSize(group1, group2):
    diff = group1.mean() - group2.mean()
    var1 = group1.var()
    var2 = group2.var()
    n1, n2 = len(group1), len(group2)
    pooled_var = (n1 * var1 + n2 * var2) / (n1 + n2)
    d = diff / math.sqrt(pooled_var)
    return d

CohenEffectSize(purchased_customer.Age, no_purchase.Age)

#Effect Size
13.6/max(social_network["Age"])*100


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Estimated Salary (Purchase vs No Purchase)
Can you write Python code for it?

no_purchase_prob = []
for sal in sorted(no_purchase["EstimatedSalary"]):
    no_purchase_prob.append(EvalCdf(no_purchase["EstimatedSalary"], sal))

purchase_prob = []
for sal in sorted(purchased_customer["EstimatedSalary"]):
    purchase_prob.append(EvalCdf(purchased_customer["EstimatedSalary"], sal))


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Evaluating a PDF for a particular value of x is usually not useful. The result
    is not a probability; it is a probability density.
In physics, density is mass per unit of volume; in order to get a mass, you have to multiply by volume or, if the density is not constant, you have to integrate over volume.
Similarly, probability density measures probability per unit of x. In order to get a probability mass, you have to integrate over x.
Can you write Python code for it?

Xs = sorted(social_network["Age"])

mean, std = social_network["Age"].mean(), social_network["Age"].std()
PDF = stat.norm.pdf(Xs, mean, std)

step_plot(Xs, PDF, "Age", ylabel="Density")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
FalseNegRate takes data in the form of two sequences, one for each group. Each time through the loop, it simulates an experiment by drawing a random sample from each group and running a hypothesis test. Then it checks the result and counts the number of false negatives.
Resample takes a sequence and draws a sample with the same length, with replacement:
Can you write Python code for it?

data = purchased_customer.Age.values, no_purchase.Age.values
neg_rate = FalseNegRate(data)

neg_rate


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
For an explanation of why S^2 is biased, and a proof that (Sn−1)^2 is unbiased, http://wikipedia.org/wiki/Bias_of_an_estimator.
The biggest problem with this estimator is that its name and symbol are used inconsistently. The name “sample variance” can refer to either S^2 or (Sn−1)^2, and the symbol S^2 is used for either or both.
Here is a function that simulates the estimation game and tests the perfor- mance of S^2 and (Sn−1)^2:
Can you write Python code for it?

def Estimate2(n=7, m=1000):
    mu = 0
    sigma = 1
    estimates1 = []
    estimates2 = []
    for _ in range(m):
        xs = [random.gauss(mu, sigma) for i in range(n)]
        biased = np.var(xs)
        unbiased = np.var(xs, ddof=1)
        estimates1.append(biased)
        estimates2.append(unbiased)
        print('mean error biased', MeanError(estimates1, sigma**2))
        print('mean error unbiased', MeanError(estimates2, sigma**2))


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Histogram
One of the best ways to describe a variable is to report the values that appear in the dataset and how many times each value appears. This description is called the distribution of the variable.
The most common representation of a distribution is a histogram, which is a graph that shows the frequency of each value. In this context, “frequency” means the number of times the value appears.
Can you write Python code for it?

#Convert series into a list
age = list(social_network["Age"])

# Create dict format for age, it helps in building histogram easily.
# The result is a dictionary that maps from values to frequencies.
hist = {}
for a in age:
    hist[a] = hist.get(a,0) + 1

#Same as dict format above.
counter = Counter(age)

#The result is a Counter object, which is a subclass of dictionary.

#To loop through the values in order, you can use the built-in function sorted:
for val in sorted(counter):
    print(val, counter[val])

# Use items() to iterate over the dict/counter.
for value, freq in counter.items():
    print(value, freq)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
I convert the arguments to pandas Series objects so I can use rank, which computes the rank for each value and returns a Series. Then I use Corr to compute the correlation of the ranks.
I could also use Series.corr directly and specify Spearman’s method:
Can you write Python code for it?

def SpearmanCorr(xs, ys):
    xs = pd.Series(xs)
    ys = pd.Series(ys)
    return xs.corr(ys, method='spearman')

SpearmanCorr(mall_customer["Age"], mall_customer["Annual Income (k$)"])

SpearmanCorr(mall_customer["Annual Income (k$)"], mall_customer["Spending Score (1-100)"])

SpearmanCorr(mall_customer["Age"], mall_customer["Spending Score (1-100)"])

SpearmanCorr(social_network["Age"], social_network["EstimatedSalary"])

Corr(social_network["Age"], social_network["EstimatedSalary"])


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
In each iteration, we choose n values from a normal distribution with the given parameters, and compute the sample mean, xbar. We run 1000 simulations and then compute the distribution, cdf, of the estimates. The result is shown in Figure. This distribution is called the sampling distribution of the estimator. It shows how much the estimates would vary if we ran the experiment over and over.
The mean of the sampling distribution is pretty close to the hypothetical value of μ, which means that the experiment yields the right answer, on average. After 1000 tries, the lowest result is 82 kg, and the highest is 98 kg. This range suggests that the estimate might be off by as much as 8 kg.
There are two common ways to summarize the sampling distribution:
* Standard error (SE) is a measure of how far we expect the estimate to be off, on average. For each simulated experiment, we compute the error, x ̄ − μ, and then compute the root mean squared error (RMSE). In this example, it is roughly 2.5 kg.
* A confidence interval (CI) is a range that includes a given fraction of the sampling distribution. For example, the 90% confidence interval is the range from the 5th to the 95th percentile. In this example, the 90% CI is (86, 94) kg.
Standard errors and confidence intervals are the source of much confusion:

People often confuse standard error and standard deviation. Remember that standard deviation describes variability in a measured quantity; in this example, the standard deviation of gorilla weight is 7.5 kg. Standard error describes variability in an estimate. In this example, the standard error of the mean, based on a sample of 9 measurements, is 2.5 kg.

One way to remember the difference is that, as sample size increases, standard error gets smaller; standard deviation does not.
* People often think that there is a 90% probability that the actual pa- rameter, μ, falls in the 90% confidence interval. Sadly, that is not true. If you want to make a claim like that, you have to use Bayesian methods (see my book, Think...
Can you write Python code for it?

def Estimate3(n=7, m=1000):
    lam = 2
    means = []
    medians = []
    for _ in range(m):
        xs = np.random.exponential(1.0/lam, n)
        L = 1 / np.mean(xs)
        Lm = math.log(2) / pd.Series(xs).median()
        means.append(L)
        medians.append(Lm)
    print('rmse L', RMSE(means, lam))
    print('rmse Lm', RMSE(medians, lam))
    print('mean error L', MeanError(means, lam))
    print('mean error Lm', MeanError(medians, lam))


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
In my age group, denoted M4049 for “male between 40 and 49 years of age”, I came in 26th out of 256. So my percentile rank in my age group was 90%.
If I am still running in 10 years (and I hope I am), I will be in the M5059 division. Assuming that my percentile rank in my division is the same, how much slower should I expect to be?
I can answer that question by converting my percentile rank in M4049 to a position in M5059. Here’s the code:
Can you write Python code for it?

def PercentileToPosition(percentile, field_size):
    beat = percentile * field_size / 100.0
    position = field_size - beat + 1
    return position


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
It’s a good idea to look at a figure like this to assess whether the relationship is linear and whether the fitted line seems like a good model of the relationship.
Another useful test is to plot the residuals. A residuals function below
Can you write Python code for it?

def Residuals(xs, ys, inter, slope):
    xs = np.asarray(xs)
    ys = np.asarray(ys)
    res = ys - (inter + slope * xs)
    return res


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Kernel Density Estimation
Kernel density estimation (KDE) is an algorithm that takes a sample and finds an appropriately smooth PDF that fits the data.
https://en.wikipedia.org/wiki/Kernel_density_estimation
Can you write Python code for it?

sample = [random.gauss(mean, std) for i in range(500)]

Kernel_density_estimate = stat.gaussian_kde(sample)

sample_pdf = Kernel_density_estimate.evaluate(sorted(social_network["Age"]))

step_plot(Xs, PDF, "Age", ylabel="Density")
step_plot(Xs, sample_pdf, "Age", ylabel="Density")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Least square fit between salary and experience
Can you write Python code for it?

regression_data = pd.read_csv("Salary_Data.csv")

inter, slope = LeastSquares(regression_data["YearsExperience"], regression_data["Salary"])

fit_xs, fit_ys = FitLine(regression_data["YearsExperience"], inter, slope)

print("intercept: ", inter)
print("Slope: ", slope)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
LeastSquares takes sequences xs and ys and returns the estimated parameters inter and slope. For details on how it works, see http://wikipedia.org/wiki/Numerical_methods_for_linear_least_squares.
FitLine, which takes inter and slope and re- turns the fitted line for a sequence of xs.
Can you write Python code for it?

def FitLine(xs, inter, slope):
    fit_xs = np.sort(xs)
    fit_ys = inter + slope * fit_xs
    return fit_xs, fit_ys


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Notice that when log_o is greater than 0, o is greater than 1 and p is greater than 0.5.
The likelihood of an outcome is p when y==1 and 1-p when y==0. For example, if we think the probability of a boy is 0.8 and the outcome is a boy, the likelihood is 0.8; if the outcome is a girl, the likelihood is 0.2. We can compute that like this:
Can you write Python code for it?

likes = y * p + (1-y) * (1-p)
print(likes)

#The overall likelihood of the data is the product of likes:
like = np.prod(likes)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
One apparent feature in these plots is a gap around November 2013. It’s possible that data collection was not active during this time, or the data might not be available. We will consider ways to deal with this missing data later.
Visually, it looks like the price of high quality cannabis is declining during this period, and the price of medium quality is increasing. The price of low quality might also be increasing, but it is harder to tell, since it seems to be more volatile. Keep in mind that quality data is reported by volunteers, so trends over time might reflect changes in how participants apply these labels.
Linear regression
Although there are methods specific to time series analysis, for many prob- lems a simple way to get started is by applying general-purpose tools like linear regression. The following function takes a DataFrame of daily prices and computes a least squares fit, returning the model and results objects from StatsModels:
Can you write Python code for it?

def RunLinearModel(daily):
    model = smf.ols('ppg ~ years', data=daily)
    results = model.fit()
    return model, results

def SummarizeResults(results):
    """Prints the most important parts of linear regression results:
    results: RegressionResults object
    """
    for name, param in results.params.items():
        pvalue = results.pvalues[name]
        print('%s   %0.3g   (%.3g)' % (name, param, pvalue))

    try:
        print('R^2 %.4g' % results.rsquared)
        ys = results.model.endog
        print('Std(ys) %.4g' % ys.std())
        print('Std(res) %.4g' % results.resid.std())
        print('\n')
    except AttributeError:
        print('R^2 %.4g' % results.prsquared)
        print('\n')

#Then we can iterate through the qualities and fit a model to each:
for name, daily in dailies.items():
    model, results = RunLinearModel(daily)
    print(name)
    SummarizeResults(results)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Overlapping data points look darker, so darkness is proportional to density. In this version of the plot we can see two details that were not apparent before: vertical clusters at Annual income 57k$.
Jittering: https://blogs.sas.com/content/iml/2011/07/05/jittering-to-prevent-overplotting-in-statistical-graphics.html
Can you write Python code for it?

plt.scatter(mall_customer["Age"], mall_customer["Annual Income (k$)"],alpha=0.2)
plt.grid()
plt.ylabel("Annual Income")
plt.xlabel("Age")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
PMF - Probability Mass Function
Purchased
Can you write Python code for it?

pmf_age_purchased = {}
for age in purchased_customer["Age"].value_counts().index:
    pmf_age_purchased[age] = purchased_customer[purchased_customer["Age"]==age]["Age"].count() / purchased_customer["Age"].shape[0]

#The Pmf is normalized so total probability is 1.
sum(list(pmf_age_purchased.values()))


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Pareto Distribution https://en.wikipedia.org/wiki/Pareto_distribution
Preferential Attachment https://en.wikipedia.org/wiki/Preferential_attachment
Probability Distribution Function
The derivative of CDF is PDF
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Percentile ranks are useful for comparing measurements across different groups. For example, people who compete in foot races are usually grouped by age and gender. To compare people in different age groups, you can convert race times to percentile ranks.
A few years ago I ran the James Joyce Ramble 10K in Dedham MA; I finished in 42:44, which was 97th in a field of 1633. I beat or tied 1537 runners out of 1633, so my percentile rank in the field is 94%.
More generally, given position and field size, we can compute percentile rank:
Can you write Python code for it?

def PositionToPercentile(position, field_size):
    beat = field_size - position + 1
    percentile = 100.0 * beat / field_size
    return percentile


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
PlotFittedValues makes a scatter plot of the data points and a line plot of the fitted values. Plot shows the results for high quality cannabis. The model seems like a good linear fit for the data; nevertheless, linear regression is not the most appropriate choice for this data:

First, there is no reason to expect the long-term trend to be a line or any other simple function. In general, prices are determined by supply and demand, both of which vary over time in unpredictable ways.
Second, the linear regression model gives equal weight to all data, recent and past. For purposes of prediction, we should probably give more weight to recent data.
Finally, one of the assumptions of linear regression is that the residuals are uncorrelated noise. With time series data, this assumption is often false because successive values are correlated.

Moving Average
Most time series analysis is based on the modeling assumption that the ob- served series is the sum of three components:
* Trend: A smooth function that captures persistent changes.
* Seasonality: Periodic variation, possibly including daily, weekly,
monthly, or yearly cycles.
* Noise: Random variation around the long-term trend.
Regression is one way to extract the trend from a series, as we saw in the previous section. But if the trend is not a simple function, a good alternative is a moving average. A moving average divides the series into overlapping regions, called windows, and computes the average of the values in each window.
One of the simplest moving averages is the rolling mean, which computes the mean of the values in each window. For example, if the window size is 3, the rolling mean computes the mean of values 0 through 2, 1 through 3, 2 through 4, etc.
pandas provides rolling_mean, which takes a Series and a window size and returns a new Series.
Can you write Python code for it?

series = pd.Series(np.arange(10))

moving_avg = series.rolling(3).mean()


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Plotting
Can you write Python code for it?

plt.hist(age)
plt.xlabel("Age")
plt.ylabel("Freq")

purchased_customer = social_network[social_network["Purchased"]==1]

plt.hist(purchased_customer["Age"])
plt.xlabel("Age")
plt.ylabel("Freq")

social_network.head()

no_purchase = social_network[social_network["Purchased"]==0]

no_purchase.Age.mean()

purchased_customer.Age.mean()


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Pmf and Hist are almost the same thing, except that a Pmf maps values to floating-point probabilities, rather than integer frequencies. If the sum of the probabilities is 1, the Pmf is normalized. Pmf provides Normalize, which computes the sum of the probabilities and divides through by a factor
https://en.wikipedia.org/wiki/Moment_of_inertia
Skewness
Skewness is a property that describes the shape of a distribution. If the distribution is symmetric around its central tendency, it is unskewed. If the values extend farther to the right, it is “right skewed” and if the values extend left, it is “left skewed.”
This use of “skewed” does not have the usual connotation of “biased.” Skewness only describes the shape of the distribution; it says nothing about whether the sampling process might have been biased.
A way to evaluate the asymmetry of a distribution is to look at the relationship between the mean and median. Extreme values have more effect on the mean than the median, so in a distribution that skews left, the mean is less than the median. In a distribution that skews right, the mean is greater.
Pearson’s median skewness coefficient is a measure of skewness based on the difference between the sample mean and median:
                                        gp = 3(x ̄ − m)/S

Where x ̄ is the sample mean, m is the median, and S is the standard deviation.
The sign of the skewness coefficient indicates whether the distribution skews left or right, but other than that, they are hard to interpret. Sample skewness is less robust; that is, it is more susceptible to outliers. As a result it is less reliable when applied to skewed distributions, exactly when it would be most relevant.
Pearson’s median skewness is based on a computed mean and variance, so it is also susceptible to outliers, but since it does not depend on a third moment, it is somewhat more robust.
Relationship between two variables
Two variables are related if knowing one gives you information about the other. For example, height and weight are...
Can you write Python code for it?

mall_customer = pd.read_csv("Mall_Customers.csv")

mall_customer.isnull().sum()


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Quantiles: https://en.wikipedia.org/wiki/Quantile
Can you write Python code for it?