I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Replication
The hypothesis testing process I demonstrated in this above, strictly speaking, good practice.
First, I performed multiple tests. If you run one hypothesis test, the chance of a false positive is about 1 in 20, which might be acceptable. But if you run 20 tests, you should expect at least one false positive, most of the time.
Second, I used the same dataset for exploration and testing. If you explore a large dataset, find a surprising effect, and then test whether it is significant, you have a good chance of generating a false positive.
To compensate for multiple tests, you can adjust the p-value threshold (see https://en.wikipedia.org/wiki/Holm-Bonferroni_method). Or you can address both problems by partitioning the data, using one set for exploration and the other for testing.
In some fields these practices are required or at least encouraged. But it is also common to address these problems implicitly by replicating published results. Typically the first paper to report a new result is considered ex- ploratory. Subsequent papers that replicate the result with new data are considered confirmatory.
Linear Least Squares
Correlation coefficients measure the strength and sign of a relationship, but not the slope. There are several ways to estimate the slope; the most common is a linear least squares fit. A “linear fit” is a line intended to model the relationship between variables. A “least squares” fit is one that minimizes the mean squared error (MSE) between the line and the data.
Suppose we have a sequence of points, ys, that we want to express as a function of another sequence xs. If there is a linear relationship between xs and ys with intercept inter and slope slope,
we expect each y[i] to be inter + slope * x[i].

But unless the correlation is perfect, this prediction is only approximate. The vertical deviation from the line, or residual, is
res = ys - (inter + slope * xs)

The residuals might be due to random factors like measurement error, or non- random factors that are unknown. For...
Can you write Python code for it?

#Implementation of Linear Least square

def LeastSquares(xs, ys):
    meanx, varx = pd.Series(xs).mean(), pd.Series(xs).var()
    meany = pd.Series(ys).mean()
    slope = Cov(xs, ys, meanx, meany) / varx
    inter = meany - slope * meanx
    return inter, slope


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Residuals takes sequences xs and ys and estimated parameters inter and slope. It returns the differences between the actual values and the fitted line.
Can you write Python code for it?

residuals = list(Residuals(regression_data["YearsExperience"], regression_data["Salary"], inter, slope))

regression_data["Residuals"] = residuals

bins = np.arange(0, 15, 2)
indices = np.digitize(regression_data.YearsExperience, bins)
groups = regression_data.groupby(indices)
for i, group in groups:
    print(i, len(group))

year_exps = [group.YearsExperience.mean() for i, group in groups]

age_residuals = defaultdict(list)
for i, grp in groups:
    for res in grp["Residuals"]:
        age_residuals[i].append(EvalCdf(grp["Residuals"], res))

age_residuals

for percent in [75, 50, 25]:
    residue = [Percentile(age_residuals[k], percent) for k,v in age_residuals.items()]
    label = '%dth' %percent
    plt.plot(year_exps, residue)
    plt.xlabel("Experience Year")
    plt.ylabel("Residuals")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Sampling Distributions
Suppose you are a scientist studying gorillas in a wildlife preserve. You want to know the average weight of the adult female gorillas in the preserve. To weigh them, you have to tranquilize them, which is dangerous, expensive, and possibly harmful to the gorillas. But if it is important to obtain this information, it might be acceptable to weigh a sample of 9 gorillas. Let’s assume that the population of the preserve is well known, so we can choose a representative sample of adult females. We could use the sample mean, x ̄, to estimate the unknown population mean, μ.
Having weighed 9 female gorillas, you might find x ̄ = 90 kg and sample standard deviation, S = 7.5 kg. The sample mean is an unbiased estimator of μ, and in the long run it minimizes MSE. So if you report a single estimate that summarizes the results, you would report 90 kg.
But how confident should you be in this estimate? If you only weigh n = 9 gorillas out of a much larger population, you might be unlucky and choose the 9 heaviest gorillas (or the 9 lightest ones) just by chance. Variation in the estimate caused by random selection is called sampling error.
Sampling Error
To quantify sampling error, we can simulate the sampling process with hypothetical values of μ and σ, and see how much x ̄ varies.
Since we don’t know the actual values of μ and σ in the population, we’ll use the estimates x ̄ and S. So the question we answer is: “If the actual values of μ and σ were 90 kg and 7.5 kg, and we ran the same experiment many times, how much would the estimated mean, x ̄, vary?”
Can you write Python code for it?

def SimulateSample(mu=90, sigma=7.5, n=9, m=1000):
    means = []
    for j in range(m):
        xs = np.random.normal(mu, sigma, n)
        xbar = np.mean(xs)
        means.append(xbar)

    return sorted(means)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
So using SAT score to predict IQ reduces RMSE from 15 points to 10.4 points. A correlation of 0.72 yields a reduction in RMSE of only 31%.
If you see a correlation that looks impressive, remember that R2 is a better indicator of reduction in MSE, and reduction in RMSE is a better indicator of predictive power.
Testing a linear model
The effect of years of experience is high on salary. So is it possible that the apparent relationship is due to chance? There are several ways we might test the results of a linear fit.
One option is to test whether the apparent reduction in MSE is due to chance. In that case, the test statistic is R2 and the null hypothesis is that there is no relationship between the variables. We can simulate the null hypothesis by permutation. In fact, because R2 = ρ2, a one-sided test of R2 is equivalent to a two-sided test of ρ. We’ve already done that test, and found p < 0.001, so we conclude that the apparent relationship between experience and salary is statistically significant.
Another approach is to test whether the apparent slope is due to chance. The null hypothesis is that the slope is actually zero; in that case we can model the salary as random variations around their mean. Try out hypothesis as before!
Can you write Python code for it?

Corr(regression_data["YearsExperience"], regression_data["Salary"])

def TestStatistic(data):
    exp, sal = data
    _, slope = LeastSquares(exp, sal)
    return slope
def MakeModel(data):
    _, sals = data
    ybar = sals.mean()
    res = sals - ybar
    return ybar, res
def RunModel(data):
    exp, _ = data
    sals = ybar + np.random.permutation(res)
    return exp, sals

ybar, res = MakeModel(data)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Some of the characteristics we might want to report are:

central tendency: Do the values tend to cluster around a particular point?
modes: Is there more than one cluster?
spread: How much variability is there in the values?
tails: How quickly do the probabilities drop off as we move away from the modes?
outliers: Are there extreme values far from the modes?

Why and Why not "Mean" can be used for central tendenancy
Sometimes the mean is a good description of a set of values. For example, apples are all pretty much the same size (at least the ones sold in supermar- kets). So if I buy 6 apples and the total weight is 3 pounds, it would be a reasonable summary to say they are about a half pound each.
But pumpkins are more diverse. Suppose I grow several varieties in my gar- den, and one day I harvest three decorative pumpkins that are 1 pound each, two pie pumpkins that are 3 pounds each, and one Atlantic Giant⃝R pumpkin that weighs 591 pounds. The mean of this sample is 100 pounds, but if I told you “The average pumpkin in my garden is 100 pounds,” that would be misleading. In this example, there is no meaningful average because there is no typical pumpkin.
Can you write Python code for it?

print("Variance: ",purchased_customer.Age.var())
print("Standard Deviation: ",purchased_customer.Age.std())

print("Variance: ",no_purchase.Age.var())
print("Standard Deviation:",no_purchase.Age.std())


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Testing Correlation
This framework can also test correlations. For example, in the NSFG data set, the correlation between customer's Age and his estimated salary is about 0.11. It seems like older customers have more salary. But could this effect be due to chance?
For the test statistic, I use Pearson’s correlation, but Spearman’s would work as well. If we had reason to expect positive correlation, we would do a one-sided test. But since we have no such reason, I’ll do a two-sided test using the absolute value of correlation.
The null hypothesis is that there is no correlation between customers age and his salary. By shuffling the observed values, we can simulate a world where the distributions of age and salary are the same, but where the variables are unrelated:
Can you write Python code for it?

Corr(social_network["Age"], social_network["EstimatedSalary"])

def TestStatistic(data):
    xs, ys = data
    test_stat = abs(Corr(xs, ys))
    return test_stat

def RunModel(data):
    xs, ys = data
    xs = np.random.permutation(xs)
    return xs, ys


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The DataFrame has a row for each reported transaction and the following
columns:

city: string city name.
state: two-letter state abbreviation.
price: price paid in dollars.
amount: quantity purchased in grams.
quality: high, medium, or low quality, as reported by the purchaser. • date: date of report, presumed to be shortly after date of purchase. • ppg: price per gram, in dollars.
state.name: string state name.
lat: approximate latitude of the transaction, based on city name.
lon: approximate longitude of the transaction.

Each transaction is an event in time, so we could treat this dataset as a time series. But the events are not equally spaced in time; the number of transactions reported each day varies from 0 to several hundred. Many methods used to analyze time series require the measurements to be equally spaced, or at least things are simpler if they are.
In order to demonstrate these methods, I divide the dataset into groups by reported quality, and then transform each group into an equally spaced series by computing the mean daily price per gram.
Can you write Python code for it?

def GroupByQualityAndDay(transactions):
    groups = transactions.groupby('quality')
    dailies = {}
    for name, group in groups:
        dailies[name] = GroupByDay(group)
    return dailies


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The data are represented as a list of frequencies: the observed values are [8, 9, 19, 5, 8, 11]; the expected frequencies are all 10. The test statistic is the sum of the absolute differences
The null hypothesis is that the die is fair, so we simulate that by drawing random samples from values. RunModel uses Hist to compute and return the list of frequencies.
Can you write Python code for it?

data = [8, 9, 19, 5, 8, 11]
actual_diff = TestStatistic(data)

def calculate_pvalue(data, iters=1000):
    test_stats = [TestStatistic(RunModel(data))
                           for _ in range(iters)]
    count = sum(1 for x in test_stats if x >= actual_diff)
    return sorted(test_stats),count / iters

test_stats, pval = calculate_pvalue(data)

pval


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The data are represented as sequences of exp and sals. The test statistic is the slope estimated by LeastSquares. The model of the null hypothesis is represented by the mean sals of all employees and the deviations from the mean. To generate simulated data, we permute the deviations and add them to the mean.
Can you write Python code for it?

data = regression_data.YearsExperience.values, regression_data.Salary.values
actual_diff =  TestStatistic(data)

def calculate_pvalue(data, iters=1000):
    test_stats = [TestStatistic(RunModel(data))
                           for _ in range(iters)]
    count = sum(1 for x in test_stats if x >= actual_diff)
    return sorted(test_stats),count / iters

test_stats, pval = calculate_pvalue(data)

pval


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The estimated intercept and slope are 27465.89 and 9134.96 salary per year. These values are hard to interpret in this form: the intercept is the expected salary of an employee, who has 0 year experience, like salary for fresher.
Can you write Python code for it?

plt.scatter(regression_data["YearsExperience"], regression_data["Salary"])
plt.plot(fit_xs, fit_ys)
plt.xlabel("Experience")
plt.ylabel("Salary")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The estimated slopes indicate that the price of high quality cannabis dropped by about 71 cents per year during the observed interval; for medium quality it increased by 28 cents per year, and for low quality it increased by 57 cents per year. These estimates are all statistically significant with very small p-values.
The R2 value for high quality cannabis is 0.44, which means that time as an explanatory variable accounts for 44% of the observed variability in price. For the other qualities, the change in price is smaller, and variability in prices is higher, so the values of R2 are smaller (but still statistically significant).
Can you write Python code for it?

#The following code plots the observed prices and the fitted values:
def PlotFittedValues(model, results, label=''):
    years = model.exog[:,1]
    values = model.endog
    plt.scatter(years, values, s=15, label=label)
    plt.plot(years, results.fittedvalues, label='model')
    plt.xlabel("years")
    plt.ylabel("ppg")

PlotFittedValues(model, results)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The first line computes a date range that includes every day from the be- ginning to the end of the observed interval. The second line creates a new DataFrame with all of the data from daily, but including rows for all dates, filled with nan.
Can you write Python code for it?

#Now we can plot the rolling mean like this:
#The window size is 30, so each value in roll_mean is the mean of 30 values from reindexed.ppg.
roll_mean = reindexed.ppg.rolling(30).mean()
plt.plot(roll_mean.index, roll_mean)
plt.xticks(rotation=30)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The first two values are nan; the next value is the mean of the first three elements, 0, 1, and 2. The next value is the mean of 1, 2, and 3. And so on.
Before we can apply rolling mean to the cannabis data, we have to deal with missing values. There are a few days in the observed interval with no reported transactions for one or more quality categories, and a period in 2013 when data collection was not active.
In the DataFrames we have used so far, these dates are absent; the index skips days with no data. For the analysis that follows, we need to represent this missing data explicitly. We can do that by “reindexing” the DataFrame:
Can you write Python code for it?

dates = pd.date_range(dailies["high"].index.min(), dailies["high"].index.max())
reindexed = dailies["high"].reindex(dates)

#dailies["high"].index

reindexed.shape


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The p-value for this data is 0.13, which means that if the die is fair we expect to see the observed total deviation, or more, about 13% of the time. So the apparent effect is not statistically significant.
Chi-squared tests
In the previous section we used total deviation as the test statistic. But for testing proportions it is more common to use the chi-squared statistic:
                χ2 = 􏰄 SUMMATION (Oi − Ei)^2 / Ei

Where Oi are the observed frequencies and Ei are the expected frequencies. Here’s the Python code:
Can you write Python code for it?

def TestStatistic(self, data):
        observed = data
        n = sum(observed)
        expected = np.ones(6) * n / 6
        test_stat = sum((observed - expected)**2 / expected)
        return test_stat


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The parameter, transactions, is a DataFrame that contains columns date
and ppg. We select these two columns, then group by date.
The result, grouped, is a map from each date to a DataFrame that contains prices reported on that date. aggregate is a GroupBy method that iterates through the groups and applies a function to each column of the group; in this case there is only one column, ppg. So the result of aggregate is a DataFrame with one row for each date and one column, ppg.
Dates in these DataFrames are stored as NumPy datetime64 objects, which are represented as 64-bit integers in nanoseconds. For some of the analyses coming up, it will be convenient to work with time in more human-friendly units, like years. So GroupByDay adds a column named date by copying the index, then adds years, which contains the number of years since the first transaction as a floating-point number.
The resulting DataFrame has columns ppg, date, and years.
Can you write Python code for it?

dailies = GroupByQualityAndDay(mj_clean)

plt.figure(figsize=(6,8))
plt.subplot(3, 1, 1)
for i, (k, v) in enumerate(dailies.items()):
    plt.subplot(3, 1, i+1)
    plt.title(k)
    plt.scatter(dailies[k].index, dailies[k].ppg, s=10)
    plt.xticks(rotation=30)
    plt.ylabel("Price per gram")
    plt.xlabel("Months")
    plt.tight_layout()


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The result is about 0.059, which means that if the coin is fair, we expect to see a difference as big as 30 about 5.9% of the time.
Interpreting the Results
How should we interpret this result? By convention, 5% is the threshold of statistical significance. If the p-value is less than 5%, the effect is considered significant; otherwise it is not.
But the choice of 5% is arbitrary, and (as we will see later) the p-value depends on the choice of the test statistics and the model of the null hypothesis. So p-values should not be considered precise measurements.
I recommend interpreting p-values according to their order of magnitude: if the p-value is less than 1%, the effect is unlikely to be due to chance; if it is greater than 10%, the effect can plausibly be explained by chance. P-values between 1% and 10% should be considered borderline. So in this example I conclude that the data do not provide strong evidence that the coin is biased or not
DiffMeansPermute
Testing a difference in means
One of the most common effects to test is a difference in mean between two groups. In the NSFG data, we saw that the mean age for purchasing customer is slightly longer, and the mean estimated_salary of purchasing customer is more than other. Now we will see if those effects are statistically significant.
For these examples, the null hypothesis is that the distributions for the two groups are the same. One way to model the null hypothesis is by permutation; that is, we can take values for purchasing customer and others and shuffle them, treating the two groups as one big group:
Can you write Python code for it?

def TestStatistic(data):
    group1, group2 = data
    test_stat = abs(group1.mean() - group2.mean())
    return test_stat

def MakeModel(data):
    group1, group2 = data
    n, m = len(group1), len(group2)
    pool = np.hstack((group1, group2))
    #print(pool.shape)
    return pool, n

def RunModel(pool, n):
    np.random.shuffle(pool)
    data = pool[:n], pool[n:]
    return data

def sample_generator(data):
    pool, n = MakeModel(data)
    return RunModel(pool, n)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
The result pvalue is about 0.0, which means that we expect to see a difference as big as the observed effect about 0% of the time. So this effect is statistically significant.
If we run the same analysis with estimated salary, the computed p-value is 0; after 1000 attempts, the simulation never yields an effect as big as the observed difference, 18564.80. So we would report p < 0.001, and conclude that the difference in estimated salary is statistically significant.
Other statistics Test
Choosing the best test statistic depends on what question you are trying to address. For example, if the relevant question is whether age are different for purchasing customer, then it makes sense to test the absolute difference in means, as we did in the previous section.
If we had some reason to think that purchasing customer are likely to be older, then we would not take the absolute value of the difference; instead we would use this test statistic:
DiffMeansOneSided
Can you write Python code for it?

def TestStatistic(data):
    group1, group2 = data
    test_stat = group1.mean() - group2.mean()
    return test_stat

def MakeModel(data):
    group1, group2 = data
    n, m = len(group1), len(group2)
    pool = np.hstack((group1, group2))
    #print(pool.shape)
    return pool, n

def RunModel(pool, n):
    np.random.shuffle(pool)
    data = pool[:n], pool[n:]
    return data


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
There were 171 people in M5059, so I would have to come in between 17th and 18th place to have the same percentile rank. The finishing time of the 17th runner in M5059 was 46:05, so that’s the time I will have to beat to maintain my percentile rank.
Modeling Distributions
Exponential Distribution
The CDF of the exponential distribution is
                        CDF(x) = 1 − e^(−λx)

The parameter, λ, determines the shape of the distribution.In the real world, exponential distributions come up when we look at a series of events and measure the times between events, called interarrival times. If the events are equally likely to occur at any time, the distribution of interarrival times tends to look like an exponential distribution.
Can you write Python code for it?

babyboom = pd.read_csv('babyboom.dat',sep="    ", header=None)

babyboom.columns = ["time", "gender", "weight", "minutes"]

diffs = list(babyboom.minutes.diff())

e_cdf = []
l = 0.5
def exponential_distribution(x):
    e_cdf.append(1 -  math.exp(-1* l * x))


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
This is a one-sided test because the hypothesis is that the standard deviation for customer purchasing is high, not just different. The p-value is 0.23, which is not statistically significant.
Can you write Python code for it?

data = purchased_customer.Age.values, no_purchase.Age.values
ht = sample_generator(data)
actual_diff =  TestStatistic(data)

def calculate_pvalue(data, iters=1000):
    test_stats = [TestStatistic(sample_generator(data))
                           for _ in range(iters)]
    count = sum(1 for x in test_stats if x >= actual_diff)
    return sorted(test_stats),count / iters

actual_diff

test_stats, pval =  calculate_pvalue(ht)

cdfs = [EvalCdf(test_stats,ts) for ts in test_stats]
plt.step(sorted(test_stats),cdfs)
plt.xlabel("test statistics")
plt.ylabel("CDF")

pval


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
To select a row by label, you can use the loc attribute, which returns a Series
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Under No purchase curve(Blue), the curve remains flat after 90K with minor bilps after that. But under purchase curve(orange), the curve keeps the steps increasing even after 90K, which suggest people with more salary have more purchasing power. And all above 50%le have 90K or more.
Can you write Python code for it?

step_plot(sorted(no_purchase["EstimatedSalary"]), no_purchase_prob, "Estimated Salary")
step_plot(sorted(purchased_customer["EstimatedSalary"]), purchase_prob, "Estimated Salary")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Valuable Functions in Pandas
Can you write Python code for it?

### Social Network Ads
social_network = pd.read_csv("Social_Network_Ads.csv")

social_network.head()

social_network.shape


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
Var(res) is the MSE of your guesses using the model, Var(ys) is the MSE without it. So their ratio is the fraction of MSE that remains if you use the model, and R2 is the fraction of MSE the model eliminates.
There is a simple relationship between the coefficient of determination and Pearson’s coefficient of correlation: R2 = ρ2. For example, if ρ is 0.8 or -0.8, R2 = 0.64.
Although ρ and R2 are often used to quantify the strength of a relationship, they are not easy to interpret in terms of predictive power. In my opinion, Std(res) is the best representation of the quality of prediction, especially if it is presented in relation to Std(ys).
For example, when people talk about the validity of the SAT (a standardized test used for college admission in the U.S.) they often talk about correlations between SAT scores and other measures of intelligence.
According to one study, there is a Pearson correlation of ρ = 0.72 between total SAT scores and IQ scores, which sounds like a strong correlation. But R2 = ρ2 = 0.52, so SAT scores account for only 52% of variance in IQ.
Can you write Python code for it?

#IQ scores are normalized with Std(ys) = 15, so
var_ys = 15**2
rho = 0.72
r2 = rho**2
var_res = (1 - r2) * var_ys
std_res = math.sqrt(var_res)
print(std_res)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
When I run this experiment with λ = 2, the RMSE of L is 1.1. For the median-based estimator Lm, RMSE is 2.2. We can’t tell from this experiment whether L minimizes MSE, but at least it seems better than Lm.
Sadly, it seems that both estimators are biased. For L the mean error is 0.39; for Lm it is 0.54. And neither converges to 0 as m increases. It turns out that x ̄ is an unbiased estimator of the mean of the distribution, 1/λ, but L is not an unbiased estimator of λ. The values changes with each call to the function.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
With lag=0, acf computes the correlation of the series with itself, which is always 1.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
With lag=1, the serial correlations for the de-trended data are -0.022 for high quality, -0.015 for medium, and 0.036 for low. These values are small, indicating that there is little or no one-day serial correlation in this series.
Can you write Python code for it?

ewma = reindexed.ppg.ewm(30).mean()
resid = reindexed.ppg - ewma
corr = SerialCorr(resid, 7)
print(corr)

ewma = reindexed.ppg.ewm(30).mean()
resid = reindexed.ppg - ewma
corr = SerialCorr(resid, 30)
print(corr)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
at this point we can tentatively conclude that there are no substantial seasonal patterns in these series, at least not with these lags.
Autocorrelation
If you think a series might have some serial correlation, but you don’t know which lags to test, you can test them all! The autocorrelation function is a function that maps from lag to the serial correlation with the given lag. “Autocorrelation” is another name for serial correlation, used more often when the lag is not 1.
StatsModels, which we used for linear regression, also provides functions for time series analysis, including acf, which computes the autocorrelation function:
Can you write Python code for it?

import statsmodels.tsa.stattools as smtsa
acf = smtsa.acf(resid, nlags=120, unbiased=True)

acf[0], acf[1], acf[45], acf[60]


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
data is a pair of sequences, one for each group.
The test statistic is the absolute difference in the means.
MakeModel records the sizes of the groups, n and m, and combines the groups into one NumPy array, pool.
RunModel simulates the null hypothesis by shuffling the pooled values and splitting them into two groups with sizes n and m. As always, the return value from RunModel has the same format as the observed data.
Can you write Python code for it?

purchased_customer.dropna(inplace=True)
no_purchase.dropna(inplace=True)

data = purchased_customer.Age.values, no_purchase.Age.values
ht = sample_generator(data)
actual_diff =  TestStatistic(data)

def calculate_pvalue(data, iters=1000):
    test_stats = [TestStatistic(sample_generator(data))
                           for _ in range(iters)]
    count = sum(1 for x in test_stats if x >= actual_diff)
    return sorted(test_stats),count / iters

test_stats, pval =  calculate_pvalue(ht)

cdfs = [EvalCdf(test_stats,ts) for ts in test_stats]
plt.step(sorted(test_stats),cdfs)
plt.xlabel("test statistics")
plt.ylabel("CDF")


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
data is a pair of sequences. TestStatistic computes the absolute value of Pearson’s correlation. RunModel shuffles the xs and returns simulated data.
Can you write Python code for it?

data = social_network.Age.values, social_network.EstimatedSalary.values
actual_diff = TestStatistic(data)

def calculate_pvalue(data, iters=1000):
    test_stats = [TestStatistic(RunModel(data))
                           for _ in range(iters)]
    count = sum(1 for x in test_stats if x >= actual_diff)
    return sorted(test_stats),count / iters

test_stats, pval =  calculate_pvalue(data)

pval


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
describe() - It gives the basic statistical metrics like mean etc.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
estimates is a list of estimates; actual is the actual value being estimated. In practice, of course, we don’t know actual; if we did, we wouldn’t have to estimate it. The purpose of this experiment is to compare the performance of the two estimators.
When I ran this code, the RMSE of the sample mean was 0.38, which means that if we use x ̄ to estimate the mean of this distribution, based on a sample with n = 7, we should expect to be off by 0.38 on average. Using the median to estimate the mean yields RMSE 0.45, which confirms that x ̄ yields lower RMSE, at least for this example.
Minimizing MSE is a nice property, but it’s not always the best strategy. For example, suppose we are estimating the distribution of wind speeds at a building site. If the estimate is too high, we might overbuild the structure, increasing its cost. But if it’s too low, the building might collapse. Because cost as a function of error is not symmetric, minimizing MSE is not the best strategy.
As another example, suppose I roll three six-sided dice and ask you to predict the total. If you get it exactly right, you get a prize; otherwise you get nothing. In this case the value that minimizes MSE is 10.5, but that would be a bad guess, because the total of three dice is never 10.5. For this game, you want an estimator that has the highest chance of being right, which is a maximum likelihood estimator (MLE). If you pick 10 or 11, your chance of winning is 1 in 8, and that’s the best you can do.
Estimate Variance
I’m thinking of a distribution. It’s a normal distribution, and here’s a (familiar) sample:
[-0.441, 1.774, -0.101, -1.138, 2.975, -2.138]
What do you think is the variance, σ2, of my distribution? Again, the obvious choice is to use the sample variance, S^2, as an estimator.
                S^2 = 1/n SUMMATION􏰄 ( x i − x ̄ )^2

For large samples, S^2 is an adequate estimator, but for small samples it tends to be too low. Because of this unfortunate property, it is called a biased estimator. An estimator is unbiased if the...
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
groupby is a DataFrame method that returns a GroupBy object, groups; used in a for loop, it iterates the names of the groups and the DataFrames that represent them. Since the values of quality are low, medium, and high, we get three groups with those names.
The loop iterates through the groups and calls GroupByDay, which computes the daily average price and returns a new DataFrame:
Can you write Python code for it?

def GroupByDay(transactions, func=np.mean):
    grouped = transactions[['date', 'ppg']].groupby('date')
    daily = grouped.aggregate(func)
    daily['date'] = daily.index
    start = daily.date[0]
    one_year = np.timedelta64(1, 'Y')
    daily['years'] = (daily.date - start) / one_year
    return daily


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
isnull - It finds the number of values in each column with value as null.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
mu and sigma are the hypothetical values of the parameters. n is the sample size, the number of gorillas we measured. m is the number of times we run the simulation.
Can you write Python code for it?

means = SimulateSample()

cdfs = [EvalCdf(means,m) for m in means]
plt.step(sorted(means),cdfs)

ci_5 = Percentile(means, 5)
ci_95 = Percentile(means, 95)
print(ci_5, ci_95)
stderr = RMSE(means, 90)

stderr


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
pvalues is a Series that maps from variable names to the associated p-values, so we can check whether the estimated slope is statistically significant:
The p-value associated with agepreg is 1.14e-20, which is less than 0.001, as
expected.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
results.rsquared contains R2, which is 0.0047. results also provides f_pvalue, which is the p-value associated with the model as a whole, similar to testing whether R2 is statistically significant.
And results provides resid, a sequence of residuals, and fittedvalues, a sequence of fitted values corresponding to agepreg.
The results object provides summary(), which represents the results in a readable format.
print(results.summary())

Betting Pool https://en.wikipedia.org/wiki/Betting_pool
Theory
Linear regression can be generalized to handle other kinds of dependent vari- ables. If the dependent variable is boolean, the generalized model is called logistic regression. If the dependent variable is an integer count, it’s called Poisson regression.
Suppose a friend of yours is pregnant and you want to predict whether the baby is a boy or a girl. You could use data from the NSFG to find factors that affect the “sex ratio”, which is conventionally defined to be the probability of having a boy.
If you encode the dependent variable numerically, for example 0 for a girl and 1 for a boy, you could apply ordinary least squares, but there would be problems. The linear model might be something like this:
y = β0 + β1x1 + β2x2 + ε

Where y is the dependent variable, and x1 and x2 are explanatory variables.
Then we could find the parameters that minimize the residuals.
The problem with this approach is that it produces predictions that are hard to interpret. Given estimated parameters and values for x1 and x2, the model might predict y = 0.5, but the only meaningful values of y are 0 and 1.
It is tempting to interpret a result like that as a probability; for example, we might say that a respondent with particular values of x1 and x2 has a 50% chance of having a boy. But it is also possible for this model to predict y = 1.1 or y = −0.1, and those are not valid probabilities.
Logistic regression avoids this problem by expressing predictions in terms of odds rather than probabilities. If you are not familiar with odds, “odds...
Can you write Python code for it?

y = np.array([0, 1, 0, 1])
x1 = np.array([0, 0, 0, 1])
x2 = np.array([0, 1, 1, 1])

#And we start with the initial guesses
β0 = -1.5,
β1 = 2.8,
β2 = 1.1

beta = [-1.5, 2.8, 1.1]

#Then for each row we can compute log_o:
log_o = beta[0] + beta[1] * x1 + beta[2] * x2

#convert from log odds to probabilities:
o = np.exp(log_o)

p = o / (o+1)


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
sort_index() - It sorts the Series by index, so the values appear in order.
Can you write Python code for it?


I am working on the file "Statistics/Pandas and ThinkStat.ipynb".
The first task is:
statsmodels provides two interfaces (APIs); the “formula” API uses strings to identify the dependent and explanatory variables. It uses a syntax called patsy; in this example, the ~ operator separates the dependent variable on the left from the explanatory variables on the right.
smf.ols takes the formula string and the DataFrame, regression_data, and returns an OLS object that represents the model. The name ols stands for “ordinary least squares.”
The fit method fits the model to the data and returns a RegressionResults object that contains the results.
The results are also available as attributes. params is a Series that maps from variable names to their parameters, so we can get the intercept and slope like this:
Can you write Python code for it?

inter = results.params['Intercept']
slope = results.params['YearsExperience']
slope_pvalue = results.pvalues['YearsExperience']
print(slope_pvalue)


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
Choosing $K$: The Gap Statistic
There is one last method worth covering for picking $K$, the so-called Gap statistic. The computation for the gap statistic builds on the sum-of-squares established in the Elbow method discussion, and compares it to the sum-of-squares of a "null distribution," that is, a random set of points with no clustering. The estimate for the optimal number of clusters $K$ is the value for which $\log{SS}$ falls the farthest below that of the reference distribution:
$$G_k = E_n^*{\log SS_k} - \log SS_k$$
In other words a good clustering yields a much larger difference between the reference distribution and the clustered data. The reference distribution is a Monte Carlo (randomization) procedure that constructs $B$ random distributions of points within the bounding box (limits) of the original data and then applies K-means to this synthetic distribution of data points.. $E_n^*{\log SS_k}$ is just the average $SS_k$ over all $B$ replicates. We then compute the standard deviation $\sigma_{SS}$ of the values of $SS_k$ computed from the $B$ replicates of the reference distribution and compute
$$s_k = \sqrt{1+1/B}\sigma_{SS}$$
Finally, we choose $K=k$ such that $G_k \geq G_{k+1} - s_{k+1}$.
Aside: Choosing $K$ when we Have Labels
Unsupervised learning expects that we do not have the labels. In some situations, we may wish to cluster data that is labeled. Computing the optimal number of clusters is much easier if we have access to labels. There are several methods available. We will not go into the math or details since it is rare to have access to the labels, but we provide the names and references of these measures.

Adjusted Rand Index
Mutual Information
V-Measure
Fowlkes–Mallows index

See this article for more information about these metrics.
Visualizing Clusters using PCA
How do we visualize clusters? If we only had two features, we could likely plot the data as is. But we have 100 data points each containing 32 features (dimensions). Principal Component Analysis (PCA) will help us reduce...
Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
Choosing K: The Silhouette Method
There exists another method that measures how well each datapoint $x_i$ "fits" its assigned cluster and also how poorly it fits into other clusters. This is a different way of looking at the same objective. Denote $a_{x_i}$ as the average distance from $x_i$ to all other points within its own cluster $k$. The lower the value, the better. On the other hand $b_{x_i}$ is the minimum average distance from $x_i$ to points in a different cluster, minimized over clusters. That is, compute separately for each cluster the average distance from $x_i$ to the points within that cluster, and then take the minimum. The silhouette $s(x_i)$ is defined as
$$s(x_i) = \frac{b_{x_i} - a_{x_i}}{\max{\left( a_{x_i}, b_{x_i}\right)}}$$
The silhouette score is computed on every datapoint in every cluster. The silhouette score ranges from -1 (a poor clustering) to +1 (a very dense clustering) with 0 denoting the situation where clusters overlap. Some criteria for the silhouette coefficient is provided in the table below.

| Range       | Interpretation                                |
|-------------|-----------------------------------------------|
| 0.71 - 1.0  | A strong structure has been found.            |
| 0.51 - 0.7  | A reasonable structure has been found.        |
| 0.26 - 0.5  | The structure is weak and could be artificial.|
| < 0.25      | No substantial structure has been found.      |

Source: http://www.stat.berkeley.edu/~spector/s133/Clus.html
Fortunately, scikit-learn provides a function to compute this for us (phew!) called sklearn.metrics.silhouette_score. Take a look at this article on picking $K$ in scikit-learn, as it will help you in the next exercise set.

Checkup Exercise Set III

Exercise: Using the documentation for the `silhouette_score` function above, construct a series of silhouette plots like the ones in the article linked above.

Exercise: Compute the average silhouette score for each...
Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
K-Means Clustering
Recall that in K-Means Clustering we want to maximize the distance between centroids and minimize the distance between data points and the respective centroid for the cluster they are in. True evaluation for unsupervised learning would require labeled data; however, we can use a variety of intuitive metrics to try to pick the number of clusters K. We will introduce two methods: the Elbow method, the Silhouette method and the gap statistic.
Choosing K: The Elbow Sum-of-Squares Method
The first method looks at the sum-of-squares error in each cluster against $K$. We compute the distance from each data point to the center of the cluster (centroid) to which the data point was assigned.
$$SS = \sum_k \sum_{x_i \in C_k} \sum_{x_j \in C_k} \left( x_i - x_j \right)^2 = \sum_k \sum_{x_i \in C_k} \left( x_i - \mu_k \right)^2$$
where $x_i$ is a point, $C_k$ represents cluster $k$ and $\mu_k$ is the centroid for cluster $k$. We can plot SS vs. $K$ and choose the elbow point in the plot as the best value for $K$. The elbow point is the point at which the plot starts descending much more slowly.

Checkup Exercise Set II

Exercise:

 What values of $SS$ do you believe represent better clusterings? Why?

 Create a numpy matrix `x_cols` with only the columns representing the offers (i.e. the 0/1 colums)

 Write code that applies the [`KMeans`](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html) clustering method from scikit-learn to this matrix.

 Construct a plot showing $SS$ for each $K$ and pick $K$ using this plot. For simplicity, test $2 \le K \le 10$.

 Make a bar chart showing the number of points in each cluster for k-means under the best $K$.

 What challenges did you experience using the Elbow method to pick $K$?

Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
Other Clustering Algorithms
k-means is only one of a ton of clustering algorithms. Below is a brief description of several clustering algorithms, and the table provides references to the other clustering algorithms in scikit-learn.

Affinity Propagation does not require the number of clusters $K$ to be known in advance! AP uses a "message passing" paradigm to cluster points based on their similarity.

Spectral Clustering uses the eigenvalues of a similarity matrix to reduce the dimensionality of the data before clustering in a lower dimensional space. This is tangentially similar to what we did to visualize k-means clusters using PCA. The number of clusters must be known a priori.

Ward's Method applies to hierarchical clustering. Hierarchical clustering algorithms take a set of data and successively divide the observations into more and more clusters at each layer of the hierarchy. Ward's method is used to determine when two clusters in the hierarchy should be combined into one. It is basically an extension of hierarchical clustering. Hierarchical clustering is divisive, that is, all observations are part of the same cluster at first, and at each successive iteration, the clusters are made smaller and smaller. With hierarchical clustering, a hierarchy is constructed, and there is not really the concept of "number of clusters." The number of clusters simply determines how low or how high in the hierarchy we reference and can be determined empirically or by looking at the dendogram.

Agglomerative Clustering is similar to hierarchical clustering but but is not divisive, it is agglomerative. That is, every observation is placed into its own cluster and at each iteration or level or the hierarchy, observations are merged into fewer and fewer clusters until convergence. Similar to hierarchical clustering, the constructed hierarchy contains all possible numbers of clusters and it is up to the analyst to pick the number by reviewing statistics or the dendogram.

DBSCAN is based on point density rather than...
Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
We see that the first dataset contains information about each offer such as the month it is in effect and several attributes about the wine that the offer refers to: the variety, minimum quantity, discount, country of origin and whether or not it is past peak. The second dataset in the second worksheet contains transactional data -- which offer each customer responded to.
Can you write Python code for it?

df_transactions = pd.read_excel("./WineKMC.xlsx", sheetname=1)
df_transactions.columns = ["customer_name", "offer_id"]
df_transactions['n'] = 1
df_transactions.head()


I am working on the file "Statistics_Exercises/Mini_Project_Clustering.ipynb".
The first task is:
What we've done is we've taken those columns of 0/1 indicator variables, and we've transformed them into a 2-D dataset. We took one column and arbitrarily called it x and then called the other y. Now we can throw each point into a scatterplot. We color coded each point based on it's cluster so it's easier to see them.

Exercise Set V

As we saw earlier, PCA has a lot of other uses. Since we wanted to visualize our data in 2 dimensions, restricted the number of dimensions to 2 in PCA. But what is the true optimal number of dimensions?

Exercise: Using a new PCA object shown in the next cell, plot the `explained_variance_` field and look for the elbow point, the point where the curve's rate of descent seems to slow sharply. This value is one possible value for the optimal number of dimensions. What is it?

Can you write Python code for it?

#your turn
# Initialize a new PCA model with a default number of components.
import sklearn.decomposition
pca = sklearn.decomposition.PCA()
pca.fit(X)

# Do the rest on your own :)


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
A Motivating Example Using sklearn: Heights and Weights
We'll use a dataset of heights and weights of males and females to hone our understanding of classifiers. We load the data into a dataframe and plot it.
Can you write Python code for it?

dflog = pd.read_csv("data/01_heights_weights_genders.csv")
dflog.head()


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
A Walkthrough of the Math Behind Logistic Regression
Setting up Some Demo Code
Let's first set some code up for classification that we will need for further discussion on the math. We first set up a function cv_optimize which takes a classifier clf, a grid of hyperparameters (such as a complexity parameter or regularization parameter) implemented as a dictionary parameters, a training set (as a samples x features array) Xtrain, and a set of labels ytrain. The code takes the traning set, splits it into n_folds parts, sets up n_folds folds, and carries out a cross-validation by splitting the training set into a training and validation section for each foldfor us. It prints the best value of the parameters, and retuens the best classifier to us.
Can you write Python code for it?

def cv_optimize(clf, parameters, Xtrain, ytrain, n_folds=5):
    gs = sklearn.model_selection.GridSearchCV(clf, param_grid=parameters, cv=n_folds)
    gs.fit(Xtrain, ytrain)
    print("BEST PARAMS", gs.best_params_)
    best = gs.best_estimator_
    return best


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Black Box Grid Search in sklearn
Scikit-learn, as with many other Python packages, provides utilities to perform common operations so you do not have to do it manually. It is important to understand the mechanics of each operation, but at a certain point, you will want to use the utility instead to save time...

Checkup Exercise Set IV

Exercise: Use scikit-learn's [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html) tool to perform cross validation and grid search.

* Instead of writing your own loops above to iterate over the model parameters, can you use GridSearchCV to find the best model over the training set?
* Does it give you the same best value of `C`?
* How does this model you've obtained perform on the test set?
Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Discriminative vs Generative Classifier
Logistic regression is what is known as a discriminative classifier as we learn a soft boundary between/among classes. Another paradigm is the generative classifier where we learn the distribution of each class. For more examples of generative classifiers, look here.
Let us plot the probabilities obtained from predict_proba, overlayed on the samples with their true labels:
Can you write Python code for it?

plt.figure()
ax = plt.gca()
points_plot_prob(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, psize=20, alpha=0.1);


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
In the figure here showing the results of the logistic regression, we plot the actual labels of both the training(circles) and test(squares) samples. The 0's (females) are plotted in red, the 1's (males) in blue. We also show the classification boundary, a line (to the resolution of a grid square). Every sample on the red background side of the line will be classified female, and every sample on the blue side, male. Notice that most of the samples are classified well, but there are misclassified people on both sides, as evidenced by leakage of dots or squares of one color ontothe side of the other color. Both test and traing accuracy are about 92%.
The Probabilistic Interpretaion
Remember we said earlier that if $h > 0.5$ we ought to identify the sample with $y=1$? One way of thinking about this is to identify $h(\v{w}\cdot\v{x})$ with the probability that the sample is a '1' ($y=1$). Then we have the intuitive notion that lets identify a sample as 1 if we find that the probabilty of being a '1' is $\ge 0.5$.
So suppose we say then that the probability of $y=1$ for a given $\v{x}$ is given by $h(\v{w}\cdot\v{x})$?
Then, the conditional probabilities of $y=1$ or $y=0$ given a particular sample's features $\v{x}$ are:
\begin{eqnarray}
P(y=1 | \v{x}) &=& h(\v{w}\cdot\v{x}) \
P(y=0 | \v{x}) &=& 1 - h(\v{w}\cdot\v{x}).
\end{eqnarray}
These two can be written together as
$$P(y|\v{x}, \v{w}) = h(\v{w}\cdot\v{x})^y \left(1 - h(\v{w}\cdot\v{x}) \right)^{(1-y)} $$
Then multiplying over the samples we get the probability of the training $y$ given $\v{w}$ and the $\v{x}$:
$$P(y|\v{x},\v{w}) = P({y_i} | {\v{x}i}, \v{w}) = \prod{y_i \in \cal{D}} P(y_i|\v{x_i}, \v{w}) = \prod_{y_i \in \cal{D}} h(\v{w}\cdot\v{x_i})^{y_i} \left(1 - h(\v{w}\cdot\v{x_i}) \right)^{(1-y_i)}$$
Why use probabilities? Earlier, we talked about how the regression function $f(x)$ never gives us the $y$ exactly, because of noise. This hold for classification too. Even with identical features, a different sample may be classified differently.
We said...
Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Logistic Regression: The Math
We could approach classification as linear regression, there the class, 0 or 1, is the target variable $y$. But this ignores the fact that our output $y$ is discrete valued, and futhermore, the $y$ predicted by linear regression will in general take on values less than 0 and greater than 1. Additionally, the residuals from the linear regression model will not be normally distributed. This violation means we should not use linear regression.
But what if we could change the form of our hypotheses $h(x)$ instead?
The idea behind logistic regression is very simple. We want to draw a line in feature space that divides the '1' samples from the '0' samples, just like in the diagram above. In other words, we wish to find the "regression" line which divides the samples. Now, a line has the form $w_1 x_1 + w_2 x_2 + w_0 = 0$ in 2-dimensions. On one side of this line we have
$$w_1 x_1 + w_2 x_2 + w_0 \ge 0,$$
and on the other side we have
$$w_1 x_1 + w_2 x_2 + w_0 < 0.$$
Our classification rule then becomes:
\begin{eqnarray}
y = 1 &\mbox{if}& \v{w}\cdot\v{x} \ge 0\
y = 0 &\mbox{if}& \v{w}\cdot\v{x} < 0
\end{eqnarray}
where $\v{x}$ is the vector ${1,x_1, x_2,...,x_n}$ where we have also generalized to more than 2 features.
What hypotheses $h$ can we use to achieve this? One way to do so is to use the sigmoid function:
$$h(z) = \frac{1}{1 + e^{-z}}.$$
Notice that at $z=0$ this function has the value 0.5. If $z > 0$, $h > 0.5$ and as $z \to \infty$, $h \to 1$. If $z < 0$, $h < 0.5$ and as $z \to -\infty$, $h \to 0$. As long as we identify any value of $y > 0.5$ as 1, and any $y < 0.5$ as 0, we can achieve what we wished above.
This function is plotted below:
Can you write Python code for it?

h = lambda z: 1. / (1 + np.exp(-z))
zs=np.arange(-5, 5, 0.1)
plt.plot(zs, h(zs), alpha=0.5);


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Remember that the form of data we will use always is

with the "response" or "label" $y$ as a plain array of 0s and 1s for binary classification. Sometimes we will also see -1 and +1 instead. There are also multiclass classifiers that can assign an observation to one of $K > 2$ classes and the labe may then be an integer, but we will not be discussing those here.
y = [1,1,0,0,0,1,0,1,0....].

Checkup Exercise Set I

 Exercise: Create a scatter plot of Weight vs. Height

 Exercise: Color the points differently by Gender

Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Logistic_Regression.ipynb".
The first task is:
So we then come up with our rule by identifying:
$$z = \v{w}\cdot\v{x}.$$
Then $h(\v{w}\cdot\v{x}) \ge 0.5$ if $\v{w}\cdot\v{x} \ge 0$ and $h(\v{w}\cdot\v{x}) \lt 0.5$ if $\v{w}\cdot\v{x} \lt 0$, and:
\begin{eqnarray}
y = 1 &if& h(\v{w}\cdot\v{x}) \ge 0.5\
y = 0 &if& h(\v{w}\cdot\v{x}) \lt 0.5.
\end{eqnarray}
We will show soon that this identification can be achieved by minimizing a loss in the ERM framework called the log loss :
$$ R_{\cal{D}}(\v{w}) = - \sum_{y_i \in \cal{D}} \left ( y_i \log(h(\v{w}\cdot\v{x})) + ( 1 - y_i) \log(1 - h(\v{w}\cdot\v{x})) \right )$$
We will also add a regularization term:
$$ R_{\cal{D}}(\v{w}) = - \sum_{y_i \in \cal{D}} \left ( y_i \log(h(\v{w}\cdot\v{x})) + ( 1 - y_i) \log(1 - h(\v{w}\cdot\v{x})) \right ) + \frac{1}{C} \v{w}\cdot\v{w},$$
where $C$ is the regularization strength (equivalent to $1/\alpha$ from the Ridge case), and smaller values of $C$ mean stronger regularization. As before, the regularization tries to prevent features from having terribly high weights, thus implementing a form of feature selection.
How did we come up with this loss? We'll come back to that, but let us see how logistic regression works out.
Can you write Python code for it?

dflog.head()

clf_l, Xtrain_l, ytrain_l, Xtest_l, ytest_l  = do_classify(LogisticRegression(),
                                                           {"C": [0.01, 0.1, 1, 10, 100]},
                                                           dflog, ['Weight', 'Height'], 'Gender','Male')

plt.figure()
ax=plt.gca()
points_plot(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, alpha=0.2);


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set VI

Exercise: Why does this method work? What does the probability for each row in the identity matrix represent

The above exercise is an example of feature selection. There are many other feature selection methods. A list of feature selection methods available in sklearn is here. The most common feature selection technique for text mining is the chi-squared $\left( \chi^2 \right)$ method.
Prediction Errors
We can see mis-predictions as well.
Can you write Python code for it?

x, y = make_xy(critics, vectorizer)

prob = clf.predict_proba(x)[:, 0]
predict = clf.predict(x)

bad_rotten = np.argsort(prob[y == 0])[:5]
bad_fresh = np.argsort(prob[y == 1])[-5:]

print("Mis-predicted Rotten quotes")
print('---------------------------')
for row in bad_rotten:
    print(critics[y == 0].quote.iloc[row])
    print("")

print("Mis-predicted Fresh quotes")
print('--------------------------')
for row in bad_fresh:
    print(critics[y == 1].quote.iloc[row])
    print("")


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set VII: Predicting the Freshness for a New Review

Exercise:

 Using your best trained classifier, predict the freshness of the following sentence: *'This movie is not remarkable, touching, or superb in any way'*

 Is the result what you'd expect? Why (not)?

Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:

Exercise Set VIII: Enrichment

There are several additional things we could try. Try some of these as exercises:

 Build a Naive Bayes model where the features are n-grams instead of words. N-grams are phrases containing n words next to each other: a bigram contains 2 words, a trigram contains 3 words, and 6-gram contains 6 words. This is useful because "not good" and "so good" mean very different things. On the other hand, as n increases, the model does not scale well since the feature set becomes more sparse.

 Try a model besides Naive Bayes, one that would allow for interactions between words -- for example, a Random Forest classifier.

 Try adding supplemental features -- information about genre, director, cast, etc.

 Use word2vec or [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) to group words into topics and use those topics for prediction.

 Use TF-IDF weighting instead of word counts.

Exercise: Try a few of these ideas to improve the model (or any other ideas of your own). Implement here and report on the result.

Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
Aside: TF-IDF Weighting for Term Importance
TF-IDF stands for
Term-Frequency X Inverse Document Frequency.
In the standard CountVectorizer model above, we used just the term frequency in a document of words in our vocabulary. In TF-IDF, we weight this term frequency by the inverse of its popularity in all documents. For example, if the word "movie" showed up in all the documents, it would not have much predictive value. It could actually be considered a stopword. By weighing its counts by 1 divided by its overall frequency, we downweight it. We can then use this TF-IDF weighted features as inputs to any classifier. TF-IDF is essentially a measure of term importance, and of how discriminative a word is in a corpus. There are a variety of nuances involved in computing TF-IDF, mainly involving where to add the smoothing term to avoid division by 0, or log of 0 errors. The formula for TF-IDF in scikit-learn differs from that of most textbooks:
$$\mbox{TF-IDF}(t, d) = \mbox{TF}(t, d)\times \mbox{IDF}(t) = n_{td} \log{\left( \frac{\vert D \vert}{\vert d : t \in d \vert} + 1 \right)}$$
where $n_{td}$ is the number of times term $t$ occurs in document $d$, $\vert D \vert$ is the number of documents, and $\vert d : t \in d \vert$ is the number of documents that contain $t$
Can you write Python code for it?

# http://scikit-learn.org/dev/modules/feature_extraction.html#text-feature-extraction
# http://scikit-learn.org/dev/modules/classes.html#text-feature-extraction-ref
from sklearn.feature_extraction.text import TfidfVectorizer
tfidfvectorizer = TfidfVectorizer(min_df=1, stop_words='english')
Xtfidf=tfidfvectorizer.fit_transform(critics.quote)


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
Explore
Can you write Python code for it?

n_reviews = len(critics)
n_movies = critics.rtid.unique().size
n_critics = critics.critic.unique().size

print("Number of reviews: {:d}".format(n_reviews))
print("Number of critics: {:d}".format(n_critics))
print("Number of movies:  {:d}".format(n_movies))

df = critics.copy()
df['fresh'] = df.fresh == 'fresh'
grp = df.groupby('critic')
counts = grp.critic.count()  # number of reviews by each critic
means = grp.fresh.mean()     # average freshness for each critic

means[counts > 100].hist(bins=10, edgecolor='w', lw=1)
plt.xlabel("Average Rating per critic")
plt.ylabel("Number of Critics")
plt.yticks([0, 2, 4, 6, 8, 10]);


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
Picking Hyperparameters for Naive Bayes and Text Maintenance
We need to know what value to use for $\alpha$, and we also need to know which words to include in the vocabulary. As mentioned earlier, some words are obvious stopwords. Other words appear so infrequently that they serve as noise, and other words in addition to stopwords appear so frequently that they may also serve as noise.
First, let's find an appropriate value for min_df for the CountVectorizer. min_df can be either an integer or a float/decimal. If it is an integer, min_df represents the minimum number of documents a word must appear in for it to be included in the vocabulary. If it is a float, it represents the minimum percentage of documents a word must appear in to be included in the vocabulary. From the documentation:

min_df: When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None.

Exercise Set III

Exercise: Construct the cumulative distribution of document frequencies (df). The $x$-axis is a document count $x_i$ and the $y$-axis is the percentage of words that appear less than $x_i$ times. For example, at $x=5$, plot a point representing the percentage or number of words that appear in 5 or fewer documents.

Exercise: Look for the point at which the curve begins climbing steeply. This may be a good value for `min_df`. If we were interested in also picking `max_df`, we would likely pick the value where the curve starts to plateau. What value did you choose?

Can you write Python code for it?


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
The parameter $\alpha$ is chosen to be a small value that simply avoids having zeros in the probability computations. This value can sometimes be chosen arbitrarily with domain expertise, but we will use K-fold cross validation. In K-fold cross-validation, we divide the data into $K$ non-overlapping parts. We train on $K-1$ of the folds and test on the remaining fold. We then iterate, so that each fold serves as the test fold exactly once. The function cv_score performs the K-fold cross-validation algorithm for us, but we need to pass a function that measures the performance of the algorithm on each fold.
Can you write Python code for it?

from sklearn.model_selection import KFold
def cv_score(clf, X, y, scorefunc):
    result = 0.
    nfold = 5
    for train, test in KFold(nfold).split(X): # split data into train/test groups, 5 times
        clf.fit(X[train], y[train]) # fit the classifier, passed is as clf.
        result += scorefunc(clf, X[test], y[test]) # evaluate score function on held-out data
    return result / nfold # average


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
We use the log-likelihood as the score here in scorefunc. The higher the log-likelihood, the better. Indeed, what we do in cv_score above is to implement the cross-validation part of GridSearchCV.
The custom scoring function scorefunc allows us to use different metrics depending on the decision risk we care about (precision, accuracy, profit etc.) directly on the validation set. You will often find people using roc_auc, precision, recall, or F1-score as the scoring function.
Can you write Python code for it?

def log_likelihood(clf, x, y):
    prob = clf.predict_log_proba(x)
    rotten = y == 0
    fresh = ~rotten
    return prob[rotten, 0].sum() + prob[fresh, 1].sum()


I am working on the file "Statistics_Exercises/Mini_Project_Naive_Bayes.ipynb".
The first task is:
We'll cross-validate over the regularization parameter $\alpha$.
Let's set up the train and test masks first, and then we can run the cross-validation procedure.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
_, itest = train_test_split(range(critics.shape[0]), train_size=0.7)
mask = np.zeros(critics.shape[0], dtype=np.bool)
mask[itest] = True


I am working on the file "Statistics_Exercises/sliderule_dsi_inferential_statistics_exercise_3.ipynb".
The first task is:
<div class="span5 alert alert-info">
### Exercise

Include your work on the following **in this notebook and submit to your Github account**.

A. Do you agree with the above analysis and recommendations? Why or why not?

B. Provide support for your arguments and your own recommendations with a statistically sound analysis:

   1. Setup an appropriate hypothesis test.
   2. Compute and report the observed significance value (or p-value).
   3. Report statistical significance for $\alpha$ = .01.
   4. Discuss statistical significance and practical significance. Do they differ here? How does this change your recommendation to the client?
   5. Look at the scatterplot above.
      - What are the advantages and disadvantages of using this plot to convey information?
      - Construct another plot that conveys the same information in a more direct manner.

You can compose in notebook cells using Markdown:
+ In the control panel at the top, choose Cell > Cell Type > Markdown
+ Markdown syntax: http://nestacms.com/docs/creating-content/markdown-cheat-sheet
</div>

Overall, rate of readmissions is trending down with increasing number of discharges

Agree, according to regression trend line shown above

With lower number of discharges, there is a greater incidence of excess rate of readmissions (area shaded red)

Agree

With higher number of discharges, there is a greater incidence of lower rates of readmissions (area shaded green)

Agree
Can you write Python code for it?

rv =relevant_columns
print rv[rv['Number of Discharges'] < 100][['Excess Readmission Ratio']].mean()

print '\nPercent of subset with excess readmission rate > 1: ', len(rv[(rv['Number of Discharges'] < 100) & (rv['Excess Readmission Ratio'] > 1)]) / len(rv[relevant_columns['Number of Discharges'] < 100])

print '\n', rv[rv['Number of Discharges'] > 1000][['Excess Readmission Ratio']].mean()
print '\nPercent of subset with excess readmission rate > 1: ', len(rv[(rv['Number of Discharges'] > 1000) & (rv['Excess Readmission Ratio'] > 1)]) / len(rv[relevant_columns['Number of Discharges'] > 1000])


I am working on the file "Statistics_Exercises/sliderule_dsi_inferential_statistics_exercise_3.ipynb".
The first task is:
In hospitals/facilities with number of discharges < 100, mean excess readmission rate is 1.023 and 63% have excess readmission rate greater than 1

Accurate

In hospitals/facilities with number of discharges > 1000, mean excess readmission rate is 0.978 and 44% have excess readmission rate greater than 1

Correction: mean excess readmission rate is 0.979, and 44.565% have excess readmission rate > 1
Can you write Python code for it?

np.corrcoef(rv['Number of Discharges'], rv['Excess Readmission Ratio'])


I am working on the file "Statistics_Exercises/sliderule_dsi_inferential_statistics_exercise_3.ipynb".
The first task is:
Preliminary Report
Read the following results/report. While you are reading it, think about if the conclusions are correct, incorrect, misleading or unfounded. Think about what you would change or what additional analyses you would perform.
A. Initial observations based on the plot above
+ Overall, rate of readmissions is trending down with increasing number of discharges
+ With lower number of discharges, there is a greater incidence of excess rate of readmissions (area shaded red)
+ With higher number of discharges, there is a greater incidence of lower rates of readmissions (area shaded green)
B. Statistics
+ In hospitals/facilities with number of discharges < 100, mean excess readmission rate is 1.023 and 63% have excess readmission rate greater than 1
+ In hospitals/facilities with number of discharges > 1000, mean excess readmission rate is 0.978 and 44% have excess readmission rate greater than 1
C. Conclusions
+ There is a significant correlation between hospital capacity (number of discharges) and readmission rates.
+ Smaller hospitals/facilities may be lacking necessary resources to ensure quality care and prevent complications that lead to readmissions.
D. Regulatory policy recommendations
+ Hospitals/facilties with small capacity (< 300) should be required to demonstrate upgraded resource allocation for quality care to continue operation.
+ Directives and incentives should be provided for consolidation of hospitals and facilities to have a smaller number of them with higher capacity and number of discharges.
Can you write Python code for it?

# A. Do you agree with the above analysis and recommendations? Why or why not?
import seaborn as sns

relevant_columns = clean_hospital_read_df[['Excess Readmission Ratio', 'Number of Discharges']][81:-3]

sns.regplot(relevant_columns['Number of Discharges'], relevant_columns['Excess Readmission Ratio'])


I am working on the file "Statistics_block1.ipynb".
The first task is:
(without numpy explicitely)
Can you write Python code for it?

#%%timeit
N=10000000
Xinv1=np.empty(N)
for i in xrange(N):
    r=random.random()
    Xinv1[i]=Finv(r,Pmu)


I am working on the file "Statistics_block1.ipynb".
The first task is:
2. Build a Monte Carlo able to generate this PDF
[Oriol]: Nota sobre python y los bucles: Python es muy bonito y simple, pero se le atragantan un poco los bucles. Por suerte, numpy esta hecho en C, cosa que agiliza muchísimo la ejecución. Tanto en el try-teject como en el inverse function hay una comparación. Teneis que descomentar el timeit para ver la comparación de tiempo y comentarlo para poderlo ejecutar correctamente.
Try-Reject p.d.f generation
We will construct our Monte Carlo with the Try-Reject method.
Without using Numpy explicitly:
Can you write Python code for it?

#%%timeit # returns the elapsed time when executing the cell, it executes the cell more than once,
         # thus, it should only be uncommented when desired
# Montecarlo try-reject
N=10000000
# Define fmax
# The goal is to have the lowest fmax possible in order to increase the efficiency of the try-reject, knowing
# the shape of the p.d.f. this can be trivially done, because we know its maximum value must be either at 1 or -1
fmax=max(gamma(-1,Pmu),gamma(1,Pmu))

Xi1=np.empty(N) # Declaring an array instead of making it grow inside a loop speeds a lot the computation time
i=0
while i<N:
    # 1st step of the try reject, choose a point inside the area [a,b]x[0,fmax]
    r=random.random()
    x=-1+2*r
    y=random.random()*fmax
    # 2nd step, check if it is inside the p.d.f. area and save x€[a,b] to Xi1, otherwise, reject it
    if gamma(x,Pmu)>=y:
        Xi1[i]=x
        i+=1


I am working on the file "Statistics_block1.ipynb".
The first task is:
6.4. How does the $\chi^{2}$ computed above change when you change the number of bins?
For this exercise, we can just modify the number of bins in 6.3 to obtain the new results.

As we increase the number of bins, we see that the observed distribution becomes more similar to a Gaussian (e.g. number of bins = 100 for a sample of 10000). Why? The chi-squared distribution depends ONLY on one parameter, k, which corresponds to the number of bins. If the size of the sample is large enough compared to the number of bins, then when we increase k and let it go to infinity, the Central Limit Theorem applies and the chi squared distribution tends to a Gaussian.

If we set the number of bins to a very small number --again, with a sample of 10000-- (e.g. 5), the resulting distribution will not look like a Gaussian at all.

Part 7
7.1.Take one of the histograms generated in the previous step and construct the conditional probability, P(histogram|$P_{\mu}$), of obtaining the entries in the obtained histogram for a given value of $P_{\mu}$.
Known parameters:
* pdf
Steps:

Generate a sample of random numbers with Monte Carlo, with a $P_{\mu}$ that will be "unknown" for the rest of the exercise.
Generate a histogram hist of the data (observed measurements). From now on, we assume we only know the theoretical pdf and that histogram.

We know that the bins of a histogram with fixed sample size behaves as a multinomial, thus, we know the function $P_{bin}(n_{i,obs};n_{bin},P_{\mu-try})$ where $n_{i,obs}$ is the number of observations in bin i, and it depends on 2 parameters, the number of bins $n_{bin}$ and the $P_{\mu_{try}}$. Therefore, we will say that the likelihood of obtaining hist for given $P_{\mu_{try}}$ is:
$$
P(\text{hist}|P_{\mu_{try}})=\prod_{i=1}^{n_{bin}} P_{bin}(\text{hist[i]};n_{bin},P_{\mu_{try}}) =n! \prod_{i=1}^{n_{bin}} \frac{p_i^{\text{hist[i]}}}{\text{hist[i]}!}=L(\text{hist}|P_{\mu_{try}})
$$
where $n$ is the sample size, hist is the array with our observed measurements and $p_i$ is the probability of...
Can you write Python code for it?

'''STEP 1'''

pmu_7 = random.uniform(-1,1)


I am working on the file "Statistics_block1.ipynb".
The first task is:
ATENCION: Intentar demostrar LLN sin generar cada vez la muestra
[Oriol]: Ya he cambiado el código para que no resetee la muestra cada vez. Al empezar a hacerlo directament siguiendo los pasos que tenia en la cabeza he hecho esta primera celda, que de momento dejo porque creo que se entiende mejor (o almenos alguien lo puede entender mejor). Lo que hago es:
1. Generar un vector de N componentes
2. Dividirlo en m tramos (de longitud N/m cadascuno) y hacer la media de cada tramo
3. Calcular la suma cumulativa y dividirla por el numero de celda para obtener la media de los valores anteriores a este.
    - Es decir, en el vector cumulative_Pmus, el valor en la posicion i es la media de los (i+1)*N/m primeros valores de la muestra
Esta version tarda 10 segundos aprox para hacer este calculo con $N=10^8$ y $m=5\times10^6$, es decir calcular la media cada 20.
Can you write Python code for it?

#%%timeit
# Here we demonstrate the law of large numbers without reseting the sample
Pmu_41 = 0.5 #Choose a value of Pmu for this exercise
sample_size_41 = int(1e8) # Size of the sample, then, to estimate the LLN we will use the first m
                          # to calculate the mean, then the first 2m and so on
m = int(5e6) # Divide the Pmu means vector (of size vec_length_41) into m elements, thus, its legth will be
X_sample_41 = Montecarlo_inv_fun(np.array([Pmu_41]), sample_size_41)[:,0]
X_divided = np.array(np.split(X_sample_41,m))
partial_means = np.mean(X_divided,axis=1)
N_vec = np.arange(1,m+1)
cumulative_Pmus = -9*np.cumsum(partial_means)/N_vec
N_vec = sample_size_41/m*N_vec


I am working on the file "Statistics_block1.ipynb".
The first task is:
As we can see from the plot, the estimated mean resulting from the first iterations is not very accurate, but as the sample size increases, the estimated mean approaches the theoretical mean --as predicted by the Law of Large Numbers.
For each given number of trials, we will now plot the residuals between the estimated and the theoretical means.
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.loglog(sample_sizes, np.abs(estimated_Pmus-Pmu_chosen_value),'ro') #Move the two axes to logarithmic space
ax.set_title('Residuals')
ax.set_xlabel('Sample Size')
ax.set_ylabel('Estimated - Theoretical $P_{\mu}$')


I am working on the file "Statistics_block1.ipynb".
The first task is:
Both the skewness and kurtosis factors are approximately zero, which implies the distribution is Gaussian.
Part 6
Generate N random events according to the probability density function $\frac{d\Gamma}{d\cos\theta}$ and fill the obtained $\cos\theta$ values in a histogram.
Can you write Python code for it?

Pmu = 0.3
num_sample_values = 1000

Xinv6 = Montecarlo_inv_fun(np.array([Pmu]),num_sample_values)[:,0] #Run Montecarlo with Pmu as a fixed parameter

fig = plt.figure()
ax = fig.add_subplot(111)

# Create the histogram (note: it's NOT normalized)
N_bin, bins, lis = ax.hist(Xinv6,color='b',bins=10,label='Montecarlo p.d.f.')
# N_bin is an array containing the values of the histogram bins
# bins is an array with the edges of the bins.
# The last parameter will not be used here.

ax.set_title('Inverse function montecarlo generated $d\Gamma$ p.d.f.')
ax.set_xlabel(r'$\cos\theta$')
ax.set_ylabel('$d\Gamma$')
ax.legend()


I am working on the file "Statistics_block1.ipynb".
The first task is:
Calculate the moments of the N_62 vector
Can you write Python code for it?

mu_62 = np.mean(N_62)
sigma_62 = np.std(N_62)
skewness_62 =stat.skew(N_62)
kurtosis_62 =stat.kurtosis(N_62)

print '\tmean = %.6f,\n\t\
sigma = %.6f,\n\tskewness = %.6f\n\tand kurtosis = %.6f' %(mu_62,
sigma_62, skewness_62, kurtosis_62)


I am working on the file "Statistics_block1.ipynb".
The first task is:
Calculate the moments of the distribution:
Can you write Python code for it?

mu_t = np.mean(t_values)
sigma_t = np.std(t_values) # equivalent to np.sqrt(np.var(Xi1))
skewness_t =stat.skew(t_values)
kurtosis_t =stat.kurtosis(t_values)

print '\tmean = %.6f,\n\t\
sigma = %.6f,\n\tskewness = %.6f\n\tand kurtosis = %.6f' %(mu_t, sigma_t, skewness_t, kurtosis_t)


I am working on the file "Statistics_block1.ipynb".
The first task is:
Como la generacion de numeros aleatorios se hace de manera independiente, y nuesto MC trabaja con arrays bidimensionales, podemos entrar los parámetros de manera que el propio montecarlo nos devuelva la matriz Nxm del paso 2 de la explicación.
De este modo, para los mismos N i m, el código tarda 4 segundos.
Can you write Python code for it?

#%%timeit
# After I did the above cell, I realized that taking into account how we have defined the MC,
# it can be done in a simpler way thanks to the fact that the random numbers we generate are independent
Pmu_41 = 0.5 #Choose a value of Pmu for this exercise
sample_size_41 = int(1e8) # Size of the sample, then, to estimate the LLN we will use the first m
                          # to calculate the mean, then the first 2m and so on
m = int(5e6) # Divide the Pmu means vector (of size vec_length_41) into m elements, thus, its legth will be
X_sample_41 = Montecarlo_inv_fun(np.ones(sample_size_41/m)*Pmu_41, m)
partial_means = np.mean(X_sample_41,axis=1)
N_vec = np.arange(1,m+1)
cumulative_Pmus = -9*np.cumsum(partial_means)/N_vec
N_vec = sample_size_41/m*N_vec

fig = plt.figure()
ax = fig.add_subplot(111)
ax.semilogx(N_vec,cumulative_Pmus,'b-',alpha=0.7)
ax.semilogx(N_vec,np.ones(m)*Pmu_41,'k')
ax.set_title('Proof of the Law of Large Numbers')
ax.set_xlabel('Sample size')
ax.set_ylabel('Estimated $P_{\mu}$')
ax.legend(['Estimated mean','Theoretical mean'],loc='best')


I am working on the file "Statistics_block1.ipynb".
The first task is:
Compute the chi squared for each MC run.
Can you write Python code for it?

chi_squared_values = np.empty(chisquared_sample)

for i in xrange(chisquared_sample):
    hist, bin_edges = np.histogram(X_63[:,i],range=(-1,1), bins = bin_number_63) # fixing the range to -1, 1 avoids the bins to start at the minimum
    # of the sample which would be min(X_63) (around -0.98 but not -1)

    chi_squared_values[i] = np.sum((hist-mean_63)**2/(mean_63*(1-Prob_bin_63)))


I am working on the file "Statistics_block1.ipynb".
The first task is:
Estimate distribution parameters
In this section, the rellevant parameters of the p.d.f. will be estimated from the montecarlo generated sample. To be able to compare this values, the theoretical values will also be obtained:
$$
\int_{-1}^{1} \frac{1}{2}\big(1-\frac{1}{3}P_{\mu}\cos\theta\big) \cos\theta d\cos\theta = \Big[\frac{\cos^2\theta}{4}-\frac{P_{\mu}}{6}\frac{\cos^3\theta}{3}\Big]{-1}^{1}=\frac{-P{\mu}}{9}
$$
Can you write Python code for it?

mu=np.mean(Xinv2)
sigma=np.std(Xinv2) # equivalent to np.sqrt(np.var(Xi1))
skewness=stat.skew(Xinv2)
kurtosis=stat.kurtosis(Xinv2)

print 'The try-reject montecarlo generated distribution has:\n\tmean = %.6f,\n\t\
sigma = %.6f,\n\tskewness = %.6f\n\tand kurtosis = %.6f' %(mu, sigma, skewness, kurtosis)


I am working on the file "Statistics_block1.ipynb".
The first task is:
For the values to follow a Gaussian distribution, the skewness factor would have to be 0 (sknewness measures symmetry, and Gaussian is 100% symmetrical), and the kurtosis factor would also have to be 0. Since none of these parameters is 0, the distribution is not Gaussian.
5.2. Show the validity of the Cetral Limit Theorem
Can you write Python code for it?

N_clt = int(1e6)

Pmu_chosen_value = 0.5 #Choose a value of Pmu for this exercise
mu_t=-Pmu_chosen_value/9.
num_repetitions = 15 # number of times the N x num_MC_experiments MC_samples matrix is generated
num_MC_experiments = 200  # Choose the number of MC experiments to generate (it will be the # of t values in our histogram)
PmuVec_t = np.ones(num_MC_experiments)*Pmu_chosen_value  # Create vector with length = num_MC_experiments filled with a given Pmu value from which the pdf sample will be generated
clt_values = np.empty(num_repetitions*num_MC_experiments)

for repeat in xrange(num_repetitions):
    MC_samples = Montecarlo_inv_fun(PmuVec_t, N_clt) # matrix N x num_MC_experiments
    x_bar = np.mean(MC_samples, axis=0) #Get estimated mean
    s_square = np.var(MC_samples, axis=0, ddof=1) #Get estimated variance (sigma^2)
    clt_values[repeat*num_MC_experiments:(repeat+1)*num_MC_experiments] = np.sqrt(N_clt/s_square)*(x_bar-mu_t)

# Expected gaussian pdf
mu = 0
sigma = 1
dist = stat.norm(mu, sigma)
x_pdf = np.linspace(-4, 4, 1000)

#Plot distributions
fig = plt.figure()
ax = fig.add_subplot(111)
lim_prob = 0.001
grid_num = 100
x = np.linspace(stat.t.ppf(lim_prob, N_clt), stat.t.ppf(1-lim_prob, N_clt), grid_num)
ax.plot(x, stat.t.pdf(x, N_clt), 'r-', lw=5, alpha=0.5, label='t pdf (Theoretical)')
ax.hist(clt_values,color='b',normed=1,bins=30,label='Montecarlo p.d.f.')
ax.plot(x_pdf, dist.pdf(x_pdf), '-k', label = 'Gaussian')
ax.legend()


I am working on the file "Statistics_block1.ipynb".
The first task is:
Inverse function p.d.f. generation
To generate a p.d.f via the inverse method we need the inverse of the cumulative function. The cumulative function was found in the first question while checking that the p.d.f. was normalized, thus:
$$
F(\cos\theta)=\Big[\frac{1}{2}z-\frac{1}{12}P_{\mu}z^2 \Big]{-1}^{\cos\theta}=\frac{1}{2}(\cos\theta+1)-\frac{1}{12}P{\mu}(\cos^2\theta-1)
$$
$$
\frac{P_{\mu}}{12}\cos^2\theta-\frac{\cos\theta}{2}+r-\frac{1}{2}-\frac{P_{\mu}}{12}=0 \quad \rightarrow \quad F^{-1}(r)=\cos\theta=\frac{3}{P_{\mu}}\pm \frac{6}{P_{\mu}}\sqrt{\frac{1}{4}-\frac{P_{\mu}}{3}(r-\frac{1}{2}-\frac{P_{\mu}}{12})}
$$
Can you write Python code for it?

# Inverse function F^{-1}(r) implemented in Python
def Finv(r,Pmu):
    # its arguments are:
    #     r   : either int, float or np.array. Must be a value between 0 and 1
    #     Pmu : either int, float or np.array, its shape must be compatible with r in case of arrays
    cost=(3.-6.*np.sqrt(.25-Pmu/3.*(r-.5-Pmu/12)))/Pmu
    return cost


I am working on the file "Statistics_block1.ipynb".
The first task is:
Part 3
The $\frac{d\Gamma}{d\cos\theta}$ mean depends on the polarisation $P_{\mu}$ in a simple manner: theoretical mean = -$P_{\mu}$/9.
3.1. Show that the Monte Carlo predicts this dependency by changing the value of $P_{\mu}$
Can you write Python code for it?

N=int(1e7)
PmuVec=np.linspace(-1,1,30)  # array of Pmu valus for which the pdf sample will be generated
X=Montecarlo_inv_fun(PmuVec,N) # call Montecarlo_inv_fun, which returns a matrix, containing N x values
# following the pdf of each Pmu in PmuVec
mu=np.mean(X,axis=0) # estimate the mean for each Pmu, thus, the result is a vector of the same length as PmuVec
muTh=-PmuVec/9. # calculate the theoretical mean for each Pmu

fig=plt.figure(1)
ax=fig.add_subplot(111)
ax.plot(PmuVec,mu,'ro',PmuVec,muTh,'b--')
ax.set_title('$P_{\mu}$ p.d.f. dependency')
ax.set_xlabel(r'$P_{\mu}$')
ax.set_ylabel('$\mu$')
ax.legend(['Estimated mean','Theoretical mean'])


I am working on the file "Statistics_block1.ipynb".
The first task is:
Part 5
Generate several Monte Carlo experiments, each with N events.
5.1. Build for each experiment the student's t variable for $P_{\mu}$ and show that it follows the Student's t distribution
Can you write Python code for it?

N = 15 # number of samples for each montecarlo experiment
Pmu_chosen_value = 0.5 #Choose a value of Pmu for this exercise
mu_t=-Pmu_chosen_value/9.
num_repetitions = 15 # number of times the N x num_MC_experiments MC_samples matrix is generated
num_MC_experiments = 500000  # Choose the number of MC experiments to generate (it will be the # of t values in our histogram)
PmuVec_t = np.ones(num_MC_experiments)*Pmu_chosen_value  # Create vector with length = num_MC_experiments filled with a given Pmu value from which the pdf sample will be generated
t_values = np.empty(num_repetitions*num_MC_experiments)

for repeat in xrange(num_repetitions):
    MC_samples = Montecarlo_inv_fun(PmuVec_t, N) # matrix N x num_MC_experiments
    x_bar = np.mean(MC_samples, axis=0) #Get estimated mean
    s_square = np.var(MC_samples, axis=0, ddof=1) #Get estimated variance (sigma^2)
    t_values[repeat*num_MC_experiments:(repeat+1)*num_MC_experiments] = np.sqrt(N/s_square)*(x_bar-mu_t)

fig = plt.figure()
ax = fig.add_subplot(111)
lim_prob = 0.001
grid_num = 100
x = np.linspace(stat.t.ppf(lim_prob, N), stat.t.ppf(1-lim_prob, N), grid_num)
ax.plot(x, stat.t.pdf(x, N), 'r-', label='t pdf (Theoretical)')
ax.hist(t_values,color='b',normed=1, bins = 20,label='Montecarlo p.d.f.')
ax.legend(loc=2)


I am working on the file "Statistics_block1.ipynb".
The first task is:
Plot the $\chi_{^2}$ distribution.
Can you write Python code for it?

x_pdf_chisquared = np.linspace(min(chi_squared_values),max(chi_squared_values),1000)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.hist(chi_squared_values,color='b', bins=15,normed=1,label = '$\chi_{^2}$ values')
ax.plot(x_pdf_chisquared,stat.chi2.pdf(x_pdf_chisquared,bin_number_63),'r--',lw=2, label = '$\chi_{^2}$ distribution (theoretical)')

ax.set_title('Expected and Observed Distribution')
ax.set_xlabel('Chi Squared values')
ax.legend()


I am working on the file "Statistics_block1.ipynb".
The first task is:
Plot the binominal distribution on top the histogram for bin bin_number
Can you write Python code for it?

ki=np.arange(num_sample)
fig = plt.figure()
ax = fig.add_subplot(111)
ax.hist(N_61,color='b',bins=10,normed=1,label='Montecarlo p.d.f.')
Prob_bin = Fcumulative(bin_edges[bin_number],Pmu_chosen_value)-Fcumulative(bin_edges[bin_number-1],Pmu_chosen_value)
ax.plot(ki,stat.binom.pmf(ki,num_sample,Prob_bin),'r--')

ax.set_title('Binomial distribution and Histogram for "bin_number" bin')
ax.set_xlabel('Number of entries in bin "bin_number"')
ax.set_ylabel('Number of succesful events (normalized)')
ax.legend()


I am working on the file "Statistics_block1.ipynb".
The first task is:
Plot the results of the Try-Reject Method:
Can you write Python code for it?

fig=plt.figure(2)
ax=fig.add_subplot(111)
ax.hist(Xi1,color='b',normed=1,bins=50,label='Montecarlo p.d.f.')
ax.hist(Xi2,color='g',normed=1,bins=50,label='Montecarlo p.d.f.',alpha=0.6)
ax.plot(cost,dG,'r--',linewidth=2,label='Theoretical')
ax.set_title('Try-reject Monte Carlo generated $d\Gamma$ p.d.f.')
ax.set_xlabel(r'$\cos\theta$')
ax.set_ylabel('$d\Gamma$')
ax.legend()
ax.set_ylim([0.4,0.6])


I am working on the file "Statistics_block1.ipynb".
The first task is:
We can use the cumulative function to calculate the probability that a point falls within the bin.
Can you write Python code for it?

def Fcumulative(cost,Pmu): # Cumulative function
    F=.5*(cost+1)-Pmu/12.*(cost**2-1)
    return F


I am working on the file "Statistics_block1.ipynb".
The first task is:
We define the variance as: $$Var(X)=E[(x-\mu)^{2}]$$
With the sum command, we are obtaining the variance with: $$\overline{Var}=\frac{1}{N}\sum(P_{i}-\mu)^{2}$$ where $\mu$ is the theoretical mean. In this case, we have chosen a specific value for Pmu (Pmu_chosen_value = 0.5), so the theoretical mean will simply correspond to this value.
With the np.var command, we are calculating the variance with equation: $$\overline{Var}=\frac{1}{N-1}\sum(P_{i}-\overline{x})$$
where $\bar{x}$ is the estimated mean, i.e. $\bar{x}=\frac{1}{N}\sum x_{i}$. Note that np.var accepts "delta Degrees of freedom" (ddof) which refers to the factor $1/N-\text{ddof}$. We chose $ddof = 1$ so that the division would be as 1/N-1, instead of 1/N.
Part 4
Generate a continuous series of N events using the Monte Carlo and compute the mean of the distribution and the estimated $P_{\mu}$ as 9*mean.
4.1. Show that the $P_{\mu}$ tends to the true evalue as predicted by the law of large numbers
Can you write Python code for it?

Pmu_chosen_value = 0.5 #Choose a value of Pmu for this exercise
vec_length = 50  # Choose the length of the array "get_PmuVec"
get_PmuVec = np.ones(vec_length)*Pmu_chosen_value  # Create vector with length = 50 filled with a given Pmu value from which the pdf sample will be generated
num_samples = 20
sample_sizes = np.logspace(start = 1, stop = 6.5, num = num_samples, dtype=int) #num refers to the nº of points generated between start and stop

estimated_Pmus = np.empty(num_samples)

for i,sample_size in enumerate(sample_sizes):
    get_Pmus = Montecarlo_inv_fun(get_PmuVec, sample_size)
    estimated_mean = np.mean(get_Pmus, axis = 0) #Get estimated mean
    estimated_Pmus[i] = -9*np.mean(estimated_mean)  # estimate the mean for each Pmu. The result is a vector of the same length as PmuVec

fig = plt.figure()
ax = fig.add_subplot(111)
ax.semilogx(sample_sizes,estimated_Pmus,'ro',sample_sizes,Pmu_chosen_value*np.ones(num_samples),'b--')
ax.set_title('Proof of the Law of Large Numbers')
ax.set_xlabel('Sample size')
ax.set_ylabel('Estimated $P_{\mu}$')
ax.legend(['Estimated mean','Theoretical mean'],loc=3)


I am working on the file "Statistics_block1.ipynb".
The first task is:
[Oriol] He separado la definición de $P_{\mu}$ del resto de parametros, así si no haceis un print no sabreis cual es el valor de $P_{\mu}$ y no os quejareis tanto. Ahora podeis tratar de adivinar $P_{\mu}$ mirando al histograma con el plot de la pdf teorica comentado. También he movido la definicón de bin number abajo por si alguien se aburre y quiere mirar como varia el resultado con la misma muestra pero usando bins diferentes.
Can you write Python code for it?

sample_size_n = int(1e3)
sample_7 = Montecarlo_inv_fun(np.array([pmu_7]), sample_size_n)[:,0]

bin_number_7 = 10
'''STEP 2'''

#Obtain the theoretical pdf
dG = gamma(cost,pmu_7)

#Plot the histogram and the theoretical distribution
fig = plt.figure()
ax = fig.add_subplot(111)
ax.hist(sample_7,color='b', bins = bin_number_7,normed=1, range=(-1,1), label = 'Sample values')
#ax.plot(cost, dG, label = 'Theoretical pdf')
ax.set_title('Histogram data')

''' STEP 3'''

hist_7, bin_edges_7 = np.histogram(sample_7,range=(-1,1), bins = bin_number_7)

#Define the likelihood function
def likelihood(pmu_try, hist_values):
    n_bin = len(hist_values)
    N = sum(hist_values)
    bin_edges = np.linspace(-1, 1, n_bin+1)
    Prob_bin = Fcumulative(bin_edges[1:], pmu_try) - Fcumulative(bin_edges[:-1], pmu_try)
    # This is the theoretical likelihood as defined in the comment above. Since the values that we obtain are very
    # small, we have decided to work with the logarithm of this function.
    '''
    nfact = factorial(len(hist_values))
    num = np.power(Prob_bin_7,hist_values)
    den = factorial(hist_values)
    likelihood = nfact*np.prod(num/den)
    '''
    #term1 = -np.log(factorial(N)) # note that the log likelihood is defined as -log(L)
    term2 = -np.sum(hist_values*np.log(Prob_bin))
    #term3 = -np.sum(np.log(factorial(hist_values)))
    #likelihood = term1 + term2 - term3
    return term2

# define the derivative of the likelihood function
def d_likelihood(pmu_try, hist_values):
    n_bin = len(hist_values)
    N = sum(hist_values)
    bin_edges = np.linspace(-1, 1, n_bin+1)
    Prob_bin = Fcumulative(bin_edges[1:], pmu_try) - Fcumulative(bin_edges[:-1], pmu_try)
    d_like = np.sum(hist_values/Prob_bin*(np.square(bin_edges[:-1])-np.square(bin_edges[1:])))/12.
    return d_like

pmu_grid = np.linspace(-1,1,50)
like = np.empty(len(pmu_grid))
Dlike = np.empty(len(pmu_grid))

for i,pmu in enumerate(pmu_grid):
    like[i] = likelihood(pmu, hist_7)
    Dlike[i] = d_likelihood(pmu, hist_7)
   ...


I am working on the file "Statistics_block2.ipynb".
The first task is:
### Case 2: $\sigma$ is unknown.
If we did not have sigma, we would first estimate it and use this estimate as our actual value. The resulting confidence intervals would be an estimation of the real confidence intervals shown above. The code below takes sigma as the std of the measurements.
Can you write Python code for it?

sigma_estimated = np.std(obs) # estimated value of the standar deviation of variable x
print sigma_estimated
sigma_est_estimator = sigma_estimated/np.sqrt(len(obs)) # estimated value of the standard deviation of variable mu hat

opt_delta_estimated = np.empty(len(alpha))
delta_t_student = np.empty(len(alpha))
mu_hat=np.mean(obs)
print "Mu = %.3g and Sigma = %.3g\n" %(mu, sigma_est_estimator)

for i in range(len(alpha)):
    opt_delta_estimated[i] = opt.fsolve(find_delta, sigma_est_estimator, args=(alpha[i],sigma_est_estimator))
    delta_t_student[i] = stat.t.interval(1-alpha[i],len(obs),0,sigma_est_estimator)[1]

    #if our estimated delta was the true one, our results would match the values in the alpha array.
    print "1-alpha = %.3f" %(1-alpha[i])
    print "\tGaussian approximation resulting 1-alpha = %.3f" %math.erf(opt_delta_estimated[i]/(np.sqrt(2)*sigma_estimator))
    print "\t\tDelta gaussian = ", opt_delta_estimated[i]
    print '\t\tInterval [mu-delta, mu+delta]', mu-opt_delta_estimated[i], mu+opt_delta_estimated[i]
    print "\tT-student approximation resulting 1-alpha = %.3f" %math.erf(delta_t_student[i]/(np.sqrt(2)*sigma_estimator))
    print "\t\tDelta t-student= ", delta_t_student[i]
    print '\t\tInterval [mu-delta, mu+delta]', mu-delta_t_student[i], mu+delta_t_student[i]


I am working on the file "Statistics_block2.ipynb".
The first task is:
(D)
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.hist(nu_mc_sample,color='b',normed=1,bins=15,label='Montecarlo poisson p.d.f. (Part III)')
ax.plot(nu_pdf, nu_posterior, 'g', lw=2, label = 'Bayesian Posterior')

ax.set_xlim([min(mc_sample), max(mc_sample)])
ax.set_xlabel(r'$\nu$')
ax.set_ylabel(r'$p(\nu|N_{obs})$')
ax.legend()

print len(nu_mc_sample) #this is the number of values that we select from N_mc sample


I am working on the file "Statistics_block2.ipynb".
The first task is:
(E)
Can you write Python code for it?

mu=np.mean(nu_mc_sample)
variance =np.var(nu_mc_sample) # equivalent to np.sqrt(np.var(nu_mc_sample))
skewness = stat.skew(nu_mc_sample)
kurtosis=stat.kurtosis(nu_mc_sample)

print 'The Montecarlo generated distribution has:\n\tmean = %.6f,\n\t\
variance = %.6f,\n\tskewness = %.6f\n\tand kurtosis = %.6f' %(mu, variance, skewness, kurtosis)