I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
4. Plot the artificial dataset:
Can you write Python code for it?

plt.bar(np.arange(80), data, color="#348ABD")
plt.bar(tau - 1, data[tau - 1], color="r", label="user behaviour changed")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Artificial dataset")
plt.xlim(0, 80)
plt.legend();


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
A Normal random variable can be take on any real number, but the variable is very likely to be relatively close to $\mu$. In fact, the expected value of a Normal is equal to its $\mu$ parameter:
$$ E[ X | \mu, \tau] = \mu$$
and its variance is equal to the inverse of $\tau$:
$$Var( X | \mu, \tau ) = \frac{1}{\tau}$$
Below we continue our modeling of the Challenger space craft:
Can you write Python code for it?

import pymc as pm

temperature = challenger_data[:, 0]
D = challenger_data[:, 1]  # defect or not?

# notice the`value` here. We explain why below.
beta = pm.Normal("beta", 0, 0.001, value=0)
alpha = pm.Normal("alpha", 0, 0.001, value=0)

@pm.deterministic
def p(t=temperature, alpha=alpha, beta=beta):
    return 1.0 / (1. + np.exp(beta * t + alpha))


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Above we also plotted two possible realizations of what the actual underlying system might be. Both are equally likely as any other draw. The blue line is what occurs when we average all the 20000 possible dotted lines together.
An interesting question to ask is for what temperatures are we most uncertain about the defect-probability? Below we plot the expected value line and the associated 95% intervals for each temperature.
Can you write Python code for it?

from scipy.stats.mstats import mquantiles

# vectorized bottom and top 2.5% quantiles for "confidence interval"
qs = mquantiles(p_t, [0.025, 0.975], axis=0)
plt.fill_between(t[:, 0], *qs, alpha=0.7,
                 color="#7A68A6")

plt.plot(t[:, 0], qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

plt.plot(t, mean_prob_t, lw=1, ls="--", color="k",
         label="average posterior \nprobability of defect")

plt.xlim(t.min(), t.max())
plt.ylim(-0.02, 1.02)
plt.legend(loc="lower left")
plt.scatter(temperature, D, color="k", s=50, alpha=0.5)
plt.xlabel("temp, $t$")

plt.ylabel("probability estimate")
plt.title("Posterior probability estimates given temp. $t$");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Adding a constant term $\alpha$ amounts to shifting the curve left or right (hence why it is called a bias).
Let's start modeling this in PyMC. The $\beta, \alpha$ parameters have no reason to be positive, bounded or relatively large, so they are best modeled by a Normal random variable, introduced next.
Normal distributions
A Normal random variable, denoted $X \sim N(\mu, 1/\tau)$, has a distribution with two parameters: the mean, $\mu$, and the precision, $\tau$. Those familiar with the Normal distribution already have probably seen $\sigma^2$ instead of $\tau^{-1}$. They are in fact reciprocals of each other. The change was motivated by simpler mathematical analysis and is an artifact of older Bayesian methods. Just remember: the smaller $\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\tau$ is always positive.
The probability density function of a $N( \mu, 1/\tau)$ random variable is:
$$ f(x | \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau}{2} (x-\mu)^2 \right) $$
We plot some different density functions below.
Can you write Python code for it?

import scipy.stats as stats

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1. / _tau),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1. / _tau), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Again, thinking of our data-generation model, we assign Bernoulli random variables to the 100 students: 1 implies they cheated and 0 implies they did not.
Can you write Python code for it?

true_answers = pm.Bernoulli("truths", p, size=N)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
All samples of $\beta$ are greater than 0. If instead the posterior was centered around 0, we may suspect that $\beta = 0$, implying that temperature has no effect on the probability of defect.
Similarly, all $\alpha$ posterior values are negative and far away from 0, implying that it is correct to believe that $\alpha$ is significantly less than 0.
Regarding the spread of the data, we are very uncertain about what the true parameters might be (though considering the low sample size and the large overlap of defects-to-nondefects this behaviour is perhaps expected).
Next, let's look at the expected probability for a specific value of the temperature. That is, we average over all samples from the posterior to get a likely value for $p(t_i)$.
Can you write Python code for it?

t = np.linspace(temperature.min() - 5, temperature.max() + 5, 50)[:, None]
p_t = logistic(t.T, beta_samples, alpha_samples)

mean_prob_t = p_t.mean(axis=0)

figsize(12.5, 4)

plt.plot(t, mean_prob_t, lw=3, label="average posterior \nprobability \
of defect")
plt.plot(t, p_t[0, :], ls="--", label="realization from posterior")
plt.plot(t, p_t[-2, :], ls="--", label="realization from posterior")
plt.scatter(temperature, D, color="k", s=50, alpha=0.5)
plt.title("Posterior expected value of probability of defect; \
plus realizations")
plt.legend(loc="lower left")
plt.ylim(-0.1, 1.1)
plt.xlim(t.min(), t.max())
plt.ylabel("probability")
plt.xlabel("temperature");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Although not everyone flips a second time, we can still model the possible realization of second coin-flips:
Can you write Python code for it?

second_coin_flips = pm.Bernoulli("second_flips", 0.5, size=N)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Below we add all the variables of interest to a Model container and run our black-box algorithm over the model.
Can you write Python code for it?

model = pm.Model([p, true_answers, first_coin_flips,
                  second_coin_flips, observed_proportion, observations])

# To be explained in Chapter 3!
mcmc = pm.MCMC(model)
mcmc.sample(40000, 15000)

figsize(12.5, 3)
p_trace = mcmc.trace("freq_cheating")[:]
plt.hist(p_trace, histtype="stepfilled", normed=True, alpha=0.85, bins=30,
         label="posterior distribution", color="#348ABD")
plt.vlines([.05, .35], [0, 0], [5, 5], alpha=0.3)
plt.xlim(0, 1)
plt.legend();


model = pm.Model([yes_responses, p_skewed, p])

# To Be Explained in Chapter 3!
mcmc = pm.MCMC(model)
mcmc.sample(25000, 2500)

figsize(12.5, 3)
p_trace = mcmc.trace("freq_cheating")[:]
plt.hist(p_trace, histtype="stepfilled", normed=True, alpha=0.85, bins=30,
         label="posterior distribution", color="#348ABD")
plt.vlines([.05, .35], [0, 0], [5, 5], alpha=0.2)
plt.xlim(0, 1)
plt.legend();


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Below we plot the posterior distributions for the three unknowns:
Can you write Python code for it?

p_A_samples = mcmc.trace("p_A")[:]
p_B_samples = mcmc.trace("p_B")[:]
delta_samples = mcmc.trace("delta")[:]

figsize(12.5, 10)

# histogram of posteriors

ax = plt.subplot(311)

plt.xlim(0, .1)
plt.hist(p_A_samples, histtype='stepfilled', bins=25, alpha=0.85,
         label="posterior of $p_A$", color="#A60628", normed=True)
plt.vlines(true_p_A, 0, 80, linestyle="--", label="true $p_A$ (unknown)")
plt.legend(loc="upper right")
plt.title("Posterior distributions of $p_A$, $p_B$, and delta unknowns")

ax = plt.subplot(312)

plt.xlim(0, .1)
plt.hist(p_B_samples, histtype='stepfilled', bins=25, alpha=0.85,
         label="posterior of $p_B$", color="#467821", normed=True)
plt.vlines(true_p_B, 0, 80, linestyle="--", label="true $p_B$ (unknown)")
plt.legend(loc="upper right")

ax = plt.subplot(313)
plt.hist(delta_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of delta", color="#7A68A6", normed=True)
plt.vlines(true_p_A - true_p_B, 0, 60, linestyle="--",
           label="true delta (unknown)")
plt.vlines(0, 0, 60, color="black", alpha=0.2)
plt.legend(loc="upper right");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
But something is missing. In the plot of the logistic function, the probability changes only near zero, but in our data above the probability changes around 65 to 70. We need to add a bias term to our logistic function:
$$p(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$
Some plots are below, with differing $\alpha$.
Can you write Python code for it?

def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

x = np.linspace(-4, 4, 100)

plt.plot(x, logistic(x, 1), label=r"$\beta = 1$", ls="--", lw=1)
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$", ls="--", lw=1)
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$", ls="--", lw=1)

plt.plot(x, logistic(x, 1, 1), label=r"$\beta = 1, \alpha = 1$",
         color="#348ABD")
plt.plot(x, logistic(x, 3, -2), label=r"$\beta = 3, \alpha = -2$",
         color="#A60628")
plt.plot(x, logistic(x, -5, 7), label=r"$\beta = -5, \alpha = 7$",
         color="#7A68A6")

plt.title("Logistic functon with bias, plotted for several value of $\\alpha$ bias parameter", fontsize=14)
plt.legend(loc="lower left");


def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

x = np.linspace(-4, 4, 100)

plt.plot(x, logistic(x, 1), label=r"$\beta = 1$", ls="--", lw=1)
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$", ls="--", lw=1)
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$", ls="--", lw=1)

plt.plot(x, logistic(x, 1, 1), label=r"$\beta = 1, \alpha = 1$",
         color="#348ABD")
plt.plot(x, logistic(x, 3, -2), label=r"$\beta = 3, \alpha = -2$",
         color="#A60628")
plt.plot(x, logistic(x, -5, 7), label=r"$\beta = -5, \alpha = 7$",
         color="#7A68A6")

plt.legend(loc="lower left");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Clearly, if $\tau, \lambda_1$ and $\lambda_2$ are known, then $\lambda$ is known completely, hence it is a deterministic variable.
Inside the deterministic decorator, the Stochastic variables passed in behave like scalars or Numpy arrays (if multivariable), and not like Stochastic variables. For example, running the following:
@pm.deterministic
def some_deterministic(stoch=some_stochastic_var):
    return stoch.value**2

will return an AttributeError detailing that stoch does not have a value attribute. It simply needs to be stoch**2. During the learning phase, it's the variable's value that is repeatedly passed in, not the actual variable.
Notice in the creation of the deterministic function we added defaults to each variable used in the function. This is a necessary step, and all variables must have default values.
Including observations in the Model
At this point, it may not look like it, but we have fully specified our priors. For example, we can ask and answer questions like "What does my prior distribution of $\lambda_1$ look like?"
Can you write Python code for it?

%matplotlib inline
from IPython.core.pylabtools import figsize
from matplotlib import pyplot as plt
figsize(12.5, 4)

samples = [lambda_1.random() for i in range(20000)]
plt.hist(samples, bins=70, normed=True, histtype="stepfilled")
plt.title("Prior distribution for $\lambda_1$")
plt.xlim(0, 8);


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Finally...
We wrap all the created variables into a pm.Model class. With this Model class, we can analyze the variables as a single unit. This is an optional step, as the fitting algorithms can be sent an array of the variables rather than a Model class. I may or may not use this class in future examples ;)
Can you write Python code for it?

model = pm.Model([obs, lambda_, lambda_1, lambda_2, tau])


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Had we had stronger beliefs, we could have expressed them in the prior above.
For this example, consider $p_A = 0.05$, and $N = 1500$ users shown site A, and we will simulate whether the user made a purchase or not. To simulate this from $N$ trials, we will use a Bernoulli distribution: if  $X\ \sim \text{Ber}(p)$, then $X$ is 1 with probability $p$ and 0 with probability $1 - p$. Of course, in practice we do not know $p_A$, but we will use it here to simulate the data.
Can you write Python code for it?

# set constants
p_true = 0.05  # remember, this is unknown.
N = 1500

# sample N Bernoulli random variables from Ber(0.05).
# each random variable has a 0.05 chance of being a 1.
# this is the data-generation step
occurrences = pm.rbernoulli(p_true, N)

print occurrences  # Remember: Python treats True == 1, and False == 0
print occurrences.sum()


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
I could have typed p_skewed  = 0.5*p + 0.25 instead for a one-liner, as the elementary operations of addition and scalar multiplication will implicitly create a deterministic variable, but I wanted to make the deterministic boilerplate explicit for clarity's sake.
If we know the probability of respondents saying "Yes", which is p_skewed, and we have $N=100$ students, the number of "Yes" responses is a binomial random variable with parameters N and p_skewed.
This is where we include our observed 35 "Yes" responses. In the declaration of the pm.Binomial, we include value = 35 and observed = True.
Can you write Python code for it?

yes_responses = pm.Binomial("number_cheaters", 100, p_skewed,
                            value=35, observed=True)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
If this probability is too high for comfortable decision-making, we can perform more trials on site B (as site B has less samples to begin with, each additional data point for site B contributes more inferential "power" than each additional data point for site A).
Try playing with the parameters true_p_A, true_p_B, N_A, and N_B, to see what the posterior of $\text{delta}$ looks like. Notice in all this, the difference in sample sizes between site A and site B was never mentioned: it naturally fits into Bayesian analysis.
I hope the readers feel this style of A/B testing is more natural than hypothesis testing, which has probably confused more than helped practitioners. Later in this book, we will see two extensions of this model: the first to help dynamically adjust for bad sites, and the second will improve the speed of this computation by reducing the analysis to a single equation.
An algorithm for human deceit
Social data has an additional layer of interest as people are not always honest with responses, which adds a further complication into inference. For example, simply asking individuals "Have you ever cheated on a test?" will surely contain some rate of dishonesty. What you can say for certain is that the true rate is less than your observed rate (assuming individuals lie only about not cheating; I cannot imagine one who would admit "Yes" to cheating when in fact they hadn't cheated).
To present an elegant solution to circumventing this dishonesty problem, and to demonstrate Bayesian modeling, we first need to introduce the binomial distribution.
The Binomial Distribution
The binomial distribution is one of the most popular distributions, mostly because of its simplicity and usefulness. Unlike the other distributions we have encountered thus far in the book, the binomial distribution has 2 parameters: $N$, a positive integer representing $N$ trials or number of instances of potential events, and $p$, the probability of an event occurring in a single trial. Like the Poisson distribution, it is a...
Can you write Python code for it?

figsize(12.5, 4)

import scipy.stats as stats
binomial = stats.binom

parameters = [(10, .4), (10, .9)]
colors = ["#348ABD", "#A60628"]

for i in range(2):
    N, p = parameters[i]
    _x = np.arange(N + 1)
    plt.bar(_x - 0.5, binomial.pmf(_x, N, p), color=colors[i],
            edgecolor=colors[i],
            alpha=0.6,
            label="$N$: %d, $p$: %.1f" % (N, p),
            linewidth=3)

plt.legend(loc="upper left")
plt.xlim(0, 10.5)
plt.xlabel("$k$")
plt.ylabel("$P(X = k)$")
plt.title("Probability mass distributions of binomial random variables");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
If we carry out the algorithm, the next step that occurs is the first coin-flip each student makes. This can be modeled again by sampling 100 Bernoulli random variables with $p=1/2$: denote a 1 as a Heads and 0 a Tails.
Can you write Python code for it?

first_coin_flips = pm.Bernoulli("first_flips", 0.5, size=N)
print first_coin_flips.value


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
In the random model, we can see that as the probability increases there is no clustering of defects to the right-hand side. Similarly for the constant model.
The perfect model, the probability line is not well shown, as it is stuck to the bottom and top of the figure. Of course the perfect model is only for demonstration, and we cannot infer any scientific inference from it.
Exercises
1. Try putting in extreme values for our observations in the cheating example. What happens if we observe 25 affirmative responses? 10? 50?
2. Try plotting $\alpha$ samples versus $\beta$ samples.  Why might the resulting plot look like this?
Can you write Python code for it?

# type your code here.
figsize(12.5, 4)

plt.scatter(alpha_samples, beta_samples, alpha=0.1)
plt.title("Why does the plot look like this?")
plt.xlabel(r"$\alpha$")
plt.ylabel(r"$\beta$")


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Is our model appropriate?
The skeptical reader will say "You deliberately chose the logistic function for $p(t)$ and the specific priors. Perhaps other functions or priors will give different results. How do I know I have chosen a good model?" This is absolutely true. To consider an extreme situation, what if I had chosen the function $p(t) = 1,\; \forall t$, which guarantees a defect always occurring: I would have again predicted disaster on January 28th. Yet this is clearly a poorly chosen model. On the other hand, if I did choose the logistic function for $p(t)$, but specified all my priors to be very tight around 0, likely we would have very different posterior distributions. How do we know our model is an expression of the data? This encourages us to measure the model's goodness of fit.
We can think: how can we test whether our model is a bad fit? An idea is to compare observed data (which if we recall is a fixed stochastic variable) with an artificial dataset which we can simulate. The rationale is that if the simulated dataset does not appear similar, statistically, to the observed dataset, then likely our model is not accurately represented the observed data.
Previously in this Chapter, we simulated artificial datasets for the SMS example. To do this, we sampled values from the priors. We saw how varied the resulting datasets looked like, and rarely did they mimic our observed dataset. In the current example,  we should sample from the posterior distributions to create very plausible datasets. Luckily, our Bayesian framework makes this very easy. We only need to create a new Stochastic variable, that is exactly the same as our variable that stored the observations, but minus the observations themselves. If you recall, our Stochastic variable that stored our observed data was:
observed = pm.Bernoulli( "bernoulli_obs", p, value=D, observed=True)

Hence we create:
simulated_data = pm.Bernoulli("simulation_data", p)

Let's simulate 10 000:
Can you write Python code for it?

simulated = pm.Bernoulli("bernoulli_sim", p)
N = 10000

mcmc = pm.MCMC([simulated, alpha, beta, observed])
mcmc.sample(N)

figsize(12.5, 5)

simulations = mcmc.trace("bernoulli_sim")[:]
print simulations.shape

plt.title("Simulated dataset using posterior parameters")
figsize(12.5, 6)
for i in range(4):
    ax = plt.subplot(4, 1, i + 1)
    plt.scatter(temperature, simulations[1000 * i, :], color="k",
                s=50, alpha=0.6)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
It is okay that our fictional dataset does not look like our observed dataset: the probability is incredibly small it indeed would. PyMC's engine is designed to find good parameters, $\lambda_i, \tau$, that maximize this probability.
The ability to generate artificial datasets is an interesting side effect of our modeling, and we will see that this ability is a very important method of Bayesian inference. We produce a few more datasets below:
Can you write Python code for it?

def plot_artificial_sms_dataset():
    tau = pm.rdiscrete_uniform(0, 80)
    alpha = 1. / 20.
    lambda_1, lambda_2 = pm.rexponential(alpha, 2)
    data = np.r_[pm.rpoisson(lambda_1, tau), pm.rpoisson(lambda_2, 80 - tau)]
    plt.bar(np.arange(80), data, color="#348ABD")
    plt.bar(tau - 1, data[tau - 1], color="r", label="user behaviour changed")
    plt.xlim(0, 80)

figsize(12.5, 5)
plt.title("More example of artificial datasets")
for i in range(1, 5):
    plt.subplot(4, 1, i)
    plot_artificial_sms_dataset()


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
It looks clear that the probability of damage incidents occurring increases as the outside temperature decreases. We are interested in modeling the probability here because it does not look like there is a strict cutoff point between temperature and a damage incident occurring. The best we can do is ask "At temperature $t$, what is the probability of a damage incident?". The goal of this example is to answer that question.
We need a function of temperature, call it $p(t)$, that is bounded between 0 and 1 (so as to model a probability) and changes from 1 to 0 as we increase temperature. There are actually many such functions, but the most popular choice is the logistic function.
$$p(t) = \frac{1}{ 1 + e^{ \;\beta t } } $$
In this model, $\beta$ is the variable we are uncertain about. Below is the function plotted for $\beta = 1, 3, -5$.
Can you write Python code for it?

figsize(12, 3)

def logistic(x, beta):
    return 1.0 / (1.0 + np.exp(beta * x))

x = np.linspace(-4, 4, 100)
plt.plot(x, logistic(x, 1), label=r"$\beta = 1$")
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$")
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$")
plt.title("Logistic functon plotted for several value of $\\beta$ parameter", fontsize=14)
plt.legend();


figsize(12, 3)

def logistic(x, beta):
    return 1.0 / (1.0 + np.exp(beta * x))

x = np.linspace(-4, 4, 100)
plt.plot(x, logistic(x, 1), label=r"$\beta = 1$")
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$")
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$")
plt.legend();


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Later we will see how we use this to make predictions and test the appropriateness of our models.
Example: Bayesian A/B testing
A/B testing is a statistical design pattern for determining the difference of effectiveness between two different treatments. For example, a pharmaceutical company is interested in the effectiveness of drug A vs drug B. The company will test drug A on some fraction of their trials, and drug B on the other fraction (this fraction is often 1/2, but we will relax this assumption). After performing enough trials, the in-house statisticians sift through the data to determine which drug yielded better results.
Similarly, front-end web developers are interested in which design of their website yields more sales or some other metric of interest. They will route some fraction of visitors to site A, and the other fraction to site B, and record if the visit yielded a sale or not. The data is recorded (in real-time), and analyzed afterwards.
Often, the post-experiment analysis is done using something called a hypothesis test like difference of means test or difference of proportions test. This involves often misunderstood quantities like a "Z-score" and even more confusing "p-values" (please don't ask). If you have taken a statistics course, you have probably been taught this technique (though not necessarily learned this technique). And if you were like me, you may have felt uncomfortable with their derivation -- good: the Bayesian approach to this problem is much more natural.
A Simple Case
As this is a hacker book, we'll continue with the web-dev example. For the moment, we will focus on the analysis of site A only. Assume that there is some true $0 \lt p_A \lt 1$ probability that users who, upon shown site A, eventually purchase from the site. This is the true effectiveness of site A. Currently, this quantity is unknown to us.
Suppose site A was shown to $N$ people, and $n$ people purchased from the site. One might conclude hastily that $p_A = \frac{n}{N}$. Unfortunately, the observed...
Can you write Python code for it?

import pymc as pm

# The parameters are the bounds of the Uniform.
p = pm.Uniform('p', lower=0, upper=1)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Modeling approaches
A good starting point in Bayesian modeling is to think about how your data might have been generated. Put yourself in an omniscient position, and try to imagine how you would recreate the dataset.
In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:

We started by thinking "what is the best random variable to describe this count data?" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.

Next, we think, "Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameter $\lambda$.

Do we know $\lambda$? No. In fact, we have a suspicion that there are two $\lambda$ values, one for the earlier behaviour and one for the latter behaviour. We don't know when the behaviour switches though, but call the switchpoint $\tau$.

What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilities to positive real numbers. Well the exponential distribution has a parameter too, call it $\alpha$.

Do we know what the parameter $\alpha$ might be? No. At this point, we could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here.
What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was shown in...
Can you write Python code for it?

tau = pm.rdiscrete_uniform(0, 80)
print tau


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Modeling approaches
A good starting thought to Bayesian modeling is to think about how your data might have been generated. Position yourself in an omniscient position, and try to imagine how you would recreate the dataset.
In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:

We started by thinking "what is the best random variable to describe this count data?" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.

Next, we think, "Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameter $\lambda$.

Do we know $\lambda$? No. In fact, we have a suspicion that there are two $\lambda$ values, one for the earlier behaviour and one for the latter behaviour. We don't know when the behaviour switches though, but call the switchpoint $\tau$.

What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilities to positive real numbers. Well the exponential distribution has a parameter too, call it $\alpha$.

Do we know what the parameter $\alpha$ might be? No. At this point, we could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here.
What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was...
Can you write Python code for it?


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
More PyMC Tricks
Protip: Lighter deterministic variables with Lambda class
Sometimes writing a deterministic function using the @pm.deterministic decorator can seem like a chore, especially for a small function. I have already mentioned that elementary math operations can produce deterministic variables implicitly, but what about operations like indexing or slicing? Built-in Lambda functions can handle this with the elegance and simplicity required. For example,
beta = pm.Normal("coefficients", 0, size=(N, 1))
x = np.random.randn((N, 1))
linear_combination = pm.Lambda(lambda x=x, beta=beta: np.dot(x.T, beta))

Protip: Arrays of PyMC variables
There is no reason why we cannot store multiple heterogeneous PyMC variables in a Numpy array. Just remember to set the dtype of the array to object upon initialization. For example:
Can you write Python code for it?

N = 10
x = np.empty(N, dtype=object)
for i in range(0, N):
    x[i] = pm.Exponential('x_%i' % i, (i + 1) ** 2)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Next we need a dataset. After performing our coin-flipped interviews the researchers received 35 "Yes" responses. To put this into a relative perspective, if there truly were no cheaters, we should expect to see on average 1/4 of all responses being a "Yes" (half chance of having first coin land Tails, and another half chance of having second coin land Heads), so about 25 responses in a cheat-free world. On the other hand, if all students cheated, we should expect to see approximately 3/4 of all responses be "Yes".
The researchers observe a Binomial random variable, with N = 100 and p = observed_proportion with value = 35:
Can you write Python code for it?

X = 35

observations = pm.Binomial("obs", N, observed_proportion, observed=True,
                           value=X)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Next we sort each column by the posterior probabilities:
Can you write Python code for it?

ix = np.argsort(posterior_probability)
print "probb | defect "
for i in range(len(D)):
    print "%.2f  |   %d" % (posterior_probability[ix[i]], D[ix[i]])


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Note that the above plots are different (if you can think of a cleaner way to present this, please send a pull request and answer here!).
We wish to assess how good our model is. "Good" is a subjective term of course, so results must be relative to other models.
We will be doing this graphically as well, which may seem like an even less objective method. The alternative is to use Bayesian p-values. These are still subjective, as the proper cutoff between good and bad is arbitrary. Gelman emphasises that the graphical tests are more illuminating [7] than p-value tests. We agree.
The following graphical test is a novel data-viz approach to logistic regression. The plots are called separation plots[8]. For a suite of models we wish to compare, each model is plotted on an individual separation plot. I leave most of the technical details about separation plots to the very accessible original paper, but I'll summarize their use here.
For each model, we calculate the proportion of times the posterior simulation proposed a value of 1 for a particular temperature, i.e. compute $P( \;\text{Defect} = 1 | t, \alpha, \beta )$ by averaging. This gives us the posterior probability of a defect at each data point in our dataset. For example, for the model we used above:
Can you write Python code for it?

posterior_probability = simulations.mean(axis=0)
print "posterior prob of defect | realized defect "
for i in range(len(D)):
    print "%.2f                     |   %d" % (posterior_probability[i], D[i])


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Notice that as a result of N_B < N_A, i.e. we have less data from site B, our posterior distribution of $p_B$ is fatter, implying we are less certain about the true value of $p_B$ than we are of $p_A$.
With respect to the posterior distribution of $\text{delta}$, we can see that the majority of the distribution is above $\text{delta}=0$, implying there site A's response is likely better than site B's response. The probability this inference is incorrect is easily computable:
Can you write Python code for it?

# Count the number of samples less than 0, i.e. the area under the curve
# before 0, represent the probability that site A is worse than site B.
print "Probability site A is WORSE than site B: %.3f" % \
    (delta_samples < 0).mean()

print "Probability site A is BETTER than site B: %.3f" % \
    (delta_samples > 0).mean()


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Our posterior distribution puts most weight near the true value of $p_A$, but also some weights in the tails. This is a measure of how uncertain we should be, given our observations. Try changing the number of observations, N, and observe how the posterior distribution changes.
A and B Together
A similar analysis can be done for site B's response data to determine the analogous $p_B$. But what we are really interested in is the difference between $p_A$ and $p_B$. Let's infer $p_A$, $p_B$, and $\text{delta} = p_A - p_B$, all at once. We can do this using PyMC's deterministic variables. (We'll assume for this exercise that $p_B = 0.04$, so $\text{delta} = 0.01$, $N_B = 750$ (significantly less than $N_A$) and we will simulate site B's data like we did for site A's data )
Can you write Python code for it?

import pymc as pm
figsize(12, 4)

# these two quantities are unknown to us.
true_p_A = 0.05
true_p_B = 0.04

# notice the unequal sample sizes -- no problem in Bayesian analysis.
N_A = 1500
N_B = 750

# generate some observations
observations_A = pm.rbernoulli(true_p_A, N_A)
observations_B = pm.rbernoulli(true_p_B, N_B)
print "Obs from Site A: ", observations_A[:30].astype(int), "..."
print "Obs from Site B: ", observations_B[:30].astype(int), "..."

print observations_A.mean()
print observations_B.mean()

# Set up the pymc model. Again assume Uniform priors for p_A and p_B.
p_A = pm.Uniform("p_A", 0, 1)
p_B = pm.Uniform("p_B", 0, 1)

# Define the deterministic delta function. This is our unknown of interest.
@pm.deterministic
def delta(p_A=p_A, p_B=p_B):
    return p_A - p_B

# Set of observations, in this case we have two observation datasets.
obs_A = pm.Bernoulli("obs_A", p_A, value=observations_A, observed=True)
obs_B = pm.Bernoulli("obs_B", p_B, value=observations_B, observed=True)

# To be explained in chapter 3.
mcmc = pm.MCMC([p_A, p_B, delta, obs_A, obs_B])
mcmc.sample(20000, 1000)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
PyMC is concerned with two types of programming variables: stochastic and deterministic.

stochastic variables are variables that are not deterministic, i.e., even if you knew all the values of the variables' parents (if it even has any parents), it would still be random. Included in this category are instances of classes Poisson, DiscreteUniform, and Exponential.

deterministic variables are variables that are not random if the variables' parents were known. This might be confusing at first: a quick mental check is if I knew all of variable foo's parent variables, I could determine what foo's value is.

We will detail each below.
Initializing Stochastic variables
Initializing a stochastic variable requires a name argument, plus additional parameters that are class specific. For example:
some_variable = pm.DiscreteUniform("discrete_uni_var", 0, 4)
where 0, 4 are the DiscreteUniform-specific lower and upper bound on the random variable. The PyMC docs contain the specific parameters for stochastic variables. (Or use object??, for example pm.DiscreteUniform?? if you are using IPython!)
The name attribute is used to retrieve the posterior distribution later in the analysis, so it is best to use a descriptive name. Typically, I use the Python variable's name as the name.
For multivariable problems, rather than creating a Python array of stochastic variables, addressing the size keyword in the call to a Stochastic variable creates multivariate array of (independent) stochastic variables. The array behaves like a Numpy array when used like one, and references to its value attribute return Numpy arrays.
The size argument also solves the annoying case where you may have many variables $\beta_i, \; i = 1,...,N$ you wish to model. Instead of creating arbitrary names and variables for each one, like:
beta_1 = pm.Uniform("beta_1", 0, 1)
beta_2 = pm.Uniform("beta_2", 0, 1)
...

we can instead wrap them into a single variable:
betas = pm.Uniform("betas", 0, 1, size=N)

Calling random()
We can also call on a stochastic...
Can you write Python code for it?

lambda_1 = pm.Exponential("lambda_1", 1)  # prior on first behaviour
lambda_2 = pm.Exponential("lambda_2", 1)  # prior on second behaviour
tau = pm.DiscreteUniform("tau", lower=0, upper=10)  # prior on behaviour change

print "lambda_1.value = %.3f" % lambda_1.value
print "lambda_2.value = %.3f" % lambda_2.value
print "tau.value = %.3f" % tau.value
print

lambda_1.random(), lambda_2.random(), tau.random()

print "After calling random() on the variables..."
print "lambda_1.value = %.3f" % lambda_1.value
print "lambda_2.value = %.3f" % lambda_2.value
print "tau.value = %.3f" % tau.value


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
References

[1] Dalal, Fowlkes and Hoadley (1989),JASA, 84, 945-957.
[2] German Rodriguez. Datasets. In WWS509. Retrieved 30/01/2013, from http://data.princeton.edu/wws509/datasets/#smoking.
[3] McLeish, Don, and Cyntha Struthers. STATISTICS 450/850 Estimation and Hypothesis Testing. Winter 2012. Waterloo, Ontario: 2012. Print.
[4] Fonnesbeck, Christopher. "Building Models." PyMC-Devs. N.p., n.d. Web. 26 Feb 2013. http://pymc-devs.github.com/pymc/modelbuilding.html.
[5] Cronin, Beau. "Why Probabilistic Programming Matters." 24 Mar 2013. Google, Online Posting to Google . Web. 24 Mar. 2013. https://plus.google.com/u/0/107971134877020469960/posts/KpeRdJKR6Z1.
[6] S.P. Brooks, E.A. Catchpole, and B.J.T. Morgan. Bayesian animal survival estimation. Statistical Science, 15: 357–376, 2000
[7] Gelman, Andrew. "Philosophy and the practice of Bayesian statistics." British Journal of Mathematical and Statistical Psychology. (2012): n. page. Web. 2 Apr. 2013.
[8] Greenhill, Brian, Michael D. Ward, and Audrey Sacks. "The Separation Plot: A New Visual Method for Evaluating the Fit of Binary Models." American Journal of Political Science. 55.No.4 (2011): n. page. Web. 2 Apr. 2013.
Can you write Python code for it?

from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The 95% credible interval, or 95% CI, painted in purple, represents the interval, for each temperature, that contains 95% of the distribution. For example, at 65 degrees, we can be 95% sure that the probability of defect lies between 0.25 and 0.75.
More generally, we can see that as the temperature nears 60 degrees, the CI's spread out over [0,1] quickly. As we pass 70 degrees, the CI's tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how wide the posterior distribution is.
What about the day of the Challenger disaster?
On the day of the Challenger disaster, the outside temperature was 31 degrees Fahrenheit. What is the posterior distribution of a defect occurring,  given this temperature? The distribution is plotted below. It looks almost guaranteed that the Challenger was going to be subject to defective O-rings.
Can you write Python code for it?

figsize(12.5, 2.5)

prob_31 = logistic(31, beta_samples, alpha_samples)

plt.xlim(0.995, 1)
plt.hist(prob_31, bins=1000, normed=True, histtype='stepfilled')
plt.title("Posterior distribution of probability of defect, given $t = 31$")
plt.xlabel("probability of defect occurring in O-ring");


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The call to random stores a new value into the variable's value attribute. In fact, this new value is stored in the computer's cache for faster recall and efficiency.
Warning: Don't update stochastic variables' values in-place.
Straight from the PyMC docs, we quote [4]:

Stochastic objects' values should not be updated in-place. This confuses PyMC's caching scheme... The only way a stochastic variable's value should be updated is using statements of the following form:

    A.value = new_value

The following are in-place updates and should never be used:

    A.value += 3
    A.value[2,1] = 5
    A.value.attribute = new_attribute_value

Deterministic variables
Since most variables you will be modeling are stochastic, we distinguish deterministic variables with a pymc.deterministic wrapper. (If you are unfamiliar with Python wrappers (also called decorators), that's no problem. Just prepend the pymc.deterministic decorator before the variable declaration and you're good to go. No need to know more. ) The declaration of a deterministic variable uses a Python function:
@pm.deterministic
def some_deterministic_var(v1=v1,):
     #jelly goes here.

For all purposes, we can treat the object some_deterministic_var as a variable and not a Python function.
Prepending with the wrapper is the easiest way, but not the only way, to create deterministic variables: elementary operations, like addition, exponentials etc. implicitly create deterministic variables. For example, the following returns a deterministic variable:
Can you write Python code for it?


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The line fc*t_a + (1-fc)*sc contains the heart of the Privacy algorithm. Elements in this array are 1 if and only if i) the first toss is heads and the student cheated or ii) the first toss is tails, and the second is heads, and are 0 else. Finally, the last line sums this vector and divides by float(N), produces a proportion.
Can you write Python code for it?


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The observed frequency is:
Can you write Python code for it?

# Occurrences.mean is equal to n/N.
print "What is the observed frequency in Group A? %.4f" % occurrences.mean()
print "Does this equal the true frequency? %s" % (occurrences.mean() == p_true)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The remainder of this chapter examines some practical examples of PyMC and PyMC modeling:
Example: Challenger Space Shuttle Disaster <span id="challenger"/>
On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle program ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below (see [1]):
Can you write Python code for it?

figsize(12.5, 3.5)
np.set_printoptions(precision=3, suppress=True)
challenger_data = np.genfromtxt("data/challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
# drop the NA values
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]

# plot it, as a function of temperature (the first column)
print "Temp (F), O-Ring failure?"
print challenger_data

plt.scatter(challenger_data[:, 0], challenger_data[:, 1], s=75, color="k",
            alpha=0.5)
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Fahrenheit)")
plt.title("Defects of the Space Shuttle O-Rings vs temperature")


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The snaking-line is the sorted probabilities, blue bars denote defects, and empty space (or grey bars for the optimistic readers) denote non-defects.  As the probability rises, we see more and more defects occur. On the right hand side, the plot suggests that as the posterior probability is large (line close to 1), then more defects are realized. This is good behaviour. Ideally, all the blue bars should be close to the right-hand side, and deviations from this reflect missed predictions.
The black vertical line is the expected number of defects we should observe, given this model. This allows the user to see how the total number of events predicted by the model compares to the actual number of events in the data.
It is much more informative to compare this to separation plots for other models. Below we compare our model (top) versus three others:

the perfect model, which predicts the posterior probability to be equal to 1 if a defect did occur.
a completely random model, which predicts random probabilities regardless of temperature.
a constant model:  where $P(D = 1 \; | \; t) = c, \;\; \forall t$. The best choice for $c$ is the observed frequency of defects, in this case 7/23.
Can you write Python code for it?

figsize(11., 1.25)

# Our temperature-dependent model
separation_plot(posterior_probability, D)
plt.title("Temperature-dependent model")

# Perfect model
# i.e. the probability of defect is equal to if a defect occurred or not.
p = D
separation_plot(p, D)
plt.title("Perfect model")

# random predictions
p = np.random.rand(23)
separation_plot(p, D)
plt.title("Random model")

# constant model
constant_prob = 7. / 23 * np.ones(23)
separation_plot(constant_prob, D)
plt.title("Constant-prediction model")


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The special case when $N = 1$ corresponds to the Bernoulli distribution. There is another connection between Bernoulli and Binomial random variables. If we have $X_1, X_2, ... , X_N$ Bernoulli random variables with the same $p$, then $Z = X_1 + X_2 + ... + X_N \sim \text{Binomial}(N, p )$.
The expected value of a Bernoulli random variable is $p$. This can be seen by noting the more general Binomial random variable has expected value $Np$ and setting $N=1$.
Example: Cheating among students
We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed post-exam (answering without consequence), we will receive integer $X$ "Yes I did cheat" answers. We then find the posterior distribution of $p$, given $N$, some specified prior on $p$, and observed data $X$.
This is a completely absurd model. No student, even with a free-pass against punishment, would admit to cheating. What we need is a better algorithm to ask students if they had cheated. Ideally the algorithm should encourage individuals to be honest while preserving privacy. The following proposed algorithm is a solution I greatly admire for its ingenuity and effectiveness:

In the interview process for each student, the student flips a coin, hidden from the interviewer. The student agrees to answer honestly if the coin comes up heads. Otherwise, if the coin comes up tails, the student (secretly) flips the coin again, and answers "Yes, I did cheat" if the coin flip lands heads, and "No, I did not cheat", if the coin flip lands tails. This way, the interviewer does not know if a "Yes" was the result of a guilty plea, or a Heads on a second coin toss. Thus privacy is preserved and the researchers receive honest answers.

I call this the Privacy Algorithm. One could of course argue that the interviewers are still receiving false data since some Yes's are not confessions but instead randomness, but an alternative...
Can you write Python code for it?

import pymc as pm

N = 100
p = pm.Uniform("freq_cheating", 0, 1)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
The use of the deterministic wrapper was seen in the previous chapter's text-message example.  Recall the model for $\lambda$ looked like:
$$
\lambda =
\begin{cases}
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
\end{cases}
$$
And in PyMC code:
Can you write Python code for it?

import numpy as np
n_data_points = 5  # in CH1 we had ~70 data points

@pm.deterministic
def lambda_(tau=tau, lambda_1=lambda_1, lambda_2=lambda_2):
    out = np.zeros(n_data_points)
    out[:tau] = lambda_1  # lambda before tau is lambda1
    out[tau:] = lambda_2  # lambda after tau is lambda2
    return out


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
This is how we include data into our models: initializing a stochastic variable to have a fixed value.
To complete our text message example, we fix the PyMC variable observations to the observed dataset.
Can you write Python code for it?

# We're using some fake data here
data = np.array([10, 25, 15, 20, 35])
obs = pm.Poisson("obs", lambda_, value=data, observed=True)
print obs.value


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified $P(A)$. Our next goal is to include data/evidence/observations $X$ into our model.
PyMC stochastic variables have a keyword argument observed which accepts a boolean (False by default). The keyword observed has a very simple role: fix the variable's current value, i.e. make value immutable. We have to specify an initial value in the variable's creation, equal to the observations we wish to include, typically an array (and it should be an Numpy array for speed). For example:
Can you write Python code for it?

data = np.array([10, 5])
fixed_variable = pm.Poisson("fxd", 1, value=data, observed=True)
print "value: ", fixed_variable.value
print "calling .random()"
fixed_variable.random()
print "value: ", fixed_variable.value


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
Using these variables, we can return a possible realization of the observed proportion of "Yes" responses. We do this using a PyMC deterministic variable:
Can you write Python code for it?

@pm.deterministic
def observed_proportion(t_a=true_answers,
                        fc=first_coin_flips,
                        sc=second_coin_flips):

    observed = fc * t_a + (1 - fc) * sc
    return observed.sum() / float(N)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
We can present the above data better in a figure: I've wrapped this up into a separation_plot function.
Can you write Python code for it?

from separation_plot import separation_plot

figsize(11., 1.5)
separation_plot(posterior_probability, D)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
We combine the observations into the PyMC observed variable, and run our inference algorithm:
Can you write Python code for it?

# include the observations, which are Bernoulli
obs = pm.Bernoulli("obs", p, value=occurrences, observed=True)

# To be explained in chapter 3
mcmc = pm.MCMC([p, obs])
mcmc.sample(18000, 1000)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
We have our probabilities, but how do we connect them to our observed data? A Bernoulli random variable with parameter $p$, denoted $\text{Ber}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:
$$ \text{Defect Incident, $D_i$} \sim \text{Ber}( \;p(t_i)\; ), \;\; i=1..N$$
where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the above code we had to set the values of beta and alpha to 0. The reason for this is that if beta and alpha are very large, they make p equal to 1 or 0. Unfortunately, pm.Bernoulli does not like probabilities of exactly 0 or 1, though they are mathematically well-defined probabilities. So by setting the coefficient values to 0, we set the variable p to be a reasonable starting value. This has no effect on our results, nor does it mean we are including any additional information in our prior. It is simply a computational caveat in PyMC.
Can you write Python code for it?

p.value

# connect the probabilities in `p` with our observations through a
# Bernoulli random variable.
observed = pm.Bernoulli("bernoulli_obs", p, value=D, observed=True)

model = pm.Model([observed, beta, alpha])

# Mysterious code to be explained in Chapter 3
map_ = pm.MAP(model)
map_.fit()
mcmc = pm.MCMC(model)
mcmc.sample(120000, 100000, 2)


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
We have trained our model on the observed data, now we can sample values from the posterior. Let's look at the posterior distributions for $\alpha$ and $\beta$:
Can you write Python code for it?

alpha_samples = mcmc.trace('alpha')[:, None]  # best to make them 1d
beta_samples = mcmc.trace('beta')[:, None]

figsize(12.5, 6)

# histogram of the samples:
plt.subplot(211)
plt.title(r"Posterior distributions of the variables $\alpha, \beta$")
plt.hist(beta_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\beta$", color="#7A68A6", normed=True)
plt.legend()

plt.subplot(212)
plt.hist(alpha_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\alpha$", color="#A60628", normed=True)
plt.legend();


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
We plot the posterior distribution of the unknown $p_A$ below:
Can you write Python code for it?

figsize(12.5, 4)
plt.title("Posterior distribution of $p_A$, the true effectiveness of site A")
plt.vlines(p_true, 0, 90, linestyle="--", label="true $p_A$ (unknown)")
plt.hist(mcmc.trace("p")[:], bins=25, histtype="stepfilled", normed=True)
plt.legend()


I am working on the file "Chapter2_MorePyMC/Chapter2.ipynb".
The first task is:
With regards to the above plot, we are still pretty uncertain about what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.05 to 0.35 (marked by the solid lines). This is pretty good, as a priori we had no idea how many students might have cheated (hence the uniform distribution for our prior). On the other hand, it is also pretty bad since there is a .3 length window the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency?
I would argue, yes, we have discovered something. It is implausible, according to our posterior, that there are no cheaters, i.e. the posterior assigns low probability to $p=0$. Since we started with a uniform prior, treating all values of $p$ as equally plausible, but the data ruled out $p=0$ as a possibility, we can be confident that there were cheaters.
This kind of algorithm can be used to gather private information from users and be reasonably confident that the data, though noisy, is truthful.
Alternative PyMC Model
Given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes:
\begin{align}
P(\text{"Yes"}) &=  P( \text{Heads on first coin} )P( \text{cheater} ) + P( \text{Tails on first coin} )P( \text{Heads on second coin} ) \\
& = \frac{1}{2}p + \frac{1}{2}\frac{1}{2}\\
& = \frac{p}{2} + \frac{1}{4}
\end{align}
Thus, knowing $p$ we know the probability a student will respond "Yes". In PyMC, we can create a deterministic function to evaluate the probability of responding "Yes", given $p$:
Can you write Python code for it?

p = pm.Uniform("freq_cheating", 0, 1)

@pm.deterministic
def p_skewed(p=p):
    return 0.5 * p + 0.25


I am working on the file "Chapter3-1.ipynb".
The first task is:
3.1.2 Assessing the accuracy of coefficient estimates
The particular minima that is found through least squares regression is effected by the particular sample of the population that is observed and utilized for estimating the coefficients of the underlying population model.
To see this we can generate a synthetic population based on an ideal model plus noise.  Here we can peek at the entire population (which in most settings cannot be observed).  We then take samples of this population and fit a regression to those.  We can then see how these regression lines differ from the population regression line, which is not exactly the ideal model.
Can you write Python code for it?

def model (x):
    return 3*x + 2

def population(n, noise_scale = 1):
    sample = ML_Table.runiform('x', n, -2, 2)
    noise = ML_Table.rnorm('e', n, sd=noise_scale)
    sample['Y'] = sample.apply(model, 'x') + noise['e']
    return sample

data = population(100, 2)

data.scatter('x')

ax = data.plot_fit('Y', data.linear_regression('Y').model)

data.linear_regression('Y').params

# A random sample of the population
sample = data.sample(10)
sample.plot_fit('Y', sample.linear_regression('Y').model)

nsamples = 5
ax = data.plot_fit('Y', data.linear_regression('Y').model, linewidth=3)
for s in range(nsamples):
    fit = data.sample(10).linear_regression('Y').model
    ax.plot([-2, 2], [fit(-2), fit(2)], linewidth=1)


I am working on the file "Chapter3-1.ipynb".
The first task is:
3.1.3 Assessing the Acurracy of the Model
Once we have rejected the null hypothesis in favor of the alternative hypothesis, it is natural to want to quantify the extent to which the model fits the data. The quality of a linear regression fit is typically assessed using two related quantities: the residual standard error (RSE) and the $R^2$ statistic.
The RSE provides an absolute measure of lack of fit of the model to the data.
Can you write Python code for it?

adver_model = advertising.regression_1d('Sales', 'TV')
advertising.RSE_model('Sales', adver_model, 'TV')


I am working on the file "Chapter3-1.ipynb".
The first task is:
Acquiring and seeing trends in multidimensional data
Can you write Python code for it?

# Getting the data
advertising = ML_Table.read_table("./data/Advertising.csv")
advertising.relabel(0, "id")


I am working on the file "Chapter3-1.ipynb".
The first task is:
FIGURE 3.1. For the Advertising data, the least squares fit for the regression of sales onto TV is shown. The fit is found by minimizing the sum of squared errors.
Each line segment represents an error, and the fit makes a compromise by averaging their squares. In this case a linear fit captures the essence of the relationship, although it is somewhat deficient in the left of the plot.
Can you write Python code for it?

ax = advertising.plot_fit_1d('Sales', 'TV', advertising.regression_1d('Sales', 'TV'))
_ = ax.set_xlim(-20,300)

lr = advertising.linear_regression('Sales', 'TV')

advertising.plot_fit_1d('Sales', 'TV', lr.model)

advertising.lm_summary_1d('Sales', 'TV')


I am working on the file "Chapter3-1.ipynb".
The first task is:
Figure 3.2 - RSS and least squares regression
Regression using least squares finds parameters $b0$ and $b1$ tohat minimize the RSS.  The least squares regression line estimates the population regression line.
Figure 3.2 shows (what is claimed to be) the RSS contour and surface around the regression point.  The computed analog of Figure 3.2 is shown below, with the role of $b0$ and $b1$ reversed to match the tuple returned from the regression method. The plot is the text is incorrect is some important ways.  The RSS is not radially symmetric around ($b0$, $b1$).  Lines with a larger intercept and smaller slope or vice versa are very close to the minima, i.e., the surface is nearly flat along the upper-left to lower-right diagonal, especially where fit is not very good, since the output depends on more than this one input.
Just because a process minimizes the error does not mean that the minima is sharply defined or that the error surface has no structure.  Below we go a bit beyond the text to illustrate this more fully.
Can you write Python code for it?

ax = advertising.RSS_contour('Sales', 'TV', sensitivity=0.2)

ax = advertising.RSS_wireframe('Sales', 'TV', sensitivity=0.2)

# The minima point
advertising.linear_regression('Sales', 'TV').params

# Some other points along the trough
points = [(0.042, 8.0), (0.044, 7.6), (0.050, 6.6), (0.054, 6.0)]
ax = advertising.RSS_contour('Sales', 'TV', sensitivity=0.2)
ax.plot([b0 for b1, b0 in points], [b1 for b1, b0 in points], 'ro')
[advertising.RSS('Sales', b0 + b1*advertising['TV']) for b1,b0 in points]

# Models as lines corresponding to points along the near-minimal vectors.
ax = advertising.plot_fit_1d('Sales', 'TV', advertising.linear_regression('Sales', 'TV').model)
for b1, b0 in points:
    fit = lambda x: b0 + b1*x
    ax.plot([0, 300], [fit(0), fit(300)])
_ = ax.set_xlim(-20,300)


I am working on the file "Chapter3-1.ipynb".
The first task is:
Let $\hat{y}_i =  \hat{\beta_0} + \hat{\beta_1}x_i$ be the prediction for $Y$ based on the $i$-th value of $X$.
Can you write Python code for it?

# Get the actual parameters that are captured within the model
advertising.regression_1d_params('Sales', 'TV')

# Regression yields a model.  The computational representation of a model is a function
# That can be applied to an input, 'TV' to get an estimate of an output, 'Sales'
advertise_model_tv = advertising.regression_1d('Sales', 'TV')

# Sales with no TV advertising
advertise_model_tv(0)

# Sales with 100 units TV advertising
advertise_model_tv(100)

# Here's the output of the model applied to the input data
advertise_model_tv(advertising['TV'])


I am working on the file "Chapter3-1.ipynb".
The first task is:
The $R^2$ statistic provides an alternative measure of fit. It takes the form of a proportion—the proportion of variance explained—and so it always takes on a value between 0 and 1, and is independent of the scale of Y.
To calculate $R^2$, we use the formula
$R^2 = \frac{TSS−RSS}{TSS} = 1 − \frac{RSS}{TSS}$
where $TSS = \sum (y_i - \bar{y})^2$ is the total sum of squares.
Can you write Python code for it?

advertising.R2_model('Sales', adver_model, 'TV')

# the other models of advertising suggest that there is more going on
advertising.R2_model('Sales', adver_model, 'Radio')

advertising.R2_model('Sales', adver_model, 'Newspaper')


I am working on the file "Chapter3-1.ipynb".
The first task is:
The residual is not very useful directly because balances over-estimates and under-estimates to produce the best overall estimate with the least error.  We can understand the overall goodness of fit by the residual sum of squares - RSS.
Can you write Python code for it?

# Residual Sum of Squares
RSS = sum(residual*residual)
RSS

# This is common enough that we have it provided as a method
advertising.RSS_model('Sales', advertising.regression_1d('Sales', 'TV'), 'TV')

# And we should move toward a general regression framework
advertising.RSS_model('Sales', advertising.linear_regression('Sales', 'TV').model, 'TV')


I am working on the file "Chapter3-1.ipynb".
The first task is:
The residual is the difference between the model output and the observed output
Can you write Python code for it?

residual = advertising['Sales'] - advertise_model_tv(advertising['TV'])
residual


I am working on the file "Chapter3-1.ipynb".
The first task is:
We can look at how well each of these inputs predict the output by visualizing the residuals.
The magnitude of the RSS gives a sense of the error.
Can you write Python code for it?

for mode, mdl in zip(advertising_models['Input'], advertising_models['Model']) :
    advertising.plot_fit_1d('Sales', mode, mdl)

# RSS at arbitrary point
res = lambda b0, b1: advertising.RSS('Sales', b0 + b1*advertising['TV'])

res(7.0325935491276965, 0.047536640433019729)


I am working on the file "Chapter3-2.ipynb".
The first task is:
3.2.1 Estimating the Regression Coefficients
The multiple linear regression model takes the form
$Y = β_0 + β_1X_1 +···+β_{p}X_{p} + ε$,
where $X_j$ represents the jth predictor and $β_j$ quantifies the association between that variable and the response. We interpret βj as the average effect on Y of a one unit increase in Xj, holding all other predictors fixed.
In the advertising example, this becomes
$sales=  β0 + β1×TV + β2×radio + β3×newspaper + ε$.
Can you write Python code for it?

advertising.linear_regression('Sales').params

adver_model = advertising.linear_regression('Sales').model

adver_model(0,0,0)


I am working on the file "Chapter3-2.ipynb".
The first task is:
3.2.2 Some Important questions

Is at least one of the predictors X1 , X2 , . . . , Xp useful in predicting the response?
Do all the predictors help to explain Y, or is only a subset of the predictors useful?
How well does the model fit the data?
Given a set of predictor values, what response value should we predict,
and how accurate is our prediction?

Correlation matrix
Above shows that spending on newspaper appears to have no effect on sales.  The apparent effect when looking at newspaper versus sales in isolation is capturing the tendency to spend more on newspaper when spending more on radio.
Can you write Python code for it?


I am working on the file "Chapter3-2.ipynb".
The first task is:
F-statistic
$F = \frac{(TSS - RSS)/p}{RSS/(n - p - 1)}$
When there is no relationship between the response and predictors, one would expect the F-statistic to take on a value close to 1. On the other hand, if Ha is true, then E{(TSS − RSS)/p} > σ2, so we expect F to be greater than 1.
Can you write Python code for it?

advertising.F_model('Sales', adver_model)

advertising.lm_fit('Sales', adver_model)

# Using this tool for the 1D model within the table
advertising.lm_fit('Sales', advertising.regression_1d('Sales', 'TV'), 'TV')


I am working on the file "Chapter3-2.ipynb".
The first task is:
Multiple regression inference and goodness of fit
At this point "ISL" skips over how to compute the standard error of the multiple regression parameters - relying on R to just produce the answer.  It requires some matrix notation and a numerical computation of the matrix inverse, but involves a bunch of standard terminology that is specific to the inference aspect, as opposed to the general notion in linear algebra of approximating a function over a basis.
A nice treatment can be found at this reference
Can you write Python code for it?

# response vector
Y = advertising['Sales']
labels = [lbl for lbl in advertising.labels if lbl != 'Sales']
p = len(labels) # number of parameter
n = len(Y)  # number of observations
labels

# Transform the table into a matrix
advertising.select(labels).rows

# Design matrix
X = np.array([np.append([1], row) for row in advertising.select(labels).rows])

# slope vector
b0, slopes = advertising.linear_regression('Sales').params
b = np.append([b0], slopes)

np.shape(X), np.shape(b)

# residual
res = np.dot(X, b) - advertising['Sales']

# Variance of the residual
sigma2 = sum(res**2)/(n-p-1)
sigma2

Xt = np.transpose(X)

# The matrix that needs to be inverted is only p x p
np.dot(Xt, X)

np.shape(np.dot(Xt, X))

# standard error matrix
SEM = sigma2*np.linalg.inv(np.dot(Xt, X))
SEM

# variance of the coefficients are the diagonal elements
variances = [SEM[i,i] for i in range(len(SEM))]
variances

# standard error of the coeficients
SE = [np.sqrt(v) for v in variances]
SE

# t-statistics
b/SE

advertising.linear_regression('Sales').summary()

advertising.RSS_model('Sales', adver_model)

advertising.R2_model('Sales', adver_model)


I am working on the file "Chapter3-2.ipynb".
The first task is:
Sometimes we want to test that a particular subset of q of the coefficients are zero. This corresponds to a null hypothesis
H0 : $β_{p−q+1} =β_{p−q+2} =...=β_{p} =0$
where for convenience we have put the variables chosen for omission at the end of the list. In this case we fit a second model that uses all the variables except those last q. Suppose that the residual sum of squares for that model is $RSS_0$. Then the appropriate F-statistic is
$F = \frac{(RSS_0 − RSS)/q}{RSS/(n−p−1)}$.
Can you write Python code for it?

ad2_model = ad2.linear_regression('Sales').model
ad2.lm_fit('Sales', ad2_model)

RSS0 = ad2.RSS_model('Sales', ad2_model)
RSS = advertising.RSS_model('Sales', adver_model)
((RSS0 - RSS)/1)/(advertising.num_rows - 3 - 1)


I am working on the file "Chapter3-2.ipynb".
The first task is:
Variable selection

Forward selection - start with null model and add predictors one at a time using the variable that result in the lowest RSS
Backward selection - start with all variables and iteratively remove the one with the largest P-value (smallest T-statistic)
Mixed selection - add like forward but skip ones with too high a P-value
Can you write Python code for it?

input_labels = [lbl for lbl in advertising.labels if lbl != 'Sales']
fwd_labels = ['Sales']
for lbl in input_labels:
    fwd = advertising.select(fwd_labels + [lbl])
    model = fwd.linear_regression('Sales').model
    print(lbl, fwd.RSS_model('Sales', model))


I am working on the file "Chapter3-2.ipynb".
The first task is:
Visualizing a 2D regression
Can you write Python code for it?

ad2 = advertising.drop('Newspaper')
ad2.linear_regression('Sales').summary()

# Linear model with two input variables is a plane
ad2.plot_fit('Sales', ad2.linear_regression('Sales').model, width=8)


I am working on the file "Chapter3.ipynb".
The first task is:
A function can call other functions and can even call on itself. If we wanted to create a function that calculated
${n\choose k} = \frac{n!}{k!(n-k)!}$ we could first define a function that calculated the factorial which we use to calculate the original function
Can you write Python code for it?

def factorial(x):
    result = 1
    for i in range(1, x + 1): # Remember that range(a,b) creates the list [a, a + 1, ..., a - 1]
        result *= i
    return result

def choose(n, k):
    result = factorial(n)/(factorial(k)*factorial(n - k))
    return result

print choose(10,5)


I am working on the file "Chapter3.ipynb".
The first task is:
As you can see some can be combined to mean greater than or equal and not equal. Try it out and see which combinations work and which don't. Using these we can write some simple conditional statements:
Can you write Python code for it?

if 3 > 6:
    print "{} is larger than {}".format(3, 6)
elif 3 < 5:
    print "{} is smaller than {}".format(3, 5)
elif 3 != 12:
    print "{} is not equal to {}".format(3, 12)
else:
    print "Tomatopaste"


I am working on the file "Chapter3.ipynb".
The first task is:
Importing libraries
Even though there are many functions in python from the start, there are many more we might need. The function factorial does not exist in python from the start which meant we had to write it, and it is the same with many other useful functions. Most of these have actually already been written by someone else. To include them we need to import the library that they are in. A library can be seen as a set of functions and when we import the library we make the functions available to the interpreter. The function factorial exists in the library math which we can import and use the function from there. To use a function from a library we first have to state the library and then that function with a dot in between
LIBRARY.FUNCTION
Can you write Python code for it?


I am working on the file "Chapter3.ipynb".
The first task is:
Notice that in this case we do not use the function by first friting the name of the library.
Conditional statements
Conditional statements are often called if-statements. You write them as follows:
if CONDITION:
    do this
elif ANOTER_CONDITION:
    do that
else:
    this should be done

the interpreter goes through each statement until it finds a condition that is true and executes the code in that statement. After that, the interpreter goes past the other elif and else statements. If None of the statementents are true it executes the code in the else part. When using if-statements we need to know some things about relational symbols first. These are <, >, ==, and !. < means less than, > means greater than, == means equal (not the same as = which is assignment) and ! means not.
Can you write Python code for it?


I am working on the file "Chapter3.ipynb".
The first task is:
Notice that the indentation is crucial. This is how python knows that certain lines of code belong to the loop. The standard is to use four spaces for indentation and I strongly urge you to do the same, but as long as the indentation is consistent within each block the code will run. Blocks of code can be conditions, loops or functions and they can be within other blocks of code.
Can you write Python code for it?

anotherList = [[1,2],[3,2],[3,4]]
for first in anotherList:
    for second in first: #first is a list so it can be used in a loop
        print second


I am working on the file "Chapter3.ipynb".
The first task is:
Often you want to just repeat the code a certain amount of times or do it from, say, $0$ to $n$. To do this we can use the range function.
Can you write Python code for it?

range(10)

range(1,11) #instead create a list from 1 up to but not including 11


I am working on the file "Chapter3.ipynb".
The first task is:
The return statement means that the function will return that as the answer. It is not neccessary to include it as some functions might print things instead, but it is very common for it to be included. If we wish to use the function we simply write
Can you write Python code for it?

theResult = some_function(3,1)
print theResult


I am working on the file "Chapter3.ipynb".
The first task is:
To use this to find the factorial of $54$ we could write
Can you write Python code for it?

solution = 1
for i in range(1,55):
    solution = solution * i
print solution


I am working on the file "Chapter3.ipynb".
The first task is:
We can also import specific functions from a library by writing
Can you write Python code for it?


I am working on the file "Chapter3.ipynb".
The first task is:
main()
Can you write Python code for it?

main()

main()

main()

M = [0,1,2,3]

main()

main()

# -*- coding: utf-8 -*-
#
# 最尤推定による正規分布の推定
#
# 2015/04/23 ver1.0
#

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from pandas import Series, DataFrame

from numpy.random import normal
from scipy.stats import norm

def gauss():
    fig = plt.figure()
    for c, datapoints in enumerate([2,4,10,100]): # サンプル数
        ds = normal(loc=0, scale=1, size=datapoints)
        mu = np.mean(ds)                # 平均の推定値
        sigma = np.sqrt(np.var(ds))     # 標準偏差の推定値

        subplot = fig.add_subplot(2,2,c+1)
        subplot.set_title("N=%d" % datapoints)
        # 真の曲線を表示
        linex = np.arange(-10,10.1,0.1)
        orig = norm(loc=0, scale=1)
        subplot.plot(linex, orig.pdf(linex), color='green', linestyle='--')
        # 推定した曲線を表示
        est = norm(loc=mu, scale=np.sqrt(sigma))
        label = "Sigma=%.2f" % sigma
        subplot.plot(linex, est.pdf(linex), color='red', label=label)
        subplot.legend(loc=1)
        # サンプルの表示
        subplot.scatter(ds, orig.pdf(ds), marker='o', color='blue')
        subplot.set_xlim(-4,4)
        subplot.set_ylim(0)
    fig.show()

if __name__ == '__main__':
    gauss()


I am working on the file "Chapter3.ipynb".
The first task is:
while lists are the most common to loop over, you can loop over any object that is ordered.
Can you write Python code for it?

simpletuple = (2,3,"2")
for i in simpletuple:
    print i

some_string = "Dust tastes good"
for i in some_string:
    print i


I am working on the file "Chapter3.ipynb".
The first task is:
標準偏差が小さくなるのは裾野のデータの発生確率が低いため

推定量

何らかの理屈にもとづいて推定値を計算する方法が得られた時に、その計算方法を推定量と呼ぶらしい
方法なのに量？
一致性と不偏性を持つのが良い推定量
一致性
データを大きくしていった時に真の値に近づいていくこと
一致性を持つ推定量を一致推定量というらしい
不偏性
何回か取得した推定値の平均が真の値に近づいていくこと
Can you write Python code for it?

# -*- coding: utf-8 -*-
#
# 推定量の一致性と不偏性の確認
#
# 2015/06/01 ver1.0
#

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from pandas import Series, DataFrame
from numpy.random import normal

def draw_subplot(subplot, linex1, liney1, linex2, liney2, ylim):
    subplot.set_ylim(ylim)
    subplot.set_xlim(min(linex1), max(linex1)+1)
    subplot.scatter(linex1, liney1)
    subplot.plot(linex2, liney2, color='red', linewidth=4, label="mean")
    subplot.legend(loc=0)

def bias():
    mean_linex = []
    mean_mu = []
    mean_s2 = []
    mean_u2 = []
    raw_linex = []
    raw_mu = []
    raw_s2 = []
    raw_u2 = []
    for n in np.arange(2,51): # 観測データ数Nを変化させて実行
        for c in range(2000): # 特定のNについて2000回の推定を繰り返す
            ds = normal(loc=0, scale=1, size=n)
            raw_mu.append(np.mean(ds))
            raw_s2.append(np.var(ds))
            raw_u2.append(np.var(ds)*n/(n-1))
            raw_linex.append(n)
        mean_mu.append(np.mean(raw_mu)) # 標本平均の平均
        mean_s2.append(np.mean(raw_s2)) # 標本分散の平均
        mean_u2.append(np.mean(raw_u2)) # 不偏分散の平均
        mean_linex.append(n)

    # プロットデータを40個に間引きする
    raw_linex = raw_linex[0:-1:50]
    raw_mu = raw_mu[0:-1:50]
    raw_s2 = raw_s2[0:-1:50]
    raw_u2 = raw_u2[0:-1:50]

    # 標本平均の結果表示
    fig1 = plt.figure()
    subplot = fig1.add_subplot(1,1,1)
    subplot.set_title('Sample mean')
    draw_subplot(subplot, raw_linex, raw_mu, mean_linex, mean_mu, (-1.5,1.5))

    # 標本分散の結果表示
    fig2 = plt.figure()
    subplot = fig2.add_subplot(1,1,1)
    subplot.set_title('Sample variance')
    draw_subplot(subplot, raw_linex, raw_s2, mean_linex, mean_s2, (-0.5,3.0))

    # 不偏分散の結果表示
    fig3 = plt.figure()
    subplot = fig3.add_subplot(1,1,1)
    subplot.set_title('Unbiased variance')
    draw_subplot(subplot, raw_linex, raw_u2, mean_linex, mean_u2, (-0.5,3.0))

    fig1.show()
    fig2.show()
    fig3.show()

if __name__ == '__main__':
    bias()


I am working on the file "Chapter3.ipynb".
The first task is:
標準偏差の推定値は実際より小さくなるらしい
けど、↑ではなってないｗ
Can you write Python code for it?


I am working on the file "Chapter3/3-Sets_and_Maps.ipynb".
The first task is:
Maps
Searching for data items based on unique key values is a very common application in computer science. ADT for this is often called Map or Dictionaries. It's is like database entities we store id/key for every value.
In this section we define our own Map ADT and then provide an implementation using a list, later we will implement and evaluate the map using a variety of data structures.
Map ADT implementation can be found linearmap.py.
Multi-Dimensional Arrays
Multi-dimensional arrays are with more than 2 dimensional arrays. In fact we can build as many dimensionsion as we want.
Data Organization
Array Storage:
A one-dimensional array is commonly used  to physically store arrays of higher dimensions. How can the individual elements of the 2D table be stored in the one-dimensional structure while maintaining direct access to the individual table elements?

row-major order
column-major order

In row-major order, the individual rows are stored sequentially, on at a time.

Index Computation:
Since multi-dimensional arrays are created and managed by instrictions in the programming language, accessing an individual element must also be handled by the language. Given a 2D array of size mxn and using row-major ordering, an equation can be derived to computer this offset.
index(i1, i2, .... , in) = i1 * f1 + i2 * f2 + ... + i(n-1) * f(n-1) + in * fn\
Can you write Python code for it?

def func( *args):
    print("number of arguments: ", len(args))
    _sum = 0
    for value in args:
        _sum += value

    print ("Sum of the arguments: ", _sum)

func(12)

func(5,8,2)


I am working on the file "Chapter3/Chapter3.ipynb".
The first task is:
## Plots with Bokeh
Can you write Python code for it?

from ipywidgets import interact
from bokeh.io import push_notebook
from bokeh.plotting import figure, output_file, show
from bokeh.plotting import output_notebook

output_notebook()


I am working on the file "Chapter3/Chapter3.ipynb".
The first task is:
1-D Random Number generation
generate a uniform random number and apply the inverse CDF of the distribution of the type of random variable you would like to simulate.
This example shows how to clone a disribution and compute the inverse cdf with interpolation.
Can you write Python code for it?

from scipy import stats, interpolate
from astroML.plotting import hist
from astroML.density_estimation import EmpiricalDistribution

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------å
# Create a distribution and clone it
Ndata = 1000
Nclone = 100000
np.random.seed(0)

# generate an 'observed' bimodal distribution with 10000 values
dists = (stats.norm(-1.3, 0.5), stats.norm(1.3, 0.5))
fracs = (0.6, 0.4)
x = np.hstack((d.rvs(f * Ndata) for d, f in zip(dists, fracs)))

# We can clone the distribution easily with this function
x_cloned = EmpiricalDistribution(x).rvs(Nclone)

# compute the KS test to check if they're the same
D, p = stats.ks_2samp(x, x_cloned)
print "KS test: D = %.2g; p = %.2g" % (D, p)

#------------------------------------------------------------
# For the sake of this example, we need to calculate some
#  of the partial steps used by EmpiricalDistribution

# create a cumulative distribution
x.sort()
Px_cuml = np.linspace(0, 1, Ndata)

# set up an interpolation of the inverse cumulative distribution
tck = interpolate.splrep(Px_cuml, x)

# sample evenly along the cumulative distribution, and interpolate
Px_cuml_sample = np.linspace(0, 1, 10 * Ndata)
x_sample = interpolate.splev(Px_cuml_sample, tck)

#------------------------------------------------------------
# Plot the cloned distribution and the procedure for obtaining it
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(hspace=0.3, left=0.1, right=0.95,
                    bottom=0.08, top=0.92)

indices = np.linspace(0, Ndata - 1, 20).astype(int)

# plot a histogram of the input
ax =...


I am working on the file "Chapter3/Chapter3.ipynb".
The first task is:
Correlation Coefficients
This section discusses the strengths and weaknesses of three techniques to measure to the sample correlation coefficient.

Pearson

Measurement errors are not used
Sensitive to outliers
Measures linear relationship

Spearman

Based on ranks (exactly Pearsons but on the ranks instead of the actual data).
Measures any kind of monotonic dependency

Kendall

Based on ranks (takes different between concordant and discordant pairs)
harder to compute than Spearman.
approaches normality faster than spearman

The nonparametric methods don't require knowledge of the distribution of the data to determine the distribution of the sample.
Ranks limit outliers to the value of their ranks.
Can you write Python code for it?

from astroML.stats.random import bivariate_normal
from astroML.decorators import pickle_results

# percent sign must be escaped if usetex=True
import matplotlib
if matplotlib.rcParams.get('text.usetex'):
    pct = '\%'
else:
    pct = '%'

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Set parameters for the distributions
Nbootstraps = 5000
N = 1000

sigma1 = 2.0
sigma2 = 1.0
mu = (10.0, 10.0)
alpha_deg = 45.0
alpha = alpha_deg * np.pi / 180
f = 0.01

#------------------------------------------------------------
# sample the distribution
# without outliers and with outliers
np.random.seed(0)
X = bivariate_normal(mu, sigma1, sigma2, alpha, N)

X_out = X.copy()
X_out[:int(f * N)] = bivariate_normal(mu, 2, 5,
                                      45 * np.pi / 180., int(f * N))

# true values of rho (pearson/spearman r) and tau
# tau value comes from Eq. 41 of arXiv:1011.2009
rho_true = 0.6
tau_true = 2 / np.pi * np.arcsin(rho_true)

#------------------------------------------------------------
# Create a function to compute the statistics.  Since this
#  takes a while, we'll use the "pickle_results" decorator
#  to save the results of the computation to disk
@pickle_results('fig_correlations_dump.pkl')
def compute_results(N, Nbootstraps):
    results = np.zeros((3, 2, Nbootstraps))

    for k in range(Nbootstraps):
        ind = np.random.randint(N, size=N)
        for j, data in enumerate([X, X_out]):
            x = data[ind, 0]
            y = data[ind, 1]
            for i, statistic in enumerate([stats.pearsonr,
                                           stats.spearmanr,
     ...


I am working on the file "Chapter3/Chapter3.ipynb".
The first task is:
Here is an example of computing the components of a bivariate gaussian using a sample of 1000 data values with two levels of contamination. Figure 3.23.
The robust method implemented in astroML uses the robust estimator of the variance $\sigma_{G}$ and takes into account the correlation between parameters.
Can you write Python code for it?

import numpy as np
from scipy import stats
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse
from astroML.stats import fit_bivariate_normal
from astroML.stats.random import bivariate_normal

# percent sign needs to be escaped if usetex is activated
import matplotlib
if matplotlib.rcParams.get('text.usetex'):
    pct = r'\%'
else:
    pct = r'%'

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

N = 1000

sigma1 = 2.0
sigma2 = 1.0
mu = [10, 10]
alpha_deg = 45.0
alpha = alpha_deg * np.pi / 180

#------------------------------------------------------------
# Draw N points from a multivariate normal distribution
#
#   we use the bivariate_normal function from astroML.  A more
#   general function for this is numpy.random.multivariate_normal(),
#   which requires the user to specify the full covariance matrix.
#   bivariate_normal() generates this covariance matrix for the
#   given inputs

np.random.seed(0)
X = bivariate_normal(mu, sigma1, sigma2, alpha, N)

#------------------------------------------------------------
# Create the figure showing the fits
fig = plt.figure(figsize=(15, 7.5)) ##3 times the original size
fig.subplots_adjust(left=0.1, right=0.95, wspace=0.05,
                    bottom=0.15, top=0.95)

# We'll create two figures, with two levels of contamination
for i, f in enumerate([0.05, 0.15]):
    ax = fig.add_subplot(1, 2, i + 1)

    # add outliers distributed using a bivariate normal.
    X[:int(f * N)] = bivariate_normal((10, 10), 2, 4,
                                      45 * np.pi / 180., int(f * N))
    x, y = X.T

    # compute the non-robust statistics
    (mu_nr, sigma1_nr,
     sigma2_nr,...


I am working on the file "Chapter3/Chapter3.ipynb".
The first task is:
Now we repeat the plot with seaborn
Can you write Python code for it?

import seaborn as sns
import pandas as pd
data = (x,y)
data = np.array(data)
data = data.T
df = pd.DataFrame(data, columns=["x", "y"])
g = sns.jointplot(x="x", y="y",data=df, kind="hex", marginal_kws={"bins":41*41/2}, gridsize=41*41/2, xlim=(0,2), ylim=(0,2))

cond1 = (y>=0.5-2./41.) & (y<=0.5+2./41.)
cond2 = (y>=1.0-2./41.) & (y<=1.0+2./41.)
cond3 = (y>=1.5-2./41.) & (y<=1.5+2./41.)

sns.__version__

weights = np.ones(10000)*1./10000.

g1 = sns.distplot(x[cond1],hist=True, color='cyan', kde=False, hist_kws={'weights':weights[cond1]})
g2 = sns.distplot(x[cond2],hist=True, color='green', kde=False, hist_kws={'weights':weights[cond2]})
g3 = sns.distplot(x[cond3],hist=True, color='red', kde=False, hist_kws={'weights':weights[cond3]})


I am working on the file "Chapter3/chapter3.2/README.ipynb".
The first task is:
分别画出角速度随时间变化,角度随时间变化，以及相空间轨迹
Can you write Python code for it?

# Omega
ddp(ans2Omega,["ForwardEuler"], xLabel = "t/s", yLabel = "Omega/ 1/s")
ddp(ans1Omega,["RK4"], xLabel = "t/s", yLabel = "Omega/ 1/s")
done()

# Theta
ddp(ans2Theta,["ForwardEuler"], xLabel = "t/s", yLabel = "Theta/ 1")
ddp(ans1Theta,["RK4"], xLabel = "t/s", yLabel = "Theta/ 1")
done()

# phase diag
easyPlotXYHoldON(getXiList(ans2,2),getXiList(ans2,1), "ForwardEuler","Theta/ 1","Omega/ 1/s", "Phase Diagram")
easyPlotXYHoldON(getXiList(ans1,2),getXiList(ans1,1), "RK4","Theta/ 1","Omega/ 1/s", "Phase Diagram")
done()


I am working on the file "Chapter3/chapter3.2/README.ipynb".
The first task is:
角度随时间变化（未取模）

角度随时间变化（取模）

角速度随时间变化

相图

可以看出，初始结果的微小差别会随着时间被放大，并占据主导
庞加莱截面
将庞加莱截面定义为对解的频闪提取
Can you write Python code for it?

def poincareSection(solution, omgd, phase = 0., tmin = 50):
	def mulOfTQ(t, omgd):
		n = (t - phase/omgd)/ (2*math.pi/omgd)
		return fEqQ(round(n), n)
	n = len(solution)
	sol = solution#[int(round(0.2*n)):-1]
	sectionPoints = filter(lambda state: mulOfTQ(getT(state), omgd) and getT(state) > tmin, sol)
	return sectionPoints


I am working on the file "Chapter3/chptr3.1-R.ipynb".
The first task is:
3.1: One predictor
A note on R packages
If the arm
library is not installed in your R setup, you'll need to figure out how to
install the package.  On Ubuntu 14.04 I do something like this:

Start R using sudo:

sudo R

Install the arm package using

install.packages('arm', dep = TRUE)
Read the data
Data are in the child.iq directory of the ARM_Data download-- you might have
to change the path I use below to reflect the path on your computer.
Can you write Python code for it?

%%R
# I had to import foreign to get access to read.dta
library("foreign")
kidiq <- read.dta("../../ARM_Data/child.iq/kidiq.dta")

# I won't attach kidiq-- i generally don't attach to avoid confusion(s)
#attach(kidiq)


I am working on the file "Chapter3/chptr3.1-R.ipynb".
The first task is:
First regression-- binary predictor, Pg 31
Loading the arm library -- this produces lots of warnings on the first load
that I can't figure out how suppress at this point.  However, things work just
fine despite the distraction of the warnings. Just rerun the cell to clean up
the mess. Also, let me know if you can figure out how get rid of the
warnings.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.1-R.ipynb".
The first task is:
Plot Figure 3.1, Pg 32
Add some jitter to kid_score
Can you write Python code for it?

%%R
# -- note that I use kidiq$kid_score because I did not attach kidiq above
kidscore.jitter <- jitter(kidiq$kid_score)


I am working on the file "Chapter3/chptr3.1-R.ipynb".
The first task is:
Second regression -- continuous predictor, Pg 32
Can you write Python code for it?

%%R
fit1 <- lm (kidiq$kid_score ~ kidiq$mom_iq)
display(fit1)


I am working on the file "Chapter3/chptr3.1.ipynb".
The first task is:
First regression-- binary predictor, Pg 31
Fit the regression using the non-jittered data
Can you write Python code for it?

fit0 = smf.ols('kid_score ~ mom_hs', data=kidiq).fit()
print(fit0.summary())


I am working on the file "Chapter3/chptr3.1.ipynb".
The first task is:
Plot Figure 3.1, Pg 32
A note for the python version:

I have not included jitter, in the vertical or horizontal directions.
  Instead, the data is plotted with opacity so the regions with high
  data-density can be distinguished.
Can you write Python code for it?

fig0, ax0 = plt.subplots(figsize=(8, 6))
hs_linspace = np.linspace(kidiq['mom_hs'].min(), kidiq['mom_hs'].max(), 50)

# default color cycle
colors = plt.rcParams['axes.color_cycle']

# plot points
plt.scatter(kidiq['mom_hs'], kidiq['kid_score'],
            s=60, alpha=0.5, c=colors[1])
# add fit
plt.plot(hs_linspace, fit0.params[0] + fit0.params[1] * hs_linspace,
         lw=3, c=colors[1])

plt.xlabel("Mother completed high school")
plt.ylabel("Child test score")


I am working on the file "Chapter3/chptr3.1.ipynb".
The first task is:
Read the data
Data are in the child.iq directory of the ARM_Data download-- you might have
to change the path I use below to reflect the path on your computer.
Can you write Python code for it?

kidiq  = pd.read_stata("../../ARM_Data/child.iq/kidiq.dta")
kidiq.head()


I am working on the file "Chapter3/chptr3.1.ipynb".
The first task is:
Second regression -- continuous predictor, Pg 32
Can you write Python code for it?

fit1 = smf.ols('kid_score ~ mom_iq', data=kidiq).fit()
print(fit1.summary())


I am working on the file "Chapter3/chptr3.2-R.ipynb".
The first task is:
3.2: Multiple predictors
Read the data
Data are in the child.iq directory of the ARM_Data download-- you might have
to change the path I use below to reflect the path on your computer.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.2-R.ipynb".
The first task is:
Load the arm library-- see the Chapter 3.1 notebook if you need help.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.2.ipynb".
The first task is:
Regression -- multiple predictors, Pg 33
Can you write Python code for it?

fit= smf.ols('kid_score ~ mom_hs + mom_iq', data=kidiq).fit()
print(fit.summary())


I am working on the file "Chapter3/chptr3.3-R.ipynb".
The first task is:
Figure 3.4 (a), Pg 35
Can you write Python code for it?

%%R
plot(kidiq$mom_iq, kidiq$kid_score,
     xlab="Mother IQ score", ylab="Child test score",
     pch=20, xaxt="n", yaxt="n", type="n")
curve(coef(fit)[1] + coef(fit)[2] + (coef(fit)[3] + coef(fit)[4])*x,
      add=TRUE, col="gray")
curve(coef(fit)[1] + coef(fit)[3]*x, add=TRUE)
points(kidiq$mom_iq[kidiq$mom_hs==0], kidiq$kid_score[kidiq$mom_hs==0],
       pch=20)
points(kidiq$mom_iq[kidiq$mom_hs==1], kidiq$kid_score[kidiq$mom_hs==1],
       col="gray", pch=20)
axis(1, c(80,100,120,140))
axis(2, c(20,60,100,140))


I am working on the file "Chapter3/chptr3.3-R.ipynb".
The first task is:
Figure 3.4 (b), Pg 35
Can you write Python code for it?

%%R
plot(kidiq$mom_iq, kidiq$kid_score,
     xlab="Mother IQ score", ylab="Child test score",
     pch=20, type="n", xlim=c(0,150), ylim=c(0,150))
curve(coef(fit)[1] + coef(fit)[2] + (coef(fit)[3] + coef(fit)[4])*x,
      add=TRUE, col="gray")
curve(coef(fit)[1] + coef(fit)[3]*x, add=TRUE)
points(kidiq$mom_iq[kidiq$mom_hs==0], kidiq$kid_score[kidiq$mom_hs==0],
       pch=20)
points(kidiq$mom_iq[kidiq$mom_hs==1], kidiq$kid_score[kidiq$mom_hs==1],
       col="gray", pch=20)


I am working on the file "Chapter3/chptr3.3-R.ipynb".
The first task is:
Load the arm library-- see the Chapter 3.1 notebook if you need help.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.3-R.ipynb".
The first task is:
Regression-- interactions, Pg 34
Can you write Python code for it?

%%R
fit <- lm(kidiq$kid_score ~ kidiq$mom_hs + kidiq$mom_iq + kidiq$mom_hs:kidiq$mom_iq)
display(fit)


I am working on the file "Chapter3/chptr3.4-R.ipynb".
The first task is:
3.4 Statistical inference
Read the data
Data are in the child.iq directory of the ARM_Data download-- you might have
to change the path I use below to reflect the path on your computer.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.4-R.ipynb".
The first task is:
Load the arm library-- see the Chapter 3.1 notebook if you need help.
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.4-R.ipynb".
The first task is:
Regression-- to demonstrate reports of fit, Pg38
Can you write Python code for it?

%%R
fit.3 <- lm(kidiq$kid_score ~ kidiq$mom_hs + kidiq$mom_iq)


I am working on the file "Chapter3/chptr3.4-R.ipynb".
The first task is:
Summary, Pg 38
Too much information?
Can you write Python code for it?


I am working on the file "Chapter3/chptr3.4.ipynb".
The first task is:
Now, use it:
Can you write Python code for it?