I am working on the file "Pressureless_flow.ipynb".
The first task is:
Similarly, a non-dimensionalization of the isothermal equations shows that the pressureless flow equations correspond to the large Mach number limit (i.e. when the typical flow velocity is much greater than the sound speed).
References
Delta shocks arising in pressureless flows have been studied by many authors; see Section 16.3 of (LeVeque, 2002) and references therein, as well as (LeVeque, 2004).  For an examination of the large Froude number limit in shallow water flow, see (Edwards et. al., 2008).
Exercises
(1) In the example below, the initial velocity is zero for both states.  Observe that multiplying the value of $g$ by a factor $\alpha$ is equivalent to rescaling all the velocities by $\sqrt{\alpha}$.  Prove that this scaling must hold for any Riemann solution of this system when $u_l = u_r = 0$.
Can you write Python code for it?

q_l  = State(Depth = 3.,
             Momentum = 0.)
q_r = State(Depth = 1.,
            Momentum = 0.)

plot_riemann_SW(q_l,q_r)


I am working on the file "Pressureless_flow.ipynb".
The first task is:
What happens if $u_l \ge u_r$, so that the double-rarefaction solution is unphysical?  The answer can be found by considering what happens when $g$ is small but nonzero.  Here's an example to play with.
Can you write Python code for it?

q_l  = State(Depth = 3.,
             Momentum = 1.)
q_r = State(Depth = 1.,
            Momentum = 0.1)

plot_riemann_SW(q_l,q_r)


I am working on the file "Pressureless_flow.ipynb".
The first task is:
What's going on here, physically, when $g$ is very small? At the interface, water is colliding.  With the gravitational pressure term present, this leads to the formation of outgoing shock waves that redistribute water away from the interface.  But without that term, there is no force to equilibrate the depth and each parcel of water just flows with its initial velocity, until eventually it collides with water of a different velocity.  Water from both sides accumulates at the interface, leading to what is known as a delta shock.
If you drag the time slider for different fixed values of $g$ in the last example, you'll see that since $g$ is still nonzero there are two shock waves and the near-delta-function region expands with time, but the rate of this expansion is smaller for smaller values of $g$.  For very small values, the region between the shocks cannot be resolved on the scale of the plots above.  Indeed, the width of the line plotted there significantly exaggerates the width of this region when $g$ is very small.
There are two import details we haven't yet explained: first, the delta shock is moving; second, it is growing in time.  Movement of the delta shock is necessary in order to conserve momentum; since the water from the left is arriving with a different speed than that from the right, if the delta shock were stationary then momentum would not be conserved.  Growth of the delta shock is necessary in order to conserve mass.
Let $h_\delta$ denote the mass of the delta shock.  How must $h_\delta$ grow in order to conserve mass?  Let $\hat{u}$ denote the velocity of the delta shock, with $u_l \ge \hat{u} \ge u_r$.  Then over a unit time interval, a quantity $h_l(u_l-\hat{u})$ of water arrives from the left, while $h_r(\hat{u}-u_r)$ arrives from the right.  Thus we must have
$$
    h_\delta(t) = t \left( h_l u_l - h_r u_r + \hat{u}(h_r-h_l) \right).
$$
At what speed must the water in the delta shock move in order to conserve momentum?  The rate of momentum flowing into the delta shock from the left is...
Can you write Python code for it?

q_l  = State(Depth = 1.,
             Momentum = 100.)
q_r = State(Depth = 1.,
            Momentum = 0.)

plot_riemann_SW(q_l,q_r)


I am working on the file "Prices and Returns.ipynb".
The first task is:
Alphabet, or Google, seems to be the winner in these 10 years. But comparison of stocks is difficult because of different ranges of prices. Let's try to normalize them, rebasing to 100.
Can you write Python code for it?

norm_prices = 100 * prices / prices.iloc[0, :]
norm_prices.head(10)


I am working on the file "Prices and Returns.ipynb".
The first task is:
And let's see how returns averaged in the period.
Can you write Python code for it?

yearly_avgs = yearly_returns.mean().to_frame().T
yearly_avgs.index = ['Average Yearly Return (%)']
yearly_avgs


I am working on the file "Prices and Returns.ipynb".
The first task is:
Now, a plot of rebased prices.
Can you write Python code for it?

norm_prices.plot(figsize=(10, 8), title='Stock prices ({} = 100)'.format(START));


I am working on the file "Prices and Returns.ipynb".
The first task is:
Now, let's download stock data from Yahoo!Finance, using pandas-datareader module.
Can you write Python code for it?

data = web.DataReader(STOCKS, 'google', pd.to_datetime(START), pd.to_datetime(END))


I am working on the file "Prices and Returns.ipynb".
The first task is:
Now, let's resample data into yearly percent returns.
Can you write Python code for it?

yearly_returns = prices.resample('A').last().pct_change()
yearly_returns


I am working on the file "Prices and Returns.ipynb".
The first task is:
Then, let's choose a bunch of stocks to analyze. They are: Apple (AAPL), Alphabet (former Google, GOOGL), Microsoft (MSFT), McDonald's (MCD), Coca-Cola (KO). Three technology companies and two food & beverage stocks.
Can you write Python code for it?

# List of stocks tickers
STOCKS = ['NASDAQ:AAPL', 'NASDAQ:GOOGL', 'NASDAQ:MSFT', 'NYSE:MCD', 'NYSE:KO']

# Analysis period
START = '12-30-2006'
END = '12-31-2016'


I am working on the file "Prices and Returns.ipynb".
The first task is:
Then, let's plot the returns.
Can you write Python code for it?

ax = yearly_returns.plot(figsize=(10, 8), title='Stock yearly returns', ls='', marker='o');
ax.set_xlim(ax.get_xlim()[0], ax.get_xlim()[1] + 1)
ax.axhline(0, color='gray');


I am working on the file "Prices and Returns.ipynb".
The first task is:
We always can transform percent returns into log returns:
Can you write Python code for it?


I am working on the file "Prices and Returns.ipynb".
The first task is:
We'll work only on Close prices, so let's discard the other data.
Can you write Python code for it?

prices = data.loc['Close', :, :]
prices.tail(10)


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Calculate the break even smile for the first 10 paths
Can you write Python code for it?

bevsmile = pd.DataFrame([pd.Series([opt.newton(bev, 0.3, args=(path,call,strike,r,q,T)) for strike in strikes], index=strikes) \
                         for path in paths[:20]])


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Define parameters
Can you write Python code for it?

call=1
put=-1
S0=100
r=0
q=0
vol_pricing=.2
vol_delta=.3
T=1
K=100
nsteps=500
npaths=500


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Defining the Black formula, according to:
$$C(S_0,t) = e^{-rT} \left[FN(d_1) - KN(d_2)\right]$$
Where:
$$F = S_0e^{\left(r-q\right)t}$$
$$d_1 = \frac{1}{\sigma\sqrt{t}} \left[ \ln{\left( \frac{S_0}{K} \right) } + \left( r -q + \frac{\sigma^2}{2} \right) \left(t \right) \right]$$
$$d_2 = d_1 - \sigma\sqrt{t}$$
Can you write Python code for it?

def bsv(phi,S,K,r,q,vol,t):
    if(t>0):
        # calculate F
        F=S*np.exp((r-q)*t)

        # calculate d1 and d2
        sigma_rt = vol*np.sqrt(t)
        d1=(np.log(S/K)+(r-q+vol**2/2)*t)/(sigma_rt)
        d2=d1-sigma_rt

        # calculate N(d1) and N(d2)
        Nd1=st.norm.cdf(phi*d1)
        Nd2=st.norm.cdf(phi*d2)

        # calculate delta and premium
        delta=phi*np.exp(-q*t)*Nd1
        premium=S*delta-phi*K*np.exp(-r*t)*Nd2
    else:
        delta=0
        premium=max(phi*(S-K),0)

    return [premium,delta]


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Final P&L distribution
Can you write Python code for it?

pls=pd.Series([portfsVolReal[i]['portf'][nsteps] for i in range(npaths)])
pls.plot(kind='hist',bins=10)


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Hedging using Implied Vol
Can you write Python code for it?

portfsVolImpl=[calcportfolio2(paths[i],call,K,r,q,vol_pricing,vol_pricing,T) for i in range(npaths)]
df=pd.DataFrame([pd.Series(portfsVolImpl[i]['portf'].values,index=portfsVolImpl[i]['time']) for i in range(npaths)])
df.transpose().plot(legend=None)


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Hedging using Realized Vol
P&L portfolio envolution
Can you write Python code for it?

portfsVolReal=[calcportfolio2(paths[i],call,K,r,q,vol_pricing,vol_delta,T) for i in range(npaths)]
df=pd.DataFrame([pd.Series(portfsVolReal[i]['portf'].values,index=portfsVolReal[i]['time']) for i in range(npaths)])
df.transpose().plot(legend=None)


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
Let´s test
Can you write Python code for it?

npaths=10
nsteps=100
S0=100
r=0.1
vol=0.2
T=1
test_mcpaths=mcpaths(npaths,nsteps,S0,r,vol,T)

[test_mcpaths[i].plot() for i in range(npaths)]


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
The Break Event Vol distribution
Can you write Python code for it?

bevs.plot(kind='hist',bins=20)

def calcvolreal(path,T):
    return np.std(path.values[1:] / path.values[0:-1] - 1)*np.sqrt(len(path)/T)

vols_real = [calcvolreal(paths[i],T) for i in range(npaths)]


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
function for a monte carlo path sample, based on Generalized geometric Brownian motion:
$$S(t) = S(0)\exp\left {\sigma W(t) + (r - \frac{1}{2}\sigma^2)t\right }$$
Can you write Python code for it?

def mcpath(nsteps, S0, r, vol, T):

    # generate nsteps random numbers
    sample=pd.Series(np.random.standard_normal(nsteps))

    # define time increment
    dt=T/nsteps

    # initialize vectors
    initial = np.zeros(1)
    ts=np.repeat(dt,nsteps)

    # calculate s1 and s2
    s1=(r-(vol**2)/2)*dt
    s2=vol*np.sqrt(dt)
    steps=s1*np.repeat(1,nsteps)+s2*sample
    path=steps.cumsum()
    Sj=S0*np.exp(path)

    # add initial time t=0
    S = np.concatenate(([S0],Sj))
    t = np.concatenate(([0],ts.cumsum()))

    return pd.Series(S,index=t)


I am working on the file "Pricing and Hedging - Exercise Two.ipynb".
The first task is:
generating a histogram of portfolio P&L
Can you write Python code for it?

npaths=500
pls=pd.Series([calcportfolio(mcpath(nsteps, S0, r, vol, T),call,K,r,q,vol,T)['portf'][nsteps] for i in range(npaths)])
pls.plot(kind='hist',bins=30)


I am working on the file "PrimNumber.ipynb".
The first task is:
Java sample code of Sieve of Eratosthenes
```
import java.util.List ;
import java.util.ArrayList ;
class sieve {
// sieve of Eratosthenes, find the prime numbers below given integer

static List<Integer> sieve(int n) {

    List<Integer> primes = new ArrayList<>() ;
    boolean is_prime[] = new boolean[n+1] ;

    for (int i=0; i<=n; i++) is_prime[i] = true ;
    is_prime[0] = is_prime[1] = false ;

    for (int p=2; p<=n; p++) {
        if (is_prime[p]) {
            for (int mop = p*2; mop<=n; mop += p) {
                is_prime[mop] = false ;
            }
        }
    }

    for (int p=2; p<=n; p++) {
        if (is_prime[p]) primes.add(p) ;
    }
    return primes ;
}

public static void main(String argv[]) {

    List<Integer> primes = sieve(100) ;
    for (int n: primes) {
        System.out.printf("%d ", n) ;
    }
}

}
```
Performance inprovement
2 is the only number that is prime and even. Above program can be improved if even number is properly handled.
Can you write Python code for it?

#!/usr/bin/python3
# -*- coding: utf-8 -*-

'''
find prime numbers up to MAX_NUMBER
sieve of Eratosthenes
'''

# Same as above program, but faster

MAX_NUMBER = 100

# Mark Odd number as True, Even number as False
prime = [True if i & 1 else False for i in range(MAX_NUMBER+1)]
# prime = [False, True, False, True, False, True, ....]
# prime[0] = False   # 0 is not prime
prime[1] = False   # 1 is not prime
prime[2] = True    # 2 is prime

sqrt_max = int(MAX_NUMBER ** 0.5)  # check up to root(MAX_NUMBER)

for i in range(3, sqrt_max + 1):  # Only check odd number
    if prime[i]:
        for mop in range(i*3, MAX_NUMBER+1, i+i):  # ignore even number
            # Mark the multiples of the prime to False in the table
            prime[mop] = False

num_prime = 0
for i in range(MAX_NUMBER):
    if prime[i]:
        num_prime += 1
        print(i, end=' ')
# print(num_prime, 'prime number found under', MAX_NUMBER)


I am working on the file "PrimNumber.ipynb".
The first task is:
Prime number checking (is_prime() function)
The definition of Prime Number is that it can be divided only 1 and itself. (see is_prime_simple() function below.
But in order to check if the number $N$ is prime or not, it's not necessary to check up to $N-1$, but the prime number equal or less of $\sqrt{N}$, because if N is a product of $A$ and $B$, one of the divisor is equal or less than $\sqrt{N}$.

Definition-1: Prime number is a integer bigger than 1 and can be divided only by 1 and itself
Definition-2: Composite Number is a integer bigger than 1, and not Prime number (i.e. has more than 1 divisor except 1 and itself)
Definition-3: Prime Number is not Composit number
Definition-4: Composite Number can be factorized. If a number is a composite number, it is a product of smaller Prime numbers.

Definition-5: Every integer bigger than 1 is either Prime or Composite number

Premise-1: Composite Number $N$ has a divisor equal or less than $\sqrt{N}$

Conclusion: If $N$ does not have divisor eqaul or less than $\sqrt{N}$, it's not Composite Number -> $N$ is Prime number

Premise-2: If $N$ has divisor of composite number $A$, divisor of $A$ is divisor of $N$. ( $N = A \cdot B, A= X \cdot Y \rightarrow N=X \cdot Y \cdot B$ )

From Definition-4 and Premise-2, we can conclude that it's necessary to check only prime number as a condidate of divisor

Proof of Premise-1:
If N is a Composite Number (Non-Prime-Number) and product of positive integer(bigger than 1) A and B, and if A is smaller or equal to B, then A is smaller or equal to $\sqrt{N}$
$$(N = A \cdot B) \ (1 \lt A \le B \lt N) \rightarrow (A \le \sqrt{N})$$
If $A$ is bigger than $\sqrt{N}$, then $A \cdot B \gt N$. Let's assume that $$A = \sqrt{N} + a \ B = \sqrt{N} + b \ 0 \lt a \le b$$
Then $$A \cdot B \ = (\sqrt{N} + a)(\sqrt{N} + b) \ = (\sqrt{N}^2 + (a+b)\sqrt{N} + a \cdot b) \gt N$$
Can you write Python code for it?

def is_prime_simple(n):
    if n < 2: return False
    for div in range(2,n):
        if n % div == 0:
            return False  # not Prime
    return True

BIG_PRIME = 10**8 + 7
print(2, is_prime_simple(2))
print(97, is_prime_simple(97))
print(BIG_PRIME, is_prime_simple(BIG_PRIME))


I am working on the file "Primeiros_passos_Google_Colab.ipynb".
The first task is:
Entrada de Dados
Can you write Python code for it?

nome = input('Qual é seu nome: ')
print(nome + ' seja bem vindo ao curso de Python')


I am working on the file "Primeiros_passos_Google_Colab.ipynb".
The first task is:
Executar comandos Linux
Can you write Python code for it?


I am working on the file "Primeiros_passos_Google_Colab.ipynb".
The first task is:
Instalar biblioteca
Can you write Python code for it?

!pip install requests

!pip install tensorflow==1.2


I am working on the file "Primeiros_passos_Google_Colab.ipynb".
The first task is:
Integração Drive e GitHub

Permite abrir e salvar arquivos nessas aplicações

Trabalhando de forma colaborativa

Compartilhando seu código
Adicionando e resolvendo comentários
Can you write Python code for it?


I am working on the file "Primeiros_passos_Google_Colab.ipynb".
The first task is:
Saída de dados ricas
Can you write Python code for it?

import numpy as np
from matplotlib import pyplot as plt

ys = 200 + np.random.randn(100)
x = [x for x in range(len(ys))]

plt.plot(x, ys, '-')
plt.fill_between(x, ys, 195, where=(ys > 195), facecolor='g', alpha=0.6)

plt.title("Fills and Alpha Example")
plt.show()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
4-D example
Let's try it again with more dimensions to demonstrate dimensionality reduction:
Can you write Python code for it?

# Generate some 4D data:
x = np.linspace(0,1,100)
y1 = x*-3
y2 = 5/(1+x)
y3 = x*.003
y4 = x*0.0022

dat_4d = np.vstack([y1,y2,y3,y4]).T
np.random.seed(rand_seed)
dat_n_4d = dat_4d + np.random.rand(*dat_4d.shape)*(np.array([1,2,1,4])) # add noise

# Plot data:
plt.plot(dat_n_4d)
plt.legend(['hats','shoes','knickers','potatoes'])
plt.show()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Correlation, covariance, and Standardized values
PCA can be peformed on covariance or correlation. Correlation is different because it is standardized between $ -1 < \text{corr} < 1 $.
Correlation is useful where units are difficult to compare, i.e. kg vs miles. When using covariance, different units can yield different results, but with correlation, they are normalized. The correlation is:
$$ corr(x,y) =  \frac{1}{\sigma_x \sigma_y} \frac{1}{n-1}\sum_{i=1}^{n}(x_i-\overline{x})(y_i - \overline{y})$$
where:
$$ \sigma_x = \text{standard deviation of variable x} $$
or just:
$$ corr(x,y) =  \frac{1}{\sigma_x \sigma_y} cov(x,y) $$
The correlation and covariance matrices are scaled by 1/product of the standard deviations:
$$\frac{1}{\sigma_x \sigma_Y}$$
Using just the covariances along the matrix diagonal, (and taking the square root) we obtain the standard deviations.
We can use these to create a new matrix of $\frac{1}{\sigma_x \sigma_Y}$ and multiply by the covariance matrix element-wise to obtain the correlation matrix:
Can you write Python code for it?

# Use the square root of the covariances along the diagonal to obtain the standard deviations:
dat_cov = np.cov(dat_n_4d.T)
stdevs = np.sqrt(np.diag(dat_cov))
# Then multiply such that the 1/product of the standard deviations
dat_corr = 1/stdevs[:,np.newaxis]* 1/stdevs[np.newaxis,:] * dat_cov

print('The Correlation matrix is:\n{}'.format(dat_corr))

# Via built-in functions:
dat_corr2 = np.corrcoef(dat_n_4d.T)
np.isclose(dat_corr, dat_corr2).all()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Find Covariance
Next we find the covariance, of the amount each feature varies with another:
$$ cov(x,y) = \frac{1}{n-1}\sum_{i=1}^{n}(x_i-\overline{x})(y_i - \overline{y})$$
We can do this for all features in a dataset, creating a n$x$n matrix representing how each feature co-varies with another:
$$ cov = \sum_{j=1}^{n}(x_j-\overline{x})(x_j - \overline{x})^T$$
Where: $\overline{x}$ is the expectation of x. If each value is equally weighted, then:
$$ \overline{x} = \frac{1}{n}\sum_{j=1}^{n}x_j$$
Note that the data here is plotted in row format (list of rows) vs columnar format (list of columns). Our math is performed on a per sublist basis, so a transpose is necessary:
Can you write Python code for it?

datc_n = dat_n.T

# The covariance matrix is the inner product of the mean-subtracted dataset with it's transpose:
datc_ms = (datc_n - datc_n.mean(axis=1)[:, np.newaxis]) # Mean subtract the data set:
dat_cov = datc_ms.dot(datc_ms.T)/(datc_n.shape[1]-1)

print('Covariance matrix is:\n{}'.format(dat_cov))

# Similarly, we could use a built-in numpy package:
dat_cov2 = np.cov(datc_n)

# Are these the same (check if they are within floating point error)?
np.isclose(dat_cov, dat_cov2).all()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Perform an eigen decomposition
This will determine the direction (eigenvalues or explained variance) and set of vectors (eigenvectors or components) that satisfy the relation:
$$ M\vec{\nu} = \lambda \vec{\nu}$$
Where in our case:
- $M$ is a measure of the variance of our dataset (often covariance)
- $\vec{\nu}$ is a vector
- $\lambda$ is a scalar associated with vector $\vec{\nu}$
In words, eigenvector $\vec{\nu}$ of matrix $M$ is equal to the scalar eigenvalue $\lambda$ of $\vec{\nu}$
From the perspective of PCA, the eigenvectors are the directions in terms of the original features (components) and the eigenvalues are the magnitudes of each vector (explained variance).
Usually, software packages (i.e. numpy, matlab) determine eigenvectors such that the length of each is 1. While computation by hand doesn't necessarily always work out this way, the decomposition into unitary vectors makes a bit more sense: the eigenvalues are the length of the vectors, and the vectors provide direction.
The eigenvalues are the explained variance, or relative 'importance' of the pca component, and the eigenvectors are the components, or relative weights of the original features within each explained vector.
Can you write Python code for it?

eig_vals, eig_vecs = np.linalg.eig(dat_cov)

print('Eigenvalues (explained variance):\n {}'.format(eig_vals))
print('Eigenvectors (components):\n {}'.format(eig_vecs))


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Plot explained variance:
How much of each component is the total sum of variance? This is often used as a proxy for how much 'information' is contained in each componet. Keep in mind, variance results from both signal and noise.
Can you write Python code for it?

# Visualize explained variance:
tot = sum(expl_var)
var_exp = [(i / tot)*100 for i in sorted(expl_var, reverse=True)]
cum_var_exp = np.cumsum(var_exp)

xlabels = ['{}'.format(x) for x in range(1, len(var_exp)+1)]

plt.bar(xlabels, var_exp, label='Individual Variance')
plt.plot(xlabels, cum_var_exp, label='Cumulative Variance', marker='o', color='red')

plt.ylabel('Percent Explained Variance')
plt.xlabel('Principal Component')
plt.title('Explained Variance')
plt.legend()

plt.show()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Recover the original dataset:
$$ M = Z \vec{\nu}^T$$
$$ or  $$
$$ M = M \vec{\nu}\vec{\nu}^T$$
Can you write Python code for it?

dat_n_recovered = dat_proj.dot(components) + dat_n.mean(axis=0)

plt.scatter(dat_n_recovered[:,0], dat_n_recovered[:,1])
plt.title('Feature 1 vs Feature 2')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.axis('equal')
plt.show()

# Did we recover our original dataset?
np.isclose(dat_n, dat_n_recovered).all()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Similarly, we could use the built-in function to plot the 2D projection, but we must specify the number of components during the PCA to do so (and thus couldn't plot explained variance for all dimensions). Also, only the highest variace dimensions are returned:
Can you write Python code for it?

dat_proj2d2 = PCA(2).fit(dat_n_4d).transform(dat_n_4d)
print(np.isclose(dat_proj2d, dat_proj2d2).all())


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Singular value decomposition (SVD) is a more general approach than the eigen decomposition and frequently used on the back end of software computing PCA.
SVD is another matrix factorization like eigen decomposition, but has all kinds of uses including the pseudoinverse, least squares fitting of data, and determining the rank, range, and null space of a matrix.
$$X = U \Sigma V^T $$
where
$$ U^TU = I $$
and
$$ V^TV = I $$
where:
- $X$ is our centered (mean-subtracted) dataset
- $I$ is the identity matrix
- $T$ is the conjugate transpose
SVD can be used to determine the eigenvalues and eigenvectors:
Can you write Python code for it?

dat_ms = dat_n - dat_n.mean(axis=0)
u,s,v = np.linalg.svd(dat_ms, full_matrices=False) # Note that we didn't use the row formatted data (sklearn svd_flip function expects columnar data)

# The explained variance is the square of the SVD singular matrix:
expl_var = (s ** 2) / (dat_n.shape[0] - 1)
print('Explained Variance:\n{}'.format(expl_var))

print('Components:\n {}'.format(v))


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Three of these components have nearly the same variance. While we can plot in 3d, it's easy to imagine a situation with 4 or more components each with high variance. How do we visualize them?
Can you write Python code for it?

# Perform PCA, obtain explained variance and basis components:
from sklearn.decomposition import PCA
pca_4d = PCA().fit(dat_n_4d)
expl_var_4d = pca_4d.explained_variance_
components_4d = pca_4d.components_

# Visualize explained variance:
tot = sum(expl_var_4d)
var_exp_4d = [(i / tot)*100 for i in sorted(expl_var_4d, reverse=True)]
cum_var_exp_4d = np.cumsum(var_exp_4d)

xlabels_4d = ['{}'.format(x) for x in range(1, len(var_exp_4d)+1)]

plt.bar(xlabels_4d, var_exp_4d, label='Individual Variance')
plt.plot(xlabels_4d, cum_var_exp_4d, label='Cumulative Variance', marker='o', color='red')

plt.ylabel('Percent Explained Variance')
plt.xlabel('Principal Component')
plt.title('Explained Variance')
plt.legend()
plt.show()


I am working on the file "Principal Component Analysis.ipynb".
The first task is:
Using StandardScaler:
Alternatively, instead of using the correlation matrix, one could use the standard scaler (center the data and scale by the standard devation per each feature), yielding a different but also scaled result.
Can you write Python code for it?

# Scaled/Standardized values:
dat_n_s4d = (dat_n_4d - dat_n_4d.mean(axis=0))/(dat_n_4d.std(axis=0))

# Use sklearn:
from sklearn.preprocessing import StandardScaler
dat_n_s4d22 = StandardScaler().fit_transform(dat_n_4d)

# Are they the same?
np.isclose(dat_n_s4d, dat_n_s4d22).all()


I am working on the file "Principal Components Analysis on the UCI Image Segmentation Data Set.ipynb".
The first task is:
Looking at the graph above, it looks like we could cut the number of features to 13 without losing much in the way of model accuracy.  Lets try numbers of features from 10 to 19 to check this hypothesis.
Can you write Python code for it?

for i in range(10, X.shape[1] + 1):
    pca = decomposition.PCA(n_components=i)
    pca.fit(X_train)
    X_train_pca = pca.transform(X_train)
    X_test_pca = pca.transform(X_test)

    clf.fit(X_train_pca, y_train)
    pred = clf.predict(X_test_pca)

    print 'RF Model: %.2f%% accurate with %d PCA components' % ((metrics.accuracy_score(y_test, pred) * 100.0), i)


I am working on the file "Principal Components Analysis on the UCI Image Segmentation Data Set.ipynb".
The first task is:
Prepare the data for machine learning by dividing between training and test sets.
Can you write Python code for it?

y = df['CLASS']
X = df.drop('CLASS', axis=1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=245)


I am working on the file "Print Formatting.ipynb".
The first task is:
Conversion Format methods.
It should be noted that two methods %s and %r actually convert any python object to a string using two separate methods: str() and repr(). We will learn more about these functions later on in the course, but you should note you can actually pass almost any Python object with these two methods and it will work:
Can you write Python code for it?

print 'Here is a number: %s. Here is a string: %s' %(123.1,'hi')

print 'Here is a number: %r. Here is a string: %r' %(123.1,'hi')


I am working on the file "Print Formatting.ipynb".
The first task is:
Floating Point Numbers
Floating point numbers use the format %n1.n2f where the n1 is the total minimum number of digits the string should contain (these may be filled with whitespace if the entire number does not have this many digits. The n2 placeholder stands for how many numbers to show past the decimal point. Lets see some examples:
Can you write Python code for it?

print 'Floating point numbers: %1.2f' %(13.144)

print 'Floating point numbers: %1.0f' %(13.144)

print 'Floating point numbers: %1.5f' %(13.144)

print 'Floating point numbers: %10.2f' %(13.144)

print 'Floating point numbers: %25.2f' %(13.144)


I am working on the file "Print Formatting.ipynb".
The first task is:
Multiple Formatting
Pass a tuple to the modulo symbol to place multiple formats in your print statements:
Can you write Python code for it?


I am working on the file "Print Formatting.ipynb".
The first task is:
Using the string .format() method
The best way to format objects into your strings for print statements is using the format method. The syntax is:
'String here {var1} then also {var2}'.format(var1='something1',var2='something2')

Lets see some examples:
Can you write Python code for it?

print 'This is a string with an {p}'.format(p='insert')

# Multiple times:
print 'One: {p}, Two: {p}, Three: {p}'.format(p='Hi!')

# Several Objects:
print 'Object 1: {a}, Object 2: {b}, Object 3: {c}'.format(a=1,b='two',c=12.3)


I am working on the file "PrintFormatting.ipynb".
The first task is:
%x.yf format: x->Minimum number of characters for the string, y-> number of digits after the decimal point.
Multiple formating
You can print multiple objects or values at same time.
Can you write Python code for it?

print('First: %s, Second: %1.1f and Third: %s' %(1,2.0,"three"))


I am working on the file "PrintFormatting.ipynb".
The first task is:
The values will follow order of appearance (same as in C's printf())
Conversion methods
%s and %r are 2 conversion methods, they convert any python object to a string.
%s uses str() and %r uses repr().
Can you write Python code for it?

print('This is object to string using %s & %s' %(1234,"String here!"))

print('This is object to string using %r & %r' %(4321,"String here!"))


I am working on the file "Prise de rdv automatique sur Doctolib.ipynb".
The first task is:
J'ai bien réussi à installer et utiliser Selenium. Ca marche bien !
Solution
J'aurai besoin de ces modules là :
Can you write Python code for it?

import time
from datetime import datetime
import urllib.request
import subprocess
from selenium import webdriver


I am working on the file "Prise de rdv automatique sur Doctolib.ipynb".
The first task is:
Vérifions le contenu de la page, et voir si elle affiche ce message suivant :

En raison d'une forte demande, ce centre n'a plus de disponibilités : 3362 vaccinations vont avoir lieu dans les 28 prochains jours. Réessayez prochainement ou cherchez un autre centre.
Can you write Python code for it?

print(f"Downloading '{URL}'...")
firefoxOptions = webdriver.FirefoxOptions()
firefoxOptions.headless = True
browser = webdriver.Firefox(options=firefoxOptions)

browser.get(URL)

message = "ce centre n'a plus de disponibilités"

if message in browser.page_source:
    print(f"{URL} indique :\n{message}")
else:
    print(f"{URL} n'indique pas :\n{message}\nPeut être qu'il y a des disponibilités désormais !")
    !FreeSMS.py f"{URL} semble indiquer qu'il y a des disponibilités désormais."


I am working on the file "PrisonersDilemma/.ipynb_checkpoints/ts_length-checkpoint.ipynb".
The first task is:
各セッションの期数の決め方は、以下の通り。
引用: README#共通ルール

ゲームを何期続けるかは、以下のルールによって決めます。

第1期は確率1で到来する

以降は毎期末に抽選を行い、**97%の確率でゲームを継続する**（3%の確率で終了する）
（これは、無限回繰り返しゲームにおいて現在割引価値を0.97と取ることを意味します）

平均は33.33期になります。

したがって、各セッションの期数を確率変数 $X_1, X_2, \ldots, X_{1000}$ とすると、
$$
X_1, X_2, \ldots, X_{1000} \overset{\text{i.i.d.}}{\sim} Geo(0.03)
$$
となる。ただしGeo(p)は「成功確率がpの時、初めて成功するまでの総試行回数」を表す幾何分布で、（下側）累積分布関数は
$$
P(X \leq x) = F(x) = \sum_{k=1}^{x} p(1-p)^{k-1} \hspace{20pt} x=1, 2, \ldots
$$
となる。
平均は
\begin{eqnarray}
E[X] &=& \sum_{x=1}^{\infty} x pq^{x-1}  \hspace{20pt} q =  1-p\
&=& p\left(\frac{\partial}{\partial q} \sum_{x=1}^{\infty} q^x \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{q}{1-q} \right) \
&=& p \frac{1-q+q}{(1-q)^2} \
&=& \frac{1}{p}
\end{eqnarray}
分散は
\begin{eqnarray}
E[X(X+1)] &=& \sum_{x=1}^{\infty} x(x+1) pq^{x-1}\
&=& p\left(\frac{\partial^2}{\partial q^2} \sum_{x=1}^{\infty} q^{x+1} \right) \
&=& p \left(\frac{\partial^2}{\partial q^2} \frac{q^2}{1-q} \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{2q(1-q)+q^2}{(1-q)^2} \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{2q-q^2}{(1-q)^2} \right) \
&=& p \left(\frac{(2-2q)(1-q)^2 + 2(2q-q^2)(1-q)}{(1-q)^4} \right) \
&=& p \left(\frac{2(1-q)^2 + 4q-2q^2)}{(1-q)^3} \right) \
&=& \frac{2}{p^2}
\end{eqnarray}
\begin{eqnarray}
Var(X) &=& E[X^2] - (E[X])^2 \
&=& E[X(X+1)] - E[X]  - (E[X])^2 \
&=& \frac{2}{p^2} - \frac{1}{p} - \frac{1}{p^2} \
&=& \frac{1-p}{p^2}
\end{eqnarray}
となる。
今回はp=0.03なので、$E[X] = 33.33, Var(X) = 1077.78$ となる。
確率関数と累積分布関数
確率（質量）関数と下側累積分布関数をplot（理論値）。線にしてあるが、実際は離散値（x=1, 2, 3, ...）。
Can you write Python code for it?

# settings
seed = 282
rs = np.random.RandomState(seed)
parameter = 0.03

# plot
fig = plt.figure(figsize=(20, 12))
mu = 33.333
sigma = 1077.778

# theoretical pmf
ax = plt.subplot(2, 1, 1)
plt.title("Geo(0.03)の確率関数")
x = np.arange(1, 301)
plt.plot(x, stats.geom.pmf(x, parameter), linewidth=1, color='green', label="theoretical pmf")
plt.xlabel("ts_length(X)")
plt.ylabel("probability(f)")
ax.text(35, 0.02, r'''$\mu$={0:.3f}, $\sigma^2$={1:.3f}'''.format(mu, sigma), ha = 'left', va = 'bottom', size=14)
ax.axvline(x=mu, linewidth=1, color='red', label="average")
ax.set_xlim([1, 300])
plt.legend()

# theoretical cmf
ax = plt.subplot(2, 1, 2)
plt.title("Geo(0.03)の下側分布関数")
x = np.arange(1, 301)
plt.plot(x, stats.geom.cdf(x, parameter), linewidth=1, color='blue', label="theoretical cdf")
plt.xlabel("ts_length(X)")
plt.ylabel("cumulative dens(F)")
ax.text(35, 0.55, r'''$\mu$={0:.3f}のときF={1:.3f}'''.format(mu, stats.geom.cdf(mu, parameter)), ha = 'left', va = 'bottom', size=14)
ax.axvline(x=mu, linewidth=1, color='red', label="average")
ax.set_xlim([1, 300])
plt.legend()

plt.show()


I am working on the file "PrisonersDilemma/.ipynb_checkpoints/ts_length-checkpoint.ipynb".
The first task is:
確率関数は単調減少。$P(X \leq E[X]) = F(33.33) = 0.634$ より、全体の63.4%程度が平均よりも小さい値であることが期待される。
このことは、幾何分布を適当な指数分布で近似（連続化）できるとすれば、その時間あたりの生起回数はポアソン分布$F(x) = \sum_{k=0}^x\frac{\lambda^k}{k!}e^{- \lambda}$に従うことからもわかる。今1単位時間を33.33期とすれば、1単位時間に1回以上「ゲームが終了する」事象が起こる確率は、$\lambda = 1$ のポアソン分布で$x \geq 1$となる確率なので、
\begin{eqnarray}
P(X\geq1) = 1 - F(0) = \frac{1^0}{0!}e^{- 1} = \frac{1}{e} = 0.6321
\end{eqnarray}
となって、ほぼ一致する。
和の分布
正確な和の分布
幾何分布に従う確率変数 $X_1, X_2, \ldots, X_{1000}$ の和を $S(X)$ とすると、
$$
S(X) \ {\sim} \ Negbin(n=1000, p=0.03)
$$
となる。ただしNegbin(n, p)は「成功確率がpの時、n回成功するまでの総試行回数」を表す負の二項分布で、累積分布関数は、
$$
P(Y \leq y) = F(y) = \sum_{k=n}^{y} {}{k-1}C {n-1} p^n (1-p)^{k-n} \hspace{20pt} y=n, n+1, n+2, \ldots
$$
となる。平均は $E[Y] = \frac{n}{p}$、分散は $Var(Y) = \frac{n(1-p)}{p^2}$ となる。
いま $p=0.03, n=1000$ なので、$E[S(X)] = 33333.33, Var(S(X)) = 1077777.78$ となる。
和の近似的な分布
$Var(X_i) < \infty$ より、 $X_1, X_2, \ldots, X_n$ の平均 $Y = E[X]$ の分布は、
\begin{eqnarray}
P(Y\leq y) = F(y) \overset{\text{d}}{\longrightarrow} \int_{-\infty}^{\infty} \frac{\sqrt{n}}{\sqrt{2 \pi \sigma^2}} exp\left( {-\cfrac{n(y-\mu)^2}{2\sigma^2}} \right ) \ dy
\end{eqnarray}
となって、正規分布 $N(\mu, \frac{\sigma^2}{n})$ に分布収束する。
したがって和の近似的な分布は、（$Z = nY$ と変数変換すれば良いので）
$$
S(X) = n E[X] \sim N(n\mu, n\sigma^2)
$$
となる。
$X_i$ の母平均, 母分散は $\mu = 0.03, \sigma = 1077.77\ldots$ だったので、$E[S] = 33.33, Var(S) = 1077777.78$ となる。
和の分布の①正確分布、②近似分布、③シミュレーションを比較
Can you write Python code for it?

seed = 282
np.random.seed(seed=seed)
rs = np.random.RandomState(seed)
parameter = 0.03
size = 1000
sum_max = 100000
ave = 1/parameter
var = (1-parameter)/parameter**2
sum_x = np.arange(100001)

# 「幾何分布から1000個のサンプルを取り出し、和を取る」
#  ことを10万回繰り返し、シミュレーションで分布を求める
sim_size = 100000
sums = np.zeros(sim_size, dtype=float)
for i in range(sim_size):
    sums[i] = rs.geometric(p=parameter, size=1000).sum()

# 負の二項分布 n=1000, p=0.03
# scipy.statの負の二項分布は「失敗回数」を数える分布なので、適宜修正
nbin = stats.nbinom.pmf(sum_x-1000, 1000, parameter)

# 正規分布 mu=3333.33, sigma^2=107777.78
normal_ave = size * ave
normal_var = size * var
normal = stats.norm.pdf(x=sum_x, loc=normal_ave, scale=pow(normal_var, 0.5))

# プロット
fig, ax = plt.subplots(figsize=(20, 6))
plt.title("Geo(0.03)から抽出した1000個のサンプルの和S(X)の分布")
plt.plot(sum_x, nbin, linewidth=2, color='blue', label="theoretical pmf(負の二項分布)")
plt.plot(sum_x, normal, linewidth=2, color='orange', label="approx pdf(正規分布)")
plt.hist(sums, bins=100, color='#cccccc', normed=True, label="simulation(size=100000)")
ax.set_xlim([25000, 45000])
plt.xlabel("ts_length(X)")
plt.ylabel("density(f)")
plt.legend()
plt.show()


I am working on the file "PrisonersDilemma/.ipynb_checkpoints/ts_length-checkpoint.ipynb".
The first task is:
近似分布の方が正確な分布よりもやや右側に寄っているが、ほぼ一致している。細かく統計量を見ると、
Can you write Python code for it?

# 統計量
data = np.zeros((11, 3), dtype=float)

ordered_sum = np.sort(sums)
sim_lower_percent = lambda p: (ordered_sum[sim_size*p-1]+ordered_sum[sim_size*p])/2
data[0, 0] = sums.mean()
data[1, 0] = sums.var()
data[2, 0] = sim_lower_percent(0.01)
data[3, 0] = sim_lower_percent(0.025)
data[4, 0] = sim_lower_percent(0.05)
data[5, 0] = sim_lower_percent(0.1)
data[6, 0] = np.median(sums)
data[7, 0] = sim_lower_percent(0.9)
data[8, 0] = sim_lower_percent(0.95)
data[9, 0] = sim_lower_percent(0.975)
data[10, 0] = sim_lower_percent(0.99)

data[0, 1] = stats.nbinom.mean(1000, parameter) + 1000
data[1, 1] = stats.nbinom.var(1000, parameter)
data[2, 1] = stats.nbinom.interval(0.98, 1000, parameter)[0]+1000
data[3, 1] = stats.nbinom.interval(0.95, 1000, parameter)[0]+1000
data[4, 1] = stats.nbinom.interval(0.9, 1000, parameter)[0]+1000
data[5, 1] = stats.nbinom.interval(0.8, 1000, parameter)[0]+1000
data[6, 1] = stats.nbinom.median(1000, parameter) + 1000
data[7, 1] = stats.nbinom.interval(0.8, 1000, parameter)[1]+1000
data[8, 1] = stats.nbinom.interval(0.9, 1000, parameter)[1]+1000
data[9, 1] = stats.nbinom.interval(0.95, 1000, parameter)[1]+1000
data[10, 1] = stats.nbinom.interval(0.98, 1000, parameter)[1]+1000

data[0, 2] = stats.norm.mean(loc=normal_ave, scale=pow(normal_var, 0.5))
data[1, 2] = stats.norm.var(loc=normal_ave, scale=pow(normal_var, 0.5))
data[2, 2] = stats.norm.interval(0.98, loc=normal_ave, scale=pow(normal_var, 0.5))[0]
data[3, 2] = stats.norm.interval(0.95, loc=normal_ave, scale=pow(normal_var, 0.5))[0]
data[4, 2] = stats.norm.interval(0.9, loc=normal_ave, scale=pow(normal_var, 0.5))[0]
data[5, 2] = stats.norm.interval(0.8, loc=normal_ave, scale=pow(normal_var, 0.5))[0]
data[6, 2] = stats.norm.median(loc=normal_ave, scale=pow(normal_var, 0.5))
data[7, 2] = stats.norm.interval(0.8, loc=normal_ave, scale=pow(normal_var, 0.5))[1]
data[8, 2] = stats.norm.interval(0.9, loc=normal_ave, scale=pow(normal_var, 0.5))[1]
data[9, 2] = stats.norm.interval(0.95, loc=normal_ave, scale=pow(normal_var, 0.5))[1]
data[10,...


# 総期数の仮説検定
def two_sided_p(x):
    parameter = 0.03
    med = stats.nbinom.median(1000, parameter) + 1000
    if x <= med:
        for p in np.arange(0, 1.005, 0.005):
            point = stats.nbinom.interval(p, 1000, parameter)[0]+1000
            if point <= x:
                return 1-p
    else:
        for p in np.arange(0, 1.005, 0.005):
            point = stats.nbinom.interval(p, 1000, parameter)[1]+1000
            if x <= point:
                return 1-p

    raise ValueError

def one_sided_p(x):
    parameter = 0.03
    med = stats.nbinom.median(1000, parameter)+1000
    if x <= med:
        p = stats.nbinom.cdf(x-1000, 1000, parameter)
        return p
    else:
        p = stats.nbinom.cdf(x-1000, 1000, parameter)
        return 1-p

    raise ValueError

print("尾山ゼミの総期数 S=32856")
print("両側検定でのp値:", two_sided_p(32856))
print("片側検定でのp値:", one_sided_p(32856))
print("")
print("神取ゼミの総期数 S=35510")
print("両側検定でのp値:", two_sided_p(35510))
print("片側検定でのp値:", one_sided_p(35510))


I am working on the file "PrisonersDilemma/experiment3/ts_length.ipynb".
The first task is:
各セッションの期数の決め方は、以下の通り。
引用: README#共通ルール

ゲームを何期続けるかは、以下のルールによって決めます。

第1期は確率1で到来する

以降は毎期末に抽選を行い、**97%の確率でゲームを継続する**（3%の確率で終了する）
（これは、無限回繰り返しゲームにおいて現在割引価値を0.97と取ることを意味します）

平均は33.33期になります。

したがって、各セッションの期数を確率変数 $X_1, X_2, \ldots, X_{1000}$ とすると、
$$
X_1, X_2, \ldots, X_{1000} \overset{\text{i.i.d.}}{\sim} Geo(0.03)
$$
となる。ただしGeo(p)は「成功確率がpの時、初めて成功するまでの総試行回数」を表す幾何分布で、（下側）累積分布関数は
$$
P(X \leq x) = F(x) = \sum_{k=1}^{x} p(1-p)^{k-1} \hspace{20pt} x=1, 2, \ldots
$$
となる。
平均は
\begin{eqnarray}
E[X] &=& \sum_{x=1}^{\infty} x pq^{x-1}  \hspace{20pt} q =  1-p\
&=& p\left(\frac{\partial}{\partial q} \sum_{x=1}^{\infty} q^x \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{q}{1-q} \right) \
&=& p \frac{1-q+q}{(1-q)^2} \
&=& \frac{1}{p}
\end{eqnarray}
分散は
\begin{eqnarray}
E[X(X+1)] &=& \sum_{x=1}^{\infty} x(x+1) pq^{x-1}\
&=& p\left(\frac{\partial^2}{\partial q^2} \sum_{x=1}^{\infty} q^{x+1} \right) \
&=& p \left(\frac{\partial^2}{\partial q^2} \frac{q^2}{1-q} \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{2q(1-q)+q^2}{(1-q)^2} \right) \
&=& p \left(\frac{\partial}{\partial q} \frac{2q-q^2}{(1-q)^2} \right) \
&=& p \left(\frac{(2-2q)(1-q)^2 + 2(2q-q^2)(1-q)}{(1-q)^4} \right) \
&=& p \left(\frac{2(1-q)^2 + 4q-2q^2)}{(1-q)^3} \right) \
&=& \frac{2}{p^2}
\end{eqnarray}
\begin{eqnarray}
Var(X) &=& E[X^2] - (E[X])^2 \
&=& E[X(X+1)] - E[X]  - (E[X])^2 \
&=& \frac{2}{p^2} - \frac{1}{p} - \frac{1}{p^2} \
&=& \frac{1-p}{p^2}
\end{eqnarray}
となる。
今回はp=0.03なので、$E[X] = 33.33, Var(X) = 1077.78$ となる。
確率関数と累積分布関数
確率（質量）関数と下側累積分布関数をplot（理論値）。線にしてあるが、実際は離散値（x=1, 2, 3, ...）。
Can you write Python code for it?


I am working on the file "PrisonersDilemma/experiment3/ts_length.ipynb".
The first task is:
確率関数は単調減少。$P(X \leq E[X]) = F(33.33) = 0.634$ より、全体の63.4%程度が平均よりも小さい値であることが期待される。
このことは、幾何分布を適当な指数分布で近似（連続化）できるとすれば、その時間あたりの生起回数はポアソン分布$F(x) = \sum_{k=0}^x\frac{\lambda^k}{k!}e^{- \lambda}$に従うことからもわかる。今1単位時間を33.33期とすれば、1単位時間に1回以上「ゲームが終了する」事象が起こる確率は、$\lambda = 1$ のポアソン分布で$x \geq 1$となる確率なので、
\begin{eqnarray}
P(X\geq1) = 1 - F(0) = \frac{1^0}{0!}e^{- 1} = \frac{1}{e} = 0.6321
\end{eqnarray}
となって、ほぼ一致する。
和の分布
正確な和の分布
幾何分布に従う確率変数 $X_1, X_2, \ldots, X_{1000}$ の和を $S(X)$ とすると、
$$
S(X) \ {\sim} \ Negbin(n=1000, p=0.03)
$$
となる。ただしNegbin(n, p)は「成功確率がpの時、n回成功するまでの総試行回数」を表す負の二項分布で、累積分布関数は、
$$
P(Y \leq y) = F(y) = \sum_{k=n}^{y} {}{k-1}C {n-1} p^n (1-p)^{k-n} \hspace{20pt} y=n, n+1, n+2, \ldots
$$
となる。平均は $E[Y] = \frac{n}{p}$、分散は $Var(Y) = \frac{n(1-p)}{p^2}$ となる。
いま $p=0.03, n=1000$ なので、$E[S(X)] = 33333.33, Var(S(X)) = 1077777.78$ となる。
和の近似的な分布
$Var(X_i) < \infty$ より、 $X_1, X_2, \ldots, X_n$ の平均 $Y = E[X]$ の分布は、
\begin{eqnarray}
P(Y\leq y) = F(y) \overset{\text{d}}{\longrightarrow} \int_{-\infty}^{\infty} \frac{\sqrt{n}}{\sqrt{2 \pi \sigma^2}} exp\left( {-\cfrac{n(y-\mu)^2}{2\sigma^2}} \right ) \ dy
\end{eqnarray}
となって、正規分布 $N(\mu, \frac{\sigma^2}{n})$ に分布収束する。
したがって和の近似的な分布は、（$Z = nY$ と変数変換すれば良いので）
$$
S(X) = n E[X] \sim N(n\mu, n\sigma^2)
$$
となる。
$X_i$ の母平均, 母分散は $\mu = 0.03, \sigma = 1077.77\ldots$ だったので、$E[S] = 33.33, Var(S) = 1077777.78$ となる。
和の分布の①正確分布、②近似分布、③シミュレーションを比較
Can you write Python code for it?


I am working on the file "Private/Exercise2-solution.ipynb".
The first task is:
Task
Make a scatter plot of the depth against age. You might need to downsample the number of points as there are a lot of them:
python
ages_decimated = ages[::10]
ages_decimated = etopoH[::10]
Do you see any systematic trend ?
Can you write Python code for it?

## Code here

# Now try interpolating the bathymetry to a smoothed / downsampled version of the ages.

ages_reduced = ages[::8,::8] # Try different values


I am working on the file "Private/Exercise2-solution.ipynb".
The first task is:
Task
Make a scatterplot with this data too.
Does smoothing help bring out a trend ?
You should be able to use
python
import scipy.optimize.curve_fit as curve_fit
help(curve_fit)
To see if you can fit $ \textrm{depth} = A + B\sqrt{\textrm{age}} $
Can you write Python code for it?

## Your code here

fig = plt.figure(1)
ax1 = fig.add_subplot(111)

ax1.scatter(ages2[::10], etopoH_resized[::10], linewidth=0.0)
ax1.set_xlim(0, 120)

from scipy.optimize import curve_fit

def sqrtage(age, A, B):
    return ( A + B * sqrt(age))

valid_ages = ages_reduced[ np.where(isnan(ages_reduced) == False) ]
# valid_depths =

etopoH_2 = imresize(etopoH_img, ages_reduced.shape, interp='bilinear', mode="F")
valid_depths = etopoH_2[ np.where(isnan(ages_reduced) == False) ]
popt, pcov = curve_fit(sqrtage, valid_ages, valid_depths)

fig = plt.figure(1)
ax1 = fig.add_subplot(111)

ax1.scatter(ages2[::10], etopoH_resized[::10], linewidth=0.0)
ax1.set_xlim(0, 120)
ax1.scatter(valid_ages, popt[0] + popt[1] * np.sqrt(valid_ages))


I am working on the file "Private/Exercise2-solution.ipynb".
The first task is:
Task
Make and image plot of etopoH_1 and ages (check they are the same shape and same mask)
Can you write Python code for it?


I am working on the file "Probabilistic neighbours.ipynb".
The first task is:
Barber follows a generative approach and uses kernel density estimation
(Parzen estimator) to interpret kNN as the limiting case of a mixture of Gaussians. An isotropic Gaussian of width $\sigma^2$ is placed at each data point, and a mixture is used to model each class.
The Parzen estimator
With kernel density estimation we want to approximate a PDF with a mixture of continuos probability distributions. A Parzen estimator centers a probability distribution at each data point $\textbf{x}_n$ as
$P(\textbf{x}) = \frac{1}{N} \sum_{n=1}^{N} P(\textbf{x}|\textbf{x}_{n})$
For a D dimensional $\textbf{x}$ we choose an isotropic Gaussian $P(\textbf{x}|\textbf{x}{n}) = \mathcal{N} (\textbf{x}|\textbf{x}{n}, \sigma^2 \textbf{I}_{D})$, which gives the mixture
$P(x) = \frac{1}{N} \sum_{n=1}^{N} \frac{1}{(2 \pi \sigma^2)^{D/2}}\mathcal{e}^{- (\textbf{x} - \textbf{x}_n)^2 / 2\sigma^2} $
Nearest Neighbour classification
Given classes $c = {0, 1}$,
we consider the following mixture model
$P(\textbf{x}|c=0) = \frac{1}{N_0} \sum_{n \in \textit{class 0}} \mathcal{N}(\textbf{x}| \textbf{x}n, \sigma^2\textbf{I})  =  \frac{1}{N_0} \frac{1}{(2 \pi \sigma^2)^{\frac{D}{2}}} \sum{n \in \textit{class 0}} e^{-(\textbf{x} - \textbf{x}_n)^2 / (2 \sigma^2) } $
$P(\textbf{x}|c=1) = \frac{1}{N_1} \sum_{n \in \textit{class 1}} \mathcal{N}(\textbf{x}| \textbf{x}n, \sigma^2\textbf{I}) = \frac{1}{N_1} \frac{1}{(2 \pi \sigma^2)^{\frac{D}{2}}} \sum{n \in \textit{class 1}} e^{-(\textbf{x} - \textbf{x}_n)^2 / (2 \sigma^2) } $
To classify a new instance $\textbf{x}_{*}$ we calculate the posterior for both classes and take the ratio
$\frac{P(c=0|\textbf{x}{\star})}{P(c=1|\textbf{x}{\star})} = \frac{P(\textbf{x}{\star}|c=0) P(c=0)}{P(\textbf{x}{\star}|c=1) P(c=1)}$. If this ratio is $\gt 1$ than we classify $\textbf{n}{\star}$ as class 0, otherwise as class 1. The class probabilities can be determined by maximum likelihood with the following setting $P(c) = \sum{i=0}^N \frac{N_{i}}{N}$.
TODO: proof!
To understand how this relates to the nearest...
Can you write Python code for it?

import numpy as np

def log_sum_exp(x):
    """
    Log likelihood of a Parzen estimator
    """
    a = np.max(x)
    return a + np.log(np.sum(np.exp(x-a)))

def log_mean_exp(x):
    """
    Log likelihood of a Parzen estimator
    """
    a = np.max(x)
    return a + np.log(np.mean(np.exp(x-a)))

def parzen(x, mu, sigma=1.0):
    """
    Makes a function that allows the evalution of a Parzen
    estimator where the Kernel is a normal distribution with
    stddev sigma and with points at mu.

    Parameters
    -----------
    x : numpy array
        Classification input
    mu : numpy matrix
        Contains the data points over which this distribution is based.
    sigma : scalar
        The standard deviation of the normal distribution around each data \
        point.
    Returns
    -------
    lpdf : callable
        Estimator of the log of the probability density under a point.
    """
    # z = (1 / mu.shape[0]) * (1 / np.sqrt(sigma * np.pi * 2.0))
    z = mu.shape[0] * np.sqrt(sigma * np.pi * 2.0)
    e = (-(x - mu)**2.0) / (2.0 * sigma)
    log_p = log_mean_exp(e)
    return log_p - z

sigmas = [1e-13]

priorC0 = class0_train.T.shape[0] / (class0_train.T.shape[0] + class1_train.T.shape[0])
priorC1 = class1_train.T.shape[0] / (class1_train.T.shape[0] + class1_train.T.shape[0])

for sigma in sigmas:
    correct = 0
    for x in class0_test.T:
        c0p = priorC0 * parzen(x, class0_train.T, sigma=sigma)
        c1p = priorC1 * parzen(x, class1_train.T, sigma=sigma)

        if (c0p / c1p) > 1:
            correct += 1

    print(sigma, correct, class1_test.shape[1], correct / class0_test.shape[1])

    correct = 0
    for x in class1_test.T:
        c0p = priorC0 * parzen(x, class0_train.T, sigma=sigma)
        c1p = priorC1 * parzen(x, class1_train.T, sigma=sigma)

        if (c0p / c1p) <= 1:
            correct += 1

    print(sigma, correct, class0_test.shape[1], correct / (class1_test.shape[1]))


I am working on the file "Probabilistic neighbours.ipynb".
The first task is:
In this notebook I'll show probabilistic interpretation of the nearest neighbours algorith as a mixture of Gaussians. Following Barber 2012. First I'll give an example of ... Next, I'll show how to reformulate ... using
Theory
This pretty much follows from Barber, 2012
kNN is simple to understand and implement, and often used as a baseline.
Some limitations of this approach.
* In metric based methods, how do we measure distance? euclidean distance does not account for how data is distributed
* The whole dataset needs to be stored to make a classification since the novel point must be compared to all of the train points.
* Each distance calculation can be expensive if the datapoints are high dimensional
We can reformulate the kNN as a class conditional mixture of Gaussians.
Probabilistic NN
In Barber 2012 shows a
an of the nearest neighbour method as the limiting case of a mixture model.
What follows is a solution for Exercise 158 from Barber 2012 in python.
Write a routine
SoftNearNeigh(xtrain,xtest,trainlabels,sigma)
to implement soft
nearest  neighbours,  analogous  to
nearNeigh.m
.  Here
sigma
is  the  variance
σ
2
in  equation  (14.3.1).  As
above,  the  file
NNdata.mat
contains  training  and  test  data  for  the  handwritten  digits  5  and  9.   Using
leave  one  out  cross-validation,  find  the  optimal
σ
2
and  use  this  to  compute  the  classification  accuracy  of
the  method  on  the  test  data.   Hint:  you  may  have  numerical  difficulty  with  this  method.   To  avoid  this,
consider  using  the  logarithm,  and  how  to  numerically  compute
log
(
e
a
+
e
b
)
for  large  (negative)
a
and
b
.
See also
logsumexp.m
.
Data
For this exercise we'll be using a subsent of the MNIST dataset provided in BRMLtoolkit at Barber 2012.
Can you write Python code for it?

import scipy.io as sio

nndata = sio.loadmat('/Users/gm/Downloads/BRMLtoolkit/data/NNdata.mat')

nndata


I am working on the file "Probabilistic neighbours.ipynb".
The first task is:
Problem; when (c0p / c1p) we have a tie! How should we interpret that? My assumption is to assing ties to class 1!
kNN
Can you write Python code for it?

X_train = np.append(class0_train.T, class1_train.T, axis=0)
y_train = ['a'] * class0_train.shape[1] + ['b'] * class1_train.shape[1]

X_train.shape

X_test = np.append(class0_test.T, class1_test.T, axis=0)
y_test = ['a'] * class0_test.shape[1] + ['b'] * class1_test.shape[1]

X_test.shape, len(y_test)

from sklearn.neighbors import  KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=10,
                           n_jobs=4,
                           metric='euclidean',
                           algorithm='ball_tree')
fitted = knn.fit(X_train, y_train)

?KNeighborsClassifier

y_pred = fitted.predict(X_test)

from sklearn.metrics import accuracy_score, confusion_matrix

print("Accuracy: {}", accuracy_score(y_test, y_pred))
print(confusion_matrix(y_test, y_pred))


I am working on the file "Probabilistic neighbours.ipynb".
The first task is:
We want to solve a binary classification problem.
Can you write Python code for it?

class0_train = nndata['train5']
class0_test = nndata['test5']

class1_train = nndata['train9']
class1_test = nndata['test9']


I am working on the file "ProbabilisticGenerativeModel.ipynb".
The first task is:
使用 scipy 來計算 gaussian density 的範例
Can you write Python code for it?

from scipy.stats import multivariate_normal
x = np.linspace(0, 5, 10, endpoint=False)
y = multivariate_normal.pdf(x, mean=2.5, cov=0.5)

plt.plot(x, y)
plt.show()


I am working on the file "ProbabilisticGenerativeModel.ipynb".
The first task is:
假定題目我們要來預測寶可夢的種類

為二元分類問題: 是 fying 或者是 normal
直觀的假設 HP 和 Weight 為 Strong Feature
Can you write Python code for it?

normal_f1 = pknormal['hp']
normal_f2 = pknormal['weight']

flying_f1 = pkflying['hp']
flying_f2 = pkflying['weight']

x_c1 = np.array(zip(normal_f1.tolist(),normal_f2.tolist()))
x_c2 = np.array(zip(flying_f1.tolist(),flying_f2.tolist()))
c1_mean = np.mean(x_c1,axis=0)
c2_mean = np.mean(x_c2,axis=0)
xc1_xc1 = x_c1-c1_mean
xc2_xc2 = x_c2-c2_mean

cov_c1 = np.matmul(xc1_xc1.transpose(),xc1_xc1)/c1_mean.shape[0]
cov_c2 = np.matmul(xc2_xc2.transpose(),xc2_xc2)/c2_mean.shape[0]

x = np.append(x_c1,x_c2,axis=0)

y = [0] * x_c1.shape[0] + [1]*x_c2.shape[0]


I am working on the file "ProbabilisticGenerativeModel.ipynb".
The first task is:
從下圖看來，兩個  Gaussian Density Function 所產生出來的 Probability 基準不一至
Can you write Python code for it?

cov = (cov_c1*len(x_c1) + cov_c2*len(x_c2))/(len(x_c1)+len(x_c2))
y1 = multivariate_normal.pdf(x, mean=c1_mean, cov=cov_c1)
y2 = multivariate_normal.pdf(x, mean=c2_mean, cov=cov_c2)
plt.plot(y1)
plt.plot(y2)
plt.show()

error =0
for label,c1p,c2p in zip(y,y1,y2):
    if c1p > c2p and label == 1:
        error += 1
    elif c1p < c2p and label ==0:
        error +=1
print("錯了 {} 個".format(error))


I am working on the file "Probability and Statistics in Python/Guided Project - Winning Jeopardy.ipynb".
The first task is:
Answers in questions
Can you write Python code for it?

def ans_in_q(row):
    match_count = 0
    split_answer = row["clean_answer"].split(" ")
    split_question = row["clean_question"].split(" ")

    try:
        split_answer.remove("the")
    except:
        pass

    if len(split_answer) == 0:
        return 0
    else:
        for word in split_answer:
            if word in split_question:
                match_count += 1
        return match_count / len(split_answer)

jeopardy["answer_in_question"] = jeopardy.apply(ans_in_q, axis=1)
print(jeopardy["answer_in_question"].mean())
jeopardy[jeopardy["answer_in_question"] > 0].head()

jeopardy[(jeopardy["answer_in_question"] > 0) & (jeopardy["clean_question"].apply(string.split).apply(len) > 6)].head()


I am working on the file "Probability and Statistics in Python/Guided Project - Winning Jeopardy.ipynb".
The first task is:
Low value vs high value questions
Can you write Python code for it?

def value(row):
    if row["clean_value"] > 800:
        value = 1
    else:
        value = 0
    return value

jeopardy["high_value"] = jeopardy.apply(value, axis=1)
jeopardy.head()


I am working on the file "Probability and Statistics in Python/Guided Project - Winning Jeopardy.ipynb".
The first task is:
Normalizing columns
Can you write Python code for it?

def norm_value(value):
    try:
        value = int(value.translate(None, string.punctuation))
    except:
        value = 0
    return value

jeopardy["clean_value"] = jeopardy["Value"].apply(norm_value)
jeopardy["Air Date"] = pd.to_datetime(jeopardy["Air Date"])

print(jeopardy.dtypes)
jeopardy.head()


I am working on the file "Probability and Statistics in Python/Guided Project - Winning Jeopardy.ipynb".
The first task is:
Normalizing text
Can you write Python code for it?

import string

def norm_words(words):
    words = words.lower().translate(None, string.punctuation)
    return words

jeopardy["clean_question"] = jeopardy["Question"].apply(norm_words)
jeopardy["clean_answer"] = jeopardy["Answer"].apply(norm_words)

jeopardy.head()


I am working on the file "Probability and Statistics in Python/Guided Project - Winning Jeopardy.ipynb".
The first task is:
Only 0.6% of the answers appear in the questions itself. Out of this 0.6%, a sample of the questions shows that they are all multiple choice questions, which concludes that it is very unlikely that the answer will be in the question itself.
Recycled questions
Can you write Python code for it?

jeopardy = jeopardy.sort_values(by="Air Date")

question_overlap = []
terms_used = set()

for index, row in jeopardy.iterrows():
    match_count = 0
    split_question = row["clean_question"].split(" ")
    for word in split_question:
        if len(word) < 6:
            split_question.remove(word)
    for word in split_question:
        if word in terms_used:
            match_count += 1
        terms_used.add(word)
    if len(split_question) > 0:
        match_count /= float(len(split_question))
    question_overlap.append(match_count)

jeopardy["question_overlap"] = question_overlap
print(jeopardy["question_overlap"].mean())

jeopardy.tail()


I am working on the file "Probability/.ipynb_checkpoints/Calvin-checkpoint.ipynb".
The first task is:
Now let's try the path for a single path of 2 lights and 1 wand and assuming a waiting time of 0 seconds. This is, he will not consider for how many seconds he needs to wait, he will automatically use the magic wand even if he needs to wait 1 second.
Can you write Python code for it?

path = TrafficLightPath(1, 0, 1, 2, [0, 0])
path.printPath()
print 'Total waiting time ' + str(path.computeTotalMeanWaitingTime()) + ' seconds'


I am working on the file "Probability/.ipynb_checkpoints/Calvin-checkpoint.ipynb".
The first task is:
Now, let's optimize this, let's see which would be the optimal value he needs to wait to use the magic wand. For this we compute all the possible waiting times from (0 to 80) and see where the optimal value is.
Can you write Python code for it?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

TwL = pd.Series(index=np.arange(0,80,0.1))

N_WANDS = 1
N_LIGHTS = 2

for i in TwL.index:
    path = TrafficLightPath(1, 0, N_WANDS, N_LIGHTS ,[i, np.nan])
    TwL.loc[i] = path.computeTotalMeanWaitingTime()
TwL.plot()
plt.hlines(TwL.min(), 0, TwL.index[TwL==TwL.min()][0], linestyles='--')
plt.vlines(TwL.index[TwL==TwL.min()][0], 0, TwL.min(), linestyles='--')
plt.title ('Mean waiting time')
plt.xlabel('Time to wait for using the magic wand [s]')
plt.ylabel('Total mean waiting time')
plt.show()
print 'Optimal waiting time is ' +  str(TwL.min()) + ' seconds.'


I am working on the file "Probability/Calvin.ipynb".
The first task is:
Now let's try the path for a single path of 2 lights and 1 wand and assuming a waiting time of 0 seconds. This is, he will not consider for how many seconds he needs to wait, he will automatically use the magic wand even if he needs to wait 1 second.
Can you write Python code for it?


I am working on the file "Probability/Calvin.ipynb".
The first task is:
Now, let's optimize this, let's see which would be the optimal value he needs to wait to use the magic wand. For this we compute all the possible waiting times from (0 to 80) and see where the optimal value is.
Can you write Python code for it?


I am working on the file "Probability/Calvin.ipynb".
The first task is:
One can see that the problem presents a multidimensional optimization problem, being the dimension the number of traffic lights minus 1. Now, let's optimize the case for 3 traffic lights:
Can you write Python code for it?

N_WANDS = 1
N_LIGHTS = 3

[L1vec, L2vec] = np.meshgrid(np.arange(0,80, 1), np.arange(0,80, 1))
Tw = np.zeros(L1vec.shape)
for i in range(Tw.shape[0]):
    for j in range(Tw.shape[1]):
        path = TrafficLightPath(1,0, N_WANDS, N_LIGHTS, [L1vec[i,j], L2vec[i,j], np.nan])
        Tw[i,j] = path.computeTotalMeanWaitingTime()

min_ind = np.unravel_index(Tw.argmin(), Tw.shape)
path = TrafficLightPath(1, 0, N_WANDS, N_LIGHTS,[L1vec[min_ind[0], min_ind[1]], L2vec[min_ind[0], min_ind[1]], np.nan])
if path.computeTotalMeanWaitingTime() == Tw.min():
    print 'Min time = ' + str(Tw.min())
    print 'L1 = ' + str(L1vec[min_ind[0], min_ind[1]])
    print 'L2 = ' + str(L2vec[min_ind[0], min_ind[1]])

plt.imshow(Tw)
plt.hlines(min_ind[0], 0, min_ind[1], linestyles='--')
plt.vlines(min_ind[1], 0, min_ind[0], linestyles='--')
plt.colorbar()
plt.show()
print 'Optimal waiting time is ' +  str(Tw.min()) + ' seconds.'


I am working on the file "ProbabilityBirthdaySurprise.ipynb".
The first task is:
After we have k people with different birthdays, we have 365-k possible days left.  But what we really want is the fraction of days in the year left.  Obviously, divide by the number of days in the year.  Let's do that and see where we are.
Can you write Python code for it?


I am working on the file "ProbabilityBirthdaySurprise.ipynb".
The first task is:
Ah, but remember the inferential part.  Given k people, what is the probability they all have different birthdays.  That's the product of these diminishing fractions.
Can you write Python code for it?

# each column in our table is really a sequence of values
bday["frac"]


I am working on the file "ProbabilityBirthdaySurprise.ipynb".
The first task is:
Inferential thinking.  I sit down next to a random person in class, what is the chance that I have the same birthday as that person?
Can you write Python code for it?

# what is the chance that we have different birthdays?
364/365

# what is the chance that we have the same birdthday?
1 - 364/365

# what is the chance that my birthday is different from both my neighbors?
(364/365) * (363/365)

# the same as one of them?
1 - (364/365) * (363/365)

# the same as one of my four neighbors?
1 - (364/365) * (363/365) * (362/365) * (361/365)


I am working on the file "ProbabilityBirthdaySurprise.ipynb".
The first task is:
Now that we understanding this by building it up step by step, could we put it all into one place that we might call a program for answering this question?  Sure.
Can you write Python code for it?

# altogether now like a real program
bd = Table()
bd["day"] = range(365)
bd['left']= 364-bd['day']
bd['frac']=bd['left']/365
bd["different"] = np.cumprod(bd["frac"])
bd['some same bday'] = 1-bd['different']
bd.select('some same bday').take(range(50)).plot()

bd


I am working on the file "ProbabilityBirthdaySurprise.ipynb".
The first task is:
Phew that's a lot of numbers... tables gives us a little peek, but we can always look into it for more.  Sure beats scrolling through 365 rows in excel!
So we need something to take the running product of a bunch of numbers.  These things you'll learn to just build.  But lots of folks built useful ones already.  That's a beautiful thing about computing - you can naturally build on the work of others. Here we'll use the 'cumulative product' tool from the 'numpy' library.  Don't worry you'll see that later.  The important thing it that it does what we did for 2, 3 or 4 neighbors - but for all of them.
Can you write Python code for it?

bday["different"] = np.cumprod(bday["frac"])
bday

# finally the probably that at least two people have the same birthday
bday['some same bday'] = 1-bday['different']
bday

# so 14% with just ten people.  How about the whole story
# Table.select produces a table containing only the selected columns
bday.select('some same bday').plot()

# wow, let's look at the start of that
# Table.take produces a table containing the rows taken from a Table.
bday.take(range(50))

# Table methods generally produce new tables so they compose naturally.
# Here to convey the essence of the Birthday Surprise
bday.take(range(50)).select('some same bday').plot()

# Since indexing a column by its name gives an array, it can be indexed.
bday['some same bday'][20]


I am working on the file "ProbabilityDistriputions.ipynb".
The first task is:
Hypergeometric Distribution
N: The number of items in the population.
k: The number of items in the population that are classified as successes.
n: The number of items in the sample.
x: The number of items in the sample that are classified as successes.
kCx: The number of combinations of k things, taken x at a time.
h(x; N, n, k): hypergeometric probability - the probability that an n-trial hypergeometric experiment results in exactly x successes, when the population consists of N items, k of which are classified as successes.
Can you write Python code for it?

N_values = [10, 50, 50]        # list of the number of items in the population
n_values = [5, 5, 5]           # list of the number of items in the sample
K_values = [5, 25, 3]          # list of the number of items in the population that are classified as successes

linestyles = ['o', 's', '*']   # list of different line styles
color = ['r','b','g']          # list of different line colors

fig, ax =  plt.subplots(figsize=(15, 15))   # figure handles

# zip is used in order to loop over multiple lists of same length
for (N, n, K,ls,c) in zip(N_values, n_values, K_values,linestyles,color):

    dist = hypergeom(N, n, K)     # create a hypergeometric distribution
    x = np.arange(0, n+1)         # generate array of numbers in range of 0:n with step of 1
    ax.plot(x, dist.pmf(x), c+ls,label = r'$N=%2i,  n=%2i,  K=%2i$' % (N, n,K))
    ax.vlines(x,0 ,dist.pmf(x),lw = 4,color = 'y') # plot vertical lines

plt.xlim(-1, 6)
plt.ylim(0, 0.8)
plt.grid(True)
plt.xlabel('$x$')
plt.ylabel(r'$p(x|N;n;k)$')
plt.title('Hypergeometric  Distribution')

ax.legend()
plt.show()

dist = hypergeom(N_values[0], n_values[0],K_values[0])
# Probability Mass Function
print 'Probability Mass Function of 3 is: {v}'.format(v = dist.pmf(3))

# Comulative Distribution Function
print 'Comulative Distribution function of 3 is: {v}'.format(v = dist.cdf(3))


I am working on the file "ProbabilityDistriputions.ipynb".
The first task is:
Normal Distribution
Can you write Python code for it?

ltx = '$ pdf(x,\\mu,\\sigma) = \\frac{1}{ \\sigma' + \
 '\\sqrt{2 \\pi}} e^{\\left(-\\frac{{\\left(\\mu - ' + \
 'x\\right)}^{2}}{2 \\, \\sigma^{2}}\\right)}$'

display(Latex(ltx)) # to display gaussian distribution equation

# Comulative Distribution Function
mu =0                                            # mean
sigma_values = [0.5, 1.0, 2.0]                   # list of standard deviations
linestyles = ['-', '--', ':']                    # list of different line styles
x = np.linspace(-10, 10,1000)                    # generate list of data from -10:10 with 1000 point

fig, ax = plt.subplots(figsize=(15, 5))          # figure handles

# zip is used in order to loop over multiple lists of same length
for (sigma, ls) in zip(sigma_values, linestyles):

    dist = norm(mu, sigma)  # create a normal distribution
    # plot probability mass function of each distribution with the corresponding parameters
    ax.plot(x, dist.pdf(x), ls=ls, c='black',lw=3,
             label=r'$\mu=%i,\ \sigma=%.1f$' % (mu, sigma))

plt.xlim(-5, 5)
plt.ylim(0, 0.85)

plt.xlabel('$x$')
plt.ylabel(r'$p(x|\mu,\sigma)$')
plt.title('Gaussian Distribution')
plt.grid(True)
ax.legend()

dist = norm(mu, sigma_values[0])          # initialize a distribution with mu = 0, sigma = 0.5
# Probability Density Function.
print 'Probability Density Function of 0.4 is: {v}'.format(v = dist.pdf(0.4))
ax.vlines(0.4,0,dist.pdf(0.4),colors='r', linestyles='--', lw=3)
# Comulative Distribution Function
print 'Comulative Distribution function of 0.4 is: {v}'.format(v = dist.cdf(0.4))

plt.show()

def normspec(interval = [-1,1], mu=0,sigma = 1,region = 'inside'):

    """
    This function plot the area between a given interval on the pdf of normal distribution
    also calculate the cdf probability inside or outside the given interval

    interval : a given interval to calcualate cdf inside or outside. default: interval = [-1,1]
    mu: mean of the normal distribution. default: mu=0
    sigma: standard deviation of the normal...


I am working on the file "ProbabilityDistriputions.ipynb".
The first task is:
Random Sampling
Can you write Python code for it?

# Mean
X = [1, 2, 3, 4, 5, 6, 8, 9, 10];                       # discrete random variable
f_x=[0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]; # pmf of x
expectedValue = 0;                                      # initialize expectedValue
for idx in range(len(X)):
    expectedValue += X[idx] * f_x[idx]                  # calculate expectedValue

print 'Mean of X vector is: {v}'.format(v = expectedValue)

# Standard Deviation
Var = 0;                                          # initialize Variance
for idx in range(len(X)):
    Var += ((X[idx]-expectedValue)**2) * f_x[idx] # calculate variance

SDe=sqrt(Var);                                    # calculate standard deviation
print 'Standard Deviation of X vector is: {v}'.format(v = SDe)


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
Ah, but remember the inferential part.  Given k people, what is the probability they all have different birthdays.  That's the product of these diminishing fractions.
Can you write Python code for it?


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
Inferential thinking.  I sit down next to a random person in class, what is the chance that I have the same birthday as that person?
Can you write Python code for it?


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
Now that we understanding this by building it up step by step, could we put it all into one place that we might call a program for answering this question?  Sure.
Can you write Python code for it?


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
OK, that looks like its going from 1 to 0 just as we'd expect.  And we can see how things are working together, just as we might in a spread sheet.  We can focus on the data that comes out of the computation.  In a spreadsheet this would be all spread around in the cells.  Here the computation is clearly laid out and we can see how it progresses from one step to the next by building up the table.
We might want to select just 'fraction left' to look at.
Can you write Python code for it?


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
Phew that's a lot of numbers... tables gives us a little peek, but we can always look into it for more.  Sure beats scrolling through 365 rows in excel!
So we need something to take the running product of a bunch of numbers.  These things you'll learn to just build.  But lots of folks built useful ones already.  That's a beautiful thing about computing - you can naturally build on the work of others. Here we'll use the 'cumulative product' tool from the 'numpy' library.  Don't worry you'll see that later.  The important thing it that it does what we did for 2, 3 or 4 neighbors - but for all of them.
Can you write Python code for it?


I am working on the file "Probability_Birthday_surprise.ipynb".
The first task is:
Wow, nearly 3%?
Computational thinking?  For a group of N people, what is the probability that at least two of them have the same birthday?  An algorithmic question.  Given K people with different birthdays, what is the probability that the next random person will have a different birthday than any of them?  If we can answer this single step, we have a way to answer the general question?  Yes, it is like induction.  But it is constructive too.  That's an algorithm.
Let's create a little table, kind of like what might think of as a spreadsheet, to collect all of our computational and inferential thoughts about solving this problem.
Can you write Python code for it?

bday = Table()

# lets start with numbering the days in the year (we'll start with zero).  We don't care if they are
# in calendar order, backwars, sideways or the ways that we encounter them in meeting random people
bday["day"] = range(365)

bday


I am working on the file "Problem Set (Solutions).ipynb".
The first task is:
Question: Given the data above, consider this question from the viewpoint of a data scientist/data analyst. What factors do you need to consider when tweaking model parameters?
Answers:
Gains in model accuracy/performance vs. time it takes to run.
Problem 4

Pick the best model from above, and re-train it on the dataset again. Refer to the Lecture notebook for a version of the code that may help here!
Now, use it to make predictions on the global HIV protease dataset.
Plot the global distribution of drug resistance values.
Can you write Python code for it?

# Load in the data and binarize it.
proteases = [s for s in SeqIO.parse('sequences/HIV1-protease.fasta', 'fasta') if len(s) == 99]
alignment = MultipleSeqAlignment(proteases)
proteases_df = pd.DataFrame(np.array([list(rec) for rec in alignment], str))
proteases_df.index = [s.id for s in proteases]
proteases_df.columns = [i for i in range(1, 100)]
X_global = cf.binarize_seqfeature(proteases_df)

# Train your model here, with optimized parameters for best MSE minimization.
### BEGIN
model = RandomForestRegressor()  # put your best model here.
model.fit(X_binarized, Y)
preds = model.predict(X_global)
plt.hist(preds)
### END


I am working on the file "Problem Set (Solutions).ipynb".
The first task is:
Question: Which two drugs are most correlated?
Answer: FPV and DRV
Question: Why might they be correlated? (Hint: you can look online for what they look like.)
Answer: They have very similar chemical structures.
Problem 3
Fill in the code below to plot the relationship between number of estimators (X-axis) and the MSE value for each of the estimators.

Try 10, 30, 50, 80, 100, 300, 500 and 800 estimators.
Use the ShuffleSplit iterator with cross-validation.
Use mean of at least 5 cross-validated MSE scores.
Can you write Python code for it?

# Fill in the code here to clean the data.
def return_cleaned_data(drug_name, data):
    # Select the subsets of columns of interest.
    # Fade out the drug_name and feat_cols variables
    cols_of_interest = []
    cols_of_interest.append(drug_name)
    cols_of_interest.extend(feat_cols)

    subset = data[cols_of_interest].dropna()  # fade out .dropna()
    Y = subset[drug_name]  # fade out drug_name, fade out .apply(np.log10)
    X = subset[feat_cols]
    # We call on a custom function to binarize the sequence feature matrix.
    # You can inspect the code in the custom_funcs.py file.
    lb = LabelBinarizer()
    lb.fit(list('CHIMSVAGLPTRFYWDNEQK'))

    X_binarized = pd.DataFrame()

    for col in X.columns:
        binarized_cols = lb.transform(X[col])

        for i, c in enumerate(lb.classes_):
            X_binarized[col + '_' + c] = binarized_cols[:,i]

    return X_binarized, Y

X_binarized, Y = return_cleaned_data('FPV', data)
len(X_binarized), len(Y)

num_estimators = [10, 30, 50, 80, 100, 300, 500 800]  # fill in the number of estimators to try here.
models = {'Random Forest':RandomForestRegressor,
          'Ada Boost':AdaBoostRegressor,
          'Gradient Boost':GradientBoostingRegressor,
          'Extra Trees':ExtraTreesRegressor}  # fill in the other models here

# Initialize a dictionary to hold the models' MSE values.
mses = dict()
for model_name, model in models.items():
    mses[model_name] = dict()
    for n in num_estimators:
        mses[model_name][n] = 0

# Iterate over the models, and number of estimators.
for model_name, model in models.items():
    for n_est in num_estimators:
        ### Begin Here
        print(model_name, n_est)
        # Set up the cross-validation iterator
        cv_iterator = cv.ShuffleSplit(len(X_binarized), test_size=0.3, n_iter=5)

        # Initialize the model
        m = model(n_estimators=n_est) # fill in the parameters

        # Collect the cross-validation scores. Remember that mse will be negative, and needs to
      ...


I am working on the file "Problem Set.ipynb".
The first task is:
Problem Set on Machine Learning
Problem 1
Identify an academic literature reference that descirbes the PhenoSense assay. Paste the URL to the PubMed article below, and write a 1-2 sentence summary on what is measured in the assay, and how it relates to drug resistance.
Compare and contrast it with the plaque reduction assay as mentioned in the literature - what would be one advantage of the plaque reduction assay that is lacking in PhenoSense, and vice versa?
Answer
Double-click on this cell to type in your answer. Use Markdown formatting if you'd like.
A new paragraph is delineated by having a line in between them. You can bold or italicize text.

Bulleted

Lists

are done this way.
4 spaces for indents.

Numbered

Lists
are done this way.
4 spaces for indents.
The numbering is automatically parsed!

Leave the answer at the top so Claire can know where your answer is!
Problem 2

Write code below to calculate the correlation between two drugs' resistance profiles. Identify the protease drugs for which the two drugs' resistance values are correlated.
Speculate as to why they would be correlated.
Can you write Python code for it?

# This cell loads the data and cleans it for you, and log10 transforms the drug resistance values.
# Remember to run this cell if you want to have the data loaded into memory.
DATA_HANDLE = 'drug_data/hiv-protease-data.csv'  # specify the relative path to the protease drug resistance data
N_DATA = 8  # specify the number of columns in the CSV file that are drug resistance measurements.
CONSENSUS = 'sequences/hiv-protease-consensus.fasta'  # specify the relative path to the HIV protease consensus sequence
data, drug_cols, feat_cols = cf.read_data(DATA_HANDLE, N_DATA)
consensus_map = cf.read_consensus(CONSENSUS)
data = cf.clean_data(data, feat_cols, consensus_map)
for name in drug_cols:
    data[name] = data[name].apply(np.log10)
data.head()

"""
Complete the function below to compute the correlation score.

Use the scipy.stats.pearsonr(x, y) function to find the correlation score between two arrays of things.
You do not need to type the whole name, as I have imported the pearsonr name for you, so you only have to do:

    pearsonr(x, y)

Procedure:

1. Select two columns' names to compare.
2. Make sure to drop NaN values. the pearsonr function cannot deal with NaN values.
   (Refer to the Lecture notebook if you forgot how to do this.)
3. Pass the data in to pearsonr().

"""
def corr_score(drug1, drug2):
    ### BEGIN SOLUTION
    # Get the subset of data, while dropping columns that have NaN in them.

    # Return the pearsonr score.
    return pearsonr(____________, ____________)
    ### END SOLUTION

assert corr_score('IDV', 'FPV') == (0.79921991532901282, 2.6346448659104859e-306)
assert corr_score('ATV', 'FPV') == (0.82009597442033089, 2.5199367322520278e-231)
assert corr_score('NFV', 'DRV') == (0.69148264851159791, 4.0640711263961111e-82)
assert corr_score('LPV', 'SQV') == (0.76682619729899326, 4.2705737581002648e-234)


I am working on the file "Problem Set.ipynb".
The first task is:
Question: Which two drugs are most correlated?
Answer
Question: Why might they be correlated? (Hint: you can look online for what they look like.)
Answer
Problem 3
Fill in the code below to plot the relationship between number of estimators (X-axis) and the MSE value for each of the estimators.

Try 10, 30, 50, 80, 100, 300, 500 and 800 estimators.
Use the ShuffleSplit iterator with cross-validation.
Use mean of at least 5 cross-validated MSE scores.
Can you write Python code for it?

def return_cleaned_data(drug_name, data):
    # Select the subsets of columns of interest.
    cols_of_interest = []
    cols_of_interest.append(drug_name)
    cols_of_interest.extend(feat_cols)
    subset = data[cols_of_interest].dropna()
    Y = subset[drug_name]
    X = subset[feat_cols]

    # Binarize the columns.
    lb = LabelBinarizer()
    lb.fit(list('CHIMSVAGLPTRFYWDNEQK'))

    X_binarized = pd.DataFrame()

    for col in X.columns:
        binarized_cols = lb.transform(X[col])

        for i, c in enumerate(lb.classes_):
            X_binarized[col + '_' + c] = binarized_cols[:,i]

    return X_binarized, Y

X_binarized, Y = return_cleaned_data('FPV', data)
len(X_binarized), len(Y)

num_estimators = [_________]  # fill in the list of estimators to try here.
models = {'Random Forest':RandomForestRegressor,
          }  # fill in the other models here

# Initialize a dictionary to hold the models' MSE values.
mses = dict()
for model_name, model in models.items():
    mses[model_name] = dict()
    for n in num_estimators:
        mses[model_name][n] = 0

# Iterate over the models, and number of estimators.
for model_name, model in models.items():
    for n_est in num_estimators:
        print(model_name, n_est)
        ### Begin Here

        # Set up the cross-validation iterator

        # Initialize the model

        # Collect the cross-validation scores. Remember that mse will be negative, and needs to
        # be transformed to be positive.

        ### End Here

        # Store the mean MSEs.
        mses[model_name][n_est] = np.mean(-cv_scores)

# When you're done, run the following cell to make your plot.
pd.DataFrame(mses).plot()
plt.xlabel('Num Estimators')
plt.ylabel('MSE')


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
Are they the same class?
Can you write Python code for it?

ticket_price_90 = titanic_dataframe[(titanic_dataframe['Fare'] >= percentile_expense[0]) & (titanic_dataframe['Pclass'] != 1)]

print(len(ticket_price_90))

ticket_price_5 = titanic_dataframe[(titanic_dataframe['Fare'] <= percentile_expense[1]) & (titanic_dataframe['Pclass'] < 3)]

print(len(ticket_price_5))


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
Display the relationship (i.e. make a plot) between survival rate and the quantile of the ticket price for 20 integer quantiles. Make sure you label your axes.
Can you write Python code for it?

import matplotlib.pyplot as pl
% matplotlib inline

fare_quantiles = np.percentile(titanic_dataframe.Fare, np.arange(5, 105, 5.0))
survival_quantiles = []
previous_quantile = 0
for f_q in fare_quantiles:
    people = titanic_dataframe[(previous_quantile <= titanic_dataframe.Fare) & (titanic_dataframe.Fare < f_q)]
    survival_quantiles.append(sum(people.Survived == 1) / float(len(people)))
pl.plot(np.arange(5, 105, 5.0), survival_quantiles)
pl.xlabel('Fare percentile')
pl.ylabel('Survival rate')


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
For the groups you chose, how far (in years) are the average ages from the median ages?

any Titanic Passenger
Can you write Python code for it?


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
How much more expensive was the 90th percentile ticket than the 5th percentile ticket?
Can you write Python code for it?

import numpy as np
percentile_expense = np.percentile(titanic_dataframe['Fare'], [90, 5])
percentile_expense[0] - percentile_expense[1]


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
Male survivors older than 30 from anywhere but Queenstown
Can you write Python code for it?

is_survivor = titanic_dataframe['Survived'] == 1
is_male = titanic_dataframe['Sex'] == 'male'
not_Queenstown = titanic_dataframe['Embarked'] != 'Q'
over_30 = titanic_dataframe['Age'] > 30
moniq = titanic_dataframe[is_survivor & is_male & not_Queenstown & over_30]
moniq.Age.mean()


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
The highest average ticket price was paid by passengers from which port? Null ports don't count
Can you write Python code for it?

ports = [port for port in titanic_dataframe.Embarked.unique() if port == port]
by_port = [(p, titanic_dataframe[titanic_dataframe.Embarked==p]) for p in ports]
for p, people in by_port:
    print(p, people.Fare.mean())


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
What are the columns and what do they mean?
Can you write Python code for it?


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
What's the most common...

passenger class
Can you write Python code for it?


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
Which port has passengers from the most similar class?
Can you write Python code for it?

for p, people in by_port:
    print(p, people.Pclass.std())


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
a non-surviving first-class passenger
Can you write Python code for it?

dead_rich = titanic_dataframe[(titanic_dataframe.Survived==0)&(titanic_dataframe.Pclass==1)]
dead_rich.Age.mean()


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
a survivor
Can you write Python code for it?


I am working on the file "Problem-Set-Statistics-1.ipynb".
The first task is:
port of Embarkation
Can you write Python code for it?


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
As shown in the table the Breusch–Pagan test indicates (p-value $< .05$) there is some heteroskedasticity in the data. This could greatly affect our inference since some regions have lower error than others and as such our accuracy is dependant on the input.
Part 4
Can you write Python code for it?

names = ['Intercept','female','age','educ']
multicollinearityDF = pandas.DataFrame({n : [model1.eigenvals[i]] for i, n in enumerate(names)})
multicollinearityDF


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
Part 1
Can you write Python code for it?

outliersDf = statsmodels.stats.outliers_influence.OLSInfluence(model1).summary_frame()

outliersDf.max()

outliersDf.min()


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
Part 1
From this table we can see that the the marginal effect of $age$ on $biden$ is $0.6719 - 0.0480educ$, thus it is postive for $educ < \frac{0.6719}{0.0480} \sim 14$ and negative for other values of $educ$.
Can you write Python code for it?


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
Plotting them shows that there are a fair number of influential points (the point labels are original indices of the points). We would first need to determine the criteria for classifying them as outliers. Cook's D above a certain value would be a good starting point. Once we identify them we could drop them. We could also check if they are only influential in one dimension and normalize that dimension to the mean or some other less significant values, while keeping the other values.
Part 2
Can you write Python code for it?

names = ['$\chi^2_2$', 'p-value', 'Skew', 'Kurtosis']
test = statsmodels.stats.api.jarque_bera(model1.resid)
nonNormDF = pandas.DataFrame({n : [test[i]] for i, n in enumerate(names)})
nonNormDF


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
Regression diagnostics
Can you write Python code for it?

model1 = statsmodels.formula.api.ols('biden ~ female + age + educ', data=df).fit()
print(model1.summary())


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
We can see fomt the tables above that $DFBETA$ values for some of the data points are quite signifcant and at least one has a cook's D much greater than $4/n$.
Can you write Python code for it?

fig, ax = plt.subplots(figsize = (20, 7))
fig = statsmodels.graphics.regressionplots.plot_leverage_resid2(model1, ax = ax)
plt.show()
fig, axes = plt.subplots(ncols=2, figsize = (20, 7))
outliersDf[['dfb_Intercept', 'dfb_female', 'dfb_age', 'dfb_educ', 'cooks_d']].boxplot(ax = axes[0])
axes[0].set_title('$DFBETA$ and Cook\'s D boxplots')
outliersDf[['cooks_d']].plot(ax = axes[1])
axes[1].set_title('Cook\'s D per point')
plt.show()


I am working on the file "ProblemSets/PS3/PS3.ipynb".
The first task is:
We can see from the Wald test of for this marginal effect that the null hypothesis of no effect is barely defeated for $p < .05$
Missing data
Can you write Python code for it?

fullDF = pandas.read_csv(bidenFname)

imputer = sklearn.preprocessing.Imputer(strategy='mean')
imputedMeanDF = pandas.DataFrame(imputer.fit_transform(fullDF), columns = ['biden','female','age','educ','dem','rep'])
modelMean = statsmodels.formula.api.ols('biden ~ female + age + educ', data=imputedMeanDF).fit()
print(modelMean.summary())


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
2. Removing Specific Columns
Use this to check whether the prediction gets better if some features are removed. The new data matrix is given by X_EDITED.
Can you write Python code for it?


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
3. KPCA / PCA
We tested different kernels, the best was the linear one. So the KPCA is being used as PCA. Transformed data matrix is given by X_KPCA.
Can you write Python code for it?

from sklearn.decomposition import KernelPCA

kpca = KernelPCA(n_components=9, kernel='linear')
X_KPCA = kpca.fit_transform(X)


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
Boosting
The best classifier we tested was Boosting.
Can you write Python code for it?

# Gradient Boosting
from sklearn.ensemble import GradientBoostingClassifier

model = GradientBoostingClassifier(n_estimators=50, learning_rate=1.0, max_depth=1, random_state=0)
accuracy_boost = cross_val_score(model, X, Y, cv=cross_val_k, scoring='accuracy').mean()
print('boosting', accuracy_boost)

# AdaBoostClassifier - Worse
# AdaBoostClassifier(svm.SVC(probability=True,kernel='linear'),n_estimators=50, learning_rate=1.0, algorithm='SAMME')


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
Classification accuracy

All features, n_estimators = 50

0.971785775714

Removing feature 5

0.971785775714

Removing features 5 and 7

0.971785775714

Using only features [1,6,10,11,12]

0.938050395597
KNN
Algoritmo K-nearest neighbors
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

# Get best k.

k_range = range(1,31)
k_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X, Y, cv=cross_val_k, scoring='accuracy')
    k_scores.append(scores.mean())
#print(k_scores)
max_score = max(k_scores)
print("Best k = " , (k_scores.index(max_score) + 1))
print("Score = " , (max_score ))

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(k_range, k_scores)
plt.xlabel('Value of k for KNN')
plt.ylabel('Cross-Validated Accuracy')


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
Classification accuracy
Using All features + cubic order kernel

poly_svc (degree=3) 0.899092707258

Removing features:

feature 0 removed: 0.910898262814
feature 1 removed: 0.872301771586
feature 2 removed: 0.898800309598
feature 3 removed: 0.893829549364
feature 4 removed: 0.894523993808
feature 5 removed: 0.917107413141
feature 6 removed: 0.893244754042
feature 7 removed: 0.910898262814
feature 8 removed: 0.894523993808
feature 9 removed: 0.909911420709
feature 10 removed: 0.864445734434
feature 11 removed: 0.89342750258
feature 12 removed: 0.777631578947

features [5,7] or [5,0] removed: 0.917107413141

Using 1,6,10,11,12 -> 0.910238218094

Worst feature: 5

Best features: 12, 10, 1, 6, 11, 3

KPCA and SVD:

PCA (KPCA with linear Kernel), using 9 components: 0.92138372893
X_SVD with 12 singularvalues: 0.910898262814 -> Same accuracy as removing feature 0 and/or feature 7.

SVM with different Kernels
In general, the following Kernels resulted in a worse results than the Polynomial Kernel with degree 3
Linear Kernel
Can you write Python code for it?

# SVC with linear kernel
svc = svm.SVC(kernel='linear', C=Penalty_C)
accuracy_svc = cross_val_score(svc, X, Y, cv=cross_val_k, scoring='accuracy').mean()
print('svm linear kernel:', accuracy_svc)


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
Manipulating the Data Matrix (features)
1. SVD
Computing SVD to check for linearly dependent features and/or dimensionality reduction. The new data matrix is given by X_EDITED.
Can you write Python code for it?


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
Polynomial Kernel with degree 2
Can you write Python code for it?

poly_svc = svm.SVC(kernel='poly', degree=2, C=Penalty_C)
accuracy_poly_svc = cross_val_score(poly_svc, X, Y, cv=cross_val_k, scoring='accuracy').mean()
print('poly_svc', accuracy_poly_svc)


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
RBF Kernel
Can you write Python code for it?

rbf_svc = svm.SVC(kernel='rbf', gamma='auto', C=Penalty_C)
accuracy_rbf_svc = cross_val_score(rbf_svc, X, Y, cv=cross_val_k, scoring='accuracy').mean()
print('rbf_svc', accuracy_rbf_svc)


I am working on the file "Problema 1/ML - Project 1.ipynb".
The first task is:
SVM
Can you write Python code for it?

from sklearn import svm

#Penalty parameter C of the error term.
Penalty_C = 1.0  # SVM regularization parameter


I am working on the file "Problema 2/Daniel - Julliana/.ipynb_checkpoints/Amazon-checkpoint.ipynb".
The first task is:
A seguir, retiramos os dois últimos caracteres sobrando apenas o nosso comentário. Depois, passamos ele para lowercase.
Can you write Python code for it?

TextosQuebrados = [ x[:-4] for x in vetor ]

TextosQuebrados = map(lambda X:X.lower(),TextosQuebrados)

#TextosQuebrados = [x.split(' ') for x in TextosQuebrados]

TextosQuebrados = [nltk.tokenize.word_tokenize(frase) for frase in TextosQuebrados]

#X[0]

import nltk
stopwords = nltk.corpus.stopwords.words('english')

stemmer = nltk.stem.RSLPStemmer()

dicionario = set()

for comentarios in TextosQuebrados:
    validas = [stemmer.stem(palavra) for palavra in comentarios if palavra not in stopwords and len(palavra) > 0]
    dicionario.update(validas)

totalDePalavras = len(dicionario)
tuplas = zip(dicionario, xrange(totalDePalavras))
tradutor = {palavra:indice for palavra,indice in tuplas}

def vetorizar_texto(texto, tradutor, stemmer):
    vetor = [0] * len(tradutor)
    for palavra in texto:
        if len(palavra) > 0:
            raiz = stemmer.stem(palavra)
            if raiz in tradutor:
                posicao = tradutor[raiz]
                vetor[posicao] += 1

    return vetor

vetoresDeTexto = [vetorizar_texto(texto, tradutor,stemmer) for texto in TextosQuebrados]
X = vetoresDeTexto

Y = [ x[-1:] for x in vetor ]
len(Y)

porcentagem_de_treino = 0.8

tamanho_do_treino = porcentagem_de_treino * len(Y)
tamanho_de_validacao = len(Y) - tamanho_do_treino

treino_dados = X[0:int(tamanho_do_treino)]
treino_marcacoes = Y[0:int(tamanho_do_treino)]

validacao_dados = X[int(tamanho_do_treino):]
validacao_marcacoes = Y[int(tamanho_do_treino):]

fim_de_teste = tamanho_do_treino + tamanho_de_validacao
teste_dados = X[int(tamanho_do_treino):int(fim_de_teste)]
teste_marcacoes = Y[int(tamanho_do_treino):int(fim_de_teste)]


I am working on the file "Problema 2/Daniel - Julliana/.ipynb_checkpoints/Amazon-checkpoint.ipynb".
The first task is:
Foi decidida a abordagem por poly SCV
Can you write Python code for it?

""""from sklearn import svm
from sklearn.model_selection import cross_val_score
k = 10

# Implement poly SVC
poly_svc = svm.SVC(kernel='linear')
accuracy_poly_svc = cross_val_score(poly_svc, treino_dados, treino_marcacoes, cv=k, scoring='accuracy')
print('poly_svc: ', accuracy_poly_svc.mean())""""


I am working on the file "Problema 2/Daniel - Julliana/.ipynb_checkpoints/Amazon-checkpoint.ipynb".
The first task is:
Resultado - Poly:
Os 3: Após 10 minutos rodando, foi decidido parar o teste
IMdB: 0.51750234411626805
Amazon: 0.51125019534302241
Yelp: 0.56500429754649173
Resultado - Linear:
Os 3: 0.7745982496802607 (5 minutos)
IMdB: 0.72168288013752147
Amazon: 0.78869745272698855
Yelp: 0.77492342553523996
Can you write Python code for it?

def fit_and_predict(nome, modelo, treino_dados, treino_marcacoes, teste_dados, teste_marcacoes):
	modelo.fit(treino_dados, treino_marcacoes)

	resultado = modelo.predict(teste_dados)
	acertos = (resultado == teste_marcacoes)

	total_de_acertos = sum(acertos)
	total_de_elementos = len(teste_dados)
	taxa_de_acerto = float(total_de_acertos) / float(total_de_elementos)

	print(taxa_de_acerto)
	return taxa_de_acerto

resultados = {}

from sklearn.naive_bayes import MultinomialNB
modeloMultinomial = MultinomialNB()

from sklearn.ensemble import AdaBoostClassifier
clf = AdaBoostClassifier(n_estimators=100)

resultadoMultinomial = fit_and_predict("MultinomialNB", AdaBoostClassifier(n_estimators=100), treino_dados, treino_marcacoes, teste_dados, teste_marcacoes)
resultados[resultadoMultinomial] = modeloMultinomial


I am working on the file "Problema 2/Daniel - Julliana/.ipynb_checkpoints/Amazon2-checkpoint.ipynb".
The first task is:
Foi decidida a abordagem por poly SCV
Can you write Python code for it?

""" from sklearn import svm
from sklearn.model_selection import cross_val_score
k = 10

# Implement poly SVC
poly_svc = svm.SVC(kernel='linear')
accuracy_poly_svc = cross_val_score(poly_svc, treino_dados, treino_marcacoes, cv=k, scoring='accuracy')
print('poly_svc: ', accuracy_poly_svc.mean()) """


I am working on the file "Problema 2/Daniel - Julliana/Amazon.ipynb".
The first task is:
Adaboost:
0.527454242928
Can you write Python code for it?

from sklearn.ensemble import GradientBoostingClassifier

classificador = GradientBoostingClassifier(n_estimators=100, learning_rate=1.0, max_depth=1, random_state=0).fit(treino_dados, treino_marcacoes)
resultado = fit_and_predict(classificador, treino_dados, treino_marcacoes, teste_dados, teste_marcacoes)


I am working on the file "Problema 2/Daniel - Julliana/Amazon.ipynb".
The first task is:
Com maior refinamento de dados:
MultinomialNB:
Todos: 0.808652246256
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier
classificador = AdaBoostClassifier(n_estimators=100)

resultado = fit_and_predict(classificador, treino_dados, treino_marcacoes, teste_dados, teste_marcacoes)


I am working on the file "Problema 2/Daniel - Julliana/Amazon.ipynb".
The first task is:
Depois, devemos retirar cada quebra de linha no final de cada linha, ou seja, os '\n'.
Can you write Python code for it?

vetor = [ x[:-1] for x in vetor ]

vetor = ([s.replace('&', '').replace(' - ', '').replace('.', '').replace(',', '').replace('!', '').
          replace('+', '')for s in vetor])


I am working on the file "Problema 2/Daniel - Julliana/Amazon.ipynb".
The first task is:
Foi decidida a abordagem por poly SCV
Can you write Python code for it?


I am working on the file "Problema 2/Daniel - Julliana/Amazon.ipynb".
The first task is:
GradientBoostingClassifier:
Todos: 0.77870216306156403
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

classificador = GaussianNB()
resultado = fit_and_predict(classificador, treino_dados, treino_marcacoes, teste_dados, teste_marcacoes)


I am working on the file "Problema 2/alexandre/Second Project ML.ipynb".
The first task is:
Creating Dictionary
Na variável "charsRemove" são indicados os caracteres a serem removidos do input.
Todas as palavras são transformadas para lower case e os caracteres indesejados são removidos.
Na variável "allWords", são armazenadas todas as palavras a serem utilizadas.
Can you write Python code for it?

# Treating Sentences
allSentences = []
charsSplit = "\\`*_{}[]()>#+-.!$,:&?"
charsRemove = ".,-_;\"\'"
for x in X :
    for c in charsSplit:
        x = x.replace(c, ' '+ c +' ')
    for c in charsRemove:
        x = x.replace(c, '')
    allSentences.append(x.lower() )

allWords = []
for x in allSentences :
    allWords.extend(x.split(" "))

allWords = list(set(allWords))
allWords.sort()