I am working on the file "Logistic Matrix Factorization.ipynb".
The first task is:
The likelihood function
\begin{eqnarray}
p(Y |W, H ) &=& \prod_j \left(\left(\prod_{i : Y(i,j)=1} \sigma(\sum_k W(i,k) H(k,j)) \right) \left(\prod_{i : Y(i,j)=0}(1- \sigma\left((\sum_k W(i,k) H(k,j))\right) \right) \right)
\end{eqnarray}
\begin{eqnarray}
\log p(Y |W, H ) &=& \sum_j \sum_{i : Y(i,j)=1} \log \sigma\left(\sum_k W(i,k) H(k,j)\right) + \sum_j \sum_{i : Y(i,j)=0} \log\left(1- \sigma(\sum_k W(i,k) H(k,j))\right) \
\end{eqnarray}
\begin{eqnarray}
\log \sigma(x) & = & -\log(1+e^{-x}) = x - \log(1+e^{x}) \
\log(1 - \sigma(x)) & = &  -\log({1+e^{x}})
\end{eqnarray}
Using the above properties of the sigmoid function
\begin{eqnarray}
\log p(Y |W, H ) &=& \sum_j\sum_{i : Y(i,j)=1} \left( \left(\sum_k W(i,k) H(k,j)\right) - \log \left(1 + \exp(\sum_k W(i,k) H(k,j))\right) \right) - \sum_j \sum_{i : Y(i,j)=0} \log\left( 1+ \exp\left(\sum_k W(i,k) H(k,j) \right)\right) \
&=& \sum_j \sum_{i} Y(i,j) \left(\sum_k W(i,k) H(k,j)\right)  - \sum_j \sum_{i} \log\left( 1+\exp\left(\sum_k W(i,k) H(k,j)\right)\right)
\end{eqnarray}
With missing values given as a mask matrix $M(i,j)$.
\begin{eqnarray}
\log p(Y |W, H ) &=& \sum_j \sum_{i} M(i,j) Y(i,j) \left(\sum_k W(i,k) H(k,j)\right)  - \sum_j \sum_{i} M(i,j) \log\left( 1+ \exp\left(\sum_k W(i,k) H(k,j)\right)\right)
\end{eqnarray}
\begin{eqnarray}
\frac{\partial}{\partial W(i,k)} \log p(Y |W, H ) &=& \sum_j M(i,j) Y(i,j) H(k,j)  - \sum_j M(i,j) \sigma\left(\sum_k W(i,k) H(k,j)\right) H(k, j) \
&=& \sum_j M(i,j) \left(Y(i,j) - \sigma\left(\sum_k W(i,k) H(k,j)\right) \right) H(k, j)
\end{eqnarray}
\begin{eqnarray}
\frac{\partial}{\partial H(k,j)} \log p(Y |W, H ) &=& \sum_i M(i,j) Y(i,j) W(i,k)  - \sum_j M(i,j) \sigma\left(\sum_k W(i,k) H(k,j)\right) W(i, k) \
&=& \sum_j M(i,j) \left(Y(i,j) - \sigma\left(\sum_k W(i,k) H(k,j)\right) \right) W(i, k)
\end{eqnarray}
Evaluating the gradient (In matrix notation)
$$
\frac{d\mathcal{L}(W,H)}{dW} = (M \odot (Y -\sigma(W H)))H^\top
$$
$$
\frac{d\mathcal{L}(W,H)}{dH} = W^\top (M \odot (Y -\sigma(W H)))
$$
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt

# Generate a random logistic regression problem

def sigmoid(t):
    return 1./(1+np.exp(-t))

M = 3
N = 2
K = 1

# Some random parameters
W_true = np.random.randn(M,K)
H_true = np.random.randn(K,N)

# Generate class labels
pi = sigmoid(np.dot(W_true, H_true))
Y = np.array(pi<np.random.rand(M,N),dtype=float)

#Mask = np.ones((M,N))
p_miss = 0.
Mask = np.array(p_miss<np.random.rand(M,N),dtype=float)

Mask_nan = Mask.copy()
Mask_nan[Mask==0] = np.nan

W, H = LogisticMF(Y, K, Mask)

plt.imshow(Y, interpolation='nearest', cmap=plt.cm.gray_r)
plt.title('Y (Full data)')
plt.show()
plt.imshow(Y*Mask_nan, interpolation='nearest')
plt.title('Mask*Y (Observed data)')
plt.show()
plt.imshow(sigmoid(np.dot(W,H)), interpolation='nearest')
plt.title('\sigma(W*H)')
plt.show()

plt.imshow(sigmoid(np.dot(W_true,H_true)), interpolation='nearest')
plt.title('\sigma(W_true*H_true)')
plt.show()
plt.imshow(W, interpolation='nearest')
plt.title('W')
plt.show()
plt.imshow(H, interpolation='nearest')
plt.title('H')
plt.show()

def binary_random_mask_generator(M=30, N=150, p_miss=0.2):
    Mask = np.array(np.random.rand(M,N)>p_miss,dtype=float)

    Mask_nan = Mask.copy()
    Mask_nan[Mask==0] = np.nan

    return Mask, Mask_nan

def binary_random_matrix_generator1(M=30, N=150, p_on=0.3, p_switch=0.25):
    Y = np.zeros((M,N))
    y = np.array(np.random.rand(M,1)<p_on, dtype=float)
    for i in range(N):
        if np.random.rand()<p_switch:
            y = np.array(np.random.rand(M,1)<p_on, dtype=float)

        Y[:,i] = y.reshape(1,M)

    return Y

# Generate a catalog and reuse these
def binary_random_matrix_generator2(R=10, M=30, N=150, p_on=0.3, p_switch=0.25):
    Y = np.zeros((M,N))
    Catalog = np.array(np.random.rand(M,R)<p_on, dtype=float)
    idx = np.random.choice(range(R))
    for i in range(N):
        if np.random.rand()<p_switch:
            idx = np.random.choice(range(R))

        Y[:,i] =...


I am working on the file "Logistic Regression.ipynb".
The first task is:
As the plot above shows, while our logistic regression model is not really that good -- the area under the curve is calculated to be 0.6784 -- in accordance with the results earlier, it still does better than random guessing.
Also note that, in alignment with the 0.5 threshhold used by scikit-learn by default, our true positive rate (recall) of 0.71 matches up with the true positive rate in the graph when the false positive rate is 0.5.
Checking Model Prediction Performance
We assess the quality our modeling above -- specifically, how effectively it will likely hold up when exponsed to unseen data -- by using cross-validation.
Can you write Python code for it?

fivefold = cross_val_score(lr, X, y, scoring='accuracy', cv=5)
print("Score per fold:", fivefold)
print("Mean score:", fivefold.mean())
print("Standard deviation:", fivefold.std())


I am working on the file "Logistic Regression.ipynb".
The first task is:
Given that prestige is a categorical value, we perform dummy coding to convert the values into binary variables.
Can you write Python code for it?

columns1 = ['Admit', 'GRE', 'GPA']
data1 = df[columns1]
columns2 = ['Prestige_1','Prestige_2','Prestige_3']
data2 = dummy_ranks[columns2]
data = pd.merge(data1, data2, how="outer", left_index=True, right_index=True)
data


I am working on the file "Logistic Regression.ipynb".
The first task is:
In alignment with our expectations based on our model tuning, all metrics have shown an improvement over our original model.
In comparison with our original model, the predictions for non-admits are the same, and we now have two more correctly classified admits than in the previous model, which is obviously an improvement.
Can you write Python code for it?

# Plot a new ROC curve for the revised model
fpr2, tpr2, thresholds2 = metrics.roc_curve(y_test2, newprobabilities[:,1])

results2 = pd.DataFrame({'False Positive Rate': fpr2, 'True Positive Rate': tpr2})

plt.plot(fpr,tpr,color='darkgray', label="Original Model")
plt.plot(fpr2,tpr2, color='g', label="Revised Model")
plt.plot([0, 1], [0, 1], color='gray', linestyle='--', label="Baseline (Random Guessing)")
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve for Revised Model')
plt.legend()
plt.show()

print("Area Under the Curve:", metrics.roc_auc_score(y_test2, newprobabilities[:,1]))


I am working on the file "Logistic Regression.ipynb".
The first task is:
Initial Exploratory Data Analysis
We take a look at basic summary statistics, a cross-tabulation, and a histogram to get a general idea of the contents of the data.
Can you write Python code for it?

# Basic summary of the data
df.describe()

# Generate a cross-tabulation (frequency table by default) of the factors; here we use prestige
pd.crosstab(df['Admit'], df['Prestige'], rownames=['Admission'])


I am working on the file "Logistic Regression.ipynb".
The first task is:
Logistic Regression
We will use logistic regression to predict the probability that a particular student gets into grad school.
Can you write Python code for it?

# Separate independent and dependent variables
X = data.ix[:,1:]
y = data['Admit']

# Create a logistic regression model
initial = LogisticRegression(C = 1000, random_state=0)
initial.fit(X,y)

# Check model accuracy
print("Accuracy Score:", initial.score(X,y))

# What percentage of students actually got into grad school
print("Actual probability of admission:", y.mean())


I am working on the file "Logistic Regression.ipynb".
The first task is:
Model Evaluation
We now evaluate our logistic regression using some common metrics for assessing model quality.
Can you write Python code for it?

# Check accuracy
print("Accuracy Score:", metrics.accuracy_score(y_test, predicted))


I am working on the file "Logistic Regression.ipynb".
The first task is:
Our mean score shows a slight improvement. Our standard deviation is slightly higher than that of the previous, feature-scaled model, but it is still lower than our original model.
Hyperparameter Tuning
We will check if results change based on the $\lambda$ parameter used in regularization. Note that in scikit-learn's logistic regression, $C = \frac{1}{\lambda}$.
Can you write Python code for it?

tens = [10**i for i in range(-5,6)]

for i in tens:
    if i == 1000:
        continue
    testlr = LogisticRegression(C = i, random_state=0, class_weight={0:0.505,1:0.495})
    testcrossval = cross_val_score(testlr, scaledX, y, scoring='accuracy', cv=5)
    print('For C = {}:'.format(i))
    print('  Score per fold:', testcrossval)
    print('    Mean score:', testcrossval.mean())
    print('    Standard deviation:', testcrossval.std())


I am working on the file "Logistic Regression.ipynb".
The first task is:
Preprocessing the Data
While the data is already very analysis-friendly, we still have to change the categorial variable (prestige) into a binary one to be able to create a logistic regression model.
Can you write Python code for it?

# Dummy code the rank variable
dummy_ranks = pd.get_dummies(df['Prestige'], prefix="Prestige")
dummy_ranks.head()


I am working on the file "Logistic Regression.ipynb".
The first task is:
The Dataset
The data from UCLA (found at http://www.ats.ucla.edu/stat/data/binary.csv and originally used in this example: http://www.ats.ucla.edu/stat/r/dae/logit.htm) contains 4 columns:
* admit - a binary variable describing if the student was admitted into grad school or not
* gre - the student's Graduate Record Examination (GRE) score
* gpa - the student's grade point average (GPA)
* rank - the prestige of the student's undergraduate school, ranked from 1 to 4
The columns will be renamed to "Admit," "GRE," "GPA," and "Prestige" as we import the data to make them more human-friendly. Note that "rank" is renamed to "Prestige" to avoid confusion with a method of pandas.
Can you write Python code for it?

# As we import the data, we rename the "Rank" column to "Prestige" to avoid confusion with the rank method of pandas
df = pd.read_csv("binary.csv", header = 0, names = ["Admit", 'GRE', 'GPA', 'Prestige'])
df.head()


I am working on the file "Logistic Regression.ipynb".
The first task is:
The accuracy score here is slightly (around 0.083%) better than the optimized logistic regression without the training/test split.
Using a well-chosen (completely random) subset of the data, we were able to create a model whose accuracy actually exceeded that of the model created using all of the data.
Performance Visualization
We use a confusion matrix, classification report, and ROC curve to get a better view of the performance of our model.
Can you write Python code for it?

# Print confusion matrix and classification report
print("Confusion Matrix:\n",metrics.confusion_matrix(y_test, predicted))
print("\nClassification Report:\n",metrics.classification_report(y_test,predicted))


I am working on the file "Logistic Regression.ipynb".
The first task is:
Using five-fold cross-validation on our current model results in a similar accuracy score as the one previously derived, which shows that the model we arrived at earlier is not biased toward the training set and will likely generalize well to new data.
Improving the Model
We will attempt to improve our model by using a variety of techniques, including feature scaling, class weighting, and tuning our hyperparameter C. Our current model will be treated as a baseline.
Feature Scaling
One aspect of our data is that GRE and GPA scores vary significantly in magnitude (GRE varies from 220 to 800 while GPA varies from 2.26 to 4.0, though both appear to be shaped like normal distributions). Scaling these features may improve the accuracy of our machine learning model.
Can you write Python code for it?

from sklearn import preprocessing

# Isolate columns to scale
toscale = X[['GRE','GPA']].astype(float)
scaledX = preprocessing.scale(toscale)
scaleddata = pd.DataFrame(scaledX, columns=['GRE','GPA'])

# Join scaled data with categorical rank columns
scaledX = scaleddata.join(data2)
scaledX.head()

improve1 = cross_val_score(lr, scaledX, y, scoring='accuracy', cv=5)
print("Score per fold:", improve1)
print("Mean score:", improve1.mean())
print("Standard deviation:", improve1.std())


I am working on the file "LogisticRegression.ipynb".
The first task is:
Gradient Descent for Logistic Regression: First implementation in pytorch
Can you write Python code for it?

import torch
import torch.autograd
from torch.autograd import Variable

#sigmoid_f = torch.nn.Sigmoid()
def sigmoid_f(x):
    return 1./(1. + torch.exp(-x))

X = Variable(torch.from_numpy(X_np).double())
y = Variable(torch.from_numpy(y_np.reshape(N,1)).double())

# Implementation
w = Variable(torch.ones(M,1).double(), requires_grad=True)

eta = 0.01
MAX_ITER = 100
for epoch in range(MAX_ITER):
    sig = sigmoid_f(torch.matmul(X, w))

    # Compute the loglikelihood
    LL = torch.sum(y*torch.log(sig) + (1-y)*torch.log(1-sig))
    # Compute the gradients by automated differentiation
    LL.backward()

    # The gradient ascent step
    w.data.add_(eta*w.grad.data)

    # Reset the gradients, as otherwise they are accumulated in w.grad
    w.grad.zero_()

print(w.data.numpy())

%connect_info


I am working on the file "LogisticRegression.ipynb".
The first task is:
Let's check that the quantitative variable are effectively centered.
Can you write Python code for it?

print("Centered age")
print(centered_nesarc['age_c'].describe())
print("\nCentered family income")
print(centered_nesarc['family_income_c'].describe())


I am working on the file "LogisticRegression.ipynb".
The first task is:
Optimization Frameworks

CVX -- Convex Optimization
CVX is a framework that can be used for solving convex optimization problems.
Convex optimization includes many problems of interest; for example the minimization of the negative loglikelihood of the logistic regression is a convex problem. S
Unfortunately, many important problems and interesting problems
Can you write Python code for it?

%matplotlib inline
from cvxpy import *
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt


I am working on the file "LogisticRegression.ipynb".
The first task is:
Optimization with pytorch
Can you write Python code for it?

X_np, y_np, w_true_np, M, N = generate_toy_dataset(number_of_features=3, number_of_datapoints=20)


I am working on the file "LogisticRegression.ipynb".
The first task is:
Properties of the logsumexp function
If
$$
f(z) = \text{logsumexp}(0, z) = \log(1 + \exp(z))
$$
The derivative is
$$
\frac{df(z)}{dz} = \frac{\exp(z)}{1 + \exp(z)} = \sigma(z)
$$
When $z$ is a vector, $f(z)$ is a vector. The derivative of
$$
\sum_i f(z_i) = \mathbf{1}^\top f(z)
$$
$$
\frac{d \mathbf{1}^\top f(z)}{dz} = \left(\begin{array}{c} \sigma(z_1) \ \vdots \ \sigma(z_N) \end{array} \right) \equiv \sigma(z)
$$
where the sigmoid function $\sigma$ is applied elementwise to $z$.
Properties of the sigmoid function
Note that
\begin{eqnarray}
\sigma(x) & = & \frac{e^x}{(1+e^{-x})e^x} = \frac{e^x}{1+e^{x}} \
1 - \sigma(x) & = & 1 - \frac{e^x}{1+e^{x}} = \frac{1+e^{x} - e^x}{1+e^{x}} = \frac{1}{1+e^{x}}
\end{eqnarray}
\begin{eqnarray}
\sigma'(x) & = & \frac{e^x(1+e^{x}) - e^{x} e^x}{(1+e^{x})^2} = \frac{e^x}{1+e^{x}}\frac{1}{1+e^{x}} = \sigma(x) (1-\sigma(x))
\end{eqnarray}
\begin{eqnarray}
\log \sigma(x) & = & -\log(1+e^{-x}) = x - \log(1+e^{x}) \
\log(1 - \sigma(x)) & = &  -\log({1+e^{x}})
\end{eqnarray}
Exercise: Plot the sigmoid function and its derivative.
Exercise: Show that
$\tanh(z) = 2\sigma(2z) - 1$
Solve
$$
\text{maximize}\; \mathcal{L}(w)
$$
Optimization via gradient ascent
One way for
optimization is gradient ascent
\begin{eqnarray}
w^{(\tau)} & \leftarrow & w^{(\tau-1)} + \eta \nabla_w {\cal L}
\end{eqnarray}
where
\begin{eqnarray}
\nabla_w {\cal L} & = &
\begin{pmatrix}
{\partial {\cal L}}/{\partial w_1} \
{\partial {\cal L}}/{\partial w_2} \
\vdots \
{\partial {\cal L}}/{\partial w_{D}}
\end{pmatrix}
\end{eqnarray}
is the gradient vector and $\eta$ is a learning rate.
Evaluating the gradient (Short Derivation)
$$
\mathcal{L}(w) = y^\top X w - \mathbf{1}^\top \text{logsumexp}(0, X w)
$$
$$
\frac{d\mathcal{L}(w)}{dw} = X^\top y - X^\top \sigma(X w) = X^\top (y -\sigma(X w))
$$
Evaluating the gradient (Long Derivation)
The partial derivative of the loglikelihood with respect to the $k$'th entry of the weight vector is given by the chain rule as
\begin{eqnarray}
\frac{\partial{\cal L}}{\partial...
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pylab as plt

# Generate a random logistic regression problem

def sigmoid(t):
    return np.exp(t)/(1+np.exp(t))

def generate_toy_dataset(number_of_features=3, number_of_datapoints=20, styles = ['ob', 'xr']):

    D = number_of_features
    N = number_of_datapoints

    # Some random features
    X = 2*np.random.rand(N,D)-1
    X[:,0] = 1

    # Generate a random paramater vector
    w_true = np.random.randn(D,1)

    # Generate class labels
    pi = sigmoid(np.dot(X, w_true))
    y = np.array([1 if u else 0 for u in np.random.rand(N,1) < pi]).reshape((N))

    return X, y, w_true, D, N

styles = ['ob', 'xr']
X, y, w_true, D, N = generate_toy_dataset(number_of_features=3, number_of_datapoints=20, styles=styles)

xl = -1.5; xr = 1.5; yl = -1.5; yr = 1.5

fig = plt.figure(figsize=(5,5))
plt.plot(X[y==1,1],X[y==1,2],styles[1])
plt.plot(X[y==0,1],X[y==0,2],styles[0])

ax = fig.gca()
ax.set_ylim([yl, yr])
ax.set_xlim([xl, xr])
plt.show()

# Implement Gradient Descent
w = np.random.randn(D)

# Learnig rate
eta = 0.05

W = []
MAX_ITER = 200
for epoch in range(MAX_ITER):
    W.append(w)
    dL = np.dot(X.T, y-sigmoid(np.dot(X,w)))
    w = w + eta*dL

# Implement Gradient Descent
w = np.random.randn(D)

# Learnig rate
eta = 0.05

MAX_ITER = 200
for epoch in range(MAX_ITER):
    dL = 0
    for i in range(X.shape[0]):
        dL = dL + X[i,:].T*(y[i]-sigmoid(X[i,:].dot(w)))
    w = w + eta*dL

xl = -1.5
xr = 1.5
yl = -1.5
yr = 1.5

fig = plt.figure(figsize=(5,5))
ax = fig.gca()
ax.set_ylim([yl, yr])
ax.set_xlim([xl, xr])

plt.plot(X[y==1,1],X[y==1,2],styles[1])
plt.plot(X[y==0,1],X[y==0,2],styles[0])

ln = plt.Line2D([],[],color='k')
ln_left  = plt.Line2D([],[],ls= ':', color=styles[0][1])
ln_right = plt.Line2D([],[],ls= ':',...


I am working on the file "LogisticRegression.ipynb".
The first task is:
The means are both very close to 0; confirming the centering.
Distributions visualization
The following plots shows the distribution of all 3 explanatory variables with the response variable.
Can you write Python code for it?

g = sns.factorplot(x='white_ethnicity', y='used_canabis', data=centered_nesarc,
                   kind="bar", ci=None)
g.set_xticklabels(['Non White', 'White'])
plt.xlabel('White ethnicity')
plt.ylabel('Ever used cannabis')
plt.title('Ever used cannabis dependance on the white ethnicity');

g = sns.factorplot(x='sex', y='used_canabis', data=centered_nesarc,
                   kind="bar", ci=None)
g.set_xticklabels(['Female', 'Male'])
plt.ylabel('Ever used cannabis')
plt.title('Ever used cannabis dependance on the sex');

g = sns.boxplot(x='used_canabis', y='family_income', data=centered_nesarc)
g.set_yscale('log')
g.set_xticklabels(('No', 'Yes'))
plt.xlabel('Ever used cannabis')
plt.ylabel('Family income ($)');

g = sns.boxplot(x='used_canabis', y='AGE', data=centered_nesarc)
g.set_xticklabels(('No', 'Yes'))
plt.xlabel('Ever used cannabis')
plt.ylabel('Age');


I am working on the file "LogisticRegression.ipynb".
The first task is:
對原始資料進行 Sacle to 0~1 之間，避免 weight 太劇烈變動，導至 learning rate 難以設定
Can you write Python code for it?

from sklearn.preprocessing import scale,MinMaxScaler
scaler1 = MinMaxScaler()
scaler2 = MinMaxScaler()

f1 = f2.reshape([f1.shape[0],1])
f2 = f2.reshape([f2.shape[0],1])

scaler1.fit(f1)
scaler2.fit(f2)

f1 = scaler1.transform(f1)
f2 = scaler2.transform(f2)

f1 = f1.reshape(f1.shape[0])
f2 = f2.reshape(f2.shape[0])

c = [ 'g' if i==1 else 'b' for i in y ]

plt.scatter(f1, f2, 20, c=c, alpha=0.5,
            label="Type")
plt.xlabel("Height")
plt.ylabel("Weight")
plt.legend(loc=2)
plt.show()

Y = np.array([1,1,0,0,1])
A = np.array([0.8, 0.7, 0.2, 0.1, 0.9])
A2 = np.array([0.6, 0.6, 0.2, 0.1, 0.3])

def cross_entropy(Y,A):

    # small tip 因 log(0)　會趨近負無限大，會產生 nan ，故這裡統一加上 0.00001
    Y = np.array(Y)
    A = np.array(A)
    m = len(A)
    cost = -(1.0/m) * np.sum(Y*np.log(A+0.00001) + (1-Y)*np.log(1-A+0.00001))
    return cost
# Test cross_entropy Function
print cross_entropy(Y,A)
print cross_entropy(Y,A2)
print cross_entropy(Y,Y)


I am working on the file "Logistic_Regression.ipynb".
The first task is:
I don't think this randomization step is really needed in our case, but let's stick with the ufldl tutorial here.
Can you write Python code for it?


I am working on the file "Logit/Boyd Logit 1.ipynb".
The first task is:
Data we manually extracted by clicking on the plot
Can you write Python code for it?


I am working on the file "Logit/Boyd Logit 1.ipynb".
The first task is:
Make diagnostic plots of the posteriour distributions as created using MCMC.
Can you write Python code for it?

pymc3.traceplot(trace, combined=True);

pymc3.summary(trace)

xp = np.linspace(seed_bins_centers.min(), seed_bins_centers.max(), 100)
a = trace['alpha'].mean()
b = trace['beta'].mean()
plt.plot(xp, pymc3.math.invlogit(a + b*xp).eval())
plt.scatter(seed_bins_centers, successes/n_pts, s=50);
plt.xlabel('Log Maximum Seed PSD')
plt.ylabel('Chance of >2x change\n in Core PSD')
plt.gca().ticklabel_format(useOffset=False)


I am working on the file "Logit/Boyd Logit 1.ipynb".
The first task is:
Oberved Data:

n_pts : the number of points in each of the 10 log spaced Maximum Seed PSD bins
successes : the number of events in each Maximum Seed PSD bins where the increase was >2x
seed_bins_centers : the centers of the Maximum Seed PSD bins

Setup the Bayesian model accorind to the description above. Run the Markov chain monte carlo (MCMC) to sample the posterior distributions for $\alpha$ and $\beta$
Can you write Python code for it?

for i, j in zip(successes, n_pts):
    print(i,j)

# define priors
# these are wide uninformative priors

with pymc3.Model() as model:
    alpha = pymc3.Uniform('alpha', 0,40)
    beta = pymc3.Uniform('beta', 0,40)

    # define likelihood
    p = pymc3.math.invlogit(alpha + beta*seed_bins_centers)
    print('n_pts', n_pts, 'p', p, 'successes', successes, )

    y = pymc3.Binomial('y_obs', n=n_pts, p=p, observed=successes)

    # inference
    trace = pymc3.sample(iter=501000, njobs=7, burn=1000, burn_till_tuned=True, thin=80)


I am working on the file "Logit/Boyd Logit 1.ipynb".
The first task is:
This is the answer. For the collected data there is a weak cutoff in the Maximum Seed PSD where the probability increases. This speaks to the "necessary but not sufficient" nature of the relationship. Other parameters can be added to the model to improve in the "necessary and sufficient" direction as they are theorized.
Also the deviation between the data and the model at low seed PSD speaks to this being more of a cutoff at the low end than a system with the more seed PSD the greater chance of a Core PSD change. However the model fits well at higher seed PSD which may well inform that once the threshold is met the probability does scale as a Logit system with  binomial probability.
Can you write Python code for it?

# one should be able to get estimates of the line uncertainity
ilu = np.empty((1000, len(xp)), dtype=float)
for ii, v in tqdm.tqdm(enumerate(np.random.randint(0, len(trace), 1000))):
    ilu[ii] = [pymc3.math.invlogit(trace['alpha'][v] + trace['beta'][v]*xx).eval() for xx in xp]

[pymc3.math.invlogit(trace['alpha'][v] + trace['beta'][v]*xp[i]).eval() for i in range(len(xp))]

xp = np.linspace(seed_bins_centers.min(), seed_bins_centers.max(), 100)
for v in ilu:
    plt.plot(xp, v, alpha=.01, c='r')

xp = np.linspace(seed_bins_centers.min(), seed_bins_centers.max(), 100)
a = trace['alpha'].mean()
b = trace['beta'].mean()
plt.plot(xp, pymc3.math.invlogit(a + b*xp).eval())
plt.scatter(seed_bins_centers, successes/n_pts, s=50);
plt.xlabel('Log Maximum Seed PSD')
plt.ylabel('Chance of >2x change\n in Core PSD')
plt.gca().ticklabel_format(useOffset=False)


I am working on the file "Logit/Reeves_Denton_Dst_MinL-10nTbins-pymc3.ipynb".
The first task is:
Is there a cutoff in Dst that allows slot filling?
Method
Using a statistical model of the relationship between the slot filling at 464keV vs Dst.
We model the number of slot filling events as a random sample from a binomial distribution meaning that there is a probability of the event occuring and there may be a parameter that changes the probability. This is not meant as a correlation but as a success failure creiteria.
From Wikipedia:
In probability theory and statistics, the binomial distribution with parameters n and p is the discrete probability distribution of the number of successes in a sequence of n independent yes/no experiments, each of which yields success with probability p.
$
change \sim Bin(n,p) \
logit(p) = \alpha + \beta x \
a \sim N(0,5) \
\beta \sim N(0,10)
$
where we set vague priors for $\alpha$ and $\beta$, the parameters for the logistic model.
This is the same technique used in the estimation of deaths due to a concentration of a chemical.
Can you write Python code for it?

# http://onlinelibrary.wiley.com/doi/10.1002/2016JA022652/epdf

import pymc3
import tqdm
from pprint import pprint
import numpy as np
import matplotlib.pyplot as plt
import spacepy.plot as spp
import seaborn as sns
sns.set(font_scale=1.5)


I am working on the file "Logit/Reeves_Denton_Dst_MinL-10nTbins-pymc3.ipynb".
The first task is:
Make diagnostic plots of the posteriour distributions as created using MCMC.
Can you write Python code for it?

pymc3.traceplot(trace, combined=True)

pymc3.summary(trace)

xp = np.linspace(dst_bins_centers[ind].min(), dst_bins_centers[ind].max(), 100)
a = trace['alpha'].mean()
b = trace['beta'].mean()
y_val = pymc3.math.invlogit(a + b*xp).eval()
plt.plot(xp, y_val)
plt.scatter(dst_bins_centers[ind], success[ind]/n_events[ind], s=50);
plt.xlabel('Minimum Dst')
plt.ylabel('Probability slot is filled')
plt.gca().ticklabel_format(useOffset=False)


I am working on the file "Logit/Reeves_Denton_Dst_MinL-10nTbins-pymc3.ipynb".
The first task is:
Predictions based on this model
Can you write Python code for it?

# get the minimum Dst where 99% should be successes
for percentage in [50,75,90,95,99]:
    ind99 = y_val >= percentage/100
    minDst99 = xp[ind99][0]
    print('At a minimum Dst of {0:0.0f}nT it is predicted to have a {1}% percent of a slot filling at this energy'.format(minDst99, percentage))


I am working on the file "Long-Tails.ipynb".
The first task is:
For $n>1$, the head has most of value.  As $n$ falls, the middle and tail become important.
How many styles of clothing should we carry?
We can choose expand our inventory from $N$ to $N+1$ styles of clothing.  How many additional units will we sell?
This is just the sales volume distribution $n(r)$ evaluated at $r=N+1$
\begin{equation}
\frac{d V}{d N} = \frac{(n-1) V}{(1-N^{1-n})(N+1)^n}
\end{equation}
Can you write Python code for it?

marginal_benefit = ((ns-1)*volume)/((1-inventory**(1-ns))*(inventory+1)**ns)

fig, ax = plt.subplots(figsize=(10, 6))

ax.plot(ns, marginal_benefit)
ax.set_ylabel('Additional Units Sold')
ax.set_xlabel('Power law index $n$')
ax.set_title('Marginal Benefit of Expanding Inventory')


I am working on the file "Long-Tails.ipynb".
The first task is:
How many units from the head, tail, and middle are sold?  Integrate over the sales rank distribution to get the sales volume in the head:
\begin{equation}
V_H = \int_1^{f_H N} \frac{A\, dr}{r^n} = \frac{V(N^{n-1} - f_H^{1-n})}{N^{n-1} - 1}
\end{equation}
where $f_H=0.1$
The volume in the tail is
\begin{equation}
V_T = \int_{f_T N}^N \frac{A\, dr}{r^n} = \frac{V(f_T^{1-n}-1)}{N^{n-1}-1}
\end{equation}
where $f_T=0.5$
And the middle:
\begin{equation}
V_M = \int_{f_H N}^{f_T N} \frac{A\, dr}{r^n} = \frac{V(f_H^{1-n} - f_T^{1-n})}{N^{n-1}-1}
\end{equation}
Can you write Python code for it?

f_head = 0.1
f_tail = 0.5

ns = np.linspace(0,2,100)
nm1 = ns-1.0

head = volume*(inventory**nm1 - f_head**-nm1)/(inventory**nm1-1)
middle = volume*(f_head**-nm1 - f_tail**-nm1)/(inventory**nm1-1)
tail = volume*(f_tail**-nm1 - 1)/(inventory**nm1-1)

fig, ax = plt.subplots(figsize=(10, 6))

ax.plot(ns, head, label='Head')
ax.plot(ns, middle, label='Middle')
ax.plot(ns, tail, label='Tail')
ax.legend(loc='upper left')
ax.set_ylabel('Units Sold')
ax.set_xlabel('Power law index $n$')


I am working on the file "LongRange/PolynomialFit.ipynb".
The first task is:
Define a quality-of-fit measure, $\chi^2$
Can you write Python code for it?

eq1 = Eq(chi2, Integral(w(r)*(f(r)-f_approx(r,ak))**2,r))
eq1


I am working on the file "LongRange/PolynomialFit.ipynb".
The first task is:
Generic fit for approximating with a polynomial
Can you write Python code for it?

f = Symbol('f')  # Function to approximate
f_approx = Symbol('fbar')   # Approximating function
w = Symbol('w') # weighting function
chi2 = Symbol('chi^2')

f, f_approx, w, chi2

M = Symbol('M', integer=True)
k = Symbol('k', integer=True,positive=True)
a = IndexedBase('a',(M,))  # coefficient
h = IndexedBase('h',(M,))  # basis function
ak = Symbol('a_k')  # Temporary symbols to make some derivatives easier
hk = Symbol('h_k')  #    Basis function (function of r)
hj = Symbol('h_j')
r = Symbol('r',positive=True)
j = Symbol('j',integer=True)
poly_approx = Sum(a[k]*h[k],(k,0,M))
poly_approx_j = Sum(a[j]*h[j],(j,0,M)) # replace summation variable
poly_approx


I am working on the file "LongRange/PolynomialFit.ipynb".
The first task is:
We want to minimized $\chi^2$. Take derivative wrt the coefficients ($a_k$) and set to zero.  Will get $k$ equations.
Can you write Python code for it?

eq2 = Eq(0,diff(eq1.rhs, ak))
eq2

eq3 = Eq(diff(poly_approx,ak,evaluate=False), hk)
eq3

eq4 = Eq(0, Integral(eq2.rhs.args[0].subs(diff(f_approx(r,ak),ak),hk(r)),r))
eq4

eq5 = Eq(0, Integral(eq4.rhs.args[0].subs(f_approx(r,ak), poly_approx_j),r))
eq5

eq6 = Eq(0, Integral(-eq5.rhs.args[0]/2,r))
eq6

base7 = expand(eq6.rhs.args[0])
eq7 = Eq(Integral(-base7.args[1],r),Integral(base7.args[0],r))
eq7

int7 = eq7.lhs.args[0]
eq8 = Eq(Sum(a[j]*Integral(Mul(*int7.args[1:]),r),(j,0,M)), eq7.rhs)
eq8


I am working on the file "Loops and Functions.ipynb".
The first task is:
Building a Workflow with US Census Surname Data
Using the 2010 surname data from the US Census, we will develop a workflow to accomplish the following:

Write functions to:
Request information about surnames
Tabulate data
Visualize data

Sample dataset
Decennial Census Surname Files (2010)
https://www.census.gov/data/developers/data-sets/surnames.html
https://api.census.gov/data/2010/surname.html
Citation
US Census Bureau (2016) Decennial Census Surname Files (2010) Retrieved from https://api.census.gov/data/2010/surname.jsonstructures

1. Import modules
The modules used in this exercise are popular and under active development. Follow the links for more information about methods, syntax, etc.
Requests: http://docs.python-requests.org/en/master/
JSON: https://docs.python.org/3/library/json.html
Pandas: http://pandas.pydata.org/
Matplotlib: https://matplotlib.org/
Look for information about or links to the API, developer's documentation, etc. Helpful examples are often included.
Note that we are providing an alias for Pandas and matplotlib. Whenever we need to call a method from those module, we can use the alias.
Can you write Python code for it?

# http://api.census.gov/data/2010/surname
import requests
import json
import pandas as pd
import matplotlib.pyplot as plt


I am working on the file "Loops and Functions.ipynb".
The first task is:
If we want to query data for a different surname, we can go back up a few cells, change the value of the "name" variable, then re-run all of the other cells. That works in a notebook context but it's not very portable. Instead, we will define a function that can be called later without having to re-execute a bunch of code or code blocks.
In Python, the syntax for defining a function is:
def function_name(arg1, arg2, arg3=Default_value):
    do stuff
    return something
Can you write Python code for it?

def get_surname_data(desired_variables, surname):
    name = str.upper(surname)                                      # API requires surnames in all uppercase
    base_url = 'http://api.census.gov/data/2010/surname?get='
    query_url = base_url + desired_vars + '&NAME=' + name
    name_request = requests.get(query_url)
    name_data = name_request.json()
    return name_data

# Now we can get data for any name just by changing the value of the surname variable.
# We also have the option to update the variables of interest.
desired_vars = 'RANK,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC'
surname = "Jones"

print(get_surname_data(desired_vars, surname))

# That is not nicely formatted, so we can write another function to create a more human-readable data frame

def format_surname_data(surname_data, variable_data):
    column_list = []
    for each in surname_data[0]:
        label = variable_data[each]['label']
        column_list.append(label)
    surname_dataframe = pd.DataFrame([surname_data[1]], columns=column_list)
    return surname_dataframe

# Now we can get a nice table for any name by calling both functions
desired_vars = 'RANK,COUNT,PCTWHITE,PCTAPI,PCT2PRACE,PCTAIAN,PCTBLACK,PCTHISPANIC'
surname = "Smith"

surname_data = get_surname_data(desired_vars, surname)
surname_dataframe = format_surname_data(surname_data, variable_data)

surname_dataframe


I am working on the file "Loops and Functions.ipynb".
The first task is:
This is okay - we have a master list of all of our movies. The nested lists break our collection down further by watched and unwatched, but we don't know which list is which.
One way to improve this design is to use a dictionary.
Can you write Python code for it?

# A dictionary is a mutable, unordered list of key-value pairs
# Like lists, the values in a dictionary can be composed of different data types or structures

# Re-assign my_movies to an empty dictionary
my_movies = {}

# Now add our lists of unwatched and watched movies
# Note we are using a string as the key and the list as the value
my_movies["unwatched"] = unwatched
print(my_movies)

my_movies["watched"] = watched
print(my_movies)

# We retrieve items from a dictionary using the keys.
# If we don't know the keys, we can ask for them first using the 'keys()' attribute:
print(my_movies.keys())

print(my_movies['watched'])


I am working on the file "Loops.ipynb".
The first task is:
Reverse a string
Given a string, print its reverse.
Can you write Python code for it?

string = 'abcdef'
reversed_string = ''
length = len(string)
for count in range(length):
    reversed_string = reversed_string + string[length - count - 1]
    #print (length - count - 1), reversed_string
print reversed_string


I am working on the file "Lorenz Attractor.ipynb".
The first task is:
Let's define the initial conditions of the system ${\bf r}_0 = (x_0, y_0, z_0)$, the constants $\sigma$, $\rho$ and $\beta$ and a time grid.
Can you write Python code for it?

# Initial position in space
r0 = [0.1, 0.0, 0.0]

# Constants sigma, rho and beta
sigma = 10.0
rho   = 28.0
beta  = 8.0/3.0

# Time grid
tf = 100.0
t = linspace(0, tf, int(tf*100))


I am working on the file "Lorenz Attractor.ipynb".
The first task is:
Now let's solve the differencial equations numericaly and extract the corresponding $(x, y, z)$:
Can you write Python code for it?

pos = odeint(f, r0, t)

x = pos[:, 0]
y = pos[:, 1]
z = pos[:, 2]


I am working on the file "Lorenz Differential Equations.ipynb".
The first task is:
Computing the trajectories and plotting the result
We define a function that can integrate the differential equations numerically and then plot the solutions. This function has arguments that control the parameters of the differential equation ($\sigma$, $\beta$, $\rho$), the numerical integration (N, max_time) and the visualization (angle).
Can you write Python code for it?

def solve_lorenz(N=10, angle=0.0, max_time=4.0, sigma=10.0, beta=8./3, rho=28.0):

    fig = plt.figure()
    ax = fig.add_axes([0, 0, 1, 1], projection='3d')
    ax.axis('off')

    # prepare the axes limits
    ax.set_xlim((-25, 25))
    ax.set_ylim((-35, 35))
    ax.set_zlim((5, 55))

    def lorenz_deriv(x_y_z, t0, sigma=sigma, beta=beta, rho=rho):
        """Compute the time-derivative of a Lorenz system."""
        x, y, z = x_y_z
        return [sigma * (y - x), x * (rho - z) - y, x * y - beta * z]

    # Choose random starting points, uniformly distributed from -15 to 15
    np.random.seed(1)
    x0 = -15 + 30 * np.random.random((N, 3))

    # Solve for the trajectories
    t = np.linspace(0, max_time, int(250*max_time))
    x_t = np.asarray([integrate.odeint(lorenz_deriv, x0i, t)
                      for x0i in x0])

    # choose a different color for each trajectory
    colors = plt.cm.jet(np.linspace(0, 1, N))

    for i in range(N):
        x, y, z = x_t[i,:,:].T
        lines = ax.plot(x, y, z, '-', c=colors[i])
        plt.setp(lines, linewidth=2)

    ax.view_init(30, angle)
    plt.show()

    return t, x_t


I am working on the file "Lorenz Differential Equations.ipynb".
The first task is:
Let's call the function once to view the solutions. For this set of parameters, we see the trajectories swirling around two points, called attractors.
Can you write Python code for it?


I am working on the file "LorenzEquations.ipynb".
The first task is:
Calculating the Lorenz Map
The Lorenz Map is a discrete dynamical system that attemps to predict the $n+1$ maximum $Z_{n+1}(t)$ as a function of the previous peak $Z_{n}$.  Viewing the dynamical system in either the $Y,Z$ plane or as the time series $Z(t)$, we see that there are roughly two types of behavior,  a set of growing oscillations with increasing amplitude in $Z$, separated by jumps to a smaller value of $Z$ where it appears qualitatively, that during the jumps, the larger the value of $Z_n$, the smaller the value $Z_{n+1}$ after the jump.
Can you write Python code for it?

Y = state0[:,1]
Z = state0[:,2]
plt.figure()
plt.subplot(2,1,1)
ihalf = int(len(Y)/2.)
print ihalf,len(Y)
plt.plot(Y[:ihalf],Z[:ihalf])
plt.xlabel('Y')
plt.ylabel('Z')
plt.title('Lorenz system, $Y,Z$ plane: $r={}$, $\sigma={}$,$b={}$'.format(r,sigma,b))
plt.grid()
plt.subplot(2,1,2)
plt.plot(t,Z)
plt.xlabel('$t$')
plt.ylabel('$Z(t)$')
plt.title('$Z$ time series')
plt.show()


I am working on the file "LorenzEquations.ipynb".
The first task is:
Chaos and the strange attractor $r=28$
Now we increase the Rayleigh number to  $r=28$ and the solution becomes highly time-dependent and a-periodic.
Can you write Python code for it?

# set the Rayleigh number
r = 28.0
X0 = [2.,3.,4.]
state = SolveLorenz(X0,t,sigma,r,b)
PlotLorenzXvT(state,t,sigma,r,b)
PlotLorenz3D(state,sigma,r,b)


I am working on the file "LorenzEquations.ipynb".
The first task is:
Damped Oscillation $r=10$
Now we increase the Rayleigh number to  $r=10$ which admits two steady solutions depending on initial condition.
Can you write Python code for it?

# set the Rayleigh number
r = 10.0
X0 = [2.,3.,4.]
state = SolveLorenz(X0,t,sigma,r,b)
PlotLorenzXvT(state,t,sigma,r,b)
PlotLorenz3D(state,sigma,r,b)

# now change the initial condition so X=-2
X0 = [-2.0, -3.0, 4.0]

state = SolveLorenz(X0,t,sigma,r,b)
PlotLorenzXvT(state,t,sigma,r,b)
PlotLorenz3D(state,sigma,r,b)


I am working on the file "LorenzEquations.ipynb".
The first task is:
Estimating the Liapunov exponent
Now we will calculate and plot the distance between the two trajectories on a log plot
Can you write Python code for it?

# calculate the distance between the two trajectories

delta = state1-state0
delta = np.sqrt(np.sum(delta**2,1))

# and plot them
plt.figure()
plt.semilogy(t,delta)
plt.xlabel('t')
plt.ylabel('$||\delta(t)||$')
plt.grid()

# now fit the first part with a straight line to determine the slope
# we'll pick the line between tmin and tmax to avoid initial transients and later saturation
tmin = 1.
tmax = 12.
imin = int(np.argwhere(t<tmin)[-1])
imax = int(np.argwhere(t>tmax)[0])
tfit = t[imin:imax]

p= np.polyfit(tfit,np.log(delta[imin:imax]),1)
plt.hold(True)
plt.semilogy(tfit,np.exp(p[1]+p[0]*tfit),'r')
plt.title('Liapunov Exponent Estimate $\lambda={}$'.format(p[0]))
plt.show()


I am working on the file "LorenzEquations.ipynb".
The first task is:
Exploring the Lorenz Equations
The Lorenz Equations are a 3-D dynamical system that is a simplified model of Rayleigh-Benard thermal convection.  They are derived and described in detail in Edward Lorenz' 1963 paper Deterministic Nonperiodic Flow  in the Journal of Atmospheric Science.  In their classical form they can be written
$$
    \dot{X} = \sigma( Y - X)\
    \dot{Y} = rX - Y - XZ \
    \dot{Z} = XY -b Z
$$
where $\sigma$ is the "Prandtl Number",  $r = \mathrm{Ra}/\mathrm{Ra}_c$ is a scaled "Raleigh number" and $b$ is a parameter that is related to the the aspect ratio of a convecting cell in the original derivation.
This ipython notebook, will provide some simple python routines for numerical integration and visualization of the Lorenz Equations.  The primary code is modified from the Wikipedia page
Some useful python routines
Can you write Python code for it?

def Lorenz(state,t,sigma,r,b):
  '''
  Returns the RHS of the Lorenz equations
  '''
  # unpack the state vector
  x = state[0]
  y = state[1]
  z = state[2]

  # compute state derivatives
  xd = sigma * (y-x)
  yd = (r-z)*x - y
  zd = x*y - b*z

  # return the state derivatives
  return [xd, yd, zd]

def SolveLorenz(state0,t,sigma=10.,r=28.,b=8./3.0):
    '''
    use ODEINT to integrate the lorenz equations from initial condition state0 at t=0 for
    the range of times given in the numpy array t
    '''

    Lorenz_p = lambda state,t: Lorenz(state,t,sigma,r,b)
    state = odeint(Lorenz_p, state0, t)
    return state

def PlotLorenzXvT(state,t,sigma,r,b):
    '''
    make time series plots of solutions of the Lorenz equations X(t),Y(t),Z(t)
    '''

    plt.figure()
    ax = plt.subplot(111)
    X = state[:,0]
    Y = state[:,1]
    Z = state[:,2]
    ax.plot(t,X,'r',label='X')
    ax.hold(True)
    ax.plot(t,Y,'g',label='Y')
    ax.plot(t,Z,'b',label='Z')
    ax.set_xlabel('time t')
    plt.title('Lorenz Equations: $\sigma=${}, $r=${}, $b=${}'.format(sigma,r,b))
    # Shrink current axis's height by 10% on the bottom
    box = ax.get_position()
    ax.set_position([box.x0, box.y0 + box.height * 0.1,
                 box.width, box.height * 0.9])

    # Put a legend below current axis
    ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.05),ncol=3)
    plt.show()

def PlotLorenz3D(state,sigma,r,b):
    '''
    Show 3-D Phase portrait using mplot3D
    '''
    # do some fancy 3D plotting
    fig = plt.figure()
    ax = fig.gca(projection='3d')
    ax.plot(state[:,0],state[:,1],state[:,2])
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
    plt.title('Lorenz Equations: $\sigma=${}, $r=${}, $b=${}'.format(sigma,r,b))
    plt.show()


I am working on the file "LorenzEquations.ipynb".
The first task is:
Subcritical behavior $r<1$
Here we will begin exploring the behavior of the Lorenz equations for fixed values of $\sigma$ and $b$ and just changing the Rayleigh number $r$.
We will begin with subcritical behavior $r=0.5$  which rapidly damps to a condition of no motion
Can you write Python code for it?

# Set the parameters
sigma= 10.
b = 8./3

# set the initial condition
X0 = [2.0, 3.0, 4.0]

# set the time for integration
t = np.arange(0.0, 30.0, 0.01)

# set the Rayleigh number
r = 0.5

# solve the Equations
state = SolveLorenz(X0,t,sigma,r,b)

# and Visualize as a time series
PlotLorenzXvT(state,t,sigma,r,b)

# and as a 3-D phase portrait
PlotLorenz3D(state,sigma,r,b)


I am working on the file "LorenzEquationsDerivation.ipynb".
The first task is:
Deriving the Lorenz Equations
The Lorenz Equations are a 3-D dynamical system that is a simplified model of Rayleigh-Benard thermal convection.  They are derived and described in detail in Edward Lorenz' 1963 paper Deterministic Nonperiodic Flow  in the Journal of Atmospheric Science.
Here we will just sketch out the key points of the derivation.  A more complete derivation can be found here
The key idea is that the Lorenz Equations result from a severely truncated spectral approximation to the 2-D equations for incompressible thermal convection in stream-function/vorticity form.  These equations govern the flow of a buouyant incompressible fluid with a temperature dependent density in a layer of depth $h$, that is heated from below and cooled from the top.
(Insert a movie here?)
The full coupled set of scaled PDE's describe the coupling of incompressible Navier Stokes flow with an advection-diffusion equation for temperature, and can be written in dimensionless form as,
$$
   \frac{1}{\mathrm{Pr}}\left[ \frac{\partial \omega}{\partial t} + \vec{v}\cdot\nabla\omega\right] = \nabla^2\omega + \mathrm{Ra}\frac{\partial T}{\partial x}
$$
$$
    \nabla^2 \psi = -\omega
$$
$$
    \frac{\partial T}{\partial t} + \vec{v}\cdot\nabla T = \nabla^2 T
$$
Where
$$
    \vec{v}=(u,w) = \nabla\times\psi\hat{k}=(\frac{\partial\psi}{\partial y},- \frac{\partial\psi}{\partial x})
$$
is the fluid velocity field (which in this form is exactly incompressible with $\nabla\cdot\vec{v}=0$).  $\psi$ is the "Streamfunction"  whose contours are tangent to the fluid trajectories at all times.  The vorticity,
$$
    \omega\hat{k} = \nabla\times\vec{v}
$$
measures the local rate of rotation of the fluid, and is driven by horizontal variations in temperature (actually density).
Boundary conditions for Temperature are $T=1$ on the bottom of the layer and $T=0$ on the top.  In the absence of any fluid motion (\omega=\vec{v}=0),  the temperature field is just a steady conductive ramp with
$$
    T = 1 - y
$$
Thus we can also solve...
Can you write Python code for it?

# show Ra vs a
a = np.linspace(0.01,2.)
Rc = np.pi**4*(1. + a**2)**3/a**2
plt.figure()
plt.semilogy(a,Rc)
amin = np.sqrt(1./2.)
Rcmin = np.pi**4*(1. + amin**2)**3/amin**2
plt.hold(True)
plt.semilogy(amin,Rcmin,'ro')
plt.xlabel('a')
plt.ylabel('Ra$_c$')
plt.title('Critical Rayleigh Number')
plt.show()


I am working on the file "LorenzEquationsDerivation.ipynb".
The first task is:
Spectral decomposition
We next expand the streamfunction and Temperature fields in terms of a highly truncated Fourier Series where
the streamfunction contains one cellular mode
$$
    \psi(x,y,t) = X(t)\sin(a\pi x)\sin(\pi y)
$$
and Temperature has two modes
$$
    T(x,y,t) = Y(t)\cos(a\pi x)\sin(\pi y) - Z(t)\sin(2\pi y)
$$
Where $X(t)$, $Y(t)$ and $Z(t)$ are the time dependent amplitudes of each mode.  The spatial components of each mode looks like
Can you write Python code for it?

a = np.sqrt(0.5)
x = np.linspace(0,1./a)
y = np.linspace(0.,1.)
X,Y = np.meshgrid(x,y)

psi = np.sin(a*np.pi*X)*np.sin(np.pi*Y)
theta0 = np.cos(a*np.pi*X)*np.sin(np.pi*Y)
theta1 = -np.sin(2.*np.pi*Y)

plt.figure()
plt.subplot(2,2,1)
plt.contourf(X,Y,psi)
plt.title('$\psi$')
plt.subplot(2,2,3)
plt.contourf(X,Y,theta0)
plt.title('$\\theta_0$')
plt.subplot(2,2,4)
plt.contourf(X,Y,theta1)
plt.title('$\\theta_1$')
plt.show()


I am working on the file "LorenzEquationsDerivation.ipynb".
The first task is:
and for our initial condition $X(0) = 2$, $Y(0) = 3$, $Z(0) = 4$,  the streamfunction and Temperature fields would look like
Can you write Python code for it?

plt.figure()
plt.subplot(1,2,1)
plt.contourf(X,Y,2.*psi)
plt.gca().set_aspect('equal')
plt.title('$\psi$')
plt.subplot(1,2,2)
plt.contourf(X,Y,3*theta0 + 4*theta1)
plt.gca().set_aspect('equal')
plt.title('Temperature $\\theta$')
plt.show()


I am working on the file "LowerStar.ipynb".
The first task is:
Black Hole

Using persistent homology we can mathematically confirm that a black has a hole! 🕳 #topology #blackhole pic.twitter.com/n6QeyZwwgh
— Mitchell Eithun (@mitchelleithun) April 11, 2019

Can you write Python code for it?

blackhole_original = plt.imread("BlackHole.jpg")
blackhole_grey = np.asarray(PIL.Image.fromarray(blackhole_original).convert('L'))
I = lower_star_image(blackhole_grey)
plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.imshow(blackhole_grey)
plt.colorbar()
plt.subplot(122)
plot_dgms(I)
plt.tight_layout()

plt.figure()
plt.imshow(blackhole_grey < 60)


I am working on the file "LowerStar.ipynb".
The first task is:
Lower Star Images: Gaussian Blobs Example
Can you write Python code for it?

plt.figure()
ts = np.linspace(-1, 1, 100)
x1 = np.exp(-ts**2/(0.1**2))
ts -= 0.4
x2 = np.exp(-ts**2/(0.1**2))

# Define depths of Gaussian blobs
h1 = -1
h2 = -2
h3 = -3
img = h1*x1[None, :]*x1[:, None] + h2*x1[None, :]*x2[:, None] + h3*x2[None, :]*x2[:, None]
plt.imshow(img, cmap = 'afmhot')
plt.colorbar()
plt.show()

I = lower_star_image(img)

plt.figure(figsize=(9, 4))
plt.subplot(121)
plt.imshow(img, cmap = 'afmhot')
plt.colorbar()
plt.title("Test Image")
plt.subplot(122)
plot_dgms(I)
plt.title("0D Persistence Diagram")
plt.tight_layout()
plt.show()


I am working on the file "LowerStar.ipynb".
The first task is:
Overview of 0D Persistence for Point Clouds
Can you write Python code for it?

# First setup point cloud
np.random.seed(0)
NClusters = 3
PPC = 10 #Points per cluster
N = NClusters*PPC
X = np.zeros((N, 2))
for c in range(NClusters):
    X[c*PPC:(c+1)*PPC, :] = np.random.randn(PPC, 2) + 10*np.random.randn(2)[None, :]

# Compute persistence diagram
res = ripser(X, maxdim=0)
H0 = res['dgms'][0]
D = res['dperm2all']

def on_value_change(change):
    execute_computation1()
Tauslider = widgets.FloatSlider(min=0.0, max = np.max(D), x=5,step=0.1,value=1,description=r'\(\tau :\)' ,continuous_update=False)
Tauslider.observe(on_value_change, names='value')
display(Tauslider)

fig = plt.figure(figsize=(9.5, 3))
ax1 = plt.subplot(121)
ax2 = plt.subplot(122)

def execute_computation1():
    ax1.clear()
    ax2.clear()

    # Get slider values
    cutoff = Tauslider.value

    ax1.scatter(X[:, 0], X[:, 1], 50)
    for i in range(N):
        for j in range(i+1, N):
            if D[i, j] < cutoff:
                ax1.plot(X[[i, j], 0], X[[i, j], 1], 'k', linestyle='--')
    plt.sca(ax2)
    plot_dgms(H0)
    ax2.plot([-0.1*np.max(D), np.max(D)], [cutoff, cutoff], linestyle='--')

execute_computation1()


I am working on the file "LowerStar.ipynb".
The first task is:
Piecewise Linear Lower Star Filtrations
First, we define a lower star time series filtration function.  Then, we launch into an example
Can you write Python code for it?

def lower_star_filtration(x):
    N = len(x)
    # Add edges between adjacent points in the time series, with the "distance"
    # along the edge equal to the max value of the points it connects
    I = np.arange(N-1)
    J = np.arange(1, N)
    V = np.maximum(x[0:-1], x[1::])
    # Add vertex birth times along the diagonal of the distance matrix
    I = np.concatenate((I, np.arange(N)))
    J = np.concatenate((J, np.arange(N)))
    V = np.concatenate((V, x))
    #Create the sparse distance matrix
    D = sparse.coo_matrix((V, (I, J)), shape=(N, N)).tocsr()
    return ripser(D, maxdim=0, distance_matrix=True)['dgms'][0]

np.random.seed(2)
x = np.random.randn(10)*2
x = np.round(x*5)/5.0

H0 = lower_star_filtration(x)

def on_value_change(change):
    execute_computation1()
Tauslider = widgets.FloatSlider(min=np.min(x)-0.1, max = np.max(x)+0.1, x=5,step=0.05,value=np.min(x)-0.1,description=r'\(\tau :\)' ,continuous_update=False)
Tauslider.observe(on_value_change, names='value')
display(Tauslider)

fig = plt.figure(figsize=(9.5, 3))
ax1 = plt.subplot(121)
ax2 = plt.subplot(122)

def execute_computation1():
    ax1.clear()
    ax2.clear()

    # Get slider values
    cutoff = Tauslider.value

    ax1.plot(x)
    ax1.plot([0, len(x)], [cutoff, cutoff])
    ax1.scatter(np.arange(len(x)), x)
    for i in range(len(x)):
        if x[i] <= cutoff:
            ax1.scatter([i]*2, [x[i]]*2, c='C1')
        if i < len(x)-1:
            if x[i] <= cutoff and x[i+1] <= cutoff:
                ax1.plot([i, i+1], x[i:i+2], c='C1')
    plt.sca(ax2)
    plot_dgms(H0)
    ax2.plot([np.min(x)-0.1, cutoff], [cutoff, cutoff], linestyle='--', c='C1')
    ax2.plot([cutoff, cutoff], [cutoff, np.max(x)+0.6], linestyle='--', c='C1')

execute_computation1()


I am working on the file "LowerStar.ipynb".
The first task is:
Warped Sinusoid Example
Can you write Python code for it?

np.random.seed(1)
NPeriods = 5
NSamples = 400
t = np.linspace(-0.5, NPeriods, NSamples)
x = np.sin(2*np.pi*t) + t

def on_value_change(change):
    execute_computation1()
WarpSlider = widgets.FloatSlider(min=0.1, max = 5,step=0.05,value=1,description=r'warp' ,continuous_update=False)
WarpSlider.observe(on_value_change, names='value')
display(WarpSlider)

fig = plt.figure(figsize=(9.5, 3))
ax1 = plt.subplot(121)
ax2 = plt.subplot(122)

def execute_computation1():
    ax1.clear()
    ax2.clear()

    # Get slider values
    p = WarpSlider.value
    t = np.linspace(0, 1, NSamples)**p
    t = (t*(NPeriods+0.5))-0.5
    x = np.sin(2*np.pi*t) + t

    ax1.plot(x)
    ax1.scatter(np.arange(len(x)), x)
    H0 = lower_star_filtration(x)
    plt.sca(ax2)
    plot_dgms(H0)

execute_computation1()


I am working on the file "LtaP.ipynb".
The first task is:
Step 1 Create a Google Form with these questions:
1. What is your name? [text entry]
2. What is your gender? [multiple choice]
3. What are your general science interests? [checkboxes]

I can ask for other information from the students (e.g., grade, school name) and scientists (email).
After receiving the responses, load up the CSV of responses from the Google Form by running the cell below (you'll have to change the path to your own CSV).
Can you write Python code for it?

def format_name(data):
    first_name = ['-'.join(list(map(str.capitalize,d))) for d in data['Name'].str.replace(" ", "-").str.split('-')]
    last_name = ['-'.join(list(map(str.capitalize,d))) for d in data['Last'].str.replace(" ", "-").str.split('-')]
    full_name = pd.Series([m+" "+n for m,n in zip(first_name,last_name)])

    return full_name

# Retrieve data from Google Sheet and parse using pandas dataframe
student_data = pd.read_csv("students.csv")
student_data = student_data.replace(np.nan,' ', regex=True)

# Store student information in variables.
#
# Collaboratr divided people into "learners" and "teachers" based on what they wanted to "learn" and "teach."
# Here, students are always "learners" by default and the scientists are always "teachers."
# To maintain the structure of the pandas dataframe,
# I've created blank values for what students want to "teach" and what scientists want to "learn."

### write a function that would format names (including hyphens)
student_data['Full Name'] = format_name(student_data)
student_names = student_data['Full Name']
nStudents = len(student_names)

student_learn = student_data['If I could be any type of scientist when I grow up, I would want to study:']
student_teach = pd.Series(["" for i in range (nStudents)], index=[i for i in range(nStudents)])
student_email = pd.Series(["" for i in range (nStudents)], index=[i for i in range(nStudents)])

# Store scientist information in variables.
scientist_data = pd.read_csv("scientists_1.csv")
scientist_data = scientist_data.replace(np.nan,' ', regex=True)

#drop any duplicate email entries in the data frame
drop = np.where(scientist_data.duplicated('Email')==True)[0]
temp = scientist_data.drop(scientist_data.index[drop])
scientist_data = temp

scientist_data['Full Name'] = format_name(scientist_data)
scientist_names = scientist_data['Full Name']
nScientists = len(scientist_names)

scientist_learn = pd.Series(["" for i in range (nScientists)], index=[i for i in range(nScientists)])
scientist_teach =...


I am working on the file "LtaP.ipynb".
The first task is:
Step 3: Assign scientists to students
I thought about several ways to do this. Each student has a "pool" of scientists to be assigned to based on their interests. This was a non-trivial problem. I try to have no more than 2 students assigned to each scientist, working with a limited dataset of roughly 20 scientists and 30 students. Most scientists come from astronomy/physics or psychology/neuroscience. Here are my attempts to do just that:

For each student, randomly draw from their "pool" of scientists with matching interests. This typically caused the more "underrepresented" scientists to get oversubscribed quickly, e.g., having one biologist and having many students interested in biology. This didn't help for students who had limited interests. If I couldn't match everyone up, I'd try again with different random draws. Couldn't find a solution for the conditions listed above. Maybe this would work better if we had a nScientists > nStudents.

Start with the "least popular" topic, that is the topic where the student-to-scientist ratio is smallest. Loop through the students with those interests and try to match them to a scientist. Then, we work are way up the list until we get to the most popular topic. This approach worked much better.
Can you write Python code for it?

# Insert users in graphs
for n,e,l,t in zip(names, emails, learn, teach):
    collaboratr.insert_node(G,n, email=e, learn=l.split(';'), teach=t.split(';'))

def sort_things(stu_data, sci_data):
    num_interests = {}

    for i,r in stu_data.iterrows():
        name = r['Name'].capitalize() + " " + r['Last'].capitalize()
        num_interests = { name: 1 }

    print(num_interests)
    stu_names_sorted = sorted(num_interests, key=num_interests.get)
    print(stu_names_sorted)

    interests_stu = Counter(list(itertools.chain.from_iterable(\
                [ i.split(';') for i in stu_data['If I could be any type of scientist when I grow up, I would want to study:'] ])))
    interests_sci = Counter(list(itertools.chain.from_iterable(\
                [ i.split(';') for i in sci_data['We will match you with a pen pal who has expressed an interest in at least one of the following subjects. Which topic is most relevant to your work?'] ])))

    interests_rel = { key: interests_stu[key]/interests_sci[key] for key in interests_sci.keys() }
    interests_rel_sorted = sorted(interests_rel, key=interests_rel.get)

    return interests_rel_sorted, stu_names_sorted

def assigner(assign, stu_data, sci_data, max_students=2):
    assign_one = {}
    subscriptions = { n: 0 for n in sci_data['What is your name?'] }

    interests_rel_sorted, stu_names_sorted = sort_things(stu_data, sci_data)

    for key in interests_rel_sorted:
        for name in stu_names_sorted:
            if name not in assign_one:
                if key in assign[name].keys():
                    try:
                        scientist = np.random.choice(assign[name][key])
                    except ValueError:
                        scientist = np.random.choice(scientist_data['What is your name?'])
                    assign_one[name] = scientist

                    subscriptions[scientist] += 1

                    if subscriptions[scientist]>=max_students:
                        for kk,vv in assign.items():
                       ...


I am working on the file "LucasSpanControl.ipynb".
The first task is:
A Lucas Economy
Can you write Python code for it?

E.Lucas = True
rwc_L, (Tc_L,Lc_L) = E.smallhold_eq([E.TBAR,E.LBAR],E.s)


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Compared to the original scenario the landlord with marker power in the Lucas-scneario faces a countervailing force:  if she pushes the wage too low then she makes self-managed production more attractive for a would-be medium sized farmer who is not now in production.
From the solution it would appear that the beyond a certain theta there does not appear to be an available way for the landlord to distort further. .
Can you write Python code for it?

E.Lucas = True
factor_plot(E,Xrcl,Xrl)

E.Lucas = False
factor_plot(E,Xrc,Xr)

E.Lucas = True
E.cartel_eq(0.5)

E.cartel_eq(0.6)

E.cartel_eq(0.2)
Lr


I am working on the file "LucasSpanControl.ipynb".
The first task is:
In the Lucas equilibrium there is less unskilled labor (since managers cannot be laborers) so all else equal we would expect higher wages.
Can you write Python code for it?


I am working on the file "LucasSpanControl.ipynb".
The first task is:
In this sample equilibrium the two lowest skill groups become pure laborers.
Can you write Python code for it?

plt.title('Eqn factor use')
plt.xlabel('group index')
plt.plot(x,Tc_L, marker='o', linestyle='None',label='land')
plt.plot(x,Lc_L, marker='o', linestyle='None', label='labor')
plt.legend();


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Just out of interest, how much lower would output be in the Non-Lucas economy if every household were self-sufficient.
Can you write Python code for it?

Tce = np.ones(N)*(E.TBAR/N)
Lce = np.ones(N)*(E.LBAR/N)
sum(E.prodn([Tce,Lce],E.s))

E.prodn([Tce,Lce],E.s)

sum(rwc*[E.TBAR/N, E.LBAR/N])+E.prodn([Tce,Lce],E.s)-

rwc*[Tc,Lc]


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Let's summarize the parameters as they now stand:
Can you write Python code for it?


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Lucas = True
Can you write Python code for it?

E.Lucas = True
rwcl, (Tcl,Lcl) = E.smallhold_eq([E.TBAR,E.LBAR],E.s)

Lcl

E.excessD(rwcl,Xbar,E.s)

plt.title('Competitive: Induced farm size (factor use) distribution')
plt.plot(Tcl,marker='o',label='land (Lucas)')
plt.plot(Lcl, marker='x',label='labor (Lucas)')
plt.plot(Tc, '-o',label='land ')
plt.plot(Lc, marker='x',label='labor ')
plt.legend();


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Note that now about 50% of labor is going into management. However this is an artifact of the unform distribution of skill which puts so much population in the higher skill groups.
Can you write Python code for it?


I am working on the file "LucasSpanControl.ipynb".
The first task is:
Something is still not right... labor used/demanded excees labor supply
Can you write Python code for it?

(r,w),(Tr,Lr)= E.cartel_eq(0.5)
sum(Lr),np.count_nonzero(Lr)*(E.LBAR)/E.N

sum(Lr)

fringe = E.smallhold_eq([E.TBAR,E.LBAR/2], E.s)

fringe.w

fringe.X

fringe.X[0]


I am working on the file "LucasSpanControl.ipynb".
The first task is:
When Lucas = False farms don't have to specialize in farm management.  When True then as in Lucas (1978) household labor must be dedicated to management and labor must be hired.
Assumed skill distribution
Let's assume population is uniform across the N groups but skill is rising.
Can you write Python code for it?

E.s = np.linspace(1,5,num=N)
plt.title('skill distribution')
plt.xlabel('group index')
plt.plot(x,E.s,marker='o');


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
Compute and show the 10 TLEs that best match the GPS data.
Can you write Python code for it?

tles_ITRF = np.array([tle.ITRF_position_velocity_error(t)[0] * AU_KM  for tle in tles])
rms_error = np.sqrt(np.average(np.sum((tles_ITRF - lucky7_gps[:,:3].transpose().reshape(1,3,-1))**2, axis = 1), axis = 1))
output = [(tles[j].model.satnum, tles[j].epoch.utc_datetime().strftime('%Y-%m-%d %H:%M'), rms_error[j]) for j in np.argsort(rms_error)[:10]]
display(HTML(tabulate.tabulate(output, headers = ('NORAD', 'TLE epoch', 'RMS error (km)'), tablefmt = 'html')))

x, v = tles[np.argmin(rms_error)].ITRF_position_velocity_error(t[0])[:2]
x * AU_KM

v * AU_KM / DAY_S

t[0].tai - 2430000.0

t[-1].tai - 2430000.0


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
Compute rotation matrices from ECI to VNB frames.
Can you write Python code for it?


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
Convert ECEF residuals to ECI using Astropy.
Can you write Python code for it?

r = astropy.coordinates.CartesianRepresentation(ecef_residual, xyz_axis = 1, unit = astropy.units.km)
obs_time = astropy.time.Time(t.utc_datetime())
eci_residual = np.array(astropy.coordinates.ITRS(r, obstime = obs_time).transform_to(astropy.coordinates.GCRS(obstime = obs_time)).cartesian.xyz).transpose()


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
It seems there is discontinuity in GPS measurements between measurement 329 and 330. This is probably caused by some jump in the GPS clock. The measurments after this discontinuity are rejected in GMAT orbit determination.
Can you write Python code for it?


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
More information about the discontinuity. If we look at the fractional parts of the GPS TOW timestamps, we see that they jump from 0.64 to 0.65 precisely when the discontinuity happens.
Can you write Python code for it?

lucky7_gps[:,3] % 1

np.where(np.diff(lucky7_gps[:,3] % 1))

tai_modjulian = t.tai - 2430000.0

gps_pos = tles_ITRF[np.argmin(rms_error),...]

with open('/tmp/gpsl7_data.gmd', 'w') as f:
    for j in range(tai_modjulian.size):
        pos = lucky7_gps[j,:3]
        f.write(f'{tai_modjulian[j]} GPS_PosVec 9014 0 {pos[0]} {pos[1]} {pos[2]}\n')


I am working on the file "Lucky-7 orbit.ipynb".
The first task is:
Read orbit determination report to compute residuals.
Can you write Python code for it?

with open('l7_estimation_report') as f:
    report = f.readlines()

iterations = [(j, int(l.split()[2].strip(':'))) for j,l in enumerate(report) if l.split()[1:2] == ['ITERATION']]
last_iter = max([i[1] for i in iterations])-1
start_line = [i[0] for i in iterations if i[1] == last_iter][2]
end_line = [i[0] for i in iterations if i[1] == last_iter + 1][0]
residual_lines = [l for l in report[start_line:end_line] if l.split()[0:1] == [str(last_iter)]]

ecef_position = np.array([float(l.split()[-2]) for l in residual_lines]).reshape((-1,3))
ecef_residual = lucky7_gps[:,:3] - ecef_position

plt.figure(figsize = (12,8), facecolor = 'w')
plt.plot(t.utc_datetime(), ecef_residual * 1e3, '.');
plt.ylim((-100,100))
plt.xlabel('UTC time')
plt.ylabel('Residual (m)')
plt.legend(['X', 'Y', 'Z'])
plt.title('GPS residuals with respect to precise orbit (ECEF coordinates)');


I am working on the file "Lucy/Lucy frames Bochum 2021-10-24.ipynb".
The first task is:
AOS frames
Telemetry is in Virtual Channel 1. Virtual channel 63 contains Only Idle Data.
Can you write Python code for it?

aos = [AOSFrame.parse(f) for f in frames]

collections.Counter([a.primary_header.transfer_frame_version_number for a in aos])

collections.Counter([a.primary_header.spacecraft_id for a in aos])

collections.Counter([a.primary_header.virtual_channel_id for a in aos])


I am working on the file "Lucy/Lucy frames Bochum 2021-10-24.ipynb".
The first task is:
APID 5
As found by r00t this APID has frames of fixed size containing a number of fields in tag-value format. Tags are 2 bytes, and values have different formats and sizes depending on the tag.
Can you write Python code for it?

tags = {2: Int16ub, 3: Int16ub, 15: Int32ub, 31: Int16ub, 32: Int16ub, 1202: Float64b,
        1203: Float64b, 1204: Float64b, 1205: Float64b, 1206: Float64b, 1208: Float32b,
        1209: Float32b, 1210: Float32b, 1601: Float32b, 1602: Float32b, 1603: Float32b,
        1630: Float32b, 1631: Float32b, 1632: Float32b, 17539: Float32b, 17547: Float32b,
        17548: Float32b, 21314: Int32sb, 21315: Int32sb, 21316: Int32sb, 21317: Int32sb,
        46555: Int32sb, 46980: Int16ub, 46981: Int16ub, 46982: Int16ub, 47090: Int16ub,
        47091: Int16ub, 47092: Int16ub,
       }

values = list()
for packet in vc0_by_apid[5]:
    t = timestamps([packet])[0]
    packet = packet[6+5:]  # skip primary and secondary headers
    while True:
        tag = Int16ub.parse(packet)
        packet = packet[2:]
        value = tags[tag].parse(packet)
        packet = packet[tags[tag].sizeof():]
        values.append((tag, value, t))
        if len(packet) == 0:
            break

values_keys = {v[0] for v in values}
values = {k: [(v[2], v[1]) for v in values if v[0] == k] for k in values_keys}

for k in sorted(values_keys):
    vals = values[k]
    plt.figure()
    plt.title(f'Key {k}')
    plt.plot([v[0] for v in vals], [v[1] for v in vals], '.')


I am working on the file "Lucy/Lucy frames Bochum 2021-10-24.ipynb".
The first task is:
Timestamps are contained in the Space Packet secondary header time code field. They are encoded as big-endian 32-bit integers counting the number of seconds elapsed since the J2000 epoch (2000-01-01T12:00:00).
Looking at the idle APID packets, the next byte might indicate fractional seconds (since it is still part of the secondary header rather than idle data), but it is difficult to be sure.
Can you write Python code for it?

def timestamps(packets):
    epoch = np.datetime64('2000-01-01T12:00:00')
    t = np.array([struct.unpack('>I', p[ccsds.SpacePacketPrimaryHeader.sizeof():][:4])[0]
                  for p in packets], 'uint32')
    return epoch + t * np.timedelta64(1, 's')

def load_frames(path):
    frame_size = 223 * 5 - 2
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = load_frames('lucy_frames_bochum_20211024_214614.u8')
frames.shape[0]


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-19.ipynb".
The first task is:
AOS frames
Telemetry is in Virtual Channel 1. Virtual channel 63 contains Only Idle Data.
Can you write Python code for it?


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-19.ipynb".
The first task is:
APID 5
As found by r00t this APID has frames of fixed size containing a number of fields in tag-value format. Tags are 2 bytes, and values have different formats and sizes depending on the tag.
Can you write Python code for it?


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-19.ipynb".
The first task is:
Timestamps are contained in the Space Packet secondary header time code field. They are encoded as big-endian 32-bit integers counting the number of seconds elapsed since the J2000 epoch (2000-01-01T12:00:00).
Looking at the idle APID packets, the next byte might indicate fractional seconds (since it is still part of the secondary header rather than idle data), but it is difficult to be sure.
Can you write Python code for it?

def timestamps(packets):
    epoch = np.datetime64('2000-01-01T12:00:00')
    t = np.array([struct.unpack('>I', p[ccsds.SpacePacketPrimaryHeader.sizeof():][:4])[0]
                  for p in packets], 'uint32')
    return epoch + t * np.timedelta64(1, 's')

def load_frames(path):
    frame_size = 223 * 5 - 2
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = np.concatenate((
    load_frames('lucy_frames_eb3frn_20211019_233036.u8'),
    load_frames('lucy_frames_eb3frn_20211019_235245.u8')))
frames.shape[0]


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-19.ipynb".
The first task is:
Virtual Channel 63 (Only Idle Data)
Virtual channel 63 corresponds to Only Idle Data. The transfer frame data field includes an M_PDU header with a first header pointer equal to 0x7fe, which indicates that the packet zone contains only idle data. The packet zone is filled with 0xaa's.
Can you write Python code for it?

vc63 = [a for a in aos if a.primary_header.virtual_channel_id == 63]
[a.primary_header for a in vc63[:10]]

vc63_frames = np.array([f for f, a in zip(frames, aos) if a.primary_header.virtual_channel_id == 63])

np.unique(vc63_frames[:, 6:8], axis = 0)

bytes(vc63_frames[0, 6:8]).hex()

np.unique(vc63_frames[:, 8:])

hex(170)

fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc63])

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(fc[1:], np.diff(fc)-1, '.')
plt.title("Lucy virtual channel 63 (OID) frame loss")
plt.xlabel('Virtual channel frame counter')
plt.ylabel('Lost frames');

first_part = fc < 219000
fc[first_part].size/(fc[first_part][-1]-fc[0]+1)


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-20.ipynb".
The first task is:
AOS frames
Telemetry is in Virtual Channel 1. Virtual channel 63 contains Only Idle Data.
Can you write Python code for it?


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-20.ipynb".
The first task is:
APID 5
As found by r00t this APID has frames of fixed size containing a number of fields in tag-value format. Tags are 2 bytes, and values have different formats and sizes depending on the tag.
Can you write Python code for it?


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-20.ipynb".
The first task is:
Timestamps are contained in the Space Packet secondary header time code field. They are encoded as big-endian 32-bit integers counting the number of seconds elapsed since the J2000 epoch (2000-01-01T12:00:00).
Looking at the idle APID packets, the next byte might indicate fractional seconds (since it is still part of the secondary header rather than idle data), but it is difficult to be sure.
Can you write Python code for it?

def timestamps(packets):
    epoch = np.datetime64('2000-01-01T12:00:00')
    t = np.array([struct.unpack('>I', p[ccsds.SpacePacketPrimaryHeader.sizeof():][:4])[0]
                  for p in packets], 'uint32')
    return epoch + t * np.timedelta64(1, 's')

def load_frames(path):
    frame_size = 223 * 5 - 2
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = np.concatenate((
    load_frames('lucy_frames_eb3frn_20211020_233618.u8'),
    load_frames('lucy_frames_eb3frn_20211020_235911.u8')))
frames.shape[0]


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-20.ipynb".
The first task is:
Virtual Channel 63 (Only Idle Data)
Virtual channel 63 corresponds to Only Idle Data. The transfer frame data field includes an M_PDU header with a first header pointer equal to 0x7fe, which indicates that the packet zone contains only idle data. The packet zone is filled with 0xaa's.
Can you write Python code for it?

vc63 = [a for a in aos if a.primary_header.virtual_channel_id == 63]
[a.primary_header for a in vc63[:10]]

vc63_frames = np.array([f for f, a in zip(frames, aos) if a.primary_header.virtual_channel_id == 63])

np.unique(vc63_frames[:, 6:8], axis = 0)

bytes(vc63_frames[0, 6:8]).hex()

np.unique(vc63_frames[:, 8:])

hex(170)

fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc63])

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(fc[1:], np.diff(fc)-1, '.')
plt.title("Lucy virtual channel 63 (OID) frame loss")
plt.xlabel('Virtual channel frame counter')
plt.ylabel('Lost frames');

last_part = fc > 391000
fc[last_part].size/(fc[-1]-fc[last_part][0]+1)


I am working on the file "Lucy/Lucy frames EB3FRN 2021-10-20.ipynb".
The first task is:
Virtual channel 0
Virtual channel 0 contains telemetry. There are a few active APIDs sending CCSDS Space Packets using the AOS M_PDU protocol.
Can you write Python code for it?

vc0 = [a for a in aos if a.primary_header.virtual_channel_id == 0]
[a.primary_header for a in vc0[:10]]

fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc0])

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(fc[1:], np.diff(fc)-1, '.')
plt.title("Lucy virtual channel 0 (telemetry) frame loss")
plt.xlabel('Virtual channel frame counter')
plt.ylabel('Lost frames');

last_part = fc > 391000
fc[last_part].size/(fc[-1]-fc[last_part][0]+1)

vc0_packets = list(ccsds.extract_space_packets(vc0, 49, 0))

vc0_t = timestamps(vc0_packets)

vc0_sp_headers = [ccsds.SpacePacketPrimaryHeader.parse(p) for p in vc0_packets]

vc0_apids = collections.Counter([p.APID for p in vc0_sp_headers])
vc0_apids

apid_axis = {a : k for k, a in enumerate(sorted(vc0_apids))}

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(vc0_t, [apid_axis[p.APID] for p in vc0_sp_headers], '.')
plt.yticks(ticks=range(len(apid_axis)), labels=apid_axis)
plt.xlabel('Space Packet timestamp')
plt.ylabel('APID')
plt.title('Lucy Virtual Channel 0 APID distribution');

vc0_by_apid = {apid : [p for h,p in zip(vc0_sp_headers, vc0_packets)
                       if h.APID == apid] for apid in vc0_apids}

plot_apids(vc0_by_apid)


I am working on the file "Lucy/Lucy frames EB3FRN.ipynb".
The first task is:
AOS frames
Telemetry is in Virtual Channel 1. Virtual channel 63 contains Only Idle Data.
Can you write Python code for it?


I am working on the file "Lucy/Lucy frames EB3FRN.ipynb".
The first task is:
Timestamps are contained in the Space Packet secondary header time code field. They are encoded as big-endian 32-bit integers counting the number of seconds elapsed since the J2000 epoch (2000-01-01T12:00:00).
Looking at the idle APID packets, the next byte might indicate fractional seconds (since it is still part of the secondary header rather than idle data), but it is difficult to be sure.
Can you write Python code for it?

def timestamps(packets):
    epoch = np.datetime64('2000-01-01T12:00:00')
    t = np.array([struct.unpack('>I', p[ccsds.SpacePacketPrimaryHeader.sizeof():][:4])[0]
                  for p in packets], 'uint32')
    return epoch + t * np.timedelta64(1, 's')

def load_frames(path):
    frame_size = 223 * 5 - 2
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = load_frames('lucy_frames_eb3frn_20211016_185442.u8')
frames.shape[0]


I am working on the file "Lucy/Lucy frames EB3FRN.ipynb".
The first task is:
Virtual channel 0
Virtual channel 0 contains telemetry. There are a few active APIDs sending CCSDS Space Packets using the AOS M_PDU protocol.
Can you write Python code for it?

vc0 = [a for a in aos if a.primary_header.virtual_channel_id == 0]
[a.primary_header for a in vc0[:10]]

fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc0])

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(fc[1:], np.diff(fc)-1, '.')
plt.title("Lucy virtual channel 0 (telemetry) frame loss")
plt.xlabel('Virtual channel frame counter')
plt.ylabel('Lost frames');

fc.size/(fc[-1]-fc[0]+1)

vc0_packets = list(ccsds.extract_space_packets(vc0, 49, 0))

vc0_t = timestamps(vc0_packets)

vc0_sp_headers = [ccsds.SpacePacketPrimaryHeader.parse(p) for p in vc0_packets]

vc0_apids = collections.Counter([p.APID for p in vc0_sp_headers])
vc0_apids

apid_axis = {a : k for k, a in enumerate(sorted(vc0_apids))}

plt.figure(figsize = (10, 5), facecolor = 'w')
plt.plot(vc0_t, [apid_axis[p.APID] for p in vc0_sp_headers], '.')
plt.yticks(ticks=range(len(apid_axis)), labels=apid_axis)
plt.xlabel('Space Packet timestamp')
plt.ylabel('APID')
plt.title('Lucy Virtual Channel 0 APID distribution');

vc0_by_apid = {apid : [p for h,p in zip(vc0_sp_headers, vc0_packets)
                       if h.APID == apid] for apid in vc0_apids}

plot_apids(vc0_by_apid)


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Check
Can you write Python code for it?

raw_data = {
        'subject_id': ['14', '15', '16', '17', '18'],
        'first_name': ['Sue', 'Maria', 'Sandra', 'Kate', 'Aurelie'],
        'last_name': ['Bonder', 'Black', 'Balwner', 'Brice', 'Btisan'],
        'pid': ['4', '5', '6', '7', '8'],}
df_a = pd.DataFrame(raw_data, columns = ['subject_id', 'first_name', 'last_name','pid'])
df_a

raw_data = {
        'subject_id': ['4', '5', '6', '7', '8'],
        'first_name': ['Billy', 'Brian', 'Bran', 'Bryce', 'Betty'],
        'last_name': ['Bonder', 'Black', 'Balwner', 'Brice', 'Btisan'],
        'pid': ['14', '15', '16', '17', '18'],}
df_b = pd.DataFrame(raw_data, columns = ['subject_id', 'first_name', 'last_name','pid'])
df_b

df_a.merge(df_b, left_on='pid', right_on='subject_id', how='outer', suffixes=('_me','_partner'))


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Data exploration
Shape, types, distribution, modalities and potential missing values
Can you write Python code for it?

raw_dataset.head(3)

raw_dataset_copy = raw_dataset

#merged_datasets = raw_dataset.merge(raw_dataset_copy, left_on="pid", right_on="iid")
#merged_datasets[["iid_x","gender_x","pid_y","gender_y"]].head(5)
#same_gender = merged_datasets[merged_datasets["gender_x"] == merged_datasets["gender_y"]]
#same_gender.head()

columns_by_types = raw_dataset.columns.to_series().groupby(raw_dataset.dtypes).groups

raw_dataset.dtypes.value_counts()

raw_dataset.isnull().sum().head(3)

summary = raw_dataset.describe() #.transpose()
print summary

#raw_dataset.groupby("gender").agg({"iid": pd.Series.nunique})
raw_dataset.groupby('gender').iid.nunique()

raw_dataset.groupby('career').iid.nunique().sort_values(ascending=False).head(5)

raw_dataset.groupby(["gender","match"]).iid.nunique()


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Data processing
Can you write Python code for it?

local_path = "/Users/sandrapietrowska/Documents/Trainings/luigi/data_source/"
local_filename = "Speed_Dating_Data.csv"
my_variables_selection = ["iid", "pid", "match","gender","date","go_out","sports","tvsports","exercise","dining",
                          "museums","art","hiking","gaming","clubbing","reading","tv","theater","movies",
                          "concerts","music","shopping","yoga"]

class RawSetProcessing(object):
    """
    This class aims to load and clean the dataset.
    """
    def __init__(self,source_path,filename,features):
        self.source_path = source_path
        self.filename = filename
        self.features = features

    # Load data
    def load_data(self):
        raw_dataset_df = pd.read_csv(self.source_path + self.filename)
        return raw_dataset_df

    # Select variables to process and include in the model
    def subset_features(self, df):
        sel_vars_df = df[self.features]
        return sel_vars_df

    @staticmethod
    # Remove ids with missing values
    def remove_ids_with_missing_values(df):
        sel_vars_filled_df = df.dropna()
        return sel_vars_filled_df

    @staticmethod
    def drop_duplicated_values(df):
        df = df.drop_duplicates()
        return df

    # Combine processing stages
    def combiner_pipeline(self):
        raw_dataset = self.load_data()
        subset_df = self.subset_features(raw_dataset)
        subset_no_dup_df = self.drop_duplicated_values(subset_df)
        subset_filled_df = self.remove_ids_with_missing_values(subset_no_dup_df)
        return subset_filled_df

raw_set = RawSetProcessing(local_path, local_filename, my_variables_selection)
dataset_df = raw_set.combiner_pipeline()

dataset_df.head(3)

# Number of unique participants
dataset_df.iid.nunique()

dataset_df.shape


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Feature engineering
Can you write Python code for it?

def get_partner_features(df):
    #print df[df["iid"] == 1]
    df_partner = df.copy()
    df_partner = df_partner.drop(['pid','match'], 1).drop_duplicates()
    #print df_partner.shape
    merged_datasets = df.merge(df_partner, how = "inner",left_on="pid", right_on="iid",suffixes=('_me','_partner'))
    #print merged_datasets[merged_datasets["iid_me"] == 1]
    return merged_datasets

feat_eng_df = get_partner_features(dataset_df)
feat_eng_df.head(3)


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Import data
Can you write Python code for it?

raw_dataset = pd.read_csv(source_path + "Speed_Dating_Data.csv")


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Tuning Parameters
Can you write Python code for it?

# Split the dataset in two equal parts
X_train, X_test, y_train, y_test = train_test_split(explanatory, explained, test_size=0.3, random_state=0)

parameters = [
  {'criterion': ['gini','entropy'], 'max_depth': [4,6,10,12,14],
   'min_samples_split': [10,20,30], 'min_samples_leaf': [10,15,20]
  }
]

scores = ['precision', 'recall']

dtc = tree.DecisionTreeClassifier()
clf = GridSearchCV(dtc, parameters,n_jobs=3, cv=5, refit=True)

for score in scores:
    print("# Tuning hyper-parameters for %s" % score)
    print("")

    clf = GridSearchCV(dtc, parameters, cv=5,
                       scoring='%s_macro' % score)
    clf.fit(X_train, y_train)

    print("Best parameters set found on development set:")
    print("")
    print(clf.best_params_)
    print("")
    y_true, y_pred = y_test, clf.predict(X_test)
    print(classification_report(y_true, y_pred))
    print("")

best_param_dtc = tree.DecisionTreeClassifier(criterion="entropy",min_samples_split=10,min_samples_leaf=10,max_depth=14)
best_param_dtc = best_param_dtc.fit(explanatory, explained)

best_param_dtc.feature_importances_

raw_dataset.rename(columns={"age_o":"age_of_partner","race_o":"race_of_partner"},inplace=True)


I am working on the file "Luigi_airflow_002.ipynb".
The first task is:
Variables selection
Can you write Python code for it?

#features = list(["gender","age_o","race_o","goal","samerace","imprace","imprelig","date","go_out","career_c"])
features = list(["gender","date","go_out","sports","tvsports","exercise","dining","museums","art",
                 "hiking","gaming","clubbing","reading","tv","theater","movies","concerts","music",
                 "shopping","yoga"])

suffix_me = "_me"
suffix_partner = "_partner"

#add suffix to each element of list
def process_features_names(features, suffix_1, suffix_2):
    features_me = [feat + suffix_1 for feat in features]
    features_partner = [feat + suffix_2 for feat in features]
    features_all = features_me + features_partner
    return features_all

features_model = process_features_names(features, suffix_me, suffix_partner)

explanatory = feat_eng_df[features_model]
explained = feat_eng_df[label]


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Data exploration
Shape, types, distribution, modalities and potential missing values
Can you write Python code for it?

raw_dataset.head(3)

raw_dataset_copy = raw_dataset

check1 = raw_dataset_copy[raw_dataset_copy["iid"] == 1]
check1_sel = check1[["iid", "pid", "match","gender","date","go_out","sports","tvsports","exercise","dining",
                          "museums","art","hiking","gaming","clubbing","reading","tv","theater","movies",
                          "concerts","music","shopping","yoga"]]

check1_sel.drop_duplicates().head(20)

#merged_datasets = raw_dataset.merge(raw_dataset_copy, left_on="pid", right_on="iid")
#merged_datasets[["iid_x","gender_x","pid_y","gender_y"]].head(5)
#same_gender = merged_datasets[merged_datasets["gender_x"] == merged_datasets["gender_y"]]
#same_gender.head()

columns_by_types = raw_dataset.columns.to_series().groupby(raw_dataset.dtypes).groups

raw_dataset.dtypes.value_counts()

raw_dataset.isnull().sum().head(3)

summary = raw_dataset.describe() #.transpose()
print summary

#raw_dataset.groupby("gender").agg({"iid": pd.Series.nunique})
raw_dataset.groupby('gender').iid.nunique()

raw_dataset.groupby('career').iid.nunique().sort_values(ascending=False).head(5)

raw_dataset.groupby(["gender","match"]).iid.nunique()


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Data processing
Can you write Python code for it?


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Decision Tree
Can you write Python code for it?

clf = tree.DecisionTreeClassifier(min_samples_split=20,min_samples_leaf=10,max_depth=4)
clf = clf.fit(explanatory, explained)

# Download http://www.graphviz.org/

with open("data.dot", 'w') as f:
    f = tree.export_graphviz(clf, out_file=f, feature_names= features_model, class_names="match")

import subprocess
subprocess.call(['dot', '-Tpdf', 'data.dot', '-o' 'data.pdf'])


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Feature engineering
Can you write Python code for it?

suffix_me = "_me"
suffix_partner = "_partner"

def get_partner_features(df, suffix_1, suffix_2, ignore_vars=True):
    #print df[df["iid"] == 1]
    df_partner = df.copy()
    if ignore_vars is True:
        df_partner = df_partner.drop(['pid','match'], 1).drop_duplicates()
    else:
        df_partner = df_partner.copy()
    #print df_partner.shape
    merged_datasets = df.merge(df_partner, how = "inner",left_on="pid", right_on="iid",suffixes=(suffix_1,suffix_2))
    #print merged_datasets[merged_datasets["iid_me"] == 1]
    return merged_datasets

feat_eng_df = get_partner_features(dataset_df,suffix_me,suffix_partner)
feat_eng_df.head(3)


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Import data
Can you write Python code for it?


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Test
Can you write Python code for it?

import unittest

class FeatureEngineeringTest(unittest.TestCase):
    def test_get_partner_features(self):
        """

        :return:
        """
        # Given
        raw_data_a = {
        'iid': ['1', '2', '3', '4', '5','6'],
        'first_name': ['Sue', 'Maria', 'Sandra', 'Bill', 'Brian','Bruce'],
        'sport':['foot','run','volley','basket','swim','tv'],
        'pid': ['4', '5', '6', '1', '2','3'],}

        df_a = pd.DataFrame(raw_data_a, columns = ['iid', 'first_name', 'sport','pid'])

        expected_output_values = pd.DataFrame({
                'iid_me': ['1', '2', '3', '4', '5','6'],
                'first_name_me': ['Sue', 'Maria', 'Sandra', 'Bill', 'Brian','Bruce'],
                'sport_me': ['foot','run','volley','basket','swim','tv'],
                'pid_me': ['4', '5', '6', '1', '2','3'],
                'iid_partner': ['4', '5', '6', '1', '2','3'],
                'first_name_partner': ['Bill', 'Brian','Bruce','Sue', 'Maria', 'Sandra'],
                'sport_partner': ['basket','swim','tv','foot','run','volley'],
                'pid_partner':['1', '2', '3', '4', '5','6']
            }, columns = ['iid_me','first_name_me','sport_me','pid_me',
                          'iid_partner','first_name_partner','sport_partner','pid_partner'])

        # When

        output_values = get_partner_features(df_a, "_me","_partner",ignore_vars=False)
        print output_values

        # Then

        self.assertItemsEqual(output_values, expected_output_values)

suite = unittest.TestLoader().loadTestsFromTestCase(FeatureEngineeringTest)
unittest.TextTestRunner(verbosity=2).run(suite)


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Tuning Parameters
Can you write Python code for it?


I am working on the file "Luigi_airflow_003.ipynb".
The first task is:
Variables selection
Can you write Python code for it?

#features = list(["gender","age_o","race_o","goal","samerace","imprace","imprelig","date","go_out","career_c"])
features = list(["gender","date","go_out","sports","tvsports","exercise","dining","museums","art",
                 "hiking","gaming","clubbing","reading","tv","theater","movies","concerts","music",
                 "shopping","yoga"])
label = "match"

#add suffix to each element of list
def process_features_names(features, suffix_1, suffix_2):
    print features
    print suffix_1
    print suffix_2
    features_me = [feat + suffix_1 for feat in features]
    features_partner = [feat + suffix_2 for feat in features]
    features_all = features_me + features_partner
    print features_all
    return features_all

features_model = process_features_names(features, suffix_me, suffix_partner)

feat_eng_df.head(5)

explanatory = feat_eng_df[features_model]
explained = feat_eng_df[label]


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Data exploration
Shape, types, distribution, modalities and potential missing values
Can you write Python code for it?

raw_dataset.head(3)

raw_dataset_copy = raw_dataset

check1 = raw_dataset_copy[raw_dataset_copy["iid"] == 1]
check1_sel = check1[["iid", "pid", "match","gender","date","go_out","sports","tvsports","exercise","dining",
                          "museums","art","hiking","gaming","clubbing","reading","tv","theater","movies",
                          "concerts","music","shopping","yoga"]]

check1_sel.drop_duplicates().head(20)

#merged_datasets = raw_dataset.merge(raw_dataset_copy, left_on="pid", right_on="iid")
#merged_datasets[["iid_x","gender_x","pid_y","gender_y"]].head(5)
#same_gender = merged_datasets[merged_datasets["gender_x"] == merged_datasets["gender_y"]]
#same_gender.head()

columns_by_types = raw_dataset.columns.to_series().groupby(raw_dataset.dtypes).groups

raw_dataset.dtypes.value_counts()

raw_dataset.isnull().sum().head(3)

summary = raw_dataset.describe() #.transpose()
print (summary.head())

#raw_dataset.groupby("gender").agg({"iid": pd.Series.nunique})
raw_dataset.groupby('gender').iid.nunique()

raw_dataset.groupby('career').iid.nunique().sort_values(ascending=False).head(5)

raw_dataset.groupby(["gender","match"]).iid.nunique()


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Decision Tree
Can you write Python code for it?

clf = tree.DecisionTreeClassifier(min_samples_split=20,min_samples_leaf=10,max_depth=4)
clf = clf.fit(explanatory, explained)

# Download http://www.graphviz.org/

with open("data.dot", 'w') as f:
    f = tree.export_graphviz(clf, out_file=f, feature_names= features_model, class_names="match")

subprocess.call(['dot', '-Tpdf', 'data.dot', '-o' 'data.pdf'])


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Feature engineering
Can you write Python code for it?


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Modelling
This model aims to answer the questions what is the profile of the persons regarding interests that got the most matches.
Variables:
* gender
* date (In general, how frequently do you go on dates?)
* go out (How often do you go out (not necessarily on dates)?
* sports: Playing sports/ athletics
* tvsports: Watching sports
* excersice: Body building/exercising
* dining: Dining out
* museums: Museums/galleries
* art: Art
* hiking:  Hiking/camping
* gaming: Gaming
* clubbing: Dancing/clubbing
* reading: Reading
* tv: Watching TV
* theater: Theater
* movies: Movies
* concerts: Going to concerts
* music: Music
* shopping: Shopping
* yoga: Yoga/meditation
Can you write Python code for it?

import sklearn
print (sklearn.__version__)

from sklearn import tree
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import classification_report
import subprocess


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Test
Can you write Python code for it?

import unittest
from pandas.util.testing import assert_frame_equal


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
There is a weird thing, with self.XX the code does not work. I tried self.assertEqual
Can you write Python code for it?

class FeatureEngineeringTest(unittest.TestCase):
    def test_get_partner_features(self):
        """

        :return:
        """
        # Given
        raw_data_a = {
        'iid': ['1', '2', '3', '4', '5','6'],
        'first_name': ['Sue', 'Maria', 'Sandra', 'Bill', 'Brian','Bruce'],
        'sport':['foot','run','volley','basket','swim','tv'],
        'pid': ['4', '5', '6', '1', '2','3'],}

        df_a = pd.DataFrame(raw_data_a, columns = ['iid', 'first_name', 'sport','pid'])

        expected_output_values = pd.DataFrame({
                'iid_me': ['1', '2', '3', '4', '5','6'],
                'first_name_me': ['Sue', 'Maria', 'Sandra', 'Bill', 'Brian','Bruce'],
                'sport_me': ['foot','run','volley','basket','swim','tv'],
                'pid_me': ['4', '5', '6', '1', '2','3'],
                'iid_partner': ['4', '5', '6', '1', '2','3'],
                'first_name_partner': ['Bill', 'Brian','Bruce','Sue', 'Maria', 'Sandra'],
                'sport_partner': ['basket','swim','tv','foot','run','volley'],
                'pid_partner':['1', '2', '3', '4', '5','6']
            }, columns = ['iid_me','first_name_me','sport_me','pid_me',
                          'iid_partner','first_name_partner','sport_partner','pid_partner'])

        # When

        output_values = get_partner_features(df_a, "_me","_partner",ignore_vars=False)

        # Then

        assert_frame_equal(output_values, expected_output_values)

suite = unittest.TestLoader().loadTestsFromTestCase(FeatureEngineeringTest)
unittest.TextTestRunner(verbosity=2).run(suite)


I am working on the file "Luigi_airflow_004.ipynb".
The first task is:
Variables selection
Can you write Python code for it?

#features = list(["gender","age_o","race_o","goal","samerace","imprace","imprelig","date","go_out","career_c"])
features = list(["gender","date","go_out","sports","tvsports","exercise","dining","museums","art",
                 "hiking","gaming","clubbing","reading","tv","theater","movies","concerts","music",
                 "shopping","yoga"])
label = "match"

#add suffix to each element of list
def process_features_names(features, suffix_1, suffix_2):
    features_me = [feat + suffix_1 for feat in features]
    features_partner = [feat + suffix_2 for feat in features]
    features_all = features_me + features_partner
    print (features_all)
    return features_all

features_model = process_features_names(features, suffix_me, suffix_partner)

feat_eng_df.head(5)

explanatory = feat_eng_df[features_model]
explained = feat_eng_df[label]


I am working on the file "Luigi_airflow_005.ipynb".
The first task is:
Data exploration
Shape, types, distribution, modalities and potential missing values
Can you write Python code for it?

raw_dataset.head(2)

raw_dataset_copy = raw_dataset

check1 = raw_dataset_copy[raw_dataset_copy["iid"] == 1]
check1_sel = check1[["iid", "pid", "match","gender","date","go_out","sports","tvsports","exercise","dining",
                          "museums","art","hiking","gaming","clubbing","reading","tv","theater","movies",
                          "concerts","music","shopping","yoga"]]

check1_sel.drop_duplicates().head(2)

#merged_datasets = raw_dataset.merge(raw_dataset_copy, left_on="pid", right_on="iid")
#merged_datasets[["iid_x","gender_x","pid_y","gender_y"]].head(5)
#same_gender = merged_datasets[merged_datasets["gender_x"] == merged_datasets["gender_y"]]
#same_gender.head()

columns_by_types = raw_dataset.columns.to_series().groupby(raw_dataset.dtypes).groups

raw_dataset.dtypes.value_counts()

raw_dataset.isnull().sum().head(3)

summary = raw_dataset.describe() #.transpose()
#print (summary.head())

#raw_dataset.groupby("gender").agg({"iid": pd.Series.nunique})
raw_dataset.groupby('gender').iid.nunique()

raw_dataset.groupby('career').iid.nunique().sort_values(ascending=False).head(5)

raw_dataset.groupby(["gender","match"]).iid.nunique()