I am working on the file "PCA.ipynb".
The first task is:
Then, we apply <i>PCA</i> function to the data and print eigenvalues associated with two PCs. The results indicate that the highest variance is held in the first PC with variance of ~1.83 while the second PC has variance of ~0.16.
Can you write Python code for it?

# Applying PCA
V,Ypca,D = PCA(X)            # perform PCA, where V, Ypca and D are
                             # eigenvectors, new transformation and eigenvalues,
                             # respectively
print("Eigenvalues D: %s" % D)


I am working on the file "PCA.ipynb".
The first task is:
We can also browse throught the reconstructions for other digits. Once again, interact becomes handy.
Can you write Python code for it?

@interact(i=(0, 10))
def show_pca_digits(i=1):
    plt.figure(figsize=(4,4))
    actual_sample = X[i].reshape(28,28)
    reconst_sample = (reconst[i, :] * std + mu).reshape(28, 28)
    plt.imshow(np.hstack([actual_sample, reconst_sample]), cmap='gray')
    plt.show()


I am working on the file "PCA.ipynb".
The first task is:
We can also compare the running time for PCA and PCA_high_dim directly. Spend some time and think about what this plot means. We mentioned in lectures that PCA_high_dim are advantageous when
we have dataset size $N$ < data dimension $D$. Although our plot for the two running times does not intersect exactly at $N = D$, it does show the trend.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.set(xlabel='number of datapoints', ylabel='run time')
ax.errorbar(times0[:, 0], times0[:, 1], times0[:, 2], label="PCA", linewidth=2)
ax.errorbar(times1[:, 0], times1[:, 1], times1[:, 2], label="PCA_high_dim", linewidth=2)
ax.legend();


I am working on the file "PCA.ipynb".
The first task is:
We can visualize the original, 4D iris data of $X$ by using the first $k$ eigenvectors of $X$, projecting the original data into a reduced-rank $k$-dimensional principal component space.
\begin{align}
  T_{rank=k} &= X V_{rank=k}
\end{align}
Can you write Python code for it?

# let's try using the first 2 principal components
k = 2

# divide T by label
irises = [T[T.species=='setosa'],
          T[T.species=='versicolor'],
          T[T.species=='virginica']]

# define a color-blind friendly set of quantative colors
# for each species
colors = ['#1b9e77', '#d95f02', '#7570b3']

_, (ax1, ax2) = plt.subplots(1, 2, sharey=False)

# plot principal component vis-a-vis total variance in data
ax1.plot([1,2,3,4],
         eigenvalues_normalized,
         '-o',
         color='#8da0cb',
         label='eigenvalue (normalized)',
         alpha=0.8,
         zorder=1000)

ax1.plot([1,2,3,4],
         cumvar_explained,
         '-o',
         color='#fc8d62',
         label='cum. variance explained',
         alpha=0.8,
         zorder=1000)

ax1.set_xlim(0.8, 4.2)
ax1.set_xticks([1,2,3,4])
ax1.set_xlabel('Principal component')
ax1.set_ylabel('Variance')
ax1.legend(loc='center right', fontsize=7)
ax1.grid(color='#fdfefe')
ax1.set_facecolor('#f4f6f7')

# plot the reduced-rank score matrix representation
for group, color in zip(irises, colors):
    ax2.scatter(group.pc1,
                group.pc2,
                marker='^',
                color=color,
                label=group.species,
                alpha=0.5,
                zorder=1000)
ax2.set_xlabel(r'PC 1')
ax2.set_ylabel(r'PC 2')
ax2.grid(color='#fdfefe')
ax2.set_facecolor('#f4f6f7')
ax2.legend(labels=iris.target_names, fontsize=7)

plt.suptitle(r'Fig. 1: PCA via eigen-decomposition, 2D visualization')
plt.tight_layout(pad=3.0)
plt.show()


I am working on the file "PCA.ipynb".
The first task is:
We first benchmark the time taken to compute $\boldsymbol X^T\boldsymbol X$ and $\boldsymbol X\boldsymbol X^T$. Jupyter's magic command %time is quite handy.
Next we benchmark PCA, PCA_high_dim.
Can you write Python code for it?

times0 = []
times1 = []

for datasetsize in np.arange(4, 784, step=100):
    XX = Xbar[:datasetsize]
    npc = 2
    mu, sigma = time(lambda : PCA( XX, 2))
    times0.append((datasetsize, mu, sigma))

    mu, sigma = time(lambda : PCA_high_dim(XX, 2))
    times1.append((datasetsize, mu, sigma))

times0 = np.asarray(times0)
times1 = np.asarray(times1)


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
Exploring Plot Variations
The example plots of PCA components have been shown to contrast the first principal component versus the second. We can display variations as follows:
Can you write Python code for it?

pca_scatter(0, 0)

pca_scatter(1, 3)

pca_scatter(1, 2)

pca_scatter(2, 3)


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
Note, once the data is standardized, our factor loadings change. Whereas before sepal width and petal length were the most informative features, now sepal length and petal length are most informative.
How have the eigenvalues changed?
Can you write Python code for it?

pca.explained_variance_

pca.explained_variance_ratio_

scree_plot()

var_explained()


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
Note: There are a number of criterion for accepting a principal component. The Kaiser criterion is simple to implement, but a more rigorous method known an Parallel Analysis can also be used. A program could be written to evaluate this in Python. Here is an example of the implementation in Stata.
Using the Kaiser criterion, only the first principal component is significant and should be analyzed.
Visualizing Eigenvalues
Related to the Kaiser criterion and component selection, one visual method of selection is known as a scree plot, an example of which is below.
Can you write Python code for it?

def scree_plot():
    from matplotlib.pyplot import figure, show
    from matplotlib.ticker import MaxNLocator

    ax = figure().gca()
    ax.plot(pca.explained_variance_)
    ax.xaxis.set_major_locator(MaxNLocator(integer=True))
    plt.xlabel('Principal Component')
    plt.ylabel('Eigenvalue')
    plt.axhline(y=1, linewidth=1, color='r', alpha=0.5)
    plt.title('Scree Plot of PCA: Component Eigenvalues')
    show()

scree_plot()


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
PCA - Iris Data Download
The previous result is somewhat simplified from the process you might have to go through if you downloaded data versus using the packaged format of a dataset that comes in a module such as Sklearn. To demonstrate the process, I will once again use the Iris dataset, but I will download the dataset instead.
Can you write Python code for it?

import pandas as pd

df = pd.read_csv(
    filepath_or_buffer='https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data',
    header=None,
    sep=',')

df.columns=['sepal_len', 'sepal_wid', 'petal_len', 'petal_wid', 'class']
df.dropna(how="all", inplace=True) # drops the empty line at file-end
df.to_csv("iris_download.csv", index = False)
df.head()

#Select Columns (Variables) on which to run PCA
X = df.loc[:, 'sepal_len':'petal_wid'].values
y = df.loc[:, 'class'].values

pca = PCA() #If we do not specify how many components, all are included
X_r = pca.fit(X).transform(X)

target_names = df.ix[:,4].unique()
target_names

def pca_scatter(pca1, pca2):
    plt.close
    plt.figure()
    colors = ['red', 'cyan', 'blue']
    lw = 2

    for color, target_name in zip(colors, target_names):
        plt.scatter(X_r[y == target_name, pca1], X_r[y == target_name, pca2], color=color, alpha=.6, lw=lw,
                    label=target_name)
    plt.legend(loc='best', shadow=False, scatterpoints=1)
    plt.title('PCA of IRIS Dataset: Components {} and {}'.format(pca1+1, pca2+1))
    plt.xlabel('Component {}'.format(pca1+1))
    plt.ylabel('Component {}'.format(pca2+1))
    plt.show()

pca_scatter(0, 1)


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
Selecting Components
Since the aim of PCA is data reduction, we need some criteria or we will have the same number of variables after a PCA.
One rule of thumb is that a component should not be retained unless it has an eigenvalue greater than or equal to one (the ‘Kaiser’ criterion).
How can we evaluate the eigenvalues?
This depends on the approach taken in making the calculation. While I would recommend using the default PCA algorithms in Sklearn, another approach is simply using NumPy. I demonstrate both below with the Iris data.
Can you write Python code for it?

#Using Numpy on the Previously Defined X-Y
import numpy as np
from sklearn.preprocessing import StandardScaler

#Covariance Matrix
cov_mat = np.cov(X.T)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)

print('\nEigenvalues \n%s' %eig_vals)
print('Eigenvectors \n%s' %eig_vecs)

#Using Sklearn (As Performed Above)

X_r = pca.fit(X).transform(X)
print('\nEigenvalues \n%s' %pca.explained_variance_)
print('Eigenvectors \n%s' %pca.components_)


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
So approximately 92.5% of the variance is explained by the first principal component (component 0) and about 98% is explained by the first two components.
Examining PCA Loadings
Another facet of PCA is examining the component loadings or components. This allows us to see how informative a particular variable is for the principal component. Paying attention to the loadings is particularly useful in scale construction using PCA (or PCA in combination with Cronbach's Alpha and Factor Analysis).
Can you write Python code for it?

#Component Loadings

def loadings():
    loadings = pca.components_
    results = pd.DataFrame(loadings)
    results.columns = ['Component 1', 'Component 2', 'Component 3', 'Component 4']
    results.index=['sepal_len', 'sepal_wid', 'petal_len', 'petal_wid']
    print(results)

loadings()


I am working on the file "PCA/PCA_in_Python.ipynb".
The first task is:
Variance Explained
Related to scree plots and eigenvalues is the variance explained (which is directly related to the eigenvalues). This can also be visualized.
Can you write Python code for it?

def var_explained():
    import numpy as np
    from matplotlib.pyplot import figure, show
    from matplotlib.ticker import MaxNLocator

    ax = figure().gca()
    ax.plot(np.cumsum(pca.explained_variance_ratio_))
    ax.xaxis.set_major_locator(MaxNLocator(integer=True))
    plt.xlabel('Number of Components')
    plt.ylabel('Cumulative Explained Variance')
    plt.axvline(x=1, linewidth=1, color='r', alpha=0.5)
    plt.title('Explained Variance of PCA by Component')
    show()

var_explained()


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
1. vector length (norm)
$ \| a \| = \sqrt{a^T a } = \sqrt{a_1^2 + \cdots + a_K^2} $
Can you write Python code for it?

import numpy as np
a = np.array([1, 1])
np.linalg.norm(a)


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
10. Basis Vector
벡터 공간에 속하는 벡터의 집합이 선형 독립이고 다른 모든 벡터 공간의 벡터들이 그 벡터 집합의 선형 조합으로 나타나면 그 벡터 집합을 벡터 공간의 기저 벡터(basis vector)라고 한다.
예를 들어 다음과 같은 두 벡터는 2차원 벡터 공간의 기저 벡터이다.
$
a = \begin{bmatrix}1 \ 0\end{bmatrix} ,\;\;
b = \begin{bmatrix}0 \ 1\end{bmatrix} \;\;
$
또는
$
a = \begin{bmatrix}1 \ 1\end{bmatrix} ,\;\;
b = \begin{bmatrix}2 \ 3\end{bmatrix} \;\;
$
다음과 같은 두 벡터는 2차원 벡터 공간의 기저 벡터가 될 수 없다.
$
a = \begin{bmatrix}1 \ 2\end{bmatrix} ,\;\;
b = \begin{bmatrix}2 \ 4\end{bmatrix} \;\;
$
Can you write Python code for it?

# Linearly Dependent : No Basis Vector
a = np.array([1, 2])
b = np.array([2, 4])
2*a - b


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
11. Column Space
행렬은 열 벡터의 집합으로 볼 수 있다. 이 때 열 벡터들의 조합으로 생성되는 벡터 공간을 열 공간(column space)이라고 한다.
$
A = \begin{bmatrix} 1 & 5 & 6 \ 2 & 6 & 8 \ 7 & 1 & 8 \end{bmatrix}
\;\;\;\; \rightarrow \;\;\;\;
\alpha_1 \begin{bmatrix} 1 \ 2 \ 7 \end{bmatrix} +
\alpha_2 \begin{bmatrix} 5 \ 6 \ 1 \end{bmatrix} +
\alpha_3 \begin{bmatrix} 6 \ 8 \ 8 \end{bmatrix}
\; \in \; \text{column space}
$
12. Column Rank
행렬의 열 벡터 중 서로 독립인 열 벡터의 최대 갯수를 열 랭크(column rank) 혹은 랭크(rank)라고 한다.
예를 들어 다음 행렬의 랭크는 2이다.
$
A = \begin{bmatrix} 1 & 5 & 6 \ 2 & 6 & 8 \ 3 & 11 & 14 \end{bmatrix}
$
numpy의 linalg 서브 패키지의 matrix_rank 명령으로 랭크를 계산할 수 있다.
Can you write Python code for it?


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
13. Coordinate
벡터의 성분, 즉 좌표(coordinate)는 표준 기저 벡터들에 대한 해당 벡터의 투영(projection)으로 볼 수 있다.
아래 좌표는 $(e_1, e_2) = (2,1)$ 이다.
* 좌표계를 $(e_1, e_2)$로 정의 했으므로 $(1,2)$가 아니다.
Can you write Python code for it?

e1 = np.array([1, 0])
e2 = np.array([0, 1])
a = np.array([1, 2])

plt.annotate('', xy=e1, xytext=(0,0), arrowprops=dict(facecolor='green'))
plt.annotate('', xy=e2, xytext=(0,0), arrowprops=dict(facecolor='green'))
plt.annotate('', xy=a, xytext=(0,0), arrowprops=dict(facecolor='gray'))

plt.plot(0, 0, 'ro', ms=10)
plt.plot(a[0], a[1], 'ro', ms=10)

plt.text(1.05, 1.35, "$a$", fontdict={"size": 18})
plt.text(-0.2, 0.5, "$e_1$", fontdict={"size": 18})
plt.text(0.5, -0.2, "$e_2$", fontdict={"size": 18})

plt.xticks(np.arange(-2, 4))
plt.yticks(np.arange(-1, 4))
plt.xlim(-1.5, 3.5)
plt.ylim(-0.5, 3)
plt.show()


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
2. Unit Vector
길이가 1인 벡터 : $\dfrac{x}{\| x \|}$
Can you write Python code for it?

a = np.array([1, 0])
b = np.array([0, 1])
c = np.array([1/np.sqrt(2), 1/np.sqrt(2)])
np.linalg.norm(a), np.linalg.norm(b), np.linalg.norm(c)


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
5. Orthogonal of Vector
두 벡터 $a$와 $b$가 이루는 각이 90도이면 서로 직교(orthogonal)라고 하며 $ a \perp b $로 표시한다.
서로 직교인 두 벡터의 벡터 내적(inner product, dot product)는 0이된다.
$ a^T b = b^T a = 0   \;\;\;\; \leftrightarrow \;\;\;\; a \perp b $
note

inner product : $xy = x^Ty = \begin{bmatrix}3 & 1\end{bmatrix} \begin{bmatrix}2 \ 4\end{bmatrix} = 32 + 14 = 10 $ (result is scala)
Can you write Python code for it?

a = np.array([1, 1])
b = np.array([-1, 1])
np.dot(a, b)


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
6. Projection of Vector
$ a^Tb = \| a \|\| b \|\cos\theta $
$ a^Tb = (a_1 + a_2)^Tb = a_1^Tb + a_2^Tb = a_2^Tb \;\; (a_1^Tb = 0,\;  a_1\perp b) $
$ \| a_2 \| = a^T\dfrac{b}{\|b\|} = \dfrac{a^Tb}{\|b\|} = \dfrac{b^Ta}{\|b\|} $
Can you write Python code for it?


I am working on the file "PCA/vectorspace.ipynb".
The first task is:
9. Linearly Dependent & Linearly Independent
벡터들의 선형 조합이 0이 되는 모두 0이 아닌 스칼라값들이 존재하면 그 벡터들은 선형 종속(linearly dependent)이라고 한다.
$
a = \begin{bmatrix}1 \ 2\end{bmatrix} ,\;\;
b = \begin{bmatrix}3 \ 3\end{bmatrix} \;\;
c = \begin{bmatrix}10 \ 14\end{bmatrix} \;\;
$
$
2a  + b - \frac{1}{2}c = 0
$
벡터들의 선형 조합이 0이 되는 모두 0이 아닌 스칼라값들이 존재하지 않으면 그 벡터들은 선형 독립(linearly independent)이라고 한다.
$ \alpha_1 a_1 + \cdots + \alpha_K a_K = 0  \;\;\;\; \leftrightarrow \;\;\;\; \alpha_1 = \cdots = \alpha_K = 0 $
Can you write Python code for it?

# Linearly Dependent
a = np.array([1, 2])
b = np.array([3, 3])
c = np.array([10, 14])
2*a + b - 0.5*c


I am working on the file "PCA_EOF_example.ipynb".
The first task is:
And begin by looking at the spectrum of singular values $\Sigma$.  Defining the variance as $\Sigma^2$ then we can also calculate the cumulative contribution to the total variance as
$$
    g_k = \frac{\sum_{i=0}^k \sigma_i^2}{\sum_{i=0}^n \sigma_i^2}
$$
Plotting both $\Sigma$ and $g$ shows that $\sim$ 80% of the total variance can be explained by the first 4-5 Components
Can you write Python code for it?

# plot the singular values
plt.figure()
plt.semilogy(S,'bo')
plt.grid()
plt.title('Singular Values')
plt.show()

# and cumulative percent of variance
g = np.cumsum(S*S)/np.sum(S*S)
plt.figure()
plt.plot(g,'bx-')
plt.title('% cumulative percent variance explained')
plt.grid()
plt.show()


I am working on the file "PCA_EOF_example.ipynb".
The first task is:
EOF analysis
While each profile lives in an 80 dimensional space, we would like to see if we can classify the variability in fewer components.  To begin we form a de-meaned data matrix $X$ where each row is a profile.
Can you write Python code for it?

plt.figure()
X = data - data_mean
plt.imshow(X)
plt.xlabel('Distance across axis (Km)')
plt.ylabel('Relative Spreading Rate')
plt.colorbar()
plt.show()


I am working on the file "PCA_EOF_example.ipynb".
The first task is:
For example,  the first EOF pattern is primarily a symmetric pattern with an axial high surrounded by two off axis troughs (or an axial low with two flanking highs,  the EOF's are just unit vector bases for the row-space and can be added with any positive or negative coefficient).  The Second EOF is broader and all of one sign while the third EOF encodes assymetry.
Reconstruction
Using the SVD we can also decompose each profile into a weighted linear combination of EOF's i.e.
$$
    X = U\Sigma V^T = C V^T
$$
where $C = U\Sigma$ is a matrix of coefficients  that describes the how each data row is decomposed into the relevant basis vectors.  We can then produce a k-rank truncated representation of the data by
$$
    X_k = C_k V_k^T
$$
where $C_k$ is the first $k$ columns of $C$ and $V_k$ is the first $k$ EOF's.
Here we show the original data and the reconstructed data using the first 5 EOF's
Can you write Python code for it?

# recontruct the data using the first 5 EOF's
k=5
Ck = np.dot(U[:,:k],np.diag(S[:k]))
Vtk = Vt[:k,:]
data_k = data_mean + np.dot(Ck,Vtk)

plt.figure()
plt.imshow(data_k)
plt.colorbar()
plt.title('reconstructed data')
plt.show()


I am working on the file "PCA_EOF_example.ipynb".
The first task is:
Principal Component/EOF analysis
GOAL: Demonstrate the use of the SVD to calculate principal components or "Empirical Orthogonal Functions" in a geophysical data set.  This example is modified from a paper by Chris Small (LDEO)
Small, C., 1994. A global analysis of mid-ocean ridge axial topography. Geophys J Int 116, 64–84. doi:10.1111/j.1365-246X.1994.tb02128.x
The Data
Here we will consider a set of topography profiles taken across the global mid-ocean ridge system where the Earth's tectonic plates are spreading apart.

The data consists of 156 profiles from a range of spreading rates. Each profile contains 80 samples so is in effect a vector in $R^{80}$
Can you write Python code for it?

# read the data from the csv file
data = np.genfromtxt('m80.csv', delimiter='')
data_mean =  np.mean(data,0)

# and plot out a few profiles and the mean depth.
plt.figure()
rows = [ 9,59,99]
labels = [ 'slow','medium','fast']
for i,row in enumerate(rows):
    plt.plot(data[row,:],label=labels[i])
    plt.hold(True)
plt.plot(data_mean,'k--',label='mean')
plt.xlabel('Distance across axis (km)')
plt.ylabel('Relative Elevation (m)')
plt.legend(loc='best')
plt.title('Example cross-axis topography of mid-ocean ridges')
plt.show()


I am working on the file "PCA_EOF_example.ipynb".
The first task is:
projection of data onto a subspace
We can also use the Principal Components to look at the projection of the data onto a lower dimensional space as the coefficients $C$, are simply the coordinates of our data along each principal component.  For example we can view the data in the 2-Dimensional space defined by the first 2 EOF's by simply plotting C_1 against C_2.
Can you write Python code for it?

# plot the data in the plane defined by the first two principal components
plt.figure()
plt.scatter(Ck[:,0],Ck[:,1])
plt.xlabel('$V_1$')
plt.ylabel('$V_2$')
plt.grid()
plt.title('Projection onto the first two principal components')
plt.show()

# Or consider the degree of assymetry (EOF 3) as a function of spreading rate
plt.figure()
plt.plot(Ck[:,2],'bo')
plt.xlabel('Spreading rate')
plt.ylabel('$C_3$')
plt.grid()
plt.title('Degree of assymetry')
plt.show()


I am working on the file "PD1D.ipynb".
The first task is:
Moving on the corresondence models, we repeat the steps above.  First for an equally spaced grid.
Can you write Python code for it?

fixed_length = 40
delta_x = 0.5
fixed_horizon = 3.5 * delta_x

problem3 = PD_Problem(bar_length=fixed_length, number_of_elements=(fixed_length/delta_x),
                      horizon=fixed_horizon, constitutive_model_flag='correspondence', randomization_factor=0.0)
problem3.solve(prescribed_displacement=0.1)
disp3 = problem3.get_solution()
nodes3 = problem3.get_nodes()
plt.plot(nodes3, disp3, 'k-');
plt.grid(b=True);
plt.axis('tight');
plt.xlabel('$x$ position');
plt.ylabel('Displacement');

def_grad3 = problem3.get_deformation_gradient()
plt.plot(nodes3, def_grad3, 'k-');
plt.grid(b=True);
plt.axis('tight');
plt.xlabel('$x$ position');
plt.ylabel('Deformation Gradient');
plt.ylim([1.0,1.01]);


I am working on the file "PD1D.ipynb".
The first task is:
Note the very small oscilations in the displacement field.  Just to verify below we plot the node volumes to show the randomness.
Can you write Python code for it?

plt.plot(range(len(nodes2)), problem2.lengths, 'k-');
plt.ylabel('Node volume');
plt.xlabel('Node number');


I am working on the file "PD1D.ipynb".
The first task is:
Now a randomly purturbed grid.
Can you write Python code for it?

fixed_length = 40
delta_x = 0.5
fixed_horizon = 3.5 * delta_x

problem4 = PD_Problem(bar_length=fixed_length, number_of_elements=(fixed_length/delta_x),
                      horizon=fixed_horizon, constitutive_model_flag='correspondence', randomization_factor=0.3)
problem4.solve(prescribed_displacement=0.1)
disp4 = problem4.get_solution()
nodes4 = problem4.get_nodes()
plt.plot(nodes4, disp4, 'k-');
plt.grid(b=True);
plt.axis('tight');

def_grad4 = problem4.get_deformation_gradient()
plt.plot(nodes4, def_grad4, 'k-');
plt.grid(b=True);
plt.axis();


I am working on the file "PD1D.ipynb".
The first task is:
Now plot the results
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

plt.plot(nodes1, disp1, 'k-');
plt.grid(b=True);
plt.axis('tight');
plt.ylabel('Displacement');
plt.xlabel('$x$ position');


I am working on the file "PD1D.ipynb".
The first task is:
Now we instantiate the problem using the LPS constititive model defined in the last section.
Can you write Python code for it?

fixed_length = 40
delta_x = 0.5
fixed_horizon = 3.5 * delta_x

problem1 = PD_Problem(bar_length=fixed_length, number_of_elements=(fixed_length/delta_x),
                      horizon=fixed_horizon, constitutive_model_flag='LPS', randomization_factor=0.0)


I am working on the file "PD1D.ipynb".
The first task is:
Now we will repeat the steps above, this time using a random purtabation of 30% to the interior nodes.  The nodes near the boundaries are not purturbed in any way to preserve consistant application of the boundary conditions.
Can you write Python code for it?

fixed_length = 40
delta_x = 0.5
fixed_horizon = 3.5 * delta_x

problem2 = PD_Problem(bar_length=fixed_length, number_of_elements=(fixed_length/delta_x),
                      horizon=fixed_horizon, constitutive_model_flag='LPS', randomization_factor=0.3)
problem2.solve(prescribed_displacement=0.1)
disp2 = problem2.get_solution()
nodes2 = problem2.get_nodes()
plt.plot(nodes2, disp2, 'k-');
plt.grid(b=True);
plt.axis('tight');
plt.ylabel('Displacment');
plt.xlabel('$x$ position');


I am working on the file "PD1D.ipynb".
The first task is:
Solve the problem and get the solution vectors for plotting.
Can you write Python code for it?

problem1.solve(prescribed_displacement=0.0001)
disp1 = problem1.get_solution()
nodes1 = problem1.get_nodes()


I am working on the file "PD1D.ipynb".
The first task is:
We can also get the deformation gradient and plot it as well.  Note that the deformation gradient is not used in the constitutive model in any way.  It is post processed and plotted here for comparison.
Can you write Python code for it?

def_grad1 = problem1.get_deformation_gradient()
plt.plot(nodes1, def_grad1, 'k-');
plt.grid(b=True);
plt.axis('tight');
plt.ylabel('Deformation Gradient');
plt.xlabel('$x$ position');
plt.ylim([0.99,1.01])


I am working on the file "PDBbind.ipynb".
The first task is:
Download a dataset from PDBbind and unpack (I used core-set 2016).
Can you write Python code for it?

%%bash
wget -qO- http://www.pdbbind.org.cn/download/pdbbind_v2016_core.tar.gz | tar xz

directory = './core-set/'


I am working on the file "PDBbind.ipynb".
The first task is:
If you want to check activity, you can use the sets dict.
Can you write Python code for it?


I am working on the file "PDBbind.ipynb".
The first task is:
Let's choose one target.
Can you write Python code for it?


I am working on the file "PDBbind.ipynb".
The first task is:
Target has three properties: protein, pocket and ligand. All of them are of oddt.tolkit.Molecule class.
Let's find the largest ligand.
Can you write Python code for it?

max_atoms = 0

for target in pdbbind_database:
    if max_atoms < len(target.ligand.atoms):
        max_atoms = len(target.ligand.atoms)
        largest = target

print('Target ID:', largest.id, '\nNumber of atoms:', max_atoms)

largest_ligand = largest.ligand
largest_ligand.removeh()
largest_ligand


I am working on the file "PDBbind.ipynb".
The first task is:
We will use the pdbbind class.
Can you write Python code for it?

pdbbind_database = pdbbind(home=directory,
    version='2016',
    default_set='core') # Available sets in wrapper: core, refined, general_PL (general for 2007)


I am working on the file "PDBbind.ipynb".
The first task is:
You can always check PDB ID.
Can you write Python code for it?


I am working on the file "PDBbind.ipynb".
The first task is:
You can get one target or iterate over all of them. To do it you can use PDB ID of a target or an index from list in INDEX file (INDEX_refined_data.2016).
The class has two properties: ids and activities.
Can you write Python code for it?

all_ids = pdbbind_database.ids
print('Number of targets:', len(all_ids))
print('First ten targets:', all_ids[:10])

all_activities = pdbbind_database.activities
print('First ten activities:', all_activities[:10])


I am working on the file "PDE.ipynb".
The first task is:
The solutions are unstable, using lax method and replacing $u^{n-1}j$ with $(u^{n-1}{j+1} + u^{n-1}_{j-1})/2$ will make the equations stable.
$u^{n}{j} = (u^{n-1}{j+1} + u^{n-1}{j-1})/2 - v \frac{\Delta t}{\Delta x} (u^{n-1}{j+1} - u^{n-1}_{j-1})$
This is equvalent to solving this equation:
$\frac{\partial u}{\partial t} = -v \frac{\partial u}{\partial x} + \frac{\Delta x^2}{2 \Delta t} \frac{\partial^2 u}{\partial x^2}$
This will make problem if we would like to look at the long term properties of the system.
Can you write Python code for it?


I am working on the file "PDE.ipynb".
The first task is:
This has error of order $\Delta t$ and $\Delta x^2$
$u^{n}{j} = u^{n-1}{j} - v \frac{\Delta t}{\Delta x} (u^{n-1}{j+1} - u^{n-1}{j-1})$
Can you write Python code for it?


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
And the next cell sets up a slider which controls the above graphs (it moves time along)
Can you write Python code for it?

# This runs an animation, controlled by a slider which advances time
interact(update,k=(0,t_len-1))


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
And we include some code to create the graphical user interface -- namely, sliders. You can read about them here:
http://bokeh.pydata.org/en/0.10.0/_images/notebook_interactors.png
Can you write Python code for it?


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
Comparing solutions
In principle, we want these different solution methods to be directly comparable.
So let's try this out, by computing the difference of two solution.
Here we compare the explicit f.d. method with d'Alambert's method.
Can you write Python code for it?

(u_exp,x,t)=w_solve1(.5,dx,dt,t_len,1*triangle,0*triangle)
(u_dal,x,t)=w_solve3(.5,dx,dt,t_len,1*triangle,0*triangle)

def update2(k=0):
    plot(x,u_dal[:,k]-u_exp[:,k])
    show()

interact(update2,k=(0,t_len-1))


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
Derivative initial condition
Can you write Python code for it?


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
Derivative initial condition test
Can you write Python code for it?

# We try again, but this time with the $u_t$ initial condition equal to the triangle impulse
(u,x,t)=w_solve1(.5,dx,dt,3*t_len,0*triangle,1*triangle)


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
Implicit method
Here we try an implicit method for solving the wave equation. Again from Myint-U and Debnath's book, in Section 14.5, part (B) on Hyperbolic equations.
We need to use scipy libraries as we have to solve a system of linear equation. In fact the system is tridiagonal and Toepliz, so this should be fast. I see how to use Toepliz in scipy, but I don't know how to tell it that the system is only tridiagonal. It should be possible to speed this up.
Can you write Python code for it?

# Based on Section 14.5 (B) in Myint-U and Debnath's book
def w_solve2(c2,dx,dt,t_len,u0,u1):
    x_len = np.size(u0)  # the length of u0 implicitly defines the num of points in x direction
    u = np.zeros((x_len,t_len),order='F') # output array initialized to zero
    e2 = c2*dt*dt/(dx*dx)  # Courant parameter squared (test for convergence!)
    print("CFL value is ",np.sqrt(e2))
    kern = np.array([e2, 2*(1-e2), e2]) # the convolution kernel we need for laplacian solver
    u[:,0] = u0            # t=0 initial condition
    u[:,1] = np.convolve(u0,kern/2)[1:x_len+1] + dt*u1 # t=0 derivative condition, eqn 14.4.6
    # Note the above is a cheat, we are using the explicit method to find u[:,1], Should do this implicitly
    kern2 = np.array([e2, -2*(1+e2), e2]) # the convolution kernel we need for implicit solver. It is different.
    toepk = np.zeros(x_len)  # this will hold the entries for the tridiagonal Toeplitz matrix
    toepk[0] = 2*(1+e2);
    toepk[1] = -e2
    for j in range(2,t_len):
        rhs = np.convolve(u[:,j-2],kern2)[1:x_len+1] + 4*u[:,j-1] # eqn 14.5.17
        u[:,j] = solve_toeplitz(toepk, rhs) # here is a linear system solver (hence an implicit method)
    # let's produce the x and t vectors, in case we need them
    x = np.linspace(0,dx*x_len,x_len,endpoint=False)
    t = np.linspace(0,dt*t_len,t_len,endpoint=False)
    return u,x,t

(u,x,t)=w_solve2(.5,dx,dt,t_len,1*triangle,0*triangle)

interact(update,k=(0,3*t_len-1))


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
Now we call up our wave equation solver, using the parameters above
Can you write Python code for it?

# Here we solve the wave equation, with initial position $u(x,0)$ set to the triangle waveform
(u,x,t)=w_solve1(.5,dx,dt,t_len,triangle,0*triangle)


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
We can plot the inital waveform, just to see what it looks like.
Can you write Python code for it?

plot(x,u[:,0])

def update(k=0):
    plot(x,u[:,k])
    show()


I am working on the file "PDE_Solve_widget.ipynb".
The first task is:
We can use the same update function, since nothing has changed.
Can you write Python code for it?


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Assignment and Index
Convert from factor assignment $A$ to the value index $I$ based on the cardinalities $D$.
Can you write Python code for it?

def convert_assignment_to_index(A, D):
    step = [np.prod(D[i+1:]) for i in range(len(D)-1)] + [0]
    return np.sum(np.multiply(A, step)) + A[-1]

A = [1, 0, 1]
I = convert_assignment_to_index(A, D = phi['cardinalities'])
I


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Basic Math Functions
get_member_index
get_member_index function returns the lowest indices of values in array $A$ found in array $B$.
Can you write Python code for it?

def get_member_index(A, B):
    C = []
    for v in A:
        idx = np.where(B == v)[0]
        if len(idx):
            C.append(idx[0])
    return np.array(C)

A = np.array([1,2,3])
B = np.array([2,1,3])
get_member_index(A, B)

A = np.array([1,2,3])
B = np.array([1,4,3,5,2])
get_member_index(A, B)


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Factor Product
Can you write Python code for it?


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Factor Product
factor_product function returns product of two factors $A$ and $B$ stored in $C$.
Can you write Python code for it?

def factor_product(A, B):
    variables = np.union1d(A.variables, B.variables)
    ttl_variables = len(variables)

    mapA = get_member_index(A.variables, variables)
    mapB = get_member_index(B.variables, variables)

    cardinalities = np.zeros(ttl_variables, int)
    cardinalities[mapA] = A.cardinalities
    cardinalities[mapB] = B.cardinalities

    ttl_values = np.prod(cardinalities)
    values = np.zeros(ttl_values)

    C = factor(variables = variables,
               cardinalities = cardinalities,
               values = values)

    assignments = np.array([C.get_assignment_from_index(i) for i in range(ttl_values)])
    indexA = [A.get_index_from_assignment(assignment) for assignment in assignments[:, mapA]];
    indexB = [B.get_index_from_assignment(assignment) for assignment in assignments[:, mapB]];

    values = np.multiply(A.values[indexA], B.values[indexB])
    for i in range(len(C.values)):
        C.set_value_of_assignment(assignments[i], values[i])

    return C

phi_X1_X2 = factor_product(phi_X1, phi_X2)

phi_X1_X2.get_table()


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Factor Reduction
Can you write Python code for it?

evidence = {
    'X_2': 0,
    'X_3': 1,
}

phi_X1.reduce(evidence).get_table()

phi_X2.reduce(evidence).get_table()

phi_X3.reduce(evidence).get_table()


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Imports
Can you write Python code for it?


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Joint Distribution
Can you write Python code for it?

def calculate_joint_distribution(F):
    joint = F[0]
    for i in range(1, len(F)):
        joint = factor_product(joint, F[i])
    return joint

F = [phi_X1, phi_X2, phi_X3]
phi_X1_X2_X3 = calculate_joint_distribution(F)

phi_X1_X2_X3.get_table()


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Operations
Sample factors ($\phi(X_1)$, $\phi(X_2)$, and $\phi(X_3)$).$^{[3]}$
Can you write Python code for it?

phi_X1 = factor(variables = ['X_1'],
                cardinalities = [2],
                values = [0.11, 0.89])

phi_X2 = factor(variables = ['X_1', 'X_2'],
                cardinalities = [2, 2],
                values = [0.59, 0.41, 0.22, 0.78])

phi_X3 = factor(variables = ['X_2', 'X_3'],
                cardinalities = [2, 2],
                values = [0.39, 0.61, 0.06, 0.94])

phi_X1.get_table()

phi_X2.get_table()

phi_X3.get_table()


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
Tested On
Can you write Python code for it?

import sys
print("Python %s" % sys.version)

import numpy as np
print("NumPy %s" % np.__version__)

import sklearn
from sklearn.utils.extmath import cartesian
print("scikit-learn %s" % sklearn.__version__)

import pandas as pd
print("Pandas %s" % pd.__version__)


I am working on the file "PGM-W1/Factor.ipynb".
The first task is:
The card key stores cardinality of all variables. In the example, the cardinality of variable $X_3$, $|Val(X_3)|$ is 2.
Following function converts factor data structure into the table form for displaying purpose.
Can you write Python code for it?

def convert_factor_ds_to_table(phi):
    cart = cartesian([range(d) for d in phi['cardinalities']])
    # Construct table
    df = pd.DataFrame(cart)
    df.columns = ['$%s$' % v for v in phi['variables']]
    df['$\phi(%s)$' % ','.join([v for v in phi['variables']])] = phi['values']
    return df

phi_table = convert_factor_ds_to_table(phi)

phi_table


I am working on the file "PHYS2211.Measurement.ipynb".
The first task is:
2. Wheatstone bridge measurements
Can you write Python code for it?


I am working on the file "PIHM_visualization.ipynb".
The first task is:
Download MM-PIHM input and output files from GitHub
To use the interactive example, download MM-PIHM input and output files from GitHub.
Can you write Python code for it?

!wget -c https://github.com/PSUmodeling/MM-PIHM/raw/master/output/ShaleHillsTestRun/ShaleHills.gw.dat -P output/ShaleHillsTestRun >& /dev/null
!wget -c https://github.com/PSUmodeling/MM-PIHM/raw/master/output/ShaleHillsTestRun/ShaleHills.river.flx1.dat -P output/ShaleHillsTestRun >& /dev/null
!wget -c https://github.com/PSUmodeling/MM-PIHM/raw/master/input/ShaleHills/ShaleHills.mesh -P input/ShaleHills >& /dev/null
!wget -c https://github.com/PSUmodeling/MM-PIHM/raw/master/input/ShaleHills/ShaleHills.riv -P input/ShaleHills >& /dev/null
!echo "Download completed."


I am working on the file "PIHM_visualization.ipynb".
The first task is:
Plot time series
This example plots temporal variations of river discharge (river.flx1 at river outlets).
Can you write Python code for it?

sim_time, sim_val, vname, unit = read_output(pihm_dir, simulation, output_dir, 'river.flx1')

# Plot average in time
fig, ax = plt.subplots()
#ax.plot(sim_time, np.average(sim_val, axis=1), '-o')
ax.plot(sim_time, sim_val[:, outlets], '-o')

# Set x and y axis labels
ax.set_xlabel('Date')
ax.set_ylabel('Discharge')
ax.set_xlim(datetime(2009, 1, 1), datetime(2010, 1, 1))
ax.set_ylim([0, 0.04])

# Clean up the x axis dates
ax.xaxis.set_major_locator(mdates.MonthLocator(interval=2))
ax.xaxis.set_major_formatter(mdates.DateFormatter("%m/%y"))
fig.tight_layout()  # Otherwise y-label could be slightly clipped


I am working on the file "PIHM_visualization.ipynb".
The first task is:
Use PIHM-utils to read MM-PIHM input and output files
read_mesh function
The read_mesh function reads mesh information from MM-PIHM .mesh input files:
Can you write Python code for it?

num_elem, num_nodes, tri, x, y, zmin, zmax = read_mesh(pihm_dir, simulation)


I am working on the file "PIHM_visualization.ipynb".
The first task is:
where num_elem is the number of triangular elements,
num_nodes is the number of nodes, tri is an array of triangles consisted of nodes,
x, y, zmin, and zmax are arrays of the x and y coordinates, and bottom and surface elevations of nodes, respectively.
pihm_dir is the path to the MM-PIHM directory, which should contain input and output directories,
and simulation is the name of the simulation.
read_river function
The read_river function reads river information from MM-PIHM .river input files:
Can you write Python code for it?

num_rivers, from_node, to_node, outlets = read_river(pihm_dir, simulation)


I am working on the file "PIHM_visualization.ipynb".
The first task is:
where num_rivers is the number of river segments,
from_nodes and to_nodes are arrays of from and to nodes of river segments,
and outlets is an array of river outlets.
read_output function
The read_output function reads MM-PIHM simulation output files:
Can you write Python code for it?

sim_time, sim_val, vname, unit = read_output(pihm_dir, simulation, output_dir, 'gw')


I am working on the file "PIHM_visualization.ipynb".
The first task is:
where sim_time is an array of simulation time steps,
sim_val is an array of output values containing simulation results from all model grids at all model steps,
and vname and unit are strings containing description and unit of the specific output variable.
The function requires the input of the name of output variable (i.e., gw in this example, which represents groundwater storage).
For a complete list of available output variables, please refer to the MM-PIHM User's Guide.
Plot variable spatial distribution
MM-PIHM uses irregular triangular grids.
Matplotlib tripcolor function can be used to plot MM-PIHM simulation results in space.
Can you write Python code for it?

# Import other packages needed for visualization
from datetime import datetime
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import numpy as np
from matplotlib.collections import LineCollection

# In the test run, model starts on 20 Oct, 2008. The simulation period from 20 Oct to 31 Dec 2009 is used as spin-up,
# and is excluded from visualization
sim_val = sim_val[sim_time >= datetime(2009, 1, 1), :]
sim_time = sim_time[sim_time >= datetime(2009, 1, 1)]

# Create line collection of river segments for plotting
lines = [[(x[node1], y[node1]), (x[node2], y[node2])] for (node1, node2) in zip(from_node, to_node)]
river_segments = LineCollection(
    lines,
    linewidths=5,
    colors='blue',
    alpha=0.7,
    linestyle='solid',
)

fig = plt.figure()

# Test the shape of domain to determine if portrait or landscape mode should be used
domain_xy_ratio = (np.max(x) - np.min(x)) / (np.max(y) - np.min(y))
ax = fig.add_axes([0, 0, 0.8, 1]) if domain_xy_ratio < 1.33 else fig.add_axes([0, 0.2, 1, 0.8])

# Plot output in space
tpc = ax.tripcolor(
    x, y, tri,
    facecolors=np.average(sim_val, axis=0),     # Average in time
    edgecolors='k',
    lw=0.5,
    cmap='RdBu'
)
ax.add_collection(river_segments)
ax.set_aspect('equal')

# Add color bar with variable name and unit
if domain_xy_ratio < 1.33:
    cbaxes = fig.add_axes([0.8, 0.2, 0.025, 0.6])
    cbar = fig.colorbar(tpc, cax=cbaxes)
else:
    cbaxes = fig.add_axes([0.2, 0.15, 0.6, 0.04])
    cbar = fig.colorbar(tpc, orientation='horizontal', cax=cbaxes)
cbar.set_label(label='%s (%s)' % (vname, unit))
ax.axis('off')

plt.show()


I am working on the file "PIV_092117.ipynb".
The first task is:
Autocorrelation Plot
Procedure:

-working to examine when each point in the velocity field becomes time indep.

- in development
Can you write Python code for it?


I am working on the file "PIV_092117.ipynb".
The first task is:
Control Volume Analysis
Procedure:

1). Create mean velocity field from mean u and v velocity fields ($(u^2 + v^2)^{1/2}$)

2). Integrate left side of image (control volume in)

3). Integrate right side of image (control volumne out)

4). Calculate difference (control volume delta) and divide by streamwise length of FOV
Can you write Python code for it?

## Control Volume Analysis ##
umean = np.nanmean(u[0], axis=0)
vmean = np.nanmean(v[0], axis=0)
mean_vel = np.sqrt(umean**2 + vmean**2)
#print(np.shape(mean_vel))
cv_in = np.trapz(mean_vel[:, 0], x = y)*-1
cv_out = np.trapz(mean_vel[:, -1], x = y)*-1
cv_delta = cv_out - cv_in
vel_out_y = cv_delta / (x[-1] - x[0])

plt.figure(num=None, figsize=(8, 6), dpi=100, facecolor='w', edgecolor='k')
plt.semilogx(y, mean_vel[:, 0], '-xb')
plt.semilogx(y, mean_vel[:, -1], '-xr')
plt.legend(['Control Volume In', 'Control Volume Out'])
plt.ylabel('Velocity Magnitude (m/sec)')
plt.xlabel('Wall Normal Position (m)')


I am working on the file "PIV_092117.ipynb".
The first task is:
Inner Normalized Plots
Can you write Python code for it?

##Plot Inner Normalized Data##
date = '092117'
num_tests = 3
utau = .15
legend = [r'$Re_{\theta}=$30288, Cont.', r'$Re_{\theta}=$30288, 100im', r'$Re_{\theta}=$30288, 500im']
piv_inner(date, num_tests, utau, legend)


I am working on the file "PIV_092117.ipynb".
The first task is:
Mean Velocity Plots
Can you write Python code for it?

# Plot Outer Normalized Data
date = '092117'
legend = [r'$Re_{\theta}=$30288, Cont.', r'$Re_{\theta}=$30288, 100im', r'$Re_{\theta}=$30288, 500im']
num_tests = 3
piv_outer(date, num_tests, legend)


I am working on the file "PIV_092117.ipynb".
The first task is:
Read in and Filter Datasets
Can you write Python code for it?

## DATA SET READ IN ##
#data sets taken continuously (test_0, test_3)
#Parameter set
date = '092117_0'
data_delimiter = '\t'
num_images = 10917
sizex = 128
sizey = 129
walloffset = 2 #mm
side_error = 5
#determine file name
file_name = dict()
for j in range(1, num_images+1):
    file_name[j] = '/B' + str('{0:05}'.format(j)) + '.txt'
#list name of data set folders
base_name = dict()
#List the base name for each test to be read in and analyzed, names taken directly from folder
base_name[0] = '/media/drummond/My Passport/DATA/FPF/test_092117/Cam_Date=170921_Time=120913_TR_SeqPIV_MP(1x16x16_50ov_ImgCorr)=unknown'
base_name[1] = '/media/drummond/My Passport/DATA/FPF/test_092117/Cam_Date=170921_Time=140859_TR_SeqPIV_MP(1x16x16_50ov_ImgCorr)=unknown'
[u, v, x, y, bad_im] = piv_readin(date, file_name, base_name, num_images, data_delimiter, sizex, sizey, walloffset, side_error)

## DATA SET READ IN ##
#data set taken on cycle, 100 images every 1hz (test_1)
#Parameter set
date = '092117_1'
data_delimiter = '\t'
num_images = 10907
sizex = 128
sizey = 129
walloffset = 2 #mm
side_error = 5
#determine file name
file_name = dict()
for j in range(1, num_images+1):
    file_name[j] = '/B' + str('{0:05}'.format(j)) + '.txt'
#list name of data set folders
base_name = dict()
#List the base name for each test to be read in and analyzed, names taken directly from folder
base_name[0] = '/media/drummond/My Passport/DATA/FPF/test_092117/Cam_Date=170921_Time=124152_TR_SeqPIV_MP(1x16x16_50ov_ImgCorr)=unknown'
piv_readin(date, file_name, base_name, num_images, data_delimiter, sizex, sizey, walloffset, side_error)

## DATA SET READ IN ##
#data set taken on cycle, 500 images every .5hz (test_1)
#Parameter set
date = '092117_2'
data_delimiter = '\t'
num_images = 10520
sizex = 128
sizey = 129
walloffset = 2 #mm
side_error = 5
#determine file name
file_name = dict()
for j in range(1, num_images+1):
    file_name[j] = '/B' + str('{0:05}'.format(j)) + '.txt'
#list name of data set folders
base_name = dict()
#List the base name for each test...


I am working on the file "PKS/voyager.ipynb".
The first task is:
Data shape is (1, 524288, 2), and dtype is presented as complex64 (but stored as 8-bit in file)
Axes are (channel, time sample, polarization). You can iterate over multiple blocks of data to get more time samples if required, until the end of the file.
Can you write Python code for it?


I am working on the file "PKS/voyager.ipynb".
The first task is:
Extract sidebands and zoom
Can you write Python code for it?

fc_cw   = 8575.34545  # Center freq of transmission
sb_sep  = 0.0225      # Sideband separation from carrier in MHz
sb_bw   = 0.001       # Sideband bandwidth
plot_bw = sb_bw / 2   # Plotting bandwidth
t = a.timestamps

# Calculate center freq of sidebands
fc_lsb = fc_cw - sb_sep
fc_usb = fc_cw + sb_sep

# Extract data
f_cw,   d_cw   = a.grab_data(f_start=fc_cw-plot_bw, f_stop=fc_cw+plot_bw)
f_lsb,  d_lsb  = a.grab_data(f_start=fc_lsb-plot_bw, f_stop=fc_lsb+plot_bw)
f_usb,  d_usb  = a.grab_data(f_start=fc_usb-plot_bw, f_stop=fc_usb+plot_bw)

# Plot
def plot_waterfall(t, f, d):
    t_elapsed = (t[-1] - t[0]) * 86400
    plt.imshow(d[::-1], aspect='auto', extent=(f[-1], f[0], 0, t_elapsed))
    plt.xticks(rotation=30)
    plt.xlabel("Frequency [MHz]")

plt.figure(figsize=(14, 5))
plt.subplot(1,3,1)
plt.ylabel("Elapsed time [s]")
plot_waterfall(t, f_lsb, d_lsb)
plt.subplot(1,3,2)
plot_waterfall(t, f_cw, d_cw)
plt.subplot(1,3,3)
plot_waterfall(t, f_usb, d_usb)


I am working on the file "PKS/voyager.ipynb".
The first task is:
Plotting and showing dynamic spectra, we can see the signal drifting due to doppler acceleration (LO does not correct for LSRK)
Can you write Python code for it?

plt.figure(figsize=(10, 6))
a.plot_spectrum()

plt.figure(figsize=(14, 6))
a.plot_waterfall()


I am working on the file "PKS/voyager.ipynb".
The first task is:
Raw voltage data
Raw voltage data for a 2.92 MHz subband was recorded about the Voyager signal. It is stored in Guppi RAW format, as 8-bit integers. This can also be loaded using blimpy.
Can you write Python code for it?

raw_filename = '/datax2/users/dancpr/voyager_2018.10.22.raw'
raw = bl.GuppiRaw(raw_filename)

raw_header0, raw_data0 = raw.read_next_data_block()


I am working on the file "PKS/voyager.ipynb".
The first task is:
We can compute the spectrum from these data
Can you write Python code for it?

dx = raw_data0.squeeze()[:, 0]
dy = raw_data0.squeeze()[:, 1]

n_chans = 8192

plt.figure(figsize=(12, 6))
plt.subplot(1,2,1)
Dx = np.abs(np.fft.fftshift(np.fft.fft(dx)))
Dx = Dx.reshape([n_chans, -1]).mean(axis=-1)
plt.plot(Dx)

plt.subplot(1,2,2)
Dy = np.abs(np.fft.fftshift(np.fft.fft(dy)))
Dy = Dy.reshape([n_chans, -1]).mean(axis=-1)
plt.plot(Dy)


I am working on the file "PLSQL_Neural_Network/MNIST_oracle_plsql.ipynb".
The first task is:
Create the package that runs the neural network in PL/SQL
Notes:
- The main function is MNIST.SCORE: it takes as input an image to process (p_testimage_array utl_nla_array_flt) and returns the predicted number.
- The procedure MNIST.INIT, loads the tensors from the table tensors_array into PL/SQL global variables.
Can you write Python code for it?

%%bash
sqlplus -s mnist/mnist@dbserver:1521/orcl.cern.ch  <<EOF

create or replace package mnist
as
    -- MNIST scoring enginge in PL/SQL
    -- Author: Luca.Canali@cern.ch, July 2016
    g_b0_array utl_nla_array_flt;
    g_W0_matrix utl_nla_array_flt;
    g_b1_array utl_nla_array_flt;
    g_W1_matrix utl_nla_array_flt;

    function score(p_testimage_array utl_nla_array_flt) return number;
    procedure init;
end;
/

create or replace package body mnist
as
    procedure init
    /* initialize the tensors that make up the neural network */
    as
    begin
        SELECT tensor_vals INTO g_W0_matrix FROM tensors_array WHERE name='W0';
        SELECT tensor_vals INTO g_W1_matrix FROM tensors_array WHERE name='W1';
        SELECT tensor_vals INTO g_b0_array FROM tensors_array WHERE name='b0';
        SELECT tensor_vals INTO g_b1_array FROM tensors_array WHERE name='b1';
    end;

    procedure print_debug(p_array utl_nla_array_flt)
    /* useful for debugging pourposes, prints an array to screen. requires set serveroutput on */
    as
    begin
        dbms_output.put_line('***************');
        for i in 1..p_array.count loop
            dbms_output.put_line('p_array(' || i ||') = ' || TO_CHAR(p_array(i),'9999.9999'));
        end loop;
        dbms_output.put_line('**************');
    end;

    function argmax(p_array utl_nla_array_flt)  return integer
    as
        v_index number;
        v_maxval float;
    begin
        v_index := 1;
        v_maxval := p_array(v_index);
        for i in 2..p_array.count loop
            if ( p_array(i) > v_maxval) then
                v_index := i;
                v_maxval := p_array(v_index);
            end if;
        end loop;
    return(v_index);
    end;

    function score(p_testimage_array utl_nla_array_flt) return number
    as
        v_Y0 utl_nla_array_flt;
        v_output_array utl_nla_array_flt;
    begin
        v_Y0 := g_b0_array;

        /* this is part of the computation of the hidden layer, Y0 = W0_matrix * p_test_image_array + B0 */
 ...


I am working on the file "PLSQL_Neural_Network/MNIST_oracle_plsql.ipynb".
The first task is:
Test the scoring engine with all the test images
Notes:
- from the SQL here below that the neural network and the serving engine MNIST.SCORE correctly predicts 9787 out of 10000 images, that is has an accuracy on the test set of ~98%
- The execution time for processing 10000 test images is about 2 minutes, that is ~12 ms to process each image on average
Can you write Python code for it?

%%bash
sqlplus -s mnist/mnist@dbserver:1521/orcl.cern.ch <<EOF

exec mnist.init
set timing on
select sum(decode(mnist.score(image_array), label, 1, 0)) "Images correctly identified",
       count(*) "Total number of images"
from testdata_array;

EOF


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Definition of the neural network:

The following defines a basic feed forward neural network with one hidden layer
Other standard techniques used are the definition of cross entropy as loss function and the use of gradient descent as optimizer
Can you write Python code for it?

# define and initialize the tensors

x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

W0 = tf.Variable(tf.truncated_normal([784, 100], stddev=0.1))
b0 = tf.Variable(tf.zeros([100]))

W1 = tf.Variable(tf.truncated_normal([100, 10], stddev=0.1))
b1 = tf.Variable(tf.zeros([10]))

# Feed forward neural network with one hidden layer

# y0 is the hidden layer with sigmoid activation
y0 = tf.sigmoid(tf.matmul(x, W0) + b0)

# y1 is the output layer (softmax)
# y1[n] is the predicted probability that the input image depicts number 'n'
y1 = tf.nn.softmax(tf.matmul(y0, W1) + b1)

# The the loss function is defined as cross_entropy
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y1), reduction_indices=[1]))

# train the network using gradient descent
train_step = tf.train.GradientDescentOptimizer(learning_rate=0.5).minimize(cross_entropy)

# start a TensorFlow interactive session
sess = tf.InteractiveSession()
sess.run(tf.initialize_all_variables())


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Example of how to run the neural network "manually" using the tensor values extracted into numpy arrays
Can you write Python code for it?

import numpy as np

def softmax(x):
    """Compute the softmax function on a numpy array"""
    return np.exp(x) / np.sum(np.exp(x), axis=0)

def sigmoid(x):
    """Compute the sigmoid function on a numpy array"""
    return (1 / (1 + np.exp(-x)))

testimage=testimages[0]
testlabel=testlabels[0]

hidden_layer = sigmoid(np.dot(testimage, W0_matrix) + b0_array)
predicted = np.argmax(softmax(np.dot(hidden_layer, W1_matrix) + b1_array))

print ("image label %d, predicted value by the neural network: %d" % (testlabel, predicted))


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Extracting the test images and labels as numpy arrays
Can you write Python code for it?

testlabels=tf.argmax(mnist.test.labels,1).eval()
testimages=mnist.test.images

print ("testimages is matrix of size: %s " % (testimages.shape,) )
print ("testlabels is array of size:  %s " % (testlabels.shape,) )


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Learning exercise: extract the tensors as 'manually' run the neural network scoring
In the following you can find an example of how to manually run the neural network scoring in Python using numpy. This is intended as an example to further the understanding of how the scoring engine works and opens the way for the next steps, that is the implementation of the scoring engine for Oracle using PL/SQL (see also the notebook MNIST_oracle_plsql.ipynb)
Can you write Python code for it?

# There are 2 matrices and 2 vectors used in this neural network:
W0_matrix=W0.eval()
b0_array=b0.eval()
W1_matrix=W1.eval()
b1_array=b1.eval()

print ("W0 is matrix of size: %s " % (W0_matrix.shape,) )
print ("b0 is array of size:  %s " % (b0_array.shape,) )
print ("W1 is matrix of size: %s " % (W1_matrix.shape,) )
print ("b1 is array of size:  %s " % (b1_array.shape,) )


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Train the network

The training uses 55000 images with labels
It is performed over 30000 iterations using mini batch size of 100 images
Can you write Python code for it?

batch_size = 100
train_iterations = 30000

# There are mnist.train.num_examples=55000 images in the train sample
# train in batches of 'batch_size' images at a time
# Repeat for 'train_iterations' number of iterations
# Training batches are randomly calculated as each new epoch starts

for i in range(train_iterations):
    batch = mnist.train.next_batch(100)
    train_data = feed_dict={x: batch[0], y_: batch[1]}
    train_step.run(train_data)

# Test the accuracy of the trained network
correct_prediction = tf.equal(tf.argmax(y1, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("Accuracy of the trained network over the test images: %s" %
      accuracy.eval({x: mnist.test.images, y_: mnist.test.labels}))


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Transfer of the tensors and test data into Oracle tables
For the following you should have access to a (test) Oracle database. This procedure has been tested with Oracle 11.2.0.4 and 12.1.0.2 on Linux.
To keep the test isolated you can create a dedicated user (suggested name, mnist) for the data transfer, as follows:

From a DBA account (for example the user system) execute:
SQL> create user mnist identified by mnist default tablespace users quota unlimited on users;
SQL> grant connect, create table, create procedure to mnist;
SQL> grant read, write on directory DATA_PUMP_DIR to mnist;

These are the tables that will be used in the following code to transfer the tensors and testdata:

SQL> connect mnist/mnist@ORCL
SQL> create table tensors(name varchar2(20), val_id number, val binary_float, primary key(name, val_id));
SQL> create table testdata(image_id number, label number, val_id number, val binary_float, primary key(image_id, val_id));

Open the connection to the database using cx_Oracle:
(for tips on how to install and use of cx_Oracle see also https://github.com/LucaCanali/Miscellaneous/tree/master/Oracle_Jupyter)
Can you write Python code for it?

import cx_Oracle
ora_conn = cx_Oracle.connect('mnist/mnist@dbserver:1521/orcl.cern.ch')
cursor = ora_conn.cursor()


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Transfer the matrixes W0 and W1 into the table tensors (which must be precreated as described above)
Can you write Python code for it?

i = 0
sql="insert into tensors values ('W0', :val_id, :val)"
for column in W0_matrix:
    array_values = []
    for element in column:
        array_values.append((i, float(element)))
        i += 1
    cursor.executemany(sql, array_values)

ora_conn.commit()

i = 0
sql="insert into tensors values ('W1', :val_id, :val)"
for column in W1_matrix:
    array_values = []
    for element in column:
        array_values.append((i, float(element)))
        i += 1
    cursor.executemany(sql, array_values)

ora_conn.commit()


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Transfer the test data with images and labels into the table "testdata" (the table is expected to exist on the DB, create it using the SQL described above)
Can you write Python code for it?

image_id = 0
array_values = []
sql="insert into testdata values (:image_id, :label, :val_id, :val)"
for image in testimages:
    val_id = 0
    array_values = []
    for element in image:
        array_values.append((image_id, testlabels[image_id], val_id, float(element)))
        val_id += 1
    cursor.executemany(sql, array_values)
    image_id += 1

ora_conn.commit()


I am working on the file "PLSQL_Neural_Network/MNIST_tensorflow_exp_to_oracle.ipynb".
The first task is:
Transfer the vectors b0 and b1 into the table "tensors" (the table is expected to exist on the DB, create it using the SQL described above)
Can you write Python code for it?

i = 0
sql="insert into tensors values ('b0', :val_id, :val)"
array_values = []
for element in b0_array:
    array_values.append((i, float(element)))
    i += 1
cursor.executemany(sql, array_values)

i = 0
sql="insert into tensors values ('b1', :val_id, :val)"
array_values = []
for element in b1_array:
    array_values.append((i, float(element)))
    i += 1
cursor.executemany(sql, array_values)

ora_conn.commit()


I am working on the file "PNC_01Basics.ipynb".
The first task is:
<style>
.answers {
    visibility: hidden;
}
</style>

Python Analysis for GEOS-Chem
Author: Barron H. Henderson
Attendees will learn how to leverage Python to interact with air pollution-related model and observational data. Air research and application relies on big data. In addition to the challenge presented by data size, researchers must understand a multitude of formats and meta-data standards. For example, CMAQ, CAMx, and GEOS-Chem all use different formats and different meta-data conventions. This tutorial provides format-independent and convention-independent tools.
Can you write Python code for it?

# Prepare my slides
%pylab inline
%cd working


I am working on the file "PNC_01Basics.ipynb".
The first task is:
Basic Types
int, float:

Literals 1 is int, 1. is float
64-bit precision values (i.e., float is real*8)

str:

literal '...' or "..."
multi-line uses triple quotes
indexed (aka subset) with square braces and "slices"
version 3 uses system dependent string types (use encode or decode methods0

*A method is a bound function...
Integers and Floats
Can you write Python code for it?


I am working on the file "PNC_01Basics.ipynb".
The first task is:
CHECK POINT
Change the code to operate on all times, latitude, and longitudes without a loop.
Can you write Python code for it?

out = np.zeros_like(ozone[:, 0])
for t in range(ozone.shape[0]):
    for j in range(ozone.shape[2]):
        for i in range(ozone.shape[3]):
            out[t, j, i] = ozone[t, :, j, i] * averagekernelm.T
out


I am working on the file "PNC_01Basics.ipynb".
The first task is:
Open Notebook
Intro to python/ipython/jupyter

Options
python - least burden on system
ipython - least burden on user
jupyter - least burden on next user

Running interactively python/ipython
Terminal
Integrated Development Environments
Notebook

Running saved scripts
from terminal $ python /path/to/file.py
from python `>>> execfile('/path/to/file.py')
from ipython In[1] %run /path/to/file.py
from spyder open in editor

*Windows replace $ with C:\>
Basic Functions
def foo(bar1, bar2, foo = 1, a = 2):
    return 'foo' + bar1

created with def name and call signature followed by colon
defined by commands following signature until the first unindented command

Most Useful Function... ever.
help:
- ? in ipython or jupyter
- it will return a docstring for that element or type
Really Useful Functions
type:
- All things in python are objects (even classes).
- type is a function object that returns the type of another object.
dir:
- List of methods and properties
print:
- display representation of object
Can you write Python code for it?

def foo(bar):
    return 'foo.' + bar

print(foo('bar'))

try:
    print(foo(1))
except Exception as e:
    print(e)


I am working on the file "PNC_01Basics.ipynb".
The first task is:
Which dimension is 2 long? 3 long? 4 long? 5 long?

Make an "averaging" kernel matrix
Can you write Python code for it?

averagekernel = np.array([[0.5, 0.35, 0.15],
                          [0.25, 0.5, 0.25],
                          [0.1, 0.4, 0.5]])
averagekernelm = np.matrix(averagekernel)

print(averagekernel.shape)
print(averagekernelm.shape)
print(averagekernelm.T.shape)

averagekernel = np.array([0.25, 0.5, 0.25])
averagekernelm = np.matrix(averagekernel)


I am working on the file "PNC_01Basics.ipynb".
The first task is:
ozone[0, :, 2, 3] * averagekernelm
Can you write Python code for it?


I am working on the file "PNC_01Basics.ipynb".
The first task is:
strings
Can you write Python code for it?


I am working on the file "PNC_02Figures.ipynb".
The first task is:
CHECK POINT

List three attributes the IAS variable have (hint: use -v IAS)?
-

What are IAS units?
-

What attributes does the file have?
-

Common Processing and Terminology
This section will explain many of the techniques used in the tile plot section and in all subsequent sections.

slicing in numpy
dimensional reductions
Loading data from different formats
CMAQ (already done)
CAMx, WRF, GEOS-Chem, CSV, NASA AMES, AQS
Adding coordinate variables
Using named dimensions via PseudoNetCDF
Adding derived variables via PseudoNetCDF

Time series
Make time series plots with 3 different methods from CMAQ data.

Python Environment
Python with PseudoNetCDF
Command Line Interface (terminal or DOS)
Advanced users will add observations
Advanced users will add another species on a secondary axis
Advanced users will repeat with CAMx or GEOS-Chem

Python Environment

Start easy with a 2-d line plot
Add real data
Transform data to time.
Can you write Python code for it?


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Download GEOS-Chem Benchmark
Benchmark data is available from Harvard's website as below
!curl -LO ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/v10-01/v10-01-public-release/Run0/tracerinfo.dat
!curl -LO ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/v10-01/v10-01-public-release/Run0/diaginfo.dat
!curl -LO ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/v10-01/v10-01-public-release/Run0/bpch.tar.gz
Smaller, Faster Downloads
Can you write Python code for it?


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Fill in the options below to plot May ozone at the surface
Can you write Python code for it?

!pncmap.py -f "bpch,nogroup=('IJ-AVG-\$',)" -v O3 -s layer72,? \
    bpch/ctm.bpch.v10-01-public-Run0.2013050100 map500hPa_


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Now use the command line to make the plot
Can you write Python code for it?

!pncmap.py --overlay --norm="Normalize(0, 100)" -v O3 \
 bpch_500hPa.nc dc3.nc dc3bpchmap
%ls dc3bpchmap*


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Overlay Observations and Model
Find the range of space and time covered by the observations.
This is our first look at observations from the Common Data Model perspective. Remember: dimensions, properties, and variables (which have dimensions and properties).
Make a map with just GEOS-Chem
pncmap.py is the primary map maker. Like all other pnc commands, it uses options that can be reviewed with --help. It shares all the same slicing, aggregating functions, but it also has a few options that are not included else where.
$ pncmap.py --help
Review the following options

--matplotlibrc
--figure-keywords
--axes-keywords
--plot-commands
--figformat
--norm
--colorbar-formatter
--overlay
--resolution
--no-countries, --states, --counties
--shapefile
--no-squeeze
--iter

Start by plotting the whole world for ozone at the 500 hPa
What layer is approximately 500 hPa?
Can you write Python code for it?

!pncdump.py -v etam_pressure --full-indices c -f bpch bpch/ctm.bpch.v10-01-public-Run0.2013050100


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Python with real data
Can you write Python code for it?

from PseudoNetCDF import PNC
args = PNC('--format=ffi1001', \
           'icartt/dc3-mrg60-dc8_merge_20120518_R7_thru20120622.ict')
infile = args.ifiles[0]
fdays = infile.variables['Fractional_Day']
o3 = infile.variables['O3_ESRL']
close()
plot(fdays, o3)
xlabel('Fractional Day')
ylabel('Ozone ' + o3.units)
?tick_params


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Scatter Plots

Python Environment (replace plot with scatter, utc with some other variable)
Python with PseudoNetCDF
Command Line Interface (terminal or DOS) (replace pncts.py with pncscatter.py)
Advanced users will switch from time/space paired to rank paired

Tile Plots
Make tile plots of ozone with 3 different methods from CMAQ data.

Python Environment
Python with PseudoNetCDF
Command Line Interface (terminal or DOS)
Advanced users will overlay observations
Advanced users will repeat with CAMx or GEOS-Chem

Python only
Can you write Python code for it?

from PseudoNetCDF import PNC
args = PNC("--format=bpch,vertgrid='GEOS-5-NATIVE',nogroup=('IJ-AVG-$',)",
           "bpch/ctm.bpch.v10-01-public-Run0.2013050100")
infile = args.ifiles[0]
latb = infile.variables['latitude_bounds']
lonb = infile.variables['longitude_bounds']
ozone = infile.variables['O3'][:, 0].mean(0)

lone = np.append(lonb[0,0], lonb[:, 1])
late = np.append(latb[0,0], latb[:, 1])
LON, LAT = np.meshgrid(lone, late)

close()
pcolor(LON, LAT, ozone, vmin = 20, vmax = 65)
colorbar();


I am working on the file "PNC_02Figures.ipynb".
The first task is:
Unzip june too.
CHECK POINT:
List the size of of the file you downloaded
ANSWERS Hidden
<div style="visibility: hidden">
ls -lh
ls -lh icartt
```
296M bpch/ctm.bpch.v10-01-public-Run0.2013050100
296M bpch/ctm.bpch.v10-01-public-Run0.2013060100
31M icartt/dc3-mrg60-dc8_merge_20120518_R7_thru20120622.ict
```
</div>

Intro to the Common Data Model

files and groups
dimensions
properties
variables
Conventions
IOAPI and WRF-IOAPI
COARDS
Climate Forecasting (CF) Conventions
Conceptualizing any data set as CDM

pncdump returns the common data language
--header : don't show data
-f ffi1001 : format is NASA AMES ffi1001
-v VAR1,VAR2,...,VARN : only show these variables
For archiving, I am only dumping 3 variables. Try omitting -v
Can you write Python code for it?

!pncdump.py --header -v Fractional_Day,UTC,O3_ESRL -f ffi1001 icartt/dc3-mrg60-dc8_merge_20120518_R7_thru20120622.ict


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
1 Include More Variables
Can you write Python code for it?

from scipy.spatial import KDTree
from PseudoNetCDF import PNC
# Read Model Data
margs = PNC('-c', "layer73,valid,0.5,0.5", \
            '--format=bpch,nogroup=("IJ-AVG-$","PEDGE-$"),'+
            'vertgrid="GEOS-5-NATIVE"', \
            "bpch/ctm.bpch.v10-01-public-Run0.2013050100")
mfile = margs.ifiles[0]
mlon = mfile.variables['longitude']
mlat = mfile.variables['latitude']
mpres = mfile.variables['PSURF']
mO3 = mfile.variables['O3']

# Read Observations
oargs = PNC("-f", "ffi1001", "--rename=v,O3_ESRL,O3", \
            "--variables=Fractional_Day,O3_ESRL,LATITUDE,LONGITUDE,PRESSURE", \
            "--expr=latitude=LATITUDE;"+\
            "longitude=np.where(LONGITUDE>180,LONGITUDE-360,LONGITUDE);"+
            "time=Fractional_Day*24*3600;time.units=\"seconds since 2011-12-31\"",
            "icartt/dc3-mrg60-dc8_merge_20120518_R7_thru20120622.ict")
ofile = oargs.ifiles[0]
olons = ofile.variables['longitude']
olats = ofile.variables['latitude']
opres = ofile.variables['PRESSURE']
oO3 = ofile.variables['O3']


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
ICARTT Evaluation
Author: Barron H. Henderson
This presentation will teach basics of spatial interpolation and statistical evaluation using Python with PseudoNetCDF, numpy, and scipy.
Can you write Python code for it?


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Lon/Lat First
Can you write Python code for it?

MLON, MLAT = np.meshgrid(mlon, mlat)
tree = KDTree(np.ma.array([MLAT.ravel(), MLON.ravel()]).T)

dists, idxs = tree.query(np.ma.array([olats, olons]).T)

latidxs, lonidxs = np.unravel_index(idxs, MLAT.shape)
meO3 = mO3[:, :, latidxs, lonidxs]
meO3.shape

dp = opres[None, None,:] - mpres[:, :, latidxs, lonidxs]
pidx = np.abs(dp).argmin(1)[0]
moO3 = meO3[:, pidx, np.arange(pidx.size)][0]
moO3.shape


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Non-Parametric Tests

Spearman Correlation (spearmanr)
Mann Whitney U (mannwhitneyu)
Wilcoxon signed-rank test (wilcoxon)
Can you write Python code for it?

from scipy.stats.mstats import mannwhitneyu

mwr = mannwhitneyu(ovals, mvals)
mwr


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Perform 2-d nearest neighbor interpolation
Can you write Python code for it?

from scipy.spatial import KDTree
tree = KDTree(np.ma.array([MLAT.ravel(), MLON.ravel()]).T)
dists, idxs = tree.query(np.ma.array([olats, olons]).T)
latidxs, lonidxs = np.unravel_index(idxs, MLAT.shape)
plt.close()
mpO3 = mO3[:, :, latidxs, lonidxs]
mpO3.shape

moO3 = mpO3[:, 22]
moO3.shape


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Plot Paired Time-Means
Can you write Python code for it?

from scipy.stats.mstats import linregress
lr = linregress(ovals[:], mvals[:])

plt.scatter(ovals, mvals)
plt.ylabel('Model'); plt.xlabel('Obs')
plt.axis('square')
x = np.array([0, 350])
plt.xlim(*x); plt.ylim(*x)
plt.plot(x, lr.slope * x + lr.intercept, ls = '--')
plt.title('r=%.2f; p=%.2f' % (lr.rvalue, lr.pvalue));


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Plot the results
Can you write Python code for it?

from scipy.stats.mstats import linregress
lr = linregress(oO3[:], moO3[:])

plt.scatter(oO3[:], moO3)
x = np.array([0, 500])
plt.plot(x, x, ls = '-', lw = 3, color = 'k')
plt.axis('square')
plt.xlim(*x)
plt.ylim(*x)
plt.ylabel('Model')
plt.xlabel('Obs')
plt.plot(x, lr.slope * x + lr.intercept, ls = '--')
plt.text(200, 500, 'r=%.2f; p=%.2f' % (lr.rvalue, lr.pvalue));


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Questions

Are KDTree and NearestNDInterpolator the same?
How different would an analysis be if we had:
used LinearNDInterpolator
would time normalization be necessary

Testing Populations

Typical with normal assumption
Non-parametric tests

Parametric Tests

Pearson Correlation (pearsonr)
Students t-test (ttest_ind)
Shapiro-Wilk (shapiro)
Can you write Python code for it?

from scipy.stats.mstats import ttest_ind

ttr = ttest_ind(ovals, mvals)
ttr


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Read Model Data
Can you write Python code for it?

margs = PNC('--format=bpch,nogroup=("IJ-AVG-$","PEDGE-$"),vertgrid="GEOS-5-NATIVE"', \
            "bpch/ctm.bpch.v10-01-public-Run0.2013050100")
mfile = margs.ifiles[0]
mlon = mfile.variables['longitude']
mlat = mfile.variables['latitude']
MLON, MLAT = np.meshgrid(mlon, mlat)
mO3 = mfile.variables['O3']


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Read Observations
Can you write Python code for it?

oargs = PNC("-f", "ffi1001",
            "--rename=v,O3_ESRL,O3",
            "--variables=Fractional_Day,O3_ESRL,LATITUDE,LONGITUDE,PRESSURE",
            "--rename=v,LATITUDE,latitude",
            "--expr=longitude=np.where(LONGITUDE>180,LONGITUDE-360,LONGITUDE);",
            "--expr=time=Fractional_Day*24*3600;time.units=\"seconds since 2011-12-31\"",
            "icartt/dc3-mrg60-dc8_merge_20120518_R7_thru20120622.ict")
ofile = oargs.ifiles[0]
olons = ofile.variables['longitude']
olats = ofile.variables['latitude']
oO3 = ofile.variables['O3']


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Remove Assymetric Variability

Benchmark has 1 mean-value all months
DC3 has 1 per minute
Can you write Python code for it?

idxs = [i for i in zip(pidx, latidxs, lonidxs)]
uniqueset = set(idxs)
idxs = np.array(idxs)
ovals = []
mvals = []
for idx in uniqueset:
    oi = (idxs == idx).all(1)
    ovals.append(oO3[oi].mean())
    mvals.append(mO3[0][idx])
print(len(ovals), len(mvals))


I am working on the file "PNC_03DC3Eval.ipynb".
The first task is:
Scipy has a spatial library
Can you write Python code for it?

import scipy.spatial
?scipy.spatial

?scipy.spatial.KDTree


I am working on the file "PNC_03Evaluation.ipynb".
The first task is:
CHECK  POINT
What should the bounding box be as a WKT Polygon?
ANSWERS Hidden
<div style="visibility: hidden">

"POLYGON ((llcrnrlon llcrnrlat, lrcrnrlon lrcrnrlat, urcrnrlon urcrnrlat, ulcrnrlon ulcrnrlat, llcrnrlon llcrnrlat))"

</div>

Download and Process
Can you write Python code for it?


I am working on the file "PNC_03Evaluation.ipynb".
The first task is:
Extract GEOS-Chem at AQS
Can you write Python code for it?

!pncgen -O -f "bpch,vertgrid='GEOS-5-NATIVE',nogroup=('IJ-AVG-$',)" \
        --extract-file AQS_DATA_20130501-20130701.nc --stack=time -v O3 -s layer72,0 \
        bpch/ctm.bpch.v10-01-public-Run0.2013050100 \
        bpch/ctm.bpch.v10-01-public-Run0.2013050100 \
        bpch_aqs_extract.nc

!pncdump.py --header bpch_aqs_extract.nc

!pnceval.py --help

%%bash
pnceval.py --funcs NO,NP,NOP,MO,MP,MB,RMSE,IOA,AC -v O3 \
--pnc " --expr O3=Ozone*1000;O3.units=\'ppb\' -r time,mean AQS_DATA_20130501-20130701.nc" \
--pnc " -r time,mean bpch_aqs_extract.nc"

from PseudoNetCDF import pnceval
help(pnceval)


I am working on the file "PNC_03Evaluation.ipynb".
The first task is:
Reproduced in Python

Read in files
Get variables
Calculate MB, RMSE
Modify
Repeat except for month specific results
Repeat except for site specific results

Reproduction Provided
Can you write Python code for it?

from PseudoNetCDF import PNC, pnceval

aqs = PNC("--reduce=time,mean", "--expr=O3=Ozone*1000", "AQS_DATA_20130501-20130701.nc")
geos = PNC("--reduce=time,mean", "bpch_aqs_extract.nc")
aqso3 = aqs.ifiles[0].variables['O3']
geoso3 = geos.ifiles[0].variables['O3']
print(aqso3.shape)
print(geoso3.shape)
print(pnceval.RMSE(aqso3, geoso3))


I am working on the file "PNC_03Evaluation.ipynb".
The first task is:
Review Output
Can you write Python code for it?


I am working on the file "PNC_03Evaluation.ipynb".
The first task is:
wktpolygon
This option is most relevant for regional extractions.
Can you write Python code for it?

from shapely.wkt import loads
geom = loads("POLYGON ((30 10, 40 35, 20 40, 10 20, 30 10))")
x, y = geom.exterior.xy
plt.plot(x, y, ls = '-', marker = 'o')


I am working on the file "PNTerms/AngularMomentum.ipynb".
The first task is:
(Look for spin-squared terms.)
The spin-orbit terms in the angular momentum are complete to 3.5pN.  These terms come from Eq. (4.7) of Bohé et al. (2012):
Can you write Python code for it?

# Lower-order terms are 0
AngularMomentum_Spin.AddDerivedVariable('L_SO_3',
    (-(35*S_ell + 15*Sigma_ell*delta)/(6*M**2))*ellHat
    + ((S_n + Sigma_n*delta)/(2*M**2))*nHat
    + (-(3*S_lambda + Sigma_lambda*delta)/M**2)*lambdaHat,
    datatype=ellHat.datatype)
# L_SO_4 is 0
AngularMomentum_Spin.AddDerivedVariable('L_SO_5',
    (7*(61*S_ell*nu - 99*S_ell + 30*Sigma_ell*delta*nu - 27*Sigma_ell*delta)/(72*M**2))*ellHat
    + ((-19*S_n*nu + 33*S_n - 10*Sigma_n*delta*nu + 33*Sigma_n*delta)/(24*M**2))*nHat
    + ((18*S_lambda*nu - 21*S_lambda + 8*Sigma_lambda*delta*nu - 3*Sigma_lambda*delta)/(6*M**2))*lambdaHat,
    datatype=ellHat.datatype)
# L_SO_6 is 0
AngularMomentum_Spin.AddDerivedVariable('L_SO_7',
    ((-29*S_ell*nu**2 + 1101*S_ell*nu - 405*S_ell - 15*Sigma_ell*delta*nu**2 + 468*Sigma_ell*delta*nu
      - 81*Sigma_ell*delta)/(16*M**2))*ellHat
    + ((11*S_n*nu**2 - 1331*S_n*nu + 183*S_n + 5*Sigma_n*delta*nu**2 - 734*Sigma_n*delta*nu
        + 183*Sigma_n*delta)/(48*M**2))*nHat
    + ((-32*S_lambda*nu**2 + 2*S_lambda*nu - 174*S_lambda - 16*Sigma_lambda*delta*nu**2 - 79*Sigma_lambda*delta*nu
        - 12*Sigma_lambda*delta)/(24*M**2))*lambdaHat,
    datatype=ellHat.datatype)


I am working on the file "PNTerms/AngularMomentum.ipynb".
The first task is:
Collected terms
Can you write Python code for it?

def AngularMomentumExpression(AngularMomentumTerms=[AngularMomentum_NoSpin, AngularMomentum_Spin], PNOrder=frac(7,2)):
    # We have to play some tricks with the log terms so that `horner` works
    def logterm(key,val):
        if 'lnv' in val:
            return logv
        else:
            return 1
    return L_coeff*horner(sum([key*(v**n)*logterm(key,val)
                               for Terms in AngularMomentumTerms
                               for n in range(2*PNOrder+1)
                               for key,val in Terms.items()
                               if val.endswith('_{0}'.format(n))])).subs(logv, ln(v))

# display(AngularMomentumExpression(PNOrder=frac(8,2)))


I am working on the file "PNTerms/AngularMomentum.ipynb".
The first task is:
Individual energy terms
In this notebook, every term will be multiplied by the following coefficient.
Can you write Python code for it?


I am working on the file "PNTerms/AngularMomentum.ipynb".
The first task is:
The following PNCollection objects will contain all the terms in the different parts of the binding energy.
Can you write Python code for it?

AngularMomentum_NoSpin = PNCollection()
AngularMomentum_Spin = PNCollection()


I am working on the file "PNTerms/BindingEnergy.ipynb".
The first task is:
(Is the following true?  What about this paper?)
The spin-squared terms (by which I mean both spin-spin and spin-orbit squared terms) in the energy are known only at 2pN order (from Kidder (1995) and Will and Wiseman (1996)).  They are most conveniently given in Eq. (C4) of Arun et al.
Can you write Python code for it?

# Lower-order terms are 0
BindingEnergy_Spin.AddDerivedVariable('E_SQ_4',
    (1+delta-2*nu)*(chi1chi1+chi2chi2)/4 - 3*(chi_a_ell**2+chi_s_ell**2)/2
    - delta*( chi2chi2/2 + 3*chi_a_ell*chi_s_ell ) + nu*( chi1chi2 + 6*chi_a_ell**2 ))


I am working on the file "PNTerms/BindingEnergy.ipynb".
The first task is:
Individual energy terms
In this notebook, every term will be multiplied by the following coefficient.
Can you write Python code for it?


I am working on the file "PNTerms/BindingEnergy.ipynb".
The first task is:
Note that fractions need to be entered as, e.g., frac(3,4) so that they are not converted to finite-precision decimals.
The nonspinning orbital binding energy is known through 4pN.  The expressions through 3.5pN here come from Eq. (194) of Blanchet (2006).
The 4pN term from Eq. (5.2d) of Jaranowski and Schäfer is known exactly, now that the $\nu$-linear piece is given as Eq. (32) of Bini and Damour (2013a).  The remaining terms are not known exactly, but Bini and Damour (2103b) have derived some terms, though there is incomplete information, which are noted as the constants in the following cell.  Note that, though the notation is confusing, Bini and Damour claim they did not calculate the coefficient they call $a_6^{\text{ln 1}}$; but it seems to be given in their Eq. (64).
Can you write Python code for it?


I am working on the file "PNTerms/BindingEnergy.ipynb".
The first task is:
The following PNCollection objects will contain all the terms in the different parts of the binding energy.
Can you write Python code for it?

BindingEnergy_NoSpin = PNCollection()
BindingEnergy_Spin = PNCollection()
BindingEnergy_NSTides = PNCollection()


I am working on the file "PNTerms/BindingEnergy.ipynb".
The first task is:
The tidal-coupling terms come in to the energy at relative 5pN order, and are known to 6pN order.
These terms come from Eq. (2.11) of Vines et al. (2011).  Note their unusual convention for mass ratios, where $\chi_1 = m_1/m$ in their notation; in particular, $\chi$ is not a spin parameter.  Also note that $\hat{\lambda} = \lambda_2 v^{10}/(m_1+m_2)^5$, and we need to add the coupling terms again with $1 \leftrightarrow 2$.  Finally, note the normalization difference, where the overall factor is different by $-2$.
Can you write Python code for it?

BindingEnergy_NSTides = PNCollection()
# Lower-order terms are 0
BindingEnergy_NSTides.AddDerivedConstant('E_NSTides_10', (-9*(M1/M2)*lambda2 - 9*(M2/M1)*lambda1)/M**5)
# E_NSTidal_11 is 0
BindingEnergy_NSTides.AddDerivedConstant('E_NSTides_12',
    (-frac(11,2)*(M1/M2)*(3+2*M2/M+3*(M2/M)**2)*lambda2 - frac(11,2)*(M2/M1)*(3+2*M1/M+3*(M1/M)**2)*lambda1)/M**5)


I am working on the file "PNTerms/Flux.ipynb".
The first task is:
Collected flux terms
Can you write Python code for it?

def FluxExpression(FluxTerms=[Flux_NoSpin, Flux_Spin], PNOrder=frac(7,2)):
    # We have to play some tricks with the log terms so that `horner` works
    def logterm(key,val):
        if 'lnv2' in val:
            return logv**2
        if 'lnv' in val:
            return logv
        else:
            return 1
    return Fcal_coeff*horner(sum([key*(v**n)*logterm(key,val)
                                  for Terms in FluxTerms
                                  for n in range(2*PNOrder+1)
                                  for key,val in Terms.items()
                                  if val.endswith('_{0}'.format(n))])).subs(logv, ln(v))

# display(FluxExpression())


I am working on the file "PNTerms/Flux.ipynb".
The first task is:
The tidal-coupling terms come in to the energy at relative 5pN order, and are known partially at 6pN order.
These terms come from Eq. (3.6) of Vines et al. (2011).  Note their unusual convention for mass ratios, where $\chi_1 = m_1/m$ in their notation; in particular, $\chi$ is not a spin parameter.  Also note that $\hat{\lambda} = \lambda_2 v^{10}/(m_1+m_2)^5$, and we need to add the coupling terms again with $1 \leftrightarrow 2$.  Finally, note the normalization difference, where a different overall factor is used, leading to a sign difference.
Can you write Python code for it?


I am working on the file "PNTerms/OrbitalEvolution.ipynb".
The first task is:
Next, we calculate the expansions needed for TaylorT4 and T5.  These will be the right-hand sides in our evolution equations for $dv/dt$.  TaylorT1 simply numerically evaluates a ratio of the terms imported above
Can you write Python code for it?

# Read in the high-order series expansion of a ratio of polynomials
p_Ratio = pickle.load(file('PolynomialRatios/PolynomialRatioSeries_Order{0}.dat'.format(2*PNOrbitalEvolutionOrder+1)))
p_Ratio = p_Ratio.removeO().subs('PolynomialVariable',v)

# Evaluate the flux, energy, and derivative of energy
FluxTerms = [Flux_NoSpin, Flux_Spin]
BindingEnergyTerms = [BindingEnergy_NoSpin, BindingEnergy_Spin]
for Term in FluxTerms:
    PNVariables.update(Term)
for Term in BindingEnergyTerms:
    PNVariables.update(Term)
Flux = FluxExpression(FluxTerms, PNOrbitalEvolutionOrder)
Energy = BindingEnergyExpression(BindingEnergyTerms, PNOrbitalEvolutionOrder)
dEdv = BindingEnergyDerivativeExpression(BindingEnergyTerms, PNOrbitalEvolutionOrder)

# Evaluate the energy absorption by the BHs, and make substitutions so that the Horner form is nice
AbsorptionTerms = [AlviTerms]
for Term in AbsorptionTerms:
    PNVariables.update(Term)
Absorption = AbsorptionExpression(AbsorptionTerms, PNOrbitalEvolutionOrder)

# Treat remaining log(v) terms as constants, for Taylor expansions and efficient numerical evaluation
Flux = Flux.subs(log(v), logv)
Energy = Energy.subs(log(v), logv)
dEdv = dEdv.subs(log(v), logv)
Absorption = Absorption.subs(log(v), logv)

# Get the series expansions for the numerators and denominators
FluxSeries = series(- (Flux + Absorption)/Fcal_coeff, x=v, x0=0, n=2*PNOrbitalEvolutionOrder+1).removeO()
dEdvSeries = series(dEdv/(-nu*v/2), x=v, x0=0, n=2*PNOrbitalEvolutionOrder+1).removeO()

# TaylorT4
T4Expressions = PNCollection()
NumTerms = {'Num{0}'.format(n): FluxSeries.coeff(v,n=n) for n in range(2*PNOrbitalEvolutionOrder+1)}
DenTerms = {'Den{0}'.format(n): dEdvSeries.coeff(v,n=n) for n in range(2*PNOrbitalEvolutionOrder+1)}
T4Expressions.AddDerivedConstant('dvdt_T4', (Fcal_coeff/(-nu*v/2))*\
    horner(sum([v**n*horner(N(p_Ratio.coeff(v,n=n).subs(dict(NumTerms.items() + DenTerms.items()))))
                for n in range(2*PNOrbitalEvolutionOrder+1)])))

# TaylorT5
T5Expressions = PNCollection()
NumTerms =...


I am working on the file "PNTerms/OrbitalEvolution.ipynb".
The first task is:
Now, the precession terms:
Can you write Python code for it?

PrecessionVelocities = PNCollection()

PrecessionVelocities.AddDerivedVariable('OmegaVec_chiVec_1',
                                        Precession_chiVec1Expression(PNOrbitalEvolutionOrder),
                                        datatype=ellHat.datatype)
PrecessionVelocities.AddDerivedVariable('OmegaVec_chiVec_2',
                                        Precession_chiVec2Expression(PNOrbitalEvolutionOrder),
                                        datatype=ellHat.datatype)
PrecessionVelocities.AddDerivedVariable(('OmegaVec' if UseQuaternions else 'OmegaVec_ellHat'),
                                        Precession_ellHatExpression(PNOrbitalEvolutionOrder)*nHat
                                        + ((v**3/M)*ellHat if UseQuaternions else 0),
                                        datatype=nHat.datatype)

CodeConstructor = CodeOutput.CodeConstructor(PNVariables, T1Expressions)
for Terms in BindingEnergyTerms+FluxTerms+[AlviTerms]+[Precession_ellHat, Precession_chiVec1, Precession_chiVec2]:
    CodeConstructor.AddDependencies(Terms)
CodeConstructor.AddDependencies(PrecessionVelocities)


I am working on the file "PNTerms/Precession.ipynb".
The first task is:
For the precession vector of the other spin, rather than re-entering the same things with 1 and 2 swapped, we just let python do it:
Can you write Python code for it?

for key,val in Precession_chiVec1.items():
    try:
        tmp = key.substitution.subs({delta: -delta, M1:'swap1', M2:M1, chi1_n:'swap2', chi2_n:chi1_n,
                                     chiVec1:'swap3', chiVec2:chiVec1}).subs({'swap1':M2,'swap2':chi2_n,'swap3':chiVec2})
        Precession_chiVec2.AddDerivedVariable(val.replace('OmegaVec1', 'OmegaVec2').replace('Omega1', 'Omega2'),
                                              tmp, datatype=key.datatype)
    except AttributeError:
        Precession_chiVec2.AddDerivedVariable(val.replace('OmegaVec1', 'OmegaVec2').replace('Omega1', 'Omega2'),
                                              key, datatype=key.datatype)


I am working on the file "PNTerms/Precession.ipynb".
The first task is:
In his Eqs. (2.4), Kidder (1995) summarized certain spin-spin terms:
Can you write Python code for it?

Precession_chiVec1.AddDerivedVariable('OmegaVec1_SS_1', M2**2*(-chiVec2+3*chi2_n*nHat)/M**2, datatype=nHat.datatype)
#print("WARNING: OmegaVec1_SS_1 in Precession.ipynb is disabled temporarily")


I am working on the file "PNTerms/Precession.ipynb".
The first task is:
NOTE: Is Etienne's notation consistent with others?   It seems like when he introduces other people's terms, he mixes $\hat{n}$ and $\hat{\ell}$.
Finally, in his Eq. (2.7) Racine (2008) added a quadrupole-monopole term along $\hat{n}$:
Can you write Python code for it?

Precession_chiVec1.AddDerivedVariable('OmegaVec1_QM_1', 3*nu*chi1_n*nHat, datatype=nHat.datatype)
#print("WARNING: OmegaVec1_QM_1 in Precession.ipynb is disabled temporarily")


I am working on the file "PNTerms/Precession.ipynb".
The first task is:
Precession of orbital angular velocity $\vec{\Omega}_{\hat{\ell}}$
Bohé et al. (2013) say that the precession of the orbital angular velocity is along $\hat{n}$, with magnitude (in their notation) $a_{\ell}/r\omega = \gamma\, a_{\ell} / v^3$.
NOTE: There is a 3pN gauge term in $\gamma_{\text{PN}}$ that I have simply dropped here.  It is $\ln(r/r_0')$.
The following two cells are Eqs. (4.3) and (4.4) of Bohé et al. (2013), respectively.
Can you write Python code for it?

Precession_ellHat.AddDerivedVariable('gamma_PN_coeff', v**2)

Precession_ellHat.AddDerivedConstant('gamma_PN_0', 1)
# gamma_PN_1 is 0
Precession_ellHat.AddDerivedConstant('gamma_PN_2', -nu/3 + 1)
Precession_ellHat.AddDerivedVariable('gamma_PN_3', (5*S_ell/3 + Sigma_ell*delta)/M**2)
Precession_ellHat.AddDerivedConstant('gamma_PN_4', -65*nu/12 + 1)
Precession_ellHat.AddDerivedVariable('gamma_PN_5',
    ((frac(8,9)*nu + frac(10,3))*S_ell + 2*Sigma_ell*delta)/M**2)
Precession_ellHat.AddDerivedConstant('gamma_PN_6',
    nu**3/81 + 229*nu**2/36 - 41*pi**2*nu/192 - 2203*nu/2520 + 1)
Precession_ellHat.AddDerivedVariable('gamma_PN_7',
    ((-6*nu**2 - 127*nu/12 + 5)*S_ell - 8*Sigma_ell*delta*nu**2/3 + (-61*nu/6 + 3)*Sigma_ell*delta)/M**2)

Precession_ellHat.AddDerivedVariable('a_ell_coeff', v**7/M**3)

Precession_ellHat.AddDerivedVariable('a_ell_0', 7*S_n + 3*Sigma_n*delta)
# gamma_PN_1 is 0
Precession_ellHat.AddDerivedVariable('a_ell_2', (-29*nu/3-10)*S_n + (-9*nu/2-6)*delta*Sigma_n)
# gamma_PN_3 is 0
Precession_ellHat.AddDerivedVariable('a_ell_4',
    (frac(52,9)*nu**2 + frac(59,4)*nu + frac(3,2))*S_n
     + (frac(17,6)*nu**2 + frac(73,8)*nu + frac(3,2))*delta*Sigma_n)

def Precession_ellHatExpression(PNOrder=frac(7,2)):
    OmegaVec_ellHat = (gamma_PN_coeff.substitution*a_ell_coeff.substitution/v**3)\
        *horner(sum([key*(v**n) for n in range(2*PNOrder+1)
                     for key,val in Precession_ellHat.items() if val==('gamma_PN_{0}'.format(n))]))\
        *horner(sum([key*(v**n) for n in range(2*PNOrder+1)
                     for key,val in Precession_ellHat.items() if val==('a_ell_{0}'.format(n))]))
    return OmegaVec_ellHat

# Precession_ellHatExpression()


I am working on the file "PNTerms/Precession.ipynb".
The first task is:
The following PNCollection objects will contain all the terms describing precession.
Can you write Python code for it?

Precession_ellHat  = PNCollection()
Precession_chiVec1 = PNCollection()
Precession_chiVec2 = PNCollection()


I am working on the file "PNTerms/Variables.ipynb".
The first task is:
Derived variables
Any variable that can be derived from the variables above should be put in this section.
These variables should probably be left in arbitrary form, unless a particular simplification is desired.  The substitutions dictionary should map from the general names and their definitions in terms of basic variables.  In numerical codes, their values can be calculated once per time step and then stored, so that the values do not have to be re-calculated every time they appear in an expression.
Various common combinations of the two masses:
Can you write Python code for it?

PNVariables.AddDerivedConstant('M', M1+M2)
PNVariables.AddDerivedConstant('delta', (M1-M2)/M)
PNVariables.AddDerivedConstant('nu', M1*M2/M**2)
PNVariables.AddDerivedConstant('nu__2', (M1*M2/M**2)**2)
PNVariables.AddDerivedConstant('nu__3', (M1*M2/M**2)**3)
PNVariables.AddDerivedConstant('q', M1/M2)


I am working on the file "PNTerms/Variables.ipynb".
The first task is:
Other functions of the angular velocity that find frequent use:
Can you write Python code for it?

PNVariables.AddDerivedVariable('x', v**2)
PNVariables.AddDerivedVariable('Omega_orb', (v**3)/M)
PNVariables.AddDerivedVariable('logv', log(v))


I am working on the file "PNTerms/Variables.ipynb".
The first task is:
The system's vector basis is given by $(\hat{\ell}, \hat{n}, \hat{\lambda})$, and will be computed by the code in terms of the fundamental logarithmic rotors defined above.  Here, we give all the substitutions that will be needed in the code.
Can you write Python code for it?

PNVariables.AddDerivedVariable('ellHat', array([ellHat_x, ellHat_y, ellHat_z]), datatype='std::vector<double>')

PNVariables.AddDerivedVariable('nHat', array([nHat_x, nHat_y, nHat_z]), datatype='std::vector<double>')

PNVariables.AddDerivedVariable('lambdaHat', cross(ellHat.substitution, nHat.substitution), datatype='std::vector<double>')

# Components of lambdaHat are defined in terms of components of ellHat and nHat
for i,d in zip([0,1,2],['x','y','z']):
    PNVariables.AddDerivedVariable('lambdaHat_'+d, lambdaHat.substitution[i])


I am working on the file "PNTerms/Variables.ipynb".
The first task is:
Various spin components and combinations:
Can you write Python code for it?

PNVariables.AddDerivedVariable('chiVec1', array([chi1_x, chi1_y, chi1_z]), datatype='std::vector<double>')
PNVariables.AddDerivedVariable('chiVec2', array([chi2_x, chi2_y, chi2_z]), datatype='std::vector<double>')
PNVariables.AddDerivedVariable('chi1Mag', sqrt(chi1_x**2 + chi1_y**2 + chi1_z**2))
PNVariables.AddDerivedVariable('chi2Mag', sqrt(chi2_x**2 + chi2_y**2 + chi2_z**2))

PNVariables.AddDerivedConstant('chi1chi1', dot(chiVec1.substitution, chiVec1.substitution))
PNVariables.AddDerivedVariable('chi1chi2', dot(chiVec1.substitution, chiVec2.substitution))
PNVariables.AddDerivedConstant('chi2chi2', dot(chiVec2.substitution, chiVec2.substitution))

PNVariables.AddDerivedVariable('chi1_ell',  dot(chiVec1.substitution, ellHat.substitution))
PNVariables.AddDerivedVariable('chi1_n',  dot(chiVec1.substitution, nHat.substitution))
PNVariables.AddDerivedVariable('chi1_lambda', dot(chiVec1.substitution, lambdaHat.substitution))
PNVariables.AddDerivedVariable('chi2_ell',  dot(chiVec2.substitution, ellHat.substitution))
PNVariables.AddDerivedVariable('chi2_n',  dot(chiVec2.substitution, nHat.substitution))
PNVariables.AddDerivedVariable('chi2_lambda', dot(chiVec2.substitution, lambdaHat.substitution))

PNVariables.AddDerivedConstant('sqrt1Mchi1chi1', sqrt(1-chi1chi1))
PNVariables.AddDerivedConstant('sqrt1Mchi2chi2', sqrt(1-chi2chi2))

PNVariables.AddDerivedVariable('S', chiVec1.substitution*M1**2 + chiVec2.substitution*M2**2, datatype=chiVec1.datatype)
PNVariables.AddDerivedVariable('S_ell', chi1_ell*M1**2 + chi2_ell*M2**2)
PNVariables.AddDerivedVariable('S_n', chi1_n*M1**2 + chi2_n*M2**2)
PNVariables.AddDerivedVariable('S_lambda', chi1_lambda*M1**2 + chi2_lambda*M2**2)

PNVariables.AddDerivedVariable('Sigma', M*(chiVec2.substitution*M2 - chiVec1.substitution*M1), datatype=chiVec1.datatype)
PNVariables.AddDerivedVariable('Sigma_ell', M*(chi2_ell*M2 - chi1_ell*M1))
PNVariables.AddDerivedVariable('Sigma_n', M*(chi2_n*M2 - chi1_n*M1))
PNVariables.AddDerivedVariable('Sigma_lambda', M*(chi2_lambda*M2 -...


I am working on the file "PNTerms/Variables_Q.ipynb".
The first task is:
Derived variables
Any variable that can be derived from the variables above should be put in this section.
These variables should probably be left in arbitrary form, unless a particular simplification is desired.  The substitutions dictionary should map from the general names and their definitions in terms of basic variables.  In numerical codes, their values can be calculated once per time step and then stored, so that the values do not have to be re-calculated every time they appear in an expression.
Various common combinations of the two masses:
Can you write Python code for it?


I am working on the file "POS_Example.ipynb".
The first task is:
Finally, we'll just take the product of the five most common words of the VB, VBZ and NNS tags
Can you write Python code for it?

from itertools import product

['{} {}'.format(verb[0], noun[0]) for (verb, noun) in product(verb_tags.most_common()[:5], NNS_tags.most_common()[:5])]


I am working on the file "POS_Example.ipynb".
The first task is:
Let's look at the part of speech tags for reference
Can you write Python code for it?


I am working on the file "POS_Example.ipynb".
The first task is:
Now, let's display some graphs just to visualize this frequency distribution
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

most_common_pos = tag_fd.most_common()
plt.figure(figsize=(15, 10))
plt.bar([x for x in range(len(most_common_pos))],
        [count for (pos, count) in most_common_pos],
        tick_label=['' for (pos, count) in most_common_pos]
    )
plt.show()

most_common_pos = tag_fd.most_common()
plt.figure(figsize=(15, 10))
plt.bar([x for x in range(len(most_common_pos))],
        [count for (pos, count) in most_common_pos],
        tick_label=[pos for (pos, count) in most_common_pos]
    )
plt.xticks(rotation=45)
plt.show()

most_common_pos = tag_fd.most_common()
plt.figure(figsize=(10, 10))
plt.pie([count for (pos, count) in most_common_pos], shadow=True)
plt.show()

most_common_pos = tag_fd.most_common()
plt.figure(figsize=(10, 10))
plt.pie([count for (pos, count) in most_common_pos],
        labels=[pos for (pos, count) in most_common_pos],
        autopct='%.2f%%',
        shadow=True
    )
plt.show()


I am working on the file "PRNN/Assignment/Assignment3/word2Vec.ipynb".
The first task is:
For Parts Of Speech
Can you write Python code for it?

def POSForSentence(sentence):
    text = word_tokenize(sentence)
    posSentence = nltk.pos_tag(text)
    posSentence = [y for x, y in posSentence]
    return posSentence

def getUniquePOS():
    tagdict = load('help/tagsets/upenn_tagset.pickle')
    return len(tagdict), tagdict.keys()