I am working on the file "DimensionReduction.ipynb".
The first task is:
Independent Component Analysis (ICA)
For data where the components are statistically independent (or nearly so) Independent Component Analysis (ICA) has become a popular method for separating mixed components.  The classical example is the so-called "cocktail party" problem.  This is illustrated in the following figure from Hastie, Tibshirani, and Friedman (Figure 14.27 on page 497 in my copy, so they have clearly added some stuff!).  Think of the "source signals" as two voices at a party.  You are trying to concentrate on just one voice.  What you hear is something like the "measured signals" pattern.  You could run the data through PCA and that would do an excellent job of reconstructing the signal with reduced dimensionality, but it wouldn't actually isolate the different physical components (bottom-left panel)  ICA on the other hand can (bottom-right panel).

ICA is a good choice for a complex system with relatively indepent components.  For example a galaxy is roughly a linear combination of cool stars and hot stars, and a quasar is just a galaxy with others component from an accretion disk and emission line regions.  Ideally we want "eigenvectors" that are aligned with those physical traits/regions as opposed to mathematical constructs.
The basic call to the FastICA algoirthm in Scikit-Learn looks like:
Can you write Python code for it?

# Execute this cell
import numpy as np
from sklearn.decomposition import FastICA
X = np.random.normal(size=(100,2)) # 100 objects in 2D
R = np.random.random((2,5)) # mixing matrix
X = np.dot(X,R) # 2D data in 5D space
ica = FastICA(2) # fit 2 components
ica.fit(X)
proj = ica.transform(X) # 100x2 projection of the data
comp = ica.components_ # 2x5 matrix of independent components
## sources = ica.sources_ # 100x2 matrix of sources


I am working on the file "DimensionReduction.ipynb".
The first task is:
Let's revisit the digits sample and see what PCA, NMF, and ICA do for it.
Can you write Python code for it?

# Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print grid_data
X = digits.data
y = digits.target


I am working on the file "DimensionReduction.ipynb".
The first task is:
N.B.  Dimensionality isn't just measuring $D$ parameters for $N$ objects.  It could be a spectrum with $D$ values or an image with $D$ pixels, etc.  In the book the examples used just happen to be spectra of galaxies from the SDSS project.  But we can insert the data of our choice instead.
For example: the SDSS comprises a sample of 357 million sources:
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$
yields a probability of having one of the 357 million sources reside within a unit hypersphere of 1 in 1.4$\times 10^5$.
Principal Component Analysis (PCA)
In Principal Component Analysis (PCA) we seek to take a data set like the one shown below and apply a transform to the data such that the new axes are aligned with the maximal variance of the data.  As can be seen in the Figure, this is basically just the same as doing regression by minimizing the square of the perpendicular distances to the new axes.  Note that we haven't made any changes to the data, we have just defined new axes.
Can you write Python code for it?

# Execute this cell
# Ivezic, Figure 7.2
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse

#------------------------------------------------------------
# Set parameters and draw the random sample
np.random.seed(42)
r = 0.9

sigma1 = 0.25
sigma2 = 0.08
rotation = np.pi / 6
s = np.sin(rotation)
c = np.cos(rotation)

X = np.random.normal(0, [sigma1, sigma2], size=(100, 2)).T
R = np.array([[c, -s],[s, c]])
X = np.dot(R, X)

#------------------------------------------------------------
# Plot the diagram
fig = plt.figure(figsize=(5, 5), facecolor='w')
ax = plt.axes((0, 0, 1, 1), xticks=[], yticks=[], frameon=False)

# draw axes
ax.annotate(r'$x$', (-r, 0), (r, 0),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))
ax.annotate(r'$y$', (0, -r), (0, r),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))

# draw rotated axes
ax.annotate(r'$x^\prime$', (-r * c, -r * s), (r * c, r * s),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))
ax.annotate(r'$y^\prime$', (r * s, -r * c), (-r * s, r * c),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))

# scatter points
ax.scatter(X[0], X[1], s=25, lw=0, c='k', zorder=2)

# draw lines
vnorm = np.array([s, -c])
for v in (X.T):
    d = np.dot(v, vnorm)
    v1 = v - d * vnorm
    ax.plot([v[0], v1[0]], [v[1], v1[1]], '-k')

# draw ellipses
for sigma in (1, 2, 3):
    ax.add_patch(Ellipse((0, 0), 2 * sigma * sigma1, 2 * sigma * sigma2,
                         rotation *...


I am working on the file "DimensionReduction.ipynb".
The first task is:
Note that the points are correlated along a particular direction which doesn't align with the initial choice of axes.  So, we should rotate our axes to align with this correlation.
We'll choose the rotation to maximize the ability to discriminate between the data points:
*   the first axis, or principal component, is direction of maximal variance
*   the second principal component is orthogonal to the first component and maximizes the residual variance
*   ...
PCA is a dimensional reduction process because we can generally account for nearly "all" of the variance in the data set with fewer than the original $K$ dimensions.  See more below.
We start with a data set ${x_i}$ which consists of $N$ objects for which we measure $K$ features.  We start by subtracting the mean for each feature in ${x_i}$ and write $X$ as a $N\times K$ matrix.
The covariance of this matrix is
$$C_X=\frac{1}{N-1}X^TX.$$
There are off-diagonal terms if there are correlations between the measurements (e.g., maybe two of the features are temperature dependent and the measurements were taken at the same time).
If $R$ is a projection of the data that is aligned with the maximal variance, then we have $Y= X R$ with covariance
$$ C_{Y} = R^T X^T X R = R^T C_X R.$$
$r_1$ is the first principal component of $R$, which can be derived using Langrange multipliers with the following cost function:
$$ \phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1). $$
If we take derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ and set it to 0, then we have
$$ C_Xr_1 - \lambda_1 r_1 = 0. $$
$\lambda_1$ (the largest eigenvalue of the matrix) is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ for which the eigenvalue is
$$ \lambda_1 =  r_1^T C_X r_1.$$
The columns of the full matrix, $R$ are the eigenvectors (known here as principal components).
We aren't going to go through the linear algebra more than that here.  But it would be a good group project for someone.  See the end of 7.3.1 starting at the bottom on page 294 or go through...
Can you write Python code for it?

#Example call from 7.3.2
import numpy as np
from sklearn.decomposition import PCA

X = np.random.normal(size=(100,3)) # 100 points in 3D
R = np.random.random((3,10)) # projection matrix
X = np.dot(X,R) # X is now 10-dim, with 3 intrinsic dims
pca = PCA(n_components=4) # n_components can be optionally set
pca.fit(X)
comp = pca.transform(X) # compute the subspace projection of X
mean = pca.mean_ # length 10 mean of the data
components = pca.components_ # 4x10 matrix of components


I am working on the file "DimensionReduction.ipynb".
The first task is:
Note that this works in the opposite direction too: let's say you want to find "rare" objects in 10 dimensions, where we'll define rare as <1% of the population.  Then you'll need to accept objects from 63% of the distribution in all 10 dimensions!  So are those really "rare" or are they just a particular 1% of the population?
Can you write Python code for it?

import numpy as np
p = 10**(np.log10(0.01)/10.0)
print p


I am working on the file "DimensionReduction.ipynb".
The first task is:
Note that we would need 1000 components to encode all of the variance.
Interpreting the PCA

The output eigenvectors are ordered by their associated eigenvalues
The eigenvalues reflect the variance within each eigenvector
The sum of the eigenvalues is total variance of the system
Projection of each spectrum onto the first few eigenspectra is a compression of the data

Once we have the eigenvectors, we can try to reconstruct an observed spectrum, ${x}(k)$, in the eigenvector basis, ${e}_i(k)$, as
$$ \begin{equation}
  {x}i(k) = {\mu}(k) + \sum_j^R \theta{ij} {e}_j(k).
\end{equation}
$$
That would give a full (perfect) reconstruction of the data since it uses all of the eigenvectors.  But if we truncate (i.e., $r<R$), then we will have reduced the dimensionality while still reconstructing the data with relatively little loss of information.
For example, we started with 4000x1000 floating point numbers.  If we can explain nearly all of the variance with 8 eigenvectors, then we have reduced the problem to 4000x8+8x1000 floating point numbers!
Execute the next cell to see how the reconstruction improves by adding more components.
Can you write Python code for it?

# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text,...


I am working on the file "DimensionReduction.ipynb".
The first task is:
Now let's make "scree" plots.  These plots tell us how much of the variance is explained as a function of the each eigenvector.  Our plot won't look much like Ivezic, Figure 7.5, so I've shown it below to explain where "scree" comes from.
Can you write Python code for it?

# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(121)
ax.plot(np.arange(n_components-1), evals)
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("eigenvalue ")

ax = fig.add_subplot(122)
ax.plot(np.arange(n_components-1), evals.cumsum())
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("cumulative eigenvalue")

plt.show()


I am working on the file "DimensionReduction.ipynb".
The first task is:
Now let's plot the components.  See also Ivezic, Figure 7.4.  The left hand panels are just the first 5 spectra for comparison with the first 5 PCA components, which are shown on the right.  They are ordered by the size of their eigenvalues.
Can you write Python code for it?

#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel('wavelength (Angstroms)')
    ax.plot(wavelengths, pca_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel('wavelength (Angstroms)')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()


I am working on the file "DimensionReduction.ipynb".
The first task is:
Similarly for NMF and ICA
Can you write Python code for it?

# NMF
# Complete

# ICA
from sklearn.decomposition import FastICA
# Complete


I am working on the file "DimensionReduction.ipynb".
The first task is:
The example that we have been using above is "spectral" PCA.  Some examples from the literature include:
- Francis et al. 1992
- Connolly et al. 1995
- Yip et al. 2004
One can also do PCA on features that aren't ordered (as they were for the spectra).  E.g., if you have $D$ different parameters measured for your objects.  The classic example in astronomy is
Boroson & Green 1992
Caveats II
One of the things that I don't like about PCA is that the eigenvectors are just mathematical constructs.  They often don't look anything like the spectra themselves.  Whereas it is often the case that you might expect that the components would look like, well, the physical components.  For example, quasars are fundamentally galaxies.  So, part of their flux comes from the galaxy that they live in.  But PCA doesn't return any component that looks like a typical galaxy.  Essentially this is because the components can be both positive and negative.
Non-negative Matrix Factorization (NMF)
This is where Non-negative Matrix Factorizaiton (NMF) comes in.  Here we are treating the data as a linear sum of positive-definite components.
NMF assumes  any data matrix can be factored into two matrices, $W$ and $Y$, with
$$\begin{equation}
X=W Y,
\end{equation}
$$
where both $W$ and $Y$ are nonnegative.
So, $WY$ is an approximation of $X$. Minimizing the reconstruction error $|| (X - W Y)^2 ||$,
nonnegative bases can be derived through an iterative process.
Note, however, that the iterative process does not guarantee nonlocal minima (like $K$-means and EM), but using
random initialization and cross-validation can be used to find the global minimum.
An example from the literature is Allen et al. 2008
In Scikit-Learn the NMF implementation looks like:
Can you write Python code for it?

# Execute this cell
import numpy as np
from sklearn.decomposition import NMF
X = np.random.random((100,3)) # 100 points in 3D
nmf = NMF(n_components=3)
nmf.fit(X)
proj = nmf.transform(X) # project to 3 dimension
comp = nmf.components_ # 3x10 array of components
err = nmf.reconstruction_err_ # how well 3 components capture the data


I am working on the file "DimensionReduction.ipynb".
The first task is:
This is why PCA enables dimensionality reduction.
How many components would we need to explain 99.5% of the variance?
Can you write Python code for it?

for num_feats in np.arange(1,20, dtype = int):
    # complete
print("{:d} features are needed to explain 99.5% of the variance".format(# Complete


I am working on the file "Dimensions of Python.ipynb".
The first task is:
Builtins
When you boot into a Python shell (REPL), you're not entering a blank canvas or blank slate. Lots of names are already defined, and ready for calling or using.
Can you write Python code for it?

exceptions, callables, not_callables = [], [], []
for name in dir(__builtins__):     # cycle through a bunch of strings
    obj = eval(name)               # turn strings back into objects
    if type(obj) == type and issubclass(obj, BaseException):
        exceptions.append(name)    # types of exception
    elif callable(obj):
        callables.append(name)     # callable
    else:
        not_callables.append(name) # whatever is left over

print("Exceptions\n", exceptions)
print("------")
print("Callables\n", callables)
print("------")
print("Other\n", not_callables)


I am working on the file "DimenstionalityReduction/DimensionalityReduction.ipynb".
The first task is:
Probabilistic Principle Component Analysis - PPCA
Conventional PCA can be seen as a projection while PPCA is a mapping. This mapping is from the latent space into data sapce: $ X =WZ+\epsilon $ where $Z$ is considered M dimentional latent vriable and $\epsilon$ is zero-mean isotropic noise. Implementations here closely flow PRML book by bishop pages [587-596].
Can you write Python code for it?

# Probabilistic Principal Component Analysis, Tipping and Bishop 1999
def compute_E_completeDataLL(X,W,s2):
    d, N = X.shape
    m = W.shape[1]

    M = np.dot(W.T,W)+s2*np.eye(m)
    Minv = inv(M)

    ll = 0.0
    for n in xrange(N):
        Xn = X[:, n].reshape(-1,1)

        Ezn = np.dot(np.dot(Minv,W.T),Xn)
        Ezzn = (s2)*Minv+np.dot(Ezn,Ezn.T)
        ll += 0.5 * np.trace(Ezzn)
        ll += 0.5 * d * np.log(2*np.pi*s2)
        ll += 0.5 * np.dot(Xn.T,Xn)/2.
        ll -= np.dot(np.dot(Ezn.T,W.T),Xn)/float(s2)
        ll += 0.5 * np.trace(np.dot(Ezzn,np.dot(W.T,W)))/float(s2)

    ll *= -1.0

    return ll/N
def compute_Wnew(X,W,s2):
    d, N = X.shape
    m = W.shape[1]

    M = np.dot(W.T,W)+s2*np.eye(m)
    Minv = inv(M)

    term1, term2 = 0.,0.
    for n in xrange(N):
        Xn = X[:, n].reshape(-1,1)

        Ezn = np.dot(np.dot(Minv,W.T),Xn)
        Ezzn = (s2)*Minv+np.dot(Ezn,Ezn.T)

        term1 += np.dot(Xn,Ezn.T)
        term2 += Ezzn

    Wnew = np.dot(term1,term2)
    return Wnew

def compute_s2new(X,W,s2):
    d, N = X.shape
    m = W.shape[1]

    M = np.dot(W.T,W)+s2*np.eye(m)
    Minv = inv(M)

    s2new = 0.
    for n in xrange(N):
        Xn = X[:, n].reshape(-1,1)

        Ezn = np.dot(np.dot(Minv,W.T),Xn)
        Ezzn = (s2)*Minv+np.dot(Ezn,Ezn.T)

        s2new += np.dot(Xn.T,Xn)
        s2new -= 2*np.dot(Ezn.T,np.dot(W.T,Xn))
        s2new += np.trace(np.dot(Ezzn,np.dot(W.T,W)))

    s2new = s2new/float(N*d)
    return s2new
def ppca(X,mode = 'ML',m=2,em_maxit=200,tol=1e-4):
    '''X: observed variable D*N
       Z: latent variable M*N
       m: # output dimension
       W: D*m
       M: m*m
       '''
    d, N = X.shape
    X = X - np.mean(X, axis=1).reshape(d,-1)
    I = np.eye(m);

    if mode == 'ML':

        S = np.cov(X)
        U,L,_ = svd(S)

        Um = U[:,0:m]
        Lm = L[0:m]*I# to diagonalize it

        sigma2 = np.sum(L[m+1:])/(d-m)

        Lm_temp =...


I am working on the file "DimenstionalityReduction/DimensionalityReduction.ipynb".
The first task is:
Visualization of the results
Can you write Python code for it?

#visualization of low dimensional data
def visualize_lowD_01(D_ld,N0,N1):
    D0_ld = D_ld[:,0:N0]
    D1_ld = D_ld[:,N0+1:]
    f, ax = plt.subplots()

    ax.plot(D0_ld[0,:],D0_ld[1,:],'b.')
    ax.plot(D1_ld[0,:],D1_ld[1,:],'r.')
    ax.set_title('0:blue, 1:red')

    plt.show()

m = 2 # project data into 2D

N = X.shape[1];N0 = X0.shape[1];N1 = X1.shape[1]
d = X.shape[0];
Xmu = np.mean(X, axis=1).reshape(d,-1)

U, _ = PCA(X)
Xpca = np.dot(U[:,0:m].T,X)

visualize_lowD_01(Xpca,N0,N1)


I am working on the file "DinoData.ipynb".
The first task is:
Remember how the .loc attribute uses enhanced slice notation ("enhanced" in the sense core Python does not support it).
Can you write Python code for it?

dinos.loc["Mo":"N"]

dinos.dtypes

dinos.index.is_unique

code = dinos.loc['Mtapaiasaurus'][0]

code

len(code)

dinos.info()

int(code, base=16)

0xafe4c2b017ed3996bf5f4f3b937f0ae22e649df2f620787e136ed6bd3ea32e2d


I am working on the file "DinoData.ipynb".
The first task is:
You'll notice the hashing algorithm has already been applied by the time we import this JSON data.  I'll be showing you the Python source code for scripts that aren't Jupyter Notebook scripts, for that part of the pipeline.
Can you write Python code for it?


I am working on the file "DinoDataLab.ipynb".
The first task is:
How about in descending order?
Can you write Python code for it?


I am working on the file "Diode Laser Spectroscopy/20171003_morning/Interference with SAS no Dopple/.ipynb_checkpoints/Interferometer with SAS No Doppler Analysis-checkpoint.ipynb".
The first task is:
Isolating the Maximum Values
Can you write Python code for it?

get_peak_data(ch2, [0.025, 0.030]);

get_peak_data(ch2, [0.030, 0.035]);

get_peak_data(ch2, [0.0350,0.045]);

get_peak_data(ch2, [0.049, 0.0517]);

maximum_time_positions = [0.028124, 0.03266, 0.042744, 0.05052]
maximum_voltage_positions = [0.738, 0.53, 0.716, 0.48]

# Two subplots, unpack the axes array immediately
f, (ax1, ax2) = plt.subplots(1, 2, sharey=True)
f.set_figheight(10)
f.set_figwidth(15)
ax1.set_title("Maximum Values on SAS, No Doppler")
ax1.set_ylabel("Voltage (V)")
ax1.set_xlabel("Time (s)")
ax1.plot(maximum_time_positions, maximum_voltage_positions, 'rp')
ax1.plot(time, ch2)
ax2.set_title("Maximum Values on Interferometer Spikes")
ax2.set_xlabel("Time (s)")
ax2.plot(maximum_time_positions, maximum_voltage_positions, 'rp')
ax2.plot(time, ch3, 'y-')


I am working on the file "Diode Laser Spectroscopy/20171003_morning/Interference with SAS no Dopple/Interferometer with SAS No Doppler Analysis.ipynb".
The first task is:
Isolating the Maximum Values
Can you write Python code for it?


I am working on the file "Diode Laser Spectroscopy/20171003_morning/Interference with SAS no Dopple/Interferometer with SAS No Doppler Analysis.ipynb".
The first task is:
The distance between the interferometer peaks should be constant, so finding the time separating two maximum values should tell me how much time elapses between the two peaks, which we can use as a measure of separation of frequencies.
Can you write Python code for it?


I am working on the file "Diode Laser Spectroscopy/20171003_morning/Interference with SAS no Dopple/Interferometer with SAS No Doppler Analysis.ipynb".
The first task is:
The time difference between peaks is 0.00158 seconds.
Can you write Python code for it?

differences = [0]
for i in range(1,len(maximum_time_positions)):
    differences.append(maximum_time_positions[i]-maximum_time_positions[i-1])
differences

from prettytable import PrettyTable

x = PrettyTable()

x = PrettyTable()
x.add_column("Time (s)", maximum_time_positions)
x.add_column("Voltage (V)", maximum_voltage_positions)
x.add_column("Difference (s)", [round(i, 5) for i in differences])
x.add_column("Number of Interferometer Distances Apart", [round(i, 5)/0.00158 for i in differences])
x.add_column("Separation of Features (MHz)", [round(i, 5)/0.00158 *379 for i in differences])
print(x)

file = open('SummaryTable.txt', 'w')
file.write(str(x))
file.close()


I am working on the file "Diploma_munka_notebook_0.ipynb".
The first task is:
Buffed up convolutional network

max pooling
dropouts

Some helper functions
Can you write Python code for it?

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)

def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')

depth = 32
graph = tf.Graph()

with graph.as_default():

    # Placeholders
    keep_prob = tf.placeholder(tf.float32)

    # Input data.
    tf_train_batch = tf.placeholder(tf.float32, shape=(None, image_size, image_size, num_channels))
    # The None at the shape argument means that the dimension is not defined,
    tf_train_labels = tf.placeholder(tf.float32, shape=(None, num_labels))

    # Constants
    tf_valid_dataset = tf.constant(valid_dataset_conv)
    tf_test_dataset = tf.constant(test_dataset_conv)

    # Variables.
    h_conv1_weights = weight_variable([patch_size, patch_size, num_channels, depth])
    h_conv1_biases = bias_variable([depth])

    h_conv2_weights = weight_variable([patch_size, patch_size, depth, depth * 2])
    h_conv2_biases = bias_variable([depth * 2])

    conv_image_size = image_size // 4
    fc1_weights = weight_variable([conv_image_size * conv_image_size * depth * 2, num_hidden])
    fc1_biases = bias_variable([num_hidden])

    output_softmax_weights = weight_variable([num_hidden, num_labels])
    output_softmax_biases = bias_variable([num_labels])

    #Define the model:
    # First layer, patches of 5x5 into 32 features
    h_conv1 = tf.nn.relu(conv2d(tf_train_batch, h_conv1_weights) + h_conv1_biases)
    h_pool1 = max_pool_2x2(h_conv1)

    # Second layer, patches of 5x5 into 64 features
    h_conv2 = tf.nn.relu(conv2d(h_pool1, h_conv2_weights) + h_conv2_biases)
    h_pool2 = max_pool_2x2(h_conv2)

    # Reshape into the densely connected layer
    h_pool2_flat = tf.reshape(h_pool2, [-1, conv_image_size * conv_image_size * depth...


I am working on the file "Diploma_munka_notebook_0.ipynb".
The first task is:
Load in the mnist dataset
Can you write Python code for it?

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
# The mnist images have a dimension of 28*28.
image_size = 28
# There are 10 labels.
num_labels = 10
train_dataset = mnist.train.images
train_labels = mnist.train.labels

perm = np.random.permutation(mnist.test.images.shape[0])

split_point = int(mnist.test.images.shape[0] * 0.1)
valid_dataset, test_dataset = mnist.test.images[:split_point], mnist.test.images[split_point:]
valid_labels, test_labels = mnist.test.labels[:split_point], mnist.test.labels[split_point:]


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
Table 1.  Current df2 with new columns for consolidated X, Y coordinates.

First Plot
Now, to have dipole vectors continue from one to another (as opposed to all starting at the zero), we have to transpose coordinates in a way that the origin of new vector starts at the end of the previous vector. In Python, use the function cumsum().
Let's try to build continuous vector graph from the original dataframe df2. This is equivalent to a vector decomposition.
The following cell is just a test case (can be removed later). It generates plot of Table 1 as continuing vector segments. You can skip it and go to section "Cleaning up the Vector Path".
Can you write Python code for it?

# Test area running off the branch df2b1
# Translate coordinates from X,Y,Z=0,0,0 to assure vector continuity
df2b1['newXa'] = df2b1.newX.cumsum()
df2b1['newYa'] = df2b1.newY.cumsum()
df3b = df2b1
# print df3b

# Calculate total dipole (compare with column Tot_Dip)
lastX = df3b.tail(1).newXa  # row and value
lastX = lastX.tolist()[0]  # value only
lastY = df3b.tail(1).newYa
lastY = lastY.tolist()[0]
total_dipole = sqrt(np.power(lastX, 2)+np.power(lastY, 2))
total_dipole = round(total_dipole, 2)

# Calc geometrical center of the vector polygon
cenX = df3b['newXa'].mean()
cenY = df3b['newYa'].mean()
x = df3b.newXa
y = df3b.newYa

# Plot
# Set rectangular plot dimensions to keep lengths proportional
xlow = x.min()
xhigh = x.max()
ylow = y.min()
yhigh = y.max()

def lst_sort(list):
    '''
    Sort list of floats by values.

    :type list: list of floats
    :param list: max and min x,y-coordinates

    :rtype: list of floats
    :return: sorted list of floats
    '''
    abslist = []
    for item in list:
        abslist.append(item)
    return sorted(abslist)

margins = lst_sort([xlow, ylow, xhigh, yhigh])

plt.figure()  # To generate multiple distinct plots.
plt.suptitle('Dipole Moments (D)')
ax = []
xmin = margins[0]-1 #x.min()-1
xmax = margins[3]+1
ymin = margins[0]-1
ymax = margins[3]+1
plt.ylabel('Y')
plt.xlabel('X')
plt.grid(True)
plt.xlim(xmin,xmax)
plt.ylim(ymin,ymax)
color='blue'
plt.scatter(df3b.newXa, df3b.newYa, s=80, c=color, label='NLMOs')
for j, txt in enumerate(df3b['NLMO']):
    plt.annotate(txt, (x[j]+0.1,y[j]+0.3))
ax = gca()
ax.add_patch(FancyArrowPatch((0,0),(lastX,lastY),arrowstyle='->',mutation_scale=20, color='red'))
for k in range(1,len(x)):
    ax.add_patch(FancyArrowPatch((x[k], y[k]),(x[k-1],y[k-1]),arrowstyle='<-',mutation_scale=20, color='blue'))

plt.annotate(total_dipole,
             xy=(lastX/2*0.9, lastY/2*1.1),
             color='red',
             xycoords='data',
             textcoords='offset points')

plt.annotate("+",
             xy=(cenX, cenY),
    ...


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
Cleaning up the Vector Path
Resulting plot is rather cluttered and thus we will need to arrange dipoles in some way to get a more interpretable view.
Since most molecules have center of the coordinate system placed somewhere at the center of the molecule, sorting dipole coordinates by xy-quadrant in which the dipole origin resides is a sensible approach. The following cartoon indicates how such quadrants are defined.

We will write a function that assigns each row an arbitrary quadrant depending on the signs of coordinates x and y. To further arrange directions of lines in each quadrant, we will include another function that calculates slope of the line and we will sort by the quadrant (rank) and the slope.
Can you write Python code for it?

# Function to rank XY-points into quadrants 0-3
def xyrank(a, b, c, d, e, f):
    '''
    Assign qudrant value 1,2,3,4.
    :type inputs: floats and str
    :param inputs: X, Y, Q(1), Q(2), Q(3), Q(4)
    :rtype: str
    :return: quadrant number as str
    '''
    if a > 0 and b > 0:  # [Q1]
        return c
    elif a > 0 > b:  # [Q2]
        return d
    elif a < 0 and b < 0:  # [Q3]
        return e
    else:  # [Q4] 1 a>0, b<0
        return f

# Function to calculate slope
def slope(a, b):
    if a != 0:
        slope = (1.0 * b / a)  # expects float
        return slope
    else:
        slope = 0  # for a = zero, set slope arbitrarily to 0
        return slope


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
First, a row of arbitrary values (zeroes) is inserted at the top of dataframe df (line 15; press Esc+l (lower L)) and the table values are assigned int, float, or string types, respectively.
Can you write Python code for it?

# Input file can be generated in ReadNboDip.ipynb notebook
infile = 'form_dip.csv'  # in directory ./dipoles
# Save the file path, name and extension
fullpath =  os.path.abspath(infile)
path,file=os.path.split(fullpath)
basename, extension = os.path.splitext(infile)

# Create Pandas dataframe from the *.csv file
df = pd.read_csv(infile)
# fix datatype for columns
df.convert_objects(convert_numeric=True).dtypes

# Prepare first blank row with XYZ=0 to set the point of vector origin
# set the first row to zeroes
rowex = df.loc[[0]]
# Get dataframe column headers
headers = df.columns.values.tolist()
# Fix the data types
for item in headers:
    if df.dtypes[item] == 'float64' :
        rowex[item] = 0.
    elif df.dtypes[item] == 'int64' :
        rowex[item] = 0
    else :
        rowex[item] = ''
row0 = rowex

# Reassemble data table placing row0 at the top, followed by the rest
df2 = pd.concat([row0, df]).reset_index().ix[::]
# print df2

""" Identify which column (coordinate) has constant values (orthogonal to the
molecular plane).
"""
# Prepare stds of absolute values for each x,y,z column
# Smallest stds indicate constant dimension perpendicular to 2D plane
C1 = df2["X"].abs()
C1std = C1.std()
C2 = df2["Y"].abs()
C2std = C2.std()
C3 = df2["Z"].abs()
C3std = C3.std()
# print "stds are: %.3f %.3f %.3f" % (C1std, C2std, C3std)

'''Assign X,Y coordinates only, Z=0. Any of the X,Y,Z can be constant or
zero 0 (which one may change every time).
Remap coordinates arbitrarily to X,Y with Z=[const] '''
def coord_headers(C1std, C2std, C3std):
    if C1std > 0.1 and C2std > 0.1 > C3std:
        df2['newX'] = df2["X"]
        df2['newY'] = df2["Y"]
    elif C1std > 0.1 > C2std and C3std > 0.1:
        df2['newX'] = df2["X"]
        df2['newY'] = df2["Z"]
    elif C1std < 0.1 < C2std and C3std > 0.1:
        df2['newX'] = df2["Y"]
        df2['newY'] = df2["Z"]
    else:
        df2['newX'] = df2["X"]
        df2['newY'] = df2["Y"]

# Append columns newX,newY to df2 (values are often same as the original X,Y)
#...


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
Plot Survey
For each set of qudrants, the code below will sort the dataframe by the quadrant and slope. The corresponding plot will be created and saved as .png image. Quadrant sequence is part of the filename.
Can you write Python code for it?

%%capture
# remove %%capture magic to see the plot and output

# Iterate through the list of all quadrants
global captarr
captarr = []
captperm = []
for i in range(len(permlist)):
    # Apply function to the table values in each row; make a copy first
    df21 = df2.copy()
    df21["xyRank"] = df2.apply(
        lambda row: xyrank(row['newX'], row['newY'], permlist[i][0],
                           permlist[i][1], permlist[i][2], permlist[i][3]),
        axis=1)

    # Apply function to table values in a row
    df21["slope"] = df21.apply(lambda row: slope(row['newX'], row['newY']), axis=1)

    toprow = df21[:1]  # row with zero dipole
    # Directly change rank of the first row in the dataframe
    toprow.xyRank[0] = 4
    # Isolate remaining data into rest (drop)
    rest1 = df21.drop(df21.index[[0, 0]])
    # sort rest1 by xyRank, then by slope - descending
    df3 = rest1.sort_index(by=['xyRank', 'slope'], ascending=[False, False])
    # Put it back
    df3 = pd.concat([toprow, df3]).reset_index().ix[::]
    df3 = df3.sort_index(by=['xyRank', 'slope'], ascending=[False, False])

    # Translate coordinates from xyz=0,0,0
    df3['newXa'] = df3.newX.cumsum()
    df3['newYa'] = df3.newY.cumsum()

    # Calculate total dipole
    lastX = df3.tail(1).newXa  # row and value
    lastX = lastX.tolist()[0]  # value only
    lastY = df3.tail(1).newYa
    lastY = lastY.tolist()[0]
    total_dipole = sqrt(np.power(lastX, 2) + np.power(lastY, 2))
    total_dipole = round(total_dipole, 2)

    # Calc distances of x,y and assess even distribution of points
    # around the geom center
    cenX = df3['newXa'].mean()
    cenY = df3['newYa'].mean()
    distX = abs(cenX - df3.newXa)
    distY = abs(cenY - df3.newYa)
    # distance from centroid
    distXY = np.sqrt(distX * distX + distY * distY)
    diststd = distXY.std()
    captarr.append(round(diststd, 1))
    captperm.append(permlist[i])
    print "++++++++++ Rank list is : ", permlist[i], "; \
                std of distances is:", round(diststd, 2)

    #...


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
Unfortunately, it is not obvious which sequence of quadrants will lead to the least cluttered dipole diagram. To make sure that we test all possibilities, we will evaluate all permutations of four quadrants, that is, 4! = 4 x 3 x 2 x 1  = 24
For each trial, we record an array of quadrants and standard deviation of distances of all vector origins from the graph origin (geometric center). Small stds should indicate even distribution of vectors around the origin (circle, ellipsoid, symmetrical polygon?). Standard deviation of distances together with the sequence of quadrants appear at the top of each plot. As we will see later, lowest stds are not necesarilly indicative of the "clean" graph shape.
 By no means the resulting vector plots indicate atom connectivity! It is the relative direction of dipole vectors that helps us to assess the importance and internal balance of NLMO orbitals within the molecule.
Can you write Python code for it?

# Get the list of quadrant permutations
# Four quadrants of a XY plot
lst = [0, 1, 2, 3]
set(list(lst))  # Creates a set
import itertools
permlist = set(itertools.permutations(lst))
permlist = list(permlist)  # Get permutation set into a list.


I am working on the file "Dipoles_quad_all_Jupyter.ipynb".
The first task is:
csv to Dataframe
In the next step, coordinates of NLMO dipoles are read from *.csv file. This file was generated earlier using the KNIME "Dipole_v2/3" workflow. Alternatively, iPython notebook ReadNboDip.ipynb at this Github repository creates the identical input file. "Plain" Python script (ReadNboDip.py) that parses NBO output files for the same Dipole moment summary can be downloaded from this Github repository.
In this example, this file is available from the ./dipole directory.

Can you write Python code for it?

%%capture
# Step into the directory of input .csv files using the magic %cd
# Make sure that we are not already in the 'dipoles' dir from the previous run
if os.path.isdir('dipoles'):
    %cd dipoles
else:
    %cd ..
    %cd dipoles


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
$$k_1$$ but density element seems null
Can you write Python code for it?


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
$$k_2$$ but density element seems null
Can you write Python code for it?


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
ALGEBRA & DEFINITIONS
Clifford algebra is $$Cl_{1,4}(\mathbb{R})$$
Flat space, no metric, just signature
All constants are equal to 1
Can you write Python code for it?

from sympy import *
variables = (t, x, y, z, w) = symbols('t x y z w', real=True)
print(variables)

metric=[1
        ,-1
        ,-1
        ,-1
        ,-1]

myBasis='gamma_t gamma_x gamma_y gamma_z gamma_w'

sp5d = Ga(myBasis, g=metric, coords=variables,norm=True)
(gamma_t, gamma_x, gamma_y, gamma_z, gamma_w) = sp5d.mv()
(grad, rgrad) = sp5d.grads()


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
Associative Hyperbolic Quaternions
Can you write Python code for it?


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
DIRAC
http://en.wikipedia.org/wiki/Dirac_equation
Regular
$$0=({\gamma}_0 \frac{\partial}{\partial t}+{\gamma}_1 \frac{\partial}{\partial x}+{\gamma}_2 \frac{\partial}{\partial y}+{\gamma}_3 \frac{\partial}{\partial z}+im) {\psi}$$
Adjoint
$$0={\overline{\psi}}(\frac{\partial}{\partial t}{\gamma}_0+\frac{\partial}{\partial x}{\gamma}_1+\frac{\partial}{\partial y}{\gamma}_2+\frac{\partial}{\partial z}{\gamma}_3+im)$$
Gradient definition
Can you write Python code for it?

displayWithTitle(grad,title='\overrightarrow{D} = \Sigma e_i \partial x_i')
displayWithTitle(rgrad,title='\overleftarrow{D} = \Sigma \partial x_i e_i')


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
Density element combined derivatives
See here http://www.bbk.ac.uk/tpru/BasilHiley/Bohm-Vienna.pdf
Can you write Python code for it?

K={}

K[1]=(ihquat*imag*E-imag*m+khquat*p)*(1+jhquat)
K[1].texLabel='('+ihquat.texLabel+imag.texLabel+'E-'+imag.texLabel+'m+'+khquat.texLabel+p.texLabel+')(1+'+jhquat.texLabel+')'

texLabel='('+ihquat.texLabel+imag.texLabel+'E+'+imag.texLabel+'m+'+khquat.texLabel+p.texLabel+')'+'(1-'+jhquat.texLabel+')'+imag.texLabel
K[2]=(ihquat*imag*E+imag*m+khquat*p)*(1-jhquat)*imag
K[2].texLabel=texLabel
texLabel='-'+'('+ihquat.texLabel+imag.texLabel+'E+'+imag.texLabel+'m+'+khquat.texLabel+p.texLabel+')'+imag.texLabel

def showDerivatives(f, i, k):
    displayWithTitle(k, 'k_' + str(i) + '=' + k.texLabel)
    displayWithTitle(k*k, 'k_' + str(i) + '^2')
    rho=(f*k)*(k*f)
    displayWithTitle(rho, title='\\rho')
    left=grad*rho
    right=rho*rgrad
    display(Latex('Combine regular and adjoint Dirac equations with difference'))
    displayWithTitle((left-right)/2, '1/2 (\overrightarrow{D}\\rho-\\rho\overleftarrow{D})')
    display(Latex('Combine regular and adjoint Dirac equations with addition'))
    displayWithTitle((left+right)/2, '1/2 (\overrightarrow{D}\\rho+\\rho\overleftarrow{D})')


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
Imaginary unit
Can you write Python code for it?

imag=gamma_w
imag.texLabel='i'

displayWithTitle(imag, title=imag.texLabel)
displayWithTitle((imag*imag), title=imag.texLabel+'^2')


I am working on the file "Dirac with Geometric Algebra 1,4 over the Reals.ipynb".
The first task is:
PHYSICS
The following symbols are defined :
Energy $$E \in \mathbb{R}$$
Mass $$m \in \mathbb{R}$$
Can you write Python code for it?

m, E, p_x, p_y, p_z = symbols('m E p_x p_y p_z', real=True)

rquat = [iquat, jquat, kquat]
pv =[p_x, p_y, p_z]
p = S(0)
for (dim, var) in zip(pv, rquat):
    p += var * dim
p.texLabel='\\mathbf{p}'

display(Latex('Momentum $'+p.texLabel+'$ is defined with $p_x, p_y, p_z \\in \\mathbb{R}$'))
display(Math(p.texLabel+'=p_x'+iquat.texLabel+'+p_y'+jquat.texLabel+'+p_z'+kquat.texLabel))
displayWithTitle(p, title=p.texLabel)

f=m*w-imag*(E*t+p_x*x+p_y*y+p_z*z)

displayWithTitle(f, title='f')
displayWithTitle(grad*f, title='\overrightarrow{D}f')
displayWithTitle(f*rgrad, title='f\overleftarrow{D}')
displayWithTitle(grad*f*grad*f, title='\overrightarrow{D}f\overrightarrow{D}f')
displayWithTitle(f*rgrad*f*rgrad, title='f\overleftarrow{D}f\overleftarrow{D}')
displayWithTitle((grad*f - f*rgrad)/2, title='1/2 (\overrightarrow{D}f-f\overleftarrow{D})')
displayWithTitle((grad*f + f*rgrad)/2, title='1/2 (\overrightarrow{D}f+f\overleftarrow{D})')


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Better get some new dice...!
Polling #1
Let's turn our review of the Dirichlet-Multinomial distribution to another example, concerning polling data.
In section 3.4 of BDA3 on multivariate models and, specifically the section on Multinomial Models for Categorical Data, the authors include a, little dated, example of polling data in the 1988 Presidential race between George H.W. Bush and Michael Dukakis.
Here's the setup:

1,447 likely voters were surveyed about their preferences in the upcoming presidential election
Their responses were:
Bush: 727
Dukakis: 583
Other: 137

What is the probability that more people will vote for Bush over Dukakis?
i.e. what is the difference in support for the two major candidates?

We set up the data, where $k$ represents the number of choices the respondents had:
Can you write Python code for it?


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
For models with multi-dimensional shapes, it's always good to check the shapes of the various parameters before sampling:
Can you write Python code for it?

for v in polling_model_debates.unobserved_RVs:
    print(v, v.tag.test_value.shape)


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
For this model, we'll need to set up the priors slightly differently. Instead of 1 set of thetas, we need 2, one for each survey (pre/post debate).
To do that without creating specific pre/post versions of each variable, we'll take advantage of PyMC3's shape parameter, available for most (all?) distributions.
In this case, we'll need a 2-dimensional shape parameter, representing the number of debates n_debates and the number of choices in candidates n_candidates
Can you write Python code for it?

n_debates, n_candidates = y.shape
n_debates, n_candidates


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Just looking at a simple bar plot, we suspect that we might not be dealing with a fair die!
However, students of Bayesian statistics that we are, we'd like to go further and quantify our uncertainty in the fairness of the die and calculate the probability that someone slipped us loaded dice.
Can you write Python code for it?


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Lastly, we can calculate the probability that the die is biased on $6$ by calculating the density to the right of our reference line at $0$:
Can you write Python code for it?

six_bias_perc = len(dice_trace["six_bias"][dice_trace["six_bias"]>0])/len(dice_trace["six_bias"])

print(f'P(Six is biased) = {six_bias_perc:.2%}')


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Let's compare the results visually, by plotting the posterior distributions of the pre/post debate values for % responses for Bush and the posterior for pre/post difference in Bush supporters:
Can you write Python code for it?

_, ax = plt.subplots(2,1, figsize=(10, 10))

sns.distplot(polling_trace_debates["bush_pref"][:,0], hist=False, ax=ax[0], label="Pre-Debate")
sns.distplot(polling_trace_debates["bush_pref"][:,1], hist=False, ax=ax[0], label="Post-Debate")
ax[0].set_title("% Responses for Bush vs Dukakis")
ax[0].set_xlabel("% Responses");

sns.distplot(polling_trace_debates["bush_shift"], hist=True, ax=ax[1], label="P(Bush Shift)")
ax[1].axvline(0, c='g', linestyle='dotted')
ax[1].set_title("% Shift Pre/Prior Debate")
ax[1].set_xlabel("% Shift");


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Let's sample with a slightly higher number of draws and tuning steps:
Can you write Python code for it?

with polling_model_debates:
    polling_trace_debates = pm.sample(draws=3000, tune=1500)

with polling_model_debates:
    pm.traceplot(polling_trace_debates, combined=True)


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Let's set up a simple model in PyMC3 that not only calculates the posterior probability for $theta$ (i.e. the probability for each side of the die), but also estimates the bias for throwing a $6$.
We will use Deterministic variable, in addition to our unobserved (theta) and observed (results) variables.
For the prior on $theta$, we'll use a non-informative uniform distribution, by initializing the $Dirichlet$ prior with a series of 1s for the parameter a, one for each of the k possible outcomes. This is similar to initializing a $Beta$ distribution as $Beta(1, 1)$, which corresponds to the Uniform distribution.
Can you write Python code for it?

n, y

with pm.Model() as dice_model:

    # initializes the Dirichlet distribution with a uniform prior:
    a = np.ones(k)

    theta = pm.Dirichlet("theta", a=a)

    # Since theta[5] will hold the posterior probability of rolling a 6
    # we'll compare this to the reference value p = 1/6
#     six_bias = pm.Deterministic("six_bias", theta[k-1] - p)

    results = pm.Multinomial("results", n=n, p=theta, observed=y)

dice_model


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Polling #2
As an extension to the previous model, the authors of BDA include an exercise in chapter 3.10 (Exercise 2) that presents us with polling data from the 1988 Presidential race, taking before and after the one of the debates.

Comparison of two multinomial observations: on September 25, 1988, the evening of a
presidential campaign debate, ABC News conducted a survey of registered voters in the
United States; 639 persons were polled before the debate, and 639 different persons were
polled after. The results are displayed in Table 3.2. Assume the surveys are independent
simple random samples from the population of registered voters. Model the data with
two different multinomial distributions. For $j = 1, 2$, let $\alpha_j$ be the proportion of voters
who preferred Bush, out of those who had a preference for either Bush or Dukakis at
the time of survey $j$. Plot a histogram of the posterior density for $\alpha_2 − \alpha_1$. What is the
posterior probability that there was a shift toward Bush?

Let's copy the data from the exercise and model the problem as a probabilistic model, again using PyMC3:
Can you write Python code for it?

data = pd.DataFrame([
        {"candidate": "bush", "pre": 294, "post": 288},
        {"candidate": "dukakis", "pre": 307, "post": 332},
        {"candidate": "other", "pre": 38, "post": 10}
       ], columns=["candidate", "pre", "post"])

data


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
Thus, we need to initialize a Dirichlet distribution prior with shape (2,3) and then refer to the relevant parameters by index where needed.
Can you write Python code for it?

with pm.Model() as polling_model_debates:

    # initializes the Dirichlet distribution with a uniform prior:
    shape = (n_debates, n_candidates)
    a = np.ones(shape)

    # This creates a separate Dirichlet distribution for each debate
    # where sum of probabilities across candidates = 100% for each debate
    theta = pm.Dirichlet("theta", a=a, shape=shape)

    # get the "Bush" theta for each debate, at index=0
    bush_pref = pm.Deterministic("bush_pref", theta[:, 0] * n / m)

    # to calculate probability that support for Bush shifted from debate 1 [0] to 2 [1]
    bush_shift = pm.Deterministic("bush_shift", bush_pref[1]-bush_pref[0])

    # because of the shapes of the inputs, this essentially creates 2 multinomials,
    # one for each debate
    responses = pm.Multinomial("responses", n=n, p=theta, observed=y)


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
We'll plot the posterior distributions for each $theta$ and compare it our reference value $p$ to see if the 95% HPD (Highest Posterior Density) interval includes $p = 1/6$.
Can you write Python code for it?

axes = pm.plot_posterior(dice_trace, varnames=["theta"], ref_val=np.round(p, 3))
for i, ax in enumerate(axes):
    ax.set_title(f"{i+1}")


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
We'll take a look at the means of the posteriors for theta, indicating the % of support for each candidate pre & post debate:
Can you write Python code for it?

s = ["pre", "post"]
candidates = data["candidate"].values
pd.DataFrame(polling_trace_debates["theta"].mean(axis=0), index=s, columns=candidates)


I am working on the file "Dirichlet Multinomial Example.ipynb".
The first task is:
We, again, set up a simple Dirichlet-Multinomial model and include a Deterministic variable that calculates the metric of interest - the difference in probability of respondents for Bush vs. Dukakis.
Can you write Python code for it?

with pm.Model() as polling_model:

    # initializes the Dirichlet distribution with a uniform prior:
    a = np.ones(k)

    theta = pm.Dirichlet("theta", a=a)

    bush_dukakis_diff = pm.Deterministic("bush_dukakis_diff", theta[0] - theta[1])

    likelihood = pm.Multinomial("likelihood", n=n, p=theta, observed=y)

pm.model_to_graphviz(polling_model)

with polling_model:
    polling_trace = pm.sample(draws=1000)

with polling_model:
    pm.traceplot(polling_trace, combined=True)


I am working on the file "DirichletProcess/Sunspot_example.ipynb".
The first task is:
For the sunspot model, the posterior distribution of αα is concentrated between 0.6 and 1.2, indicating that we should expect more components to contribute non-negligible amounts to the mixture than for the Old Faithful waiting time model.
Can you write Python code for it?


I am working on the file "DirichletProcess/Sunspot_example.ipynb".
The first task is:
Generating data
Can you write Python code for it?

sunspot_df = get_rdataset('sunspot.year', cache=True).data
sunspot_df.head()

sunspot_df.plot(x='time')


I am working on the file "DirichletProcess/Sunspot_example.ipynb".
The first task is:
The Dirichlet process mixture model is incredibly flexible in terms of the family of parametric component distributions {fθ | fθ∈Θ}{fθ | fθ∈Θ}. We illustrate this flexibility below by using Poisson component distributions to estimate the density of sunspots per year.
Can you write Python code for it?

# pymc3.distributions.DensityDist?

import matplotlib.pyplot as plt
import matplotlib as mpl
from pymc3 import Model, Normal, Slice
from pymc3 import sample
from pymc3 import traceplot
from pymc3.distributions import Interpolated
from theano import as_op
import theano.tensor as tt
import numpy as np
from scipy import stats
from matplotlib import pyplot as plt
import numpy as np
import pymc3 as pm
import scipy as sp
import seaborn as sns
from statsmodels.datasets import get_rdataset
from theano import tensor as tt

%matplotlib inline

%load_ext version_information

%version_information pymc3, statsmodels, pandas


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-country.ipynb".
The first task is:
Generate a bunch of histograms of the data to make sure that all of the data
is in an expected range.
with plt.style.context('https://gist.githubusercontent.com/rhiever/d0a7332fe0beebfdc3d5/raw/223d70799b48131d5ce2723cd5784f39d7a3a653/tableau10.mplstyle'):
    for column in df_infl_ctry.columns[:-2]:
        #if column in ['date']:
        #    continue
        plt.figure()
        plt.hist(df_infl_ctry[column].values)
        plt.title(column)
        #plt.savefig('{}.png'.format(column))
Can you write Python code for it?

len(df_infl_ctry)

df_infl_ctry.columns

df_infl_ctry['month_order'] = range(len(df_infl_ctry))
month_order = df_infl_ctry['month_order']
max_infl = df_infl_ctry['max'].values
min_infl = df_infl_ctry['min'].values
mean_infl = df_infl_ctry['mean'].values
mode_infl = df_infl_ctry['mode'].values
p25th = df_infl_ctry['25th'].values
p75th = df_infl_ctry['75th'].values
p10th = df_infl_ctry['10th'].values
p90th = df_infl_ctry['90th'].values
inflEA = df_infl_ctry['76451'].values

year_begin_df = df_infl_ctry[df_infl_ctry.index.month == 1]
year_begin_df;

year_beginning_indeces = list(year_begin_df['month_order'].values)
year_beginning_indeces

year_beginning_names = list(year_begin_df.index.year)
year_beginning_names

month_order

#import seaborn as sns

fig, ax1 = plt.subplots(figsize=(15, 7))

# Create the bars showing highs and lows
#plt.bar(month_order, max_infl - min_infl, bottom=min_infl,
#            edgecolor='none', color='#C3BBA4', width=1)

plt.bar(month_order, p90th - p10th, bottom=p10th,
            edgecolor='none', color='#C3BBA4', width=1)
# Create the bars showing average highs and lows
plt.bar(month_order, p75th - p25th, bottom=p25th,
            edgecolor='none', color='#9A9180', width=1);

#annotations={month_order[50]:'Dividends'}
plt.plot(month_order, inflEA, color='#5A3B49',linewidth=2 );

plt.plot(month_order, mode_infl, color='wheat',linewidth=2,alpha=.3);

plt.xticks(year_beginning_indeces,
               year_beginning_names,
               fontsize=10)

#ax2 = ax1.twiny()

plt.xticks(year_beginning_indeces,
               year_beginning_names,
               fontsize=10);

plt.xlim(-5,200)
plt.grid(False)

##ax2 = ax1.twiny()
plt.ylim(-5, 14)

#ax3 = ax1.twinx()

plt.yticks(range(-4, 15, 2), [r'{}'.format(x)
                              for x in range(-4, 15, 2)], fontsize=10);

plt.grid(axis='both', color='wheat', linewidth=1.5, alpha = .5)

plt.title('HICP innflation, annual rate of change, Jan 2000 - March 2016\n\n', fontsize=20);


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-country.ipynb".
The first task is:
df_infl_ctry.rename(columns = dic)
tt = df_infl_ctry.copy()
tt['month'] = tt.index.month
tt['year'] = tt.index.year
melted_df = pd.melt(tt,id_vars=['month','year'])
melted_df.head()
Can you write Python code for it?

df_infl_ctry['min'] = df_infl_ctry.apply(min,axis=1)
df_infl_ctry['max'] = df_infl_ctry.apply(max,axis=1)
df_infl_ctry['mean'] = df_infl_ctry.apply(np.mean,axis=1)
df_infl_ctry['mode'] = df_infl_ctry.quantile(q=0.5, axis=1)
df_infl_ctry['10th'] = df_infl_ctry.quantile(q=0.10, axis=1)
df_infl_ctry['90th'] = df_infl_ctry.quantile(q=0.90, axis=1)
df_infl_ctry['25th'] = df_infl_ctry.quantile(q=0.25, axis=1)
df_infl_ctry['75th'] = df_infl_ctry.quantile(q=0.75, axis=1)

df_infl_ctry.head()


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-country.ipynb".
The first task is:
df_infl_ctry['month'] = df_infl_ctry.index.month
df_infl_ctry['year'] = df_infl_ctry.index.year
Can you write Python code for it?


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item-ver-2.ipynb".
The first task is:
Compute annual inflation rates
Can you write Python code for it?

df_infl_items = df_ind_items.pct_change(periods=12)*100
mask_rows_infl = df_infl_items.index.year >= 2000
df_infl_items = df_infl_items[mask_rows_infl]
df_infl_items.tail()

tt = df_infl_items.copy()
tt['month'] = tt.index.month
tt['year'] = tt.index.year
tt.head()

tt.to_csv('infl_items.csv')


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item-ver-2.ipynb".
The first task is:
df_infl_items.rename(columns = dic)
tt = df_infl_items.copy()
tt['month'] = tt.index.month
tt['year'] = tt.index.year
melted_df = pd.melt(tt,id_vars=['month','year'])
melted_df.head()
Can you write Python code for it?

df_infl_items['min'] = df_infl_items.apply(min,axis=1)
df_infl_items['max'] = df_infl_items.apply(max,axis=1)
df_infl_items['mean'] = df_infl_items.apply(np.mean,axis=1)
df_infl_items['mode'] = df_infl_items.quantile(q=0.5, axis=1)
df_infl_items['10th'] = df_infl_items.quantile(q=0.10, axis=1)
df_infl_items['90th'] = df_infl_items.quantile(q=0.90, axis=1)
df_infl_items['25th'] = df_infl_items.quantile(q=0.25, axis=1)
df_infl_items['75th'] = df_infl_items.quantile(q=0.75, axis=1)

df_infl_items.tail()


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item-ver-2.ipynb".
The first task is:
df_infl_items['month'] = df_infl_items.index.month
df_infl_items['year'] = df_infl_items.index.year
Can you write Python code for it?


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item.ipynb".
The first task is:
Compute annual inflation rates
Can you write Python code for it?


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item.ipynb".
The first task is:
Generate a bunch of histograms of the data to make sure that all of the data
is in an expected range.
with plt.style.context('https://gist.githubusercontent.com/rhiever/d0a7332fe0beebfdc3d5/raw/223d70799b48131d5ce2723cd5784f39d7a3a653/tableau10.mplstyle'):
    for column in df_infl_items.columns[:-2]:
        #if column in ['date']:
        #    continue
        plt.figure()
        plt.hist(df_infl_items[column].values)
        plt.title(column)
        #plt.savefig('{}.png'.format(column))
Can you write Python code for it?

len(df_infl_items)

df_infl_items.columns

df_infl_items['month_order'] = range(len(df_infl_items))
month_order = df_infl_items['month_order']
max_infl = df_infl_items['max'].values
min_infl = df_infl_items['min'].values
mean_infl = df_infl_items['mean'].values
mode_infl = df_infl_items['mode'].values
p25th = df_infl_items['25th'].values
p75th = df_infl_items['75th'].values
p10th = df_infl_items['10th'].values
p90th = df_infl_items['90th'].values
inflEA = df_infl_items['76451'].values

year_begin_df = df_infl_items[df_infl_items.index.month == 1]
year_begin_df;

year_beginning_indeces = list(year_begin_df['month_order'].values)
year_beginning_indeces

year_beginning_names = list(year_begin_df.index.year)
year_beginning_names

month_order;

blue3 = map(to_colors, (24, 116, 205))  # 1874CD
wheat2 = map(to_colors, (238, 216, 174))  # EED8AE
wheat3 = map(to_colors, (205, 186, 150))  # CDBA96
wheat4 = map(to_colors, (139, 126, 102))  # 8B7E66
firebrick3 = map(to_colors, (205, 38, 38))  # CD2626
gray30 = map(to_colors, (77, 77, 77))  # 4D4D4D

fig, ax1 = plt.subplots(figsize=(15,7))

plt.bar(month_order, p90th - p10th, bottom=p10th,
            edgecolor='none', color='#C3BBA4', width=1);

# Create the bars showing average highs and lows
plt.bar(month_order, p75th - p25th, bottom=p25th,
            edgecolor='none', color='#9A9180', width=1);

#annotations={month_order[50]:'Dividends'}
plt.plot(month_order, inflEA, color='#5A3B49',linewidth=2 );

plt.plot(month_order, mode_infl, color='wheat',linewidth=2,alpha=.3);

plt.xticks(year_beginning_indeces,
               year_beginning_names,
               fontsize=10)

#ax2 = ax1.twiny()

plt.xlim(-5,200)
plt.grid(False)

##ax2 = ax1.twiny()
plt.ylim(-5, 10)

#ax3 = ax1.twinx()

plt.yticks(range(-4, 10, 2), [r'{}'.format(x)
                              for x in range(-4, 10, 2)], fontsize=10);

plt.grid(axis='both', color='wheat', linewidth=1.5, alpha = .5)

plt.title('HICP inflation, annual rate of change, Jan 2000 - March 2016\n\n', fontsize=20);


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item.ipynb".
The first task is:
df_infl_items.rename(columns = dic)
tt = df_infl_items.copy()
tt['month'] = tt.index.month
tt['year'] = tt.index.year
melted_df = pd.melt(tt,id_vars=['month','year'])
melted_df.head()
Can you write Python code for it?


I am working on the file "Disaggregated-Data/weather-like-plot-HICP-by-item.ipynb".
The first task is:
df_infl_items['month'] = df_infl_items.index.month
df_infl_items['year'] = df_infl_items.index.year
Can you write Python code for it?


I am working on the file "Disambiguition.ipynb".
The first task is:
Obviously, the number of papers is highly reduced. We can now check a random subset to see if the filtered papers make sense for this author.
Can you write Python code for it?

filterd_pub_df.iloc[pd.np.random.randint(0, high=134, size=20)][['title', 'publication_name']]


I am working on the file "Disambiguition.ipynb".
The first task is:
Sometimes we would obtain a list of author profiles for each author. In this case, we only have one and it is clear that the author profile is highly noisy.
In the following step, I would use the helper functions in utils to screen each paper by this author_id
Can you write Python code for it?

author_id = '7404651152'
author_id, d['affil_id_list']


I am working on the file "Disambiguition.ipynb".
The first task is:
When I was collecting data for my own research, I found that Dr. Vivek K. Singh has a very noisy profile in Scopus. Let's use this as an example.
The basic idea is to match author-affiliation pair:
- For all the paper found in the mixed profile
    - Find the focal author (in this case, Dr. Singh)
    - Look at his/her affiliation
        - Keep this paper if the affiliation is indeed where he/she is
        - If not, discard the paper
For Dr. Singh, I manually obtained his affiliation ids by searching through Scopus affiliation search. Upon obtaining that, create a dictionary containing name (first/last), affiliation name, and a list of affiliation ids. Author and affiliation names would be used to search for this author. The list of affiliation ids would be used for cleaning papers:

UC Irvine 60007278
MIT 60022195
Rutgers 60030623
Can you write Python code for it?

d = {'authfirst': 'Vivek', 'authlastname': 'Singh', 'affiliation': 'Rutgers',
     'affil_id_list': ['60030623', '60022195', '60007278']
    }
d

query = "AUTHLASTNAME({}) and AUTHFIRST({}) and AFFIL({})".format(d['authlastname'], d['authfirst'], d['affiliation'])
author_search_df = scopus.search_author(query)
author_search_df


I am working on the file "Disambiguition.ipynb".
The first task is:
Wrapper functions for disambiguity
Can you write Python code for it?

import requests, time
import pandas as pd
from bs4 import BeautifulSoup as Soup
BASE_URL = "https://api.elsevier.com/content/abstract/scopus_id/"

def _check_pub_validity(sid, author_id, author_affil_id_list, apikey):
    global BASE_URL
    r = requests.get(BASE_URL+sid, params={'apikey': apikey})
    soup = Soup(r.content, 'lxml')
    author_list = soup.find('authors').find_all('author')

    ## go through the author list to find the author first, by matching author id
    for au in author_list:
        if au['auid'] == author_id:
            ## find it and break
            break

    ## check the affiliation id: note that an author may have a list of affiliations
    this_affil_id_list = [affil_tag['id'] for affil_tag in au.find_all('affiliation')]
    ## get the affiliation id and check if there are any overlap
    if len(set(author_affil_id_list).intersection(set(this_affil_id_list))) > 0:
        return True
    return False

def check_pub_validity(scopus_obj, author_id, author_affil_id_list, apikey):
    ## first find out all pub
    pub_df = scopus_obj.search_author_publication(author_id)
    ## do this for all non-null scopus ids
    pub_df = pub_df[~pub_df.scopus_id.isnull()]
    ## list to save all eligble scopus ids
    eligible_scopus_id_list = list()
    for i, sid in enumerate(pub_df.scopus_id.values):
        if (i+1)%5==0:
            time.sleep(pd.np.random.random()+.3)
        if _check_pub_validity(sid, author_id, author_affil_id_list, apikey):
            ## if true, save it
            eligible_scopus_id_list.append(sid)
    ## finally, get a subset of the original pub_df
    filtered_pub_df = pub_df.query("scopus_id in @eligible_scopus_id_list")
    return filtered_pub_df


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Choose best regularization parameter from parabolic fit
Can you write Python code for it?

minalpha = np.arange(-12, 4.0, .1)[np.argmin(p(np.arange(-12, 4.0, .1)))]
# minalpha = np.log(alphas)[np.argmin(cv_error)] # This chooses the minimum

minalpha = np.sqrt(10)**minalpha
print(minalpha)


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Choose best size from parabolic fit
When I create neural network sizes, the first parameter $n_1$ normally distributed between 1 and 500. Its mean is ~250. The number of nodes in the second layer, $n_2$ depends on the first: it is between 1 and $n_1+1$. Also, its mean is $n_1/2$. The third layer is analogous: between 1 and $n_2/+1$ and with mean $n_2/2$. This is an empirical relationship I use to loosely "parametrize" the number of nodes in each hidden layer.
Knowing the optimal sum, I simply choose the number of nodes whose means would result in this sum, according to my empirical relationships. This gives the following optimal size:
Can you write Python code for it?

minsum = range(1, max(sizes_sum)+1)[np.argmin(p(range(1, max(sizes_sum)+1)))]
minsize = (int(minsum*4/7),int(minsum*2/7),int(minsum*1/7))
print(minsize)


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Choose regularization valus
Here we will choose the regularization value using the same approach as before. This takes a few minutes.
Can you write Python code for it?

alphas = np.append([0], np.sqrt(10)**np.arange(-10, 4.0, 1))
train_error = np.array([])
cv_error = np.array([])
train_adj_error = np.array([])
cv_adj_error = np.array([])
minerr = 1
for i, a in enumerate(alphas):
    clf = MLPClassifier(solver='lbfgs', alpha=a,
                        hidden_layer_sizes=minsize)
    clf.fit(X_train,y_train)

    # Compute errors
    conf_cv = confusion_matrix(y_cv, clf.predict(X_cv))
    conf_tr = confusion_matrix(y_train, clf.predict(X_train))

    train_error = np.append(train_error, 1-accuracy(conf_tr))
    cv_error = np.append(cv_error, 1-accuracy(conf_cv))

    train_adj_error = np.append(train_adj_error,
            1-accuracy_adjacent(conf_tr, adjacent_facies))
    cv_adj_error = np.append(cv_adj_error,
            1-accuracy_adjacent(conf_cv, adjacent_facies))
    print('[ %3d%% done ] ' % (100*(i+1)/len(alphas),), end="")
    if cv_error[-1] < minerr:
        minerr = cv_error[-1]
        print('CV error = %d%% with %g' % (100*minerr, a))
    else:
        print()


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Condition dataset
Can you write Python code for it?

correct_facies_labels = training_data['Facies'].values

feature_vectors = training_data.drop(['Formation', 'Well Name', 'Depth','Facies','FaciesLabels'], axis=1)
feature_vectors.describe()

scaler = preprocessing.StandardScaler().fit(feature_vectors)
scaled_features = scaler.transform(feature_vectors)


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Convenience functions
Can you write Python code for it?

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

    ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([])
    ax[4].set_yticklabels([]); ax[5].set_yticklabels([])
    ax[5].set_xticklabels([])
    f.suptitle('Well: %s'%logs.iloc[0]['Well Name'],...


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Load, treat and color data
Can you write Python code for it?

# Loading Data
validationFull = pd.read_csv('../validation_data_nofacies.csv')
training_data = pd.read_csv('../facies_vectors.csv')

# Treat Data
training_data.fillna(training_data.mean(),inplace=True)
training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data['Well Name'].unique()
training_data.describe()

# Color Data
# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
       '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)
make_facies_log_plot(
    training_data[training_data['Well Name'] == 'SHRIMPLIN'],
    facies_colors)


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Plot performance of neural networks vs sum of nodes
Can you write Python code for it?

sizes_sum = [ np.sum(s) for s in sizes ]
p = np.poly1d(np.polyfit(sizes_sum, cv_error, 2))
f, ax = plt.subplots(figsize=(5,5))
ax.scatter(sizes_sum, cv_error, c='k', label='Cross-validate')
ax.plot(range(1, max(sizes_sum)+1), p(range(1, max(sizes_sum)+1)))
ax.set_ylim([min(cv_error)-.1, max(cv_error)+.1])
ax.set_xlabel('Sum of nodes')
ax.set_ylabel('Error')
plt.legend()


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Test, train and cross-validate
Up to here, there have been no secrets, just reusing the standard code to load the data. Now, instead of doing the usual test/train split, I create another dataset, the cross-validate set. The split will be 60% train, 20% cross-validate and 20% test.
It which will be used as the "test set", to tune the neural network parameters. My actual test set will only be used to predict the performance of my neural network at the end.
Can you write Python code for it?

X_train, X_cv_test, y_train, y_cv_test = train_test_split(scaled_features,
        correct_facies_labels, test_size=0.4, random_state=42)
X_cv, X_test, y_cv, y_test = train_test_split(X_cv_test, y_cv_test,
        test_size=0.5, random_state=42)


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub1.ipynb".
The first task is:
Training with several model sizes
This takes a few minutes.
Can you write Python code for it?

train_error = np.array([])
cv_error = np.array([])
train_adj_error = np.array([])
cv_adj_error = np.array([])
minerr = 1
for i, s in enumerate(sizes):
    clf = MLPClassifier(solver='lbfgs', alpha=0,
                        hidden_layer_sizes=s)
    clf.fit(X_train,y_train)

    # Compute errors
    conf_cv = confusion_matrix(y_cv, clf.predict(X_cv))
    conf_tr = confusion_matrix(y_train, clf.predict(X_train))

    train_error = np.append(train_error, 1-accuracy(conf_tr))
    cv_error = np.append(cv_error, 1-accuracy(conf_cv))

    train_adj_error = np.append(train_adj_error,
            1-accuracy_adjacent(conf_tr, adjacent_facies))
    cv_adj_error = np.append(cv_adj_error,
            1-accuracy_adjacent(conf_cv, adjacent_facies))
    print('[ %3d%% done ] ' % (100*(i+1)/len(sizes),), end="")
    if cv_error[-1] < minerr:
        minerr = cv_error[-1]
        print('CV error = %d%% with' % (100*minerr,), s)
    else:
        print()


I am working on the file "DiscerningHaggis/Discerning_Haggis_Facies_Classification_sub3.ipynb".
The first task is:
Retrain and predict
Finally we train a neural network using all data available, and apply it to our blind test.
Can you write Python code for it?

clf_final = MLPClassifier(solver='lbfgs', alpha=0.1,
                        hidden_layer_sizes=(300,300,300))
clf_final.fit(scaled_features,correct_facies_labels)

validationFullsm = validationFull.copy()
validation_features = validationFullsm.drop(['Formation', 'Well Name', 'Depth'], axis=1)
scaled_validation = scaler.transform(validation_features)
validation_output = clf_final.predict(scaled_validation)
validationFull['Facies']=validation_output
validationFull.to_csv('well_data_with_facies_DH_sub3.csv')


I am working on the file "Disorder.ipynb".
The first task is:
Here I joined both datasets
Can you write Python code for it?

df2 = pd.DataFrame([disorders]).T
idx = df2.index.intersection(df.index)
df2 = df2.loc[idx]

df = pd.concat([df.loc[idx],df2], axis=1)
df.columns = ['sequence', 'secondStruct', 'disorder', 'predictions', 'iupred']

del(df2)


I am working on the file "Disorder.ipynb".
The first task is:
I have generated iupred results for the whole proteome since we didn't have the scores.
Can you write Python code for it?

def read_iupred_results(fileName):
    '''
        function read files containing the scores from iupred into
        the hash results
        INPUT:  fileName
                results (hash)
    '''
    results = {}
    f = open(fileName, "r")
    while True:
        try:
            k,v = next(f), next(f)
            k = k.strip()[1:]
            results[k] = v.strip().split(',')[:-1]
        except StopIteration:
            break
    f.close()
    return results

import os,re
files = [i for i in os.listdir('../scripts/iupred/') if re.search(".results$",i)]

disorders = {}
for i in files:
    d = read_iupred_results('../scripts/iupred/'+i)
    disorders.update(d)


I am working on the file "Disorder.ipynb".
The first task is:
Predict TADs from the proteome
Can you write Python code for it?

# load NN model and weights
from keras.models import model_from_json

# open json model and weights
with open("../models/deep_model.json", "r") as json_file:
    json_model = json_file.read()

deep_model = model_from_json(json_model)
deep_model.load_weights("../models/deep_model.h5")

# set cutoff to predict TADs in the proteome
cutoff=0.8

results = np.zeros(shape=(df.shape[0],4))
for n,prot in enumerate(df.predictions):
    results[n] = predict_motif_statistics(prot, cutoff)

results = pd.DataFrame(results, index=df.index, columns = ['length', 'start_position', 'gral_mean', 'mean_longest_region'])


I am working on the file "Disorder.ipynb".
The first task is:
These are old predictions that include sequences and scores from our deep learning model
Can you write Python code for it?

# forgot to add the confidence of secondary_structure_oredictions...
path = '../scripts/fastas/'
tmp, tnp = [], []
for f in [i for i in os.listdir(path) if i[-11:]==".output.csv"]:
    predsName = f + ".predictions.npz"

    df = pd.read_csv(path + f, index_col=0)
    tmp.append(df[['sequence','secondStruct','disorder']])
    nf = np.load(path + predsName)
    tnp.append(nf[nf.files[0]])

predictions = np.hstack(tnp)
df = pd.concat(tmp)

# finally join all fields into a single data structure to facilitate further analysis
df['predictions'] = predictions


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Base 2
Can you write Python code for it?

eps = 0.006
starting_i = 0
max_i = 9
base = 2

## maximum grid length:
1+pow(base,max_i+1)

ranges = setup_ranges(max_i,base)
solution_containing_voxels = initial_voxels(max_i,base,starting_i,eps)
add_high_res_voxels(max_i,base,starting_i,eps,solution_containing_voxels)

## Number of solutions found for each resolution:

for i in range(0,max_i+1):
    print len(solution_containing_voxels[i])

## Number of solutions found for each resolution:
plt.title('Number of voxels per step')
plt.semilogy([(len(solution_containing_voxels[i])) for i in range(0,max_i+1)])

plot_voxels(solution_containing_voxels,max_i)


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Development for method to be used in the package
Can you write Python code for it?

def setup_ranges(max_i,base,min_value = 6.,max_value = 11.):
    ranges= {}
    for i in range(max_i+1):
        ranges[i] = np.linspace(min_value,max_value,1+pow(base,i+1))
    return ranges

@timeit
def initial_voxels(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps):
    solution_containing_voxels = {}
    eps_current = eps * pow(base,max_i-starting_i)
    solution_containing_voxels[starting_i] = {}

    for i1,om1 in enumerate(ranges[starting_i]):
        for i2,om2 in enumerate(ranges[starting_i]):
            err = k_of_nu1_nu2(om1,om2)
            if abs(err) < eps_current:
                solution_containing_voxels[starting_i][i1,i2] = err
    return solution_containing_voxels

@timeit
def add_high_res_voxels(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps,solution_containing_voxels):
    for i in range(starting_i+1,max_i+1):
        eps_current = eps * pow(base,max_i-i)
        solution_containing_voxels[i] = {}
        for (i1,i2) in solution_containing_voxels[i-1]:
            step_size = int(base/2)
            max_length = pow(base,i+1)
            for i1_new in range(max(0,i1*base-step_size),min(max_length,i1*base+step_size+1)):
                for i2_new in range(max(0,i2*base-step_size),min(max_length,i2*base+step_size+1)):
                    err = k_of_nu1_nu2(ranges[i][i1_new],ranges[i][i2_new])
                    if abs(err) < eps_current:
                        solution_containing_voxels[i][i1_new,i2_new] = err

@timeit
def plot_voxels(solution_containing_voxels,i):
    voxels = np.zeros((1+pow(base,i+1),1+pow(base,i+1)))
    for (i1,i2) in solution_containing_voxels[i]:
        voxels[i1,i2] = 1
    plot_arr(voxels)

def voxel_solutions(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps):
    solution_containing_voxels = initial_voxels(ranges,k_of_nu1_nu2,max_i,
                                                base,starting_i,eps)
    add_high_res_voxels(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps,
                        solution_containing_voxels)
    return solution_containing_voxels

def...


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Different bases comparison
Base 10
Can you write Python code for it?

eps = 0.006
starting_i = 0
max_i = 2
base = 10

## maximum grid length:
1+pow(base,max_i+1)

ranges = setup_ranges(max_i,base)
solution_containing_voxels = initial_voxels(max_i,base,starting_i,eps)
add_high_res_voxels(max_i,base,starting_i,eps,solution_containing_voxels)

## Number of solutions found for each resolution:

for i in range(0,max_i+1):
    print len(solution_containing_voxels[i])

## Number of solutions found for each resolution:
plt.title('Number of voxels per step')
plt.semilogy([(len(solution_containing_voxels[i])) for i in range(0,max_i+1)])

plot_voxels(solution_containing_voxels,max_i)


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Discussion
The number of solution voxels increases by a factor of base at each step. This happens because the function being optimize is close to linear near the solutions and because we decrease eps_current by a factor of base at each step. As a result, the total number of voxels increases by a factor of base**2 at each step, but the thickness of the solution voxel surface decreases by a factor of base.
The cost of the algorithm is dominated by the last step. This is because the number of voxels increases approximately by a factor of base at each step, and the computational cost at each step is the number of voxels from the previous step multiplied by base**2. As such, the algorithm runtime is essentially the number of solution points. Notice in the above experiments the runtime was similar for different bases used.
A more careful analysis assuming a large number of points and the scaling described above gives a geometric series for the runtime, where the sum starts at the last step ($b$ stands for base and $p$ stands for number of points):
\begin{align}
b^2(\frac{p}{b} + \frac{p}{b^2} + ...)
=
p\left( b + 1 + \frac{1}{b} + ...\right)
\approx \frac{pb^2}{b-1}.
\end{align}
The other factor contributing to the runtime is breaking away from the scaling law in the above discussion.
Using the same technique for $\chi^{(3)}$.
Let's extend the above search technique to using four-wave mixing.
The problems here will be larger, so runtime may be more important. For this reason, I tested different values of epsilon at each stage. Using a smaller epsilon will improve the runtime, but may miss correct solutions. Compare the number of solutions and runtime to the methods used in the notebook Dispersion_relation_two_approaches. The complexity is essentially linear in the number of solutions, but here the constant factor may be large if epsilon is not chosen carefully at each step. For this reason one may prefer to use the other techniques for $\chi^{(3)}$ problems. The number of solutions in the method below converges...
Can you write Python code for it?

eps = 2e-4
starting_i = 0
max_i = 1
base = 10

relative_scalings = [4,4,10]

phi1_min = 30.
phi1_max = 34.
ranges1 = {}
for i in range(0,2):
    ranges1[i] = np.linspace(phi1_min,phi1_max,relative_scalings[0]*pow(base,i+1)+1)

phi2_min = -13
phi2_max = -9
ranges2 = {}
for i in range(0,2):
    ranges2[i] = np.linspace(phi2_min,phi2_max,relative_scalings[1]*pow(base,i+1)+1)

nu3_min = -26.
nu3_max = -16.
ranges3 = {}
for i in range(0,2):
    ranges3[i] = np.linspace(nu3_min,nu3_max,relative_scalings[2]*pow(base,i+1)+1)

print len(ranges1[1]),len(ranges2[1]),len(ranges3[1])

phi1, phi2 = sp.symbols('phi_1 phi_2')

ex1 = (k_symb(nu1,pol='e')+k_symb(nu2,pol='e')).expand().subs({nu1:(phi1 + phi2)/2, nu2: (phi1-phi2)/2})

ex2 = -(k_symb(nu3,pol='e')+k_symb(nu4,pol='e')).expand().subs(nu4,-phi1-nu3)

f_phi12_nu3 =ufuncify([phi1,phi2,nu3], ex1-ex2)

@timeit
def initial_voxels_4wv(max_i,base,starting_i,eps,eps_factor = None):

    if eps_factor is None:
        eps_factor = pow(base,max_i-starting_i)
    eps_current = eps * eps_factor

    solution_containing_voxels = {}
    solution_containing_voxels[starting_i] = {}

    for i1,om1 in enumerate(ranges1[starting_i]):
        for i2,om2 in enumerate(ranges2[starting_i]):
            for i3,om3 in enumerate(ranges3[starting_i]):
                err = f_phi12_nu3(om1,om2,om3)
                if abs(err) < eps_current:
                    solution_containing_voxels[starting_i][i1,i2,i3] = err
    return solution_containing_voxels

@timeit
def add_high_res_voxels_4wv(max_i,base,starting_i,eps,solution_containing_voxels):
    for i in range(starting_i+1,max_i+1):
        eps_current = eps * pow(base,max_i-i)
        solution_containing_voxels[i] = {}
        for (i1,i2,i3) in solution_containing_voxels[i-1]:
            step_size = int(base/2)
            max_length = pow(base,i+1)
            for i1_new in range(max(0,i1*base-step_size),min(relative_scalings[0]*max_length,i1*base+step_size+1)):
                for i2_new in...


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Find maximum derivative of dispersion_difference_function over a range. This could be used as a bound for $\epsilon$ to guarantee results.
Can you write Python code for it?

def find_max_der(expression,symbol,input_range):
    expr_der = sp.diff(expression,symbol)
    expr_def_func = ufuncify([symbol],expr_der)
    return max(abs(expr_def_func(input_range)))

## Apply the triangle inequality over a range of nus

nus = np.asarray([6.+ i*5e-2 for i in range(1+int(1e3))])
max_derivative = sum([find_max_der(exp,om,nus) for om,exp in zip([nu1,nu2,nu3],expressions)])

max_derivative


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Finding solutions in the voxels:
In general the nus to be considered do not lie on a grid. For this reason it is necessary to find which nus lie in each voxel.
Setup for the experiment
Can you write Python code for it?

eps = 0.006
starting_i = 0
max_i = 2
base = 10

ranges = setup_ranges(max_i,base)
solution_containing_voxels = initial_voxels(max_i,base,starting_i,eps)
add_high_res_voxels(max_i,base,starting_i,eps,solution_containing_voxels)

i = 2  ## where to draw points from.

scale = 0.1  ## scale on random noise to add to points
Delta = ranges[i][1] - ranges[i][0]  ## Delta here was made with ranges[0] spacing
range_min = ranges[i][0]
ranges_perturbed = [num+random.random()*scale for num in ranges[2]] ## make

values = [ int(round( (el - range_min) / Delta)) for el in ranges_perturbed]

def make_dict_values_to_lists_of_inputs(values,inputs):
    D = {}
    for k, v in zip(values,inputs):
        D.setdefault(k, []).append(v)
    return D

D = make_dict_values_to_lists_of_inputs(values,ranges_perturbed)

solution_nus = []
for indices in solution_containing_voxels[i].keys():
    if all([ind in D for ind in indices]): ## check all indices are in D,
        for it in itertools.product(*[D[ind] for ind in indices]): ## get all nus in the voxel.
            solution_nus.append(it)

plt.figure(figsize=(13,13))
plt.title('perturbed grid points still in the solution voxels')
plt.scatter([el[0] for el in solution_nus],[el[1] for el in solution_nus])


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Main methods used
Can you write Python code for it?

@timeit
def initial_voxels(max_i,base,starting_i,eps):
    solution_containing_voxels = {}
    eps_current = eps * pow(base,max_i-starting_i)
    solution_containing_voxels[starting_i] = {}

    for i1,om1 in enumerate(ranges[starting_i]):
        for i2,om2 in enumerate(ranges[starting_i]):
            err = k_of_nu1_nu2(om1,om2)
            if abs(err) < eps_current:
                solution_containing_voxels[starting_i][i1,i2] = err
    return solution_containing_voxels

@timeit
def add_high_res_voxels(max_i,base,starting_i,eps,solution_containing_voxels):
    for i in range(starting_i+1,max_i+1):
        eps_current = eps * pow(base,max_i-i)
        solution_containing_voxels[i] = {}
        for (i1,i2) in solution_containing_voxels[i-1]:
            step_size = int(base/2)
            max_length = pow(base,i+1)
            for i1_new in range(max(0,i1*base-step_size),min(max_length,i1*base+step_size+1)):
                for i2_new in range(max(0,i2*base-step_size),min(max_length,i2*base+step_size+1)):
                    err = k_of_nu1_nu2(ranges[i][i1_new],ranges[i][i2_new])
                    if abs(err) < eps_current:
                        solution_containing_voxels[i][i1_new,i2_new] = err

@timeit
def plot_voxels(solution_containing_voxels,i):
    voxels = np.zeros((1+pow(base,i+1),1+pow(base,i+1)))
    for (i1,i2) in solution_containing_voxels[i]:
        voxels[i1,i2] = 1
    plot_arr(voxels)

def voxel_solutions(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps):
    solution_containing_voxels = initial_voxels(ranges,k_of_nu1_nu2,max_i,
                                                base,starting_i,eps)
    add_high_res_voxels(ranges,k_of_nu1_nu2,max_i,base,starting_i,eps,
                        solution_containing_voxels)
    return solution_containing_voxels

ranges = setup_ranges(max_i,base)

solution_containing_voxels =...


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Methods for systematic search over ranges
Definitions:
base -- The number base to use, i.e. the factor to increase the grid resolution at each step.
starting_i -- index of starting step. 0 means we use a grid of size base by base.
max_i -- final index.
eps -- desired resolution at step max_i
Description
To look for solutions more efficiently, we can recursively enhance the resolution of the grid in which we are looking. At each step, decrease the cutoff eps_current by some factor (for now let's make it base). For the set of voxels in each step that are close enough to a solution of the equation, increase the resolution by a factor of base and examine the resulting smaller voxels. Continue until the last step.
Can you write Python code for it?

eps = 0.00002
starting_i = 0
max_i = 4
base = 10

min_value = 6.
max_value = 20.

@timeit
def setup_ranges(max_i,base):
    ranges= {}
    for i in range(max_i+1):
        ranges[i] = np.linspace(min_value,max_value,1+pow(base,i+1))
    return ranges


I am working on the file "Dispersion_relation_chi_2_voxels_approach.ipynb".
The first task is:
Note: How to obtain the index of $\nu_3$.
Can you write Python code for it?

i = 2

1+pow(base,i+1)

np.linspace(min_value,max_value,1+pow(base,i+1))

spacing = (max_value-min_value)/ pow(base,i+1)
spacing

num_indices_from_zero = min_value / spacing
num_indices_from_zero

ranges[i]

sample_index = solution_containing_voxels[2].keys()[1000]

sample_index

ranges[2][(sum(sample_index)+int(num_indices_from_zero))]


I am working on the file "Display CIFAR-10 Images.ipynb".
The first task is:
Load the Neon ModelZoo data for CIFAR-10
When you call CIFAR10's load_data() helper function, it will look for the CIFAR-10 data in the file path you've specified and download the file from the University of Toronto website if needed (the file is 163 MB).
According to the website, the archive contains the files data_batch_1, data_batch_2, ..., data_batch_5, as well as test_batch. Each of these files is a Python "pickled" object produced with cPickle.
These pickled batch files contain dictionaries with the images:

data is a Nx3072 numpy array of unsigned 8-bit integers (uint8) (where N is the number of images; 50,000 for train and 10,000 for validation). Each row of the array stores a 32x32 color image. The first 1024 entries contain the red channel values, the next 1024 the green, and the final 1024 the blue. The image is stored in row-major order, so that the first 32 entries of the array are the red channel values of the first row of the image.
labels is a list of N numbers (where N is the number of images; 50,000 for train and 10,000 for validation) in the range 0-9. The number at index i indicates the label of the ith image in the array data.
Can you write Python code for it?

dataset = dict()   # We'll create a dictionary for the training and testing/validation set images

pathToSaveData = './path'  # The path where neon will download and extract the data files

cifar10 = CIFAR10(path=pathToSaveData, normalize=False, whiten=False, contrast_normalize=False)
dataset['train'], dataset['validation'], numClasses = cifar10.load_data()


I am working on the file "Displaying_WesternUS_cube_data.ipynb".
The first task is:
Read in a CETB cube file and use it to get the projected extent
Can you write Python code for it?

file = "/Users/brodzik/cetb_data/v1.3/F16_SSMIS/N/cubes_WesternUS/CETB.cubefile.WesternUS.F16_SSMIS-19H-SIR-CSU-v1.3.2005.TB.nc"
f = Dataset(file, 'r', 'netCDF4')

x = f.variables['x'][:]
y = f.variables['y'][:]

# Define extent in projected coordinates.
# Use the x, y coordinate variables in the file, which give the centers of the pixels,
# and adjust these by 1/2 pixel to get the corners of the corner pixels
# We want extent as [x_min, x_max, y_min, y_max]
extent = [x[0], x[-1], y[-1], y[0]]
extent

x_res_m = np.fabs(x[1] - x[0])
y_res_m = np.fabs(y[1] - y[0])

x_res_m, y_res_m

extent = [x[0] - (x_res_m / 2.), x[-1] + (x_res_m / 2.),
          y[-1] - (y_res_m / 2.), y[0] + (y_res_m / 2.)]
extent

tb = f.variables['TB'][:]
tb.shape

f.close()

for t in np.arange(730):
    print(t, np.min(tb[t,:,:]), np.max(tb[t,:,:]))


I am working on the file "DistantSupervision.ipynb".
The first task is:
$\LaTeX$ definition block
$\newcommand{\sign}{\operatorname{sign}}$
Distant Supervision
This notebook has a few cute experiments to explore distant supervision.
Setup
In the distant supervision setting, we receive data $((x_i^a, x_i^b), y_i)$. Assuming the data is linearly separable with hyperplane $w^$, we also are given that if $y_i = \sign[\max(w^{\top} x_i^{a}, w^{*\top} x_i^{a})]$. Thus, if $y_i = -1$, we know that both $x^a_i$ and $x^b_i$ are in the negative class, but if $y_i = 1$, we only know that atleast one of $x^a_i$ or $x^b_i$ is positive.
Data generation
Consider a random (unit) weights $w^$.
We generate elements ${x_a, x_b}$, order $x_a$ by $w^ x_a$ and set $y_i$ to be $\sign[\max(w^{\top} x_a, w^{\top} x_b)]$.
Can you write Python code for it?

# Constants
D = 2
N = 100
K = 2

w = np.random.randn(D)
w = normalize(w)
theta = np.arctan2(w[0], w[1])

X = np.random.randn(N,D,K)
y = np.zeros(N)
for i in range(N):
    m = w.dot(X[i])
    X[i] = X[i][:,np.argsort(-m)]
    y[i] = np.sign(max(m))

# Visualize data
plt.plot(np.arange(-3,3), -w[0]/w[1] * np.arange(-3,3), c='black')
plt.scatter(X[y<0,0,0], X[y<0,1,0], c='r', marker='o')
plt.scatter(X[y<0,0,1], X[y<0,1,1], c='r', marker='o')
plt.scatter(X[y>0,0,0], X[y>0,1,0], c='b', marker='D')
plt.scatter(X[y>0,0,1], X[y>0,1,1], c='b', marker='D')

# Split data for convenience
X_ = np.concatenate((X[y<0,:,0],X[y<0,:,1]))
Xa = X[y>0,:,0]
Xb = X[y>0,:,1]
Xp = np.concatenate((Xa, Xb))
Na, N_, Np = y[y>0].shape[0], 2*y[y<0].shape[0], 2*y[y>0].shape[0]
Nb = Na
Na, N_, Np


I am working on the file "DistantSupervision.ipynb".
The first task is:
As expected, the naive approach does really poorly.
Proposed method
Incorporate responsibilities for the points.
$\begin{align}
\min{}& \frac{1}{2} \|\beta\|^2 + C \xi \
\textrm{subject to}
& -1(\beta^\top x_{-}) \ge 1 - \xi \
& +1(\beta^\top x_{a}) \ge y_a - \xi \
& +1(\beta^\top x_{b}) \ge y_b - \xi \
& y_a > 0, y_b > 0, y_a + y_b \ge 1
\end{align}$
Can you write Python code for it?

# Beta is the coefficients, and e is the slack.
Da, = y[y>0].shape
beta = Variable(D)
e = Variable()
loss = 0.5 * norm(beta, 2) ** 2 + 1 * e
constr = [mul_elemwise(-1,X_*beta) > 1 - e,
          Xa*beta + Xb*beta > 1 - e,
         ]
prob = Problem(Minimize(loss), constr)
print("loss", prob.solve())

w_ = np.array(beta.value).flatten()
w_ = w_ / np.linalg.norm(w_) # Until I add a 0.
print("error", np.linalg.norm(w-w_))

# Visualize data
plt.plot(np.arange(-3,3), -w[0]/w[1] * np.arange(-3,3), linestyle='dashed', c='black')
plt.plot(np.arange(-3,3), -w_[0]/w_[1] * np.arange(-3,3), c='green')
plt.scatter(X[y<0,0,0], X[y<0,1,0], c='r', marker='o')
plt.scatter(X[y<0,0,1], X[y<0,1,1], c='r', marker='o')
plt.scatter(X[y>0,0,0], X[y>0,1,0], c='b', marker='D')
plt.scatter(X[y>0,0,1], X[y>0,1,1], c='b', marker='D')