I am working on the file "Deep_learning_ND/tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set num_epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set embed_dim to the size of the embedding.
Set seq_length to the length of sequence.
Set learning_rate to the learning rate.
Set show_every_n_batches to the number of batches the neural network should print progress.
Can you write Python code for it?

# Number of Epochs
num_epochs = 50
# Batch Size
batch_size = 128
# RNN Size
rnn_size = 512
# Embedding Dimension Size
embed_dim = 300
# Sequence Length
seq_length = 32
# Learning Rate
learning_rate = 0.01
# Show stats for every n number of batches
show_every_n_batches = 50

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
save_dir = './save'


I am working on the file "Deep_learning_ND/tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)


I am working on the file "Deep_learning_ND/tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Save Parameters
Save seq_length and save_dir for generating a new TV script.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params((seq_length, save_dir))


I am working on the file "Deep_learning_ND/tv-script-generation/dlnd_tv_script_generation.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data.  If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
batches = get_batches(int_text, batch_size, seq_length)

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(num_epochs):
        state = sess.run(initial_state, {input_text: batches[0][0]})

        for batch_i, (x, y) in enumerate(batches):
            feed = {
                input_text: x,
                targets: y,
                initial_state: state,
                lr: learning_rate}
            train_loss, state, _ = sess.run([cost, final_state, train_op], feed)

            # Show every <show_every_n_batches> batches
            if (epoch_i * len(batches) + batch_i) % show_every_n_batches == 0:
                print('Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(
                    epoch_i,
                    batch_i,
                    len(batches),
                    train_loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_dir)
    print('Model Trained and Saved')


I am working on the file "DefensiveProgramming_3.ipynb".
The first task is:
And test it...
Can you write Python code for it?

test_range_overlap()

test_range_overlap_no_overlap()

test_range_overlap_one_range()


I am working on the file "DefensiveProgramming_3.ipynb".
The first task is:
But what if there is no overlap? What if they just touch?
Can you write Python code for it?

def test_range_overlap_no_overlap():
    assert range_overlap([ (0.0, 1.0), (5.0, 6.0) ]) == None
    assert range_overlap([ (0.0, 1.0), (1.0, 2.0) ]) == None


I am working on the file "DefensiveProgramming_3.ipynb".
The first task is:
Should we add to the tests?
Can you write version with fewer bugs. My attempt is below.
Can you write Python code for it?

def pairs_overlap(rangeA, rangeB):
    # Check if A starts after B ends and
    # A ends before B starts. If both are
    # false, there is an overlap.
    # We are assuming (0.0 1.0) and
    # (1.0 2.0) do not overlap. If these should
    # overlap swap >= for > and <= for <.
    overlap = not ((rangeA[0] >= rangeB[1]) or
                   (rangeA[1] <= rangeB[0]))

    return overlap

def find_overlap(rangeA, rangeB):
    # Return the overlap between range
    # A and B
    if pairs_overlap(rangeA, rangeB):
        low = max(rangeA[0], rangeB[0])
        high = min(rangeA[1], rangeB[1])
        return (low, high)
    else:
        return None

def range_overlap(ranges):
    # Return common overlap among a set of
    # [low, high] ranges.

    if len(ranges) == 1:
        # Special case of one range -
        # overlaps with itself
        return(ranges[0])
    elif len(ranges) == 2:
        # Just return from find_overlap
        return find_overlap(ranges[0], ranges[1])
    else:
        # Range of A, B, C is the
        # range of range(B,C) with
        # A, etc. Do this by recursion...
        overlap = find_overlap(ranges[-1], ranges[-2])
        if overlap is not None:
            # Chop off the end of ranges and
            # replace with the overlap
            ranges = ranges[:-2]
            ranges.append(overlap)
            # Now run again, with the smaller list.
            return range_overlap(ranges)
        else:
            return None

test_range_overlap()
test_range_overlap_one_range()
test_range_overlap_no_overlap()


I am working on the file "DefensiveProgramming_3.ipynb".
The first task is:
The write a solution - one possible one is below.
Can you write Python code for it?

def range_overlap(ranges):
    # Return common overlap among a set of [low, high] ranges.
    lowest = -1000.0
    highest = 1000.0
    for (low, high) in ranges:
        lowest = max(lowest, low)
        highest = min(highest, high)
    return (lowest, highest)


I am working on the file "DefensiveProgramming_3.ipynb".
The first task is:
What about the case of a single range?
Can you write Python code for it?

def test_range_overlap_one_range():
        assert range_overlap([ (0.0, 1.0) ]) == (0.0, 1.0)


I am working on the file "Demo 1 - Single forward problem [local].ipynb".
The first task is:
Import plotting tools from matplotlib and set format defaults
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib

%matplotlib inline

from IPython.display import set_matplotlib_formats
set_matplotlib_formats('png')
matplotlib.rcParams['savefig.dpi'] = 150 # Change this to adjust figure size

# Plotting options
font = {
    'family': 'Bitstream Vera Sans',
    'weight': 'normal',
    'size': 8,
}

matplotlib.rc('font', **font)


I am working on the file "Demo 1 - Single forward problem [local].ipynb".
The first task is:
Numerical Examples
Base configuration
In this configuration, the PML is set to a reasonable default and no free-surface conditions are imposed.
Can you write Python code for it?

from zephyr.Kernel import SeisFDFDKernel
sp = SeisFDFDKernel(systemConfig)

activeSource = 50
u, d = sp.forward(sxs[activeSource], False)
b = sp.backprop(sxs[activeSource], np.ones((len(sxs[activeSource].rxList),)))
u.shape = (nz, nx)
b.shape = (nz, nx)


I am working on the file "Demo 1 - Single forward problem [local].ipynb".
The first task is:
Plotting functions
These are convenience functions to allow us to set plotting limits in one place. Since we're defining these with closures to save effort, they need to go after the system configuration to grab things like the geometry after it's defined.
Can you write Python code for it?

sms = 4
rms = 0.5

def plotField(u):
    clip = 0.1*abs(u).max()
    plt.imshow(u.real, cmap=cm.bwr, vmin=-clip, vmax=clip)

def plotModel(v):
    lclip = 2000
    hclip = 3000
    plt.imshow(v.real, cmap=cm.jet, vmin=lclip, vmax=hclip)

def plotGeometry():

    srcpos = srcs[activeSource][::2]
    recpos = recs[:,::2]

    axistemp = plt.axis()
    plt.plot(srcpos[0], srcpos[1], 'kx', markersize=sms)
    plt.plot(recpos[:,0], recpos[:,1], 'kv', markersize=rms)
    plt.axis(axistemp)


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Array geometry
Can you write Python code for it?

srcs        = np.array([np.ones(101)*32, np.zeros(101), np.linspace(32, 232, 101)]).T
recs        = np.array([np.ones(101)*132, np.zeros(101), np.linspace(32, 232, 101)]).T
nsrc        = len(srcs)
nrec        = len(recs)
recmode     = 'fixed'

geom        = {
    'src':  srcs,
    'rec':  recs,
    'mode': 'fixed',
}


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Computed properties
Can you write Python code for it?

dims        = (nx,nz)       # tuple
rho         = np.fliplr(np.ones(dims) * density)
nfreq       = len(freqs)    # number of frequencies
nsp         = nfreq * nky   # total number of 2D subproblems
cPert       = np.zeros(dims)
cPert[(nx/2)-20:(nx/2)+20,(nz/2)-20:(nz/2)+20] = vanom
c           = np.fliplr(np.ones(dims) * velocity)
cFlat       = c
c          += np.fliplr(cPert)
cTrue       = c


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Configuration dictionary
(assembled from previous sections)
Can you write Python code for it?

# Base configuration for all subproblems
systemConfig = {
    'dx':   cellSize,       # m
    'dz':   cellSize,       # m
    'c':        c.T,        # m/s
    'rho':      rho.T,      # density
    'Q':        Q,          # can be inf
    'nx':       nx,         # count
    'nz':       nz,         # count
    'freeSurf': freeSurf,   # t r b l
    'nPML':     nPML,
    'geom':     geom,
    'cache':    cache,
    'cacheDir': cacheDir,
    'freqs':    freqs,
    'nky':      nky,
    'parFac':   parFac,
    'chunksPerWorker':  chunksPerWorker,
    'profile':  profile,
    'ensembleClear':    ensembleClear,
#     'MPI': False,
#    'Solver':   Reference('SimPEG.SolverWrapD(scipy.sparse.linalg.splu)'),#Solver,
}


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Forward modelling and backpropagation
Example (commented out) showing how to generate synthetic data using the SimPEG-style survey and problem interfaces. In this implementation, both are essentially expressions of the Dispatcher. The Dispatcher API has yet to be merged into SimPEG.
Can you write Python code for it?

# d = survey.projectFields()
# uF = problem.fields()


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Geometry, data and forward wavefield:
Can you write Python code for it?

fig = plt.figure()

ax1 = fig.add_subplot(1,3,1)
plotModel(c.T)
plotGeometry(geom)
ax1.set_title('Velocity Model')
ax1.set_xlabel('X')
ax1.set_ylabel('Z')

ax2 = fig.add_subplot(1,3,2)
plt.imshow(drt.real, cmap=cm.bwr)
ax2.set_title('Real part of d: $\omega = %0.1f$'%(freqs[freqNum],))
ax2.set_xlabel('Receiver #')
ax2.set_ylabel('Source #')

ax3 = fig.add_subplot(1,3,3)
plt.imshow(frt[srcNum].real, vmin=-clipScaleF, vmax=clipScaleF, cmap=cm.bwr)
plt.title('uF: $\omega = %0.1f$, src. %d'%(freqs[freqNum], srcNum))

fig.tight_layout()


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
However, it will block if we ask for the data or wavefields:
Can you write Python code for it?

%%time
d = sp.dPred
uF = sp.uF

d[0].shape


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Importing libraries

numpy is the de facto standard Python numerical computing library
zephyr.Dispatcher is zephyr's primary parallel remote problem interface
IPython.parallel provides parallel task control (nominally, this is to be handled inside the Dispatcher object)
Can you write Python code for it?

import numpy as np
from zephyr.Dispatcher import SeisFDFDDispatcher
from IPython.parallel import Reference


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Numerical / parallel parameters
Can you write Python code for it?

cache       = False         # whether to cache computed wavefields for a given source
cacheDir    = '.'

parFac = 2
chunksPerWorker = 0.5       # NB: parFac * chunksPerWorker = number of source array subsets
ensembleClear = False


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Parallel computations
This section runs each of the parallel computations on the remote worker nodes.
Set up problem

Create the Dispatcher object using the systemConfig dictionary as input
Spawn survey and problem interfaces, which implement the SimPEG standard properties
Generate a set of "transmitter" objects, each of which knows about its respective "receivers" (in seismic parlance, these would be "sources" and "receivers"; the term "transmitter" is more common in EM and potential fields geophysics)
Tell the dispatcher object about the transmitters
Can you write Python code for it?

%%time
sp = SeisFDFDDispatcher(systemConfig)
survey, problem = sp.spawnInterfaces()
sxs = survey.genSrc()
sp.srcs = sxs


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Properties of the model
Can you write Python code for it?

velocity    = 2500          # m/s
vanom       = 500           # m/s
density     = 2700          # units of density
Q           = 500           # can be inf


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
Results
We show the resulting data and wavefield properties.
Data selection:
Can you write Python code for it?

freqNum = 0
srcNum = 0

frt = uF[freqNum]
drt = d[freqNum]
clipScaleF = 1e-1 * abs(frt[srcNum]).max()


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
These lines define some plotting functions, which are used later.
Can you write Python code for it?

lclip = 2000
hclip = 3000
clipscale = 0.1
sms = 0.5
rms = 0.5

def plotField(u):
    clip = clipscale*abs(u).max()
    plt.imshow(u.real, cmap=cm.bwr, vmin=-clip, vmax=clip)

def plotModel(v):
    plt.imshow(v.real, cmap=cm.jet, vmin=lclip, vmax=hclip)

def plotGeometry(geom):

    srcpos = geom['src'][:,::2]
    recpos = geom['rec'][:,::2]

    axistemp = plt.axis()
    plt.plot(srcpos[:,0], srcpos[:,1], 'kx', markersize=sms)
    plt.plot(recpos[:,0], recpos[:,1], 'kv', markersize=rms)
    plt.axis(axistemp)


I am working on the file "Demo 2 - Remote parallel computation [distributed].ipynb".
The first task is:
This code runs the forward modelling on the [remote] workers. It returns asynchronously, so the code can run in the background.
Can you write Python code for it?


I am working on the file "Demo-Test.ipynb".
The first task is:
Solution 2
Can you write Python code for it?

def solution_2(A):
    addition_list = list()
    list_index = 1

    addition_list.append(A[0])
    try:
        if len(A) >= 0 and len(A) <= 100000:

            for i, int_in_arr in enumerate(A):
                # print i, " ", int_in_arr

                if int_in_arr >= -2 and int_in_arr <= 647 and type(int_in_arr) is int:
                    if i == 0: continue

                    addition_list.append(int_in_arr + addition_list[list_index - 1])

                    #print "i: ", i, "\n"

                    #print A[0:i], "\n"

                    # print math.sum(A[0:i])

                    #print A[i + 1:], "\n"

                    # print addition_list

                    list_index += 1

                else:
                    raise ValueError("one of the array element: integer out of range", int_in_arr)

        else:
            raise ValueError("array indices out of range ", len(A))
        #print A

        #print "\n", addition_list

        last_list_index = i

        list_index = 1

        while list_index != last_list_index:
            print addition_list[last_list_index]
            if A[list_index-1] == (addition_list[last_list_index] - addition_list[list_index]):
                return list_index

    except (ValueError, RuntimeError) as err:

        return err

print solution_2(Ax)


I am working on the file "Demo-Test.ipynb".
The first task is:
Solution 6 - reduced asympottic time complexity by increasing space complexity. O^n
Can you write Python code for it?

def solution_6(A):

    left_sum = 0

    right_sum = sum(A[1:])

    len_arr = len(A)

    try:
        if len_arr <= 1:
            if len_arr == 1:
                return 0
            else:
                return -1

        if left_sum == right_sum:
            return 0

        #if sum(A[:-1]) == 0:
            #return len_arr-1

        if len(A) == abs(sum(A[:])):
            if len(A) % 2:
                return len(A)/2
            else:
                return -1

        if len(A) >= 0 and len(A) <= 100000:
            for i,val in enumerate(A):
                if i == 0: continue

                right_sum -= val
                left_sum += A[i-1]

                if left_sum  == right_sum:
                    return i

                if i >= len_arr:
                    return -1

            else:
                return -1

    except (ValueError, RuntimeError) as err:

        return err

print solution_6(Ax)


I am working on the file "Demo-Test.ipynb".
The first task is:
solution 4
Can you write Python code for it?

def solution_4(A):
    addition_list = list()
    list_index = 0

    addition_list.append(A[0])
    try:
        if 0 == sum(A[1:]):
            return 0

        elif len(A) == abs(sum(A[:])):
            if len(A) % 2:
                return len(A)/2
            else:
                return -1

        elif len(A) >= 0 and len(A) <= 100000:
            for i in xrange(len(A)):
                #if type(int_in_arr) is int:
                if i == 0: continue
                if sum(A[:i]) == (sum(A[:]) - sum(A[:i+1])):
                    return i

            else:
                raise ValueError("one of the array element: integer out of range", int_in_arr)

        else:
            raise ValueError("array indices out of range ", len(A))

        return -1

    except (ValueError, RuntimeError) as err:

        return err

print solution_4(Ax)


I am working on the file "Demo-pt-br.ipynb".
The first task is:
Entendendo Linhas
Can you write Python code for it?

from geomath.line import Line

Linha = Line()

Linha.create_via_equation("1x+2y+3=0")

Linha.equation()

Linha.create(Point(0,0),Point(4,4))

Linha.equation()


I am working on the file "Demo.ipynb".
The first task is:
Activate logging for Gensim, so we can see that everything is working correctly. Gensim will, for example, complain if no C compiler is installed to let you know that Word2Vec will be awfully slow.
Can you write Python code for it?

import re
import nltk
import os.path as path
from random import shuffle
from gensim.models import Word2Vec


I am working on the file "Demo.ipynb".
The first task is:
Check that the chosen vectors are orthogonal:
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Check the convergence towards the plane with largest eigenvalues.
Can you write Python code for it?

for i in range(2):
    plt.plot(np.log10(np.abs(axs[:,-1,i])),marker='.')
plt.title("log10 of the z coordinate")


I am working on the file "Demo.ipynb".
The first task is:
Compute location in predictive embedding space
Transforms the location in the statistical embedding space into a location of the embedding space of mututal predictivity.
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Creating simple and log return  columns
Can you write Python code for it?

#adding '_' to facilitate handling the column
#sp.rename(columns={'Adj Close':'Adj_Close'}, inplace=True)
sp['p_r'] = sp.Close/sp.Close.shift()-1        #simple returns
sp['l_r'] = np.log(sp.Close/sp.Close.shift())  #log returns
sp.tail()


I am working on the file "Demo.ipynb".
The first task is:
First Digits Test
Let us see if the SPY log retunrs conform to Benford's Law
Can you write Python code for it?

f1d = bf.first_digits(sp.l_r, digs=1, decimals=8) # digs=1 for the first digit (1-9)


I am working on the file "Demo.ipynb".
The first task is:
First Two Digists
Can you write Python code for it?

f2d = bf.first_digits(sp.l_r, digs=2, decimals=8) # Note the parameter digs=2!

f2d.head()

f2d.tail()


I am working on the file "Demo.ipynb".
The first task is:
Flat space: Lorenz equation
Example from DiffMan
This is the equation
\[
(x,y,z)' = (-βx + yz, -σy + σz, -xy + ρy - z)
\]
with the values
\[
σ = 10 \qquad ρ = 28 \qquad β = 8/3
\]
The idea here is to use the translation group, so the infinitesimal vector field is
\[
\xi(x) = \begin{bmatrix} 0 & v(x)\ 0 & 0\end{bmatrix}
\]
where $v(x)\in\mathbf{R}^3$ is the Lorenz vector field above.
Can you write Python code for it?

def lorenz(x, sigma=10, beta=8./3., rho=28):
    y = x[1]
    A = np.array([[-beta, 0, y],
         [0, -sigma, sigma],
        [-y, rho, -1]])
    vf = np.dot(A,x[:3])
    xi = np.zeros([4,4])
    xi[:-1,-1] = vf # translation only
    return xi

xs = [np.array([25.,0,-20,1])]
solve(time_step(0.02)(lorenz), xs, lambda i,x: i > 20/.02)

axs = np.array(xs)

fig = plt.figure(figsize=(10,8))
ax = fig.add_subplot(111, projection='3d')
ax.plot(axs[:,0],axs[:,1],axs[:,2],marker='.')
ax.plot([axs[0,0]],[axs[0,1]],[axs[0,2]],marker='o')
#ax.view_init(90,120)


I am working on the file "Demo.ipynb".
The first task is:
IPython extension
You can also enable wurlitzer as an IPython extension,
so that it always forwards C-level output during execution:
Can you write Python code for it?

%load_ext wurlitzer

for i in range(5):
    time.sleep(1)
    printf("Hello from C, %i!" % i)


I am working on the file "Demo.ipynb".
The first task is:
IPython forwards the Python-level sys.stdout and sys.stderr,
but it leaves the process-level file descriptors that C code will write to untouched.
That means that in a context like this notebook, these functions will print to the terminal, because they are not captured:
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Import NLTK to seperate a document into sentences and import Gensim to actually train the Word2Vec models.
Can you write Python code for it?

pattern = re.compile(r'[\W\d]')

try:
    nltk.data.find("tokenizers/punkt")
except LookupError:
    punkt = nltk.download('punkt')

def init_tokenizer(lang):
    model = 'tokenizers/punkt/{}.pickle'.format(lang.lower())
    return nltk.data.load(model)

def sentences(doc):
    tokenizer = init_tokenizer("german")
    sent = tokenizer.tokenize(doc.strip())
    return sent

def token(sentence):
    letters = pattern.sub(" ", sentence)
    words = letters.lower().split()
    words = [word for word in words if len(word) > 1]
    return words

class FileCorpus:
    def __init__(self, files, encoding='UTF-8'):
        self.files = files
        self.encoding = encoding

    def __iter__(self):
        for file in self.files:
            for doc in open(file, encoding=self.encoding):
                yield doc

    def sentences(self):
        documents = [sentences(document) for document in self]
        t = [token(sentence) for document in documents for sentence in document]
        return t

def build_model(name, coll):
    sentences = FileCorpus(coll).sentences()
    shuffle(sentences)
    model = Word2Vec(sentences, workers=4, iter=10, size=100, window=2, sg=1, hs=1)
    model.save(path.join('models','{}.w2v'.format(name)))
    return model

npd = build_model('NPD', [path.join('data', file)
                          for file in ['NPD.txt', 'NPD_MV.txt', 'NPD_Sachsen.txt']])
spd = build_model('SPD', [path.join('data', file)
                          for file in ['SPD_Inland.txt', 'SPD_International.txt', 'SPD_Parteileben.txt']])
cdu = build_model('CDU', [path.join('data', file)
                          for file in ['CDU.txt', 'CDU_EU.txt', 'CDU_Fraktion.txt']])
fdp = build_model('FDP', [path.join('data', file)
                          for file in ['FDP.txt', 'FDP_Fraktion.txt']])
grüne = build_model('GRÜNE', [path.join('data', file)
                          for file in ['Grüne.txt', 'Grüne_Fraktion.txt']])
linke = build_model('LINKE', [path.join('data', file)
                          for file...


I am working on the file "Demo.ipynb".
The first task is:
Isospectral Manifold: Toda flow
An isospectral flow is an equation of the form
\[
P' = ξ(P)P - Pξ(P)
\]
where $P$ is symmetric and $ξ(P)$ is skew-symmetric.
We implement what is known as the Toda flow:
Can you write Python code for it?

def iso_field(P):
    sk = np.tril(P) - np.triu(P) # skew symmetric
    return sk


I am working on the file "Demo.ipynb".
The first task is:
Kalman filtering

Linear dynamic system, estimation, inference
Two phases:
Prediction
Update
<img src="figure/KF.png" alt="Drawing" style="width: 1000px;"/>

Scalability Issues

Computationally prohibitive for large-scale problem
Store and update covariance matrix ($N^2$) at each Kalman step
Computational cost grows quadratically with $N$
$80$ days for a typical problem size ~1 million

| Resolution       | Low           | Medium      | High             |
| ---------------- |:-------------:|:-----------:|:----------------:|
| State dimension  |N = 3245       | N = 3245 x 4| N = 3245 x 16    |
| Run time     | 1.2 min       | 19 minutes  |  4.4 hours       |
| Storage cost     | 100 MB        | 1331 MB     |  20 G            |
Scalable Kalman filtering

$\mathcal{O}(N)$: a linear computational complexity algorithm
Conventional dimension reduction algorithms (e.g.,PCA) truncates information
Lossless compression for a kernel covariance matrix
Hierarchical matrices approach efficiently explore the structure of a kernel matrix
Can you write Python code for it?

import visualizeCO2 as vCO2
vCO2.scale_barplot()


I am working on the file "Demo.ipynb".
The first task is:
Mantissas
The mantissa is the decimal part of a logarithm. In a Benford data set, the mantissas of the registries' logs are uniformly distributed, such that when ordered,they should form a straight line in the interval [0,1), with slope 1/N, N being the sample size..
Can you write Python code for it?

mant = bf.mantissas(sp.l_r, inform=True, show_plot=True)

mant.hist(bins=30, figsize=(12,5))


I am working on the file "Demo.ipynb".
The first task is:
Note that you must choose the test parameter, since there is one MAD for each test.

First Digit: 1 or 'F1D';

First Two Digits: 2 or 'F2D';

First Three Digits: 3 or 'F3D';

Second Digit: 22 or 'SD';

Last Two Digits: -2 or 'L2D';     # pithonic

Can you write Python code for it?

mad2 = bf.mad(sp.l_r, test=2, decimals=8) # test=2 : MAD for the First Two Digits
mad2

mad3 = bf.mad(sp.l_r, test=3, decimals=8) # test=3 : MAD for the First Three Digits
mad3

mad_sd = bf.mad(sp.l_r, test=22, decimals=8) # test=22 : MAD for the Second Digits
mad_sd

mad_l2d = bf.mad(sp.l_r, test=-2, decimals=8) # test=-2 : MAD for the Last Two Digits
mad_l2d


I am working on the file "Demo.ipynb".
The first task is:
Other Important Parameters

digs: only used in the First Digits function, to tell it which test to run: 1- First Digits; 2- Fist Two Digits; and 3- First Three Digits.

decimals: informs the number of decimal places to consider. Defaluts to 2, for currencies, but I set it to 8 here, since we are dealing with log returns (long floats). If the sequence is of integers, set it to 0. You may also set it to infer if you don't know exactly or if the data has registries with different number of decimal places, and it will treat every registry separately.

sign: tells which portion of the data to consider. pos: only the positive entries; neg: only the negative ones; all: all entries but zeros. Defaults to all.

inform: gives information about the test during its run, like the number of registries analysed, the number of registries discarded according to each test (ie, < 10 for the First Digits), and shows the top Z scores of the resulting DataFrame if confidence is not None.

high_Z: chooses which Z scores to be used when displaying results, according to the confidence level chosen. Defaluts to pos, which will return only values higher than the expexted frequencies; neg will return only values lower than the expexted frequencies; all will return both extremes (positive and negative); and an integer will return the first n entries, positive and negative, regardless of whether Z is higher than the confidence or not.

limit_N: sets a limit to the sample size for the calculation of the Z scores. This may be found useful if the sample is too big, due to the Z test power problem. Defaults to None.

show_plot: draws the test plot. Defaults to True. Note that if confidence is not None, the plot will highlight the bars outside the lower and upper boundaries, regardless of the high_Z value.

MAD and MSE: calculate, respectively, the Mean Absolute Deviation and the Mean Squared Error of the sample, for each test. Defaults to False. Both can be used inside the tests'...
Can you write Python code for it?

mad1 = bf.mad(sp.l_r, test=1, decimals=8) # test=1 : MAD for the First Digits
mad1


I am working on the file "Demo.ipynb".
The first task is:
Part I. The simple, pretrained CNN model
Load CNN model
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Plot deconvolved calcium data as a 2D matrix. Y-axis corresponds to the 110 neurons, X-axis to time (total ca. 180 sec).
Can you write Python code for it?

plt.figure(101)
plt.imshow(np.transpose(Ypredict[indizes,:]), aspect='auto')
plt.gray()
plt.axis('off')
plt.clim(0.2,20)


I am working on the file "Demo.ipynb".
The first task is:
Plot refined results
Can you write Python code for it?

plt.figure(103)
indizes = range(int(windowsize*before_frac),calcium_traceX.shape[0]-int(windowsize*after_frac))
plt.imshow(np.transpose(Ypredict[indizes,:]), aspect='auto')
plt.gray()
plt.clim(0.2,20)
plt.figure(104)
plt.imshow(np.transpose(tracex[indizes,:]), aspect='auto')
plt.jet()
plt.clim(0,8)

plt.figure(105)
neuron_index = 103
timet = np.arange(0,Ypredict.shape[0])/100
plt.plot(timet,Ypredict[:,neuron_index]*0.5)
plt.plot(timet,tracex[:,neuron_index]+9)
plt.xlim(36,55)
plt.ylim(-1.2, 15)
plt.xlabel('Time [sec]')


I am working on the file "Demo.ipynb".
The first task is:
Plot results
Plot calcium data as a 2D matrix. Y-axis corresponds to the 110 neurons, X-axis to time (total ca. 180 sec). <br />
It is necessary to discard the first and last few data points from the calcium traces, since the model cannot make predictions for them.
Can you write Python code for it?

%matplotlib inline
plt.figure(102)
indizes = range(int(windowsize*before_frac),calcium_traceX.shape[0]-int(windowsize*after_frac))
plt.imshow(np.transpose(tracex[indizes,:]), aspect='auto')
plt.jet()
plt.clim(0,8)
plt.axis('off')


I am working on the file "Demo.ipynb".
The first task is:
Plot the three unit vectors of the rotation matrix:
Can you write Python code for it?

fig = plt.figure(figsize=(15,10))
ax = plot_sphere()
for i in range(3):
    ax.plot(axs[:,0,i],axs[:,1,i],axs[:,2,i],lw=2,marker='.')
ax.view_init(45,80)
plt.savefig('so3quad.svg', bbox_inches='tight', pad_inches=-1.5)


I am working on the file "Demo.ipynb".
The first task is:
Plotting routine:
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D
def plot_sphere(ax=None):
    if ax is None:
        ax = plt.gcf().add_subplot(111, projection='3d')
    ax.autoscale(tight=True)
    ax.set_axis_off()
    ax.set_aspect("equal")

    u = np.linspace(0, 2 * np.pi, 100)
    v = np.linspace(0, np.pi, 100).reshape(-1,1)

    x = np.cos(u) * np.sin(v)
    y = np.sin(u) * np.sin(v)
    z = np.cos(v)
    ax.plot_wireframe(x, y, z,  rstride=4, cstride=4, color='k', alpha=.1)

    return ax


I am working on the file "Demo.ipynb".
The first task is:
Preprocess calcium time series
Preprocess data (upsample to 100 Hz, normalize offset/amplitude).<br />
The preprocessing is based on Theis et al., 2016.
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Quick start
Getting some public data, the S&P500 EFT quotes, up until Dec 2016
Can you write Python code for it?

sp = pd.read_csv('data/SPY.csv', index_col='Date', parse_dates=True)


I am working on the file "Demo.ipynb".
The first task is:
Random symmetric matrix as initial condition.
Can you write Python code for it?

#init = np.array([[-1.,1,0],[1,.5,1],[0,1,.5]])
rmat = np.random.randn(20,20)
init = rmat + rmat.T
plt.matshow(init)
plt.savefig('matinit.png', bbox_inches='tight', pad_inches=0)

from homogint.homogint import trans_adjoint
Ps = [init]
dt = .25
solve(time_step(dt)(iso_field), Ps, lambda i,x: i>30/dt, action=trans_adjoint)


I am working on the file "Demo.ipynb".
The first task is:
Results
                                    TRUE                             HIKF

<img src="figure/co2movie.gif" alt="Drawing" style="width: 800px;"/>
Filter Design
By choosing an appropriate $Q/R$ ratio to optimize the filter preformance
Can you write Python code for it?

# theta controls the trade-off between data misfit and variance in final estimates
def plot_CO2maps(theta):
    import param
    CO2 = CO2simulation(param)
    param.theta = (theta,1e-5)
    hikf, x_kf, cov_kf = simCO2.CO2_filter(CO2, param)
    fig_setting = vco2.getImgParam(param)
    vco2.plotCO2map(x_kf,cov_kf,fig_setting)
    plt.show()
    print "Theta                  Variance"
    print " %f              %f" % (theta,np.sum(cov_kf[-1]))

interact(plot_CO2maps,
         theta = FloatSlider(value=1.14, min=1.14e-3, max=1.14e1, step=1));


I am working on the file "Demo.ipynb".
The first task is:
Retrain model in a local environment
First load ground truth data, used for retraining of the model. <br />
Since the datasets of the ground truth or not equally large, it is necessary to use them accordingly for retraining in a weighted manner. To this end, the number of samples of the datasets have to be known.
Can you write Python code for it?

exec(open("elephant/1_load_data.py").read())
dataset_sizes = np.array((740777, 672625, 563966, 165048, 149976, 171687, 875247, 486090, 584103, 697437),dtype=float)
dataset_fraction_to_take = np.min(dataset_sizes)/dataset_sizes


I am working on the file "Demo.ipynb".
The first task is:
Scalable Kalman filtering for Temporal-Spatial Analysis

Seismic monitoring of CO$_2$

State estimation
Time-series data

Methodology

Kalman filter
Scalability

Result
Visit my GitHub page!

Track a CO$_2$ Plume from Sensor Measurements
Objective: monitor a CO$_2$ plume for $5$ days resulted from injecting $300$ tons of CO$_2$ at a depth of $1657m$.
<img src="figure/field_experiment.png" alt="Drawing" style="width: 600px;"/>
* The sensor measures the travel time of a seismic signal from a source to a receiver.

The presence of CO$_2$ slows down the seismic signal and causes travel time delay along a ray path.

Goal: interpret the changes in seismic signals into maps of moving CO$_2$ plume.
Can you write Python code for it?

from IPython.html.widgets import interact, interactive, fixed
from IPython.html.widgets import FloatSlider
from CO2simulation import CO2simulation
def plot_CO2plume(time):
    import param as param
    CO2 = CO2simulation(param)
    x = CO2.extract_state(int(time/3))
    data = CO2.extract_data(int(time/3))
    fig_setting = vco2.getImgParam(param)
    vco2.plotCO2_data_map(x, data, 0, 20, fig_setting)
    plt.show()

interact(plot_CO2plume,
         time = FloatSlider(value=0, min=0, max=120));


I am working on the file "Demo.ipynb".
The first task is:
Set up the solver
We will use the fourth order Rungke–Kutta–Munthe-Kaas method.
It can be described by the following transition functions (see §3 in the paper):
\begin{align}
    \theta_{1,0} &= \frac{1}{2} F_{0} \\
    \theta_{2,0} &= \frac{1}{2}F_1 - \frac{1}{8}[F_{0},F_1]\\
    \theta_{3,0} &= F_2 \\
    \theta_{4,0} &= \frac{1}{6} (F_0 + 2(F_1+F_2) + F_3) - \frac{1}{12} [F_0, F_3]
\end{align}
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
Some things happened:

It printed a DataFrame wiith the significant positive deviations, in descending order of the Z score.

In the plot, to the Benford Expected line, it added upper and lower boundaries, based on the level of confidence by the parameter. Accordingly, it changed the colors of the bars whose proportions fell lower or higher than the drawn boundaries, for better vizualisation.

The confidence parameter takes the follwoing values other than None: 80, 85, 90, 95, 99 99.9, 99.99, 99.999, 99.9999 and 99.99999.
Other tests
We can do all this with the First Three Digits, Second Digit and the Last Two Digits tests too.
Can you write Python code for it?

# First Three Digits Test, now with 99% confidence level
# digs=3 for the first three digits
f3d = bf.first_digits(sp.l_r, digs=3, decimals=8, confidence=99)

# The First Three Digits plot is better seen and zoomed in and out without the inline plotting.
# Try %matplotlib


I am working on the file "Demo.ipynb".
The first task is:
Sphere: quadrature
Example from DiffMan.
We study the solution of the equation $x'(t) = ξ(t)x(t)$, where $x$ is on the sphere, and
\[
ξ(t) = \begin{bmatrix}0 & t & -0.4\cos(t) \ -t & 0 & 0.1t \ 0.4 \cos(t) & -0.1 t & 0\end{bmatrix} \in \mathsf{so}(3)
\]
This is equivalent to consider the problem
\[
x'(t) = ω(t) \times x(t)
\]
with $ω(t) = -(0.1t,0.4\cos(t),t)$.
We use a bit of a trickery here, and use instead the autonomous vector (in block notation):
\[
\zeta(x) = \begin{bmatrix} \xi(t) & 0 &0 \ 0 & 0 & 1 \ 0 & 0 & 0 \end{bmatrix}
\]
This amounts to work with the group $\mathsf{SO(3) \times \mathbf{R}}$ instead.
Can you write Python code for it?

def timedep_field(x):
    """
    Example from Diffman manual.
    """
    J = np.zeros([5,5])
    t = x[-2]
    J[0,1] = t
    J[0,2] = -.4*np.cos(t)
    J[1,2] = .1*t
    J -= J.T
    J[-2,-1] = 1.
    return J

xs = [np.array([0.,0,1,0,1])]
dt = .02
solve(time_step(dt)(timedep_field),xs,lambda i,x:x[-2]>10)
axs = np.array(xs)

def plot2(axs):
    plt.plot(axs[0,0],axs[0,1],'o')
    plt.plot(axs[:,0], axs[:,1],marker='.')

plt.axis('equal')
plot2(axs)

fig = plt.figure(figsize=(15,10))
ax = plot_sphere()
tot = len(xs)
for i,s in enumerate([slice(0,tot//2,None), slice(tot//2,None,None)]):
    ax.plot(axs[s,0],axs[s,1],axs[s,2],lw=2,marker='.',color=['black','blue'][i], alpha=[1.,0.2][i])
ax.view_init(50,-130)
# savefig('quad.pdf')


I am working on the file "Demo.ipynb".
The first task is:
The first_digits function draws the plot (default) with bars fot the digits found frequencies and a line corresponding to the expected Benford proportions.
It also returns a DataFrame object with Counts, Found proportions and Expected values for each digit in the data studied.
Can you write Python code for it?


I am working on the file "Demo.ipynb".
The first task is:
The flow converges towards an invariant subspace. Here it converges towards the subspace containing the two largest eigenvalues:
Can you write Python code for it?

xs[-1]

axs = np.array(xs)
fig = plt.figure(figsize=(15,10))
ax = plot_sphere()
ths = np.linspace(0,2*np.pi,200)
plt.plot(np.cos(ths), np.sin(ths), np.zeros_like(ths))
for i in range(2):
    for j in [0,-1]:
        ax.plot([axs[j,0,i]],[axs[j,1,i]],[axs[j,2,i]],lw=2,marker=['o','D'][j])
        ax.plot([0.,axs[j,0,i]],[0,axs[j,1,i]],[0,axs[j,2,i]],color=['black','red'][j])
    ax.plot(axs[:,0,i],axs[:,1,i],axs[:,2,i],marker='.')
ax.view_init(30,0)
plt.savefig('oja.pdf')


I am working on the file "Demo.ipynb".
The first task is:
The flow does not change the eigenvalues (hence the name isospectral flow)
Can you write Python code for it?

import numpy.linalg as nl
eigenvalues = [np.sort(nl.eigvals(P)) for P in Ps]
aeigenvalues = np.array(eigenvalues)
deig = aeigenvalues - aeigenvalues[0]
plt.plot(deig)
plt.title("eigenvalue drift")

from IPython.html.widgets import interact
def view_matrix(i):
    plt.matshow(Ps[i])
interact(view_matrix, i=(0,len(Ps)-1,1))


I am working on the file "Demo.ipynb".
The first task is:
The flow is a continuous version of the QR algorithm, so it almost converges towards a diagonal matrix (almost because there is no shift and deflations).
Can you write Python code for it?

plt.matshow(Ps[-1])
plt.savefig('matfinal.png', bbox_inches='tight', pad_inches=0)


I am working on the file "Demo.ipynb".
The first task is:
There are also the Second Digit test, and Last Two Digits test, as shown bellow.
Can you write Python code for it?

# Second Digit Test
sd = bf.second_digit(sp.l_r, decimals=8, confidence=95)

# Last Two Digits Test
l2d = bf.last_two_digits(sp.l_r, decimals=8, confidence=90)


I am working on the file "Demo.ipynb".
The first task is:
Train for 5 epochs with changing datasubsets
Retrains the model with subsets of the ground truth dataset. <br />
The subset is selected based on the distance of the respective recordings to the computed location in the predictive embedding space (distancesX).
Can you write Python code for it?

for jjj in range(0,5):
    XX0 = np.empty((0,128,1))
    YY0 = np.empty((0,1))
    for kk in range(0,10):
        dataset_chosen = kk + 1
        datasets_to_train = {}
        IY = datasets == dataset_chosen
        datasets_to_train[dataset_chosen] = neurons[IY]
        verbosity = 0
        exec( open("elephant/3_preprocessing.py").read() )
        X = X[0:int(X.shape[0]*distancesX[kk]*dataset_fraction_to_take[kk]),:,:]
        Y = Y[0:int(Y.shape[0]*distancesX[kk]*dataset_fraction_to_take[kk]),:]
        XX0 = np.concatenate((XX0,X),axis=0)
        YY0 = np.concatenate((YY0,Y),axis=0)

    learning_rate = 0.0033
    model.optimizer.lr.assign(learning_rate)
    model.fit(XX0, YY0, batch_size=batch_size, epochs=1)


I am working on the file "Demo.ipynb".
The first task is:
We choose a random starting point. It amounts to choose two orthogonal vectors of length one.
Can you write Python code for it?

def normalize(x):
    nx = np.sqrt(np.sum(np.square(x)))
    return x/nx

def rand_sphere_point():
    u,v = np.random.rand(2)
    phi = u*2*np.pi
    theta = np.arccos(2*v-1)
    sth = np.sin(theta)
    return np.array([sth*np.cos(phi), sth*np.sin(phi), np.cos(theta)])
r1 = rand_sphere_point()
r1_ = rand_sphere_point()
r2 = normalize(np.cross(r1,r1_))
x0 = np.array([r1,r2]).T


I am working on the file "Demo.ipynb".
The first task is:
We define a simple solver function that we use to solve the examples.
Can you write Python code for it?

def solve(vf,xs,stopping,action=None, maxit=10000):
    "Simple solver with stopping condition. The list xs is modified **in place**."
    for i in range(maxit):
        if stopping(i,xs[-1]):
            break
        xs.append(rkmk4.step(vf, xs[-1], action=action))


I am working on the file "Demo.ipynb".
The first task is:
We solve it with initial condition $x(0) = 1.$, $x'(0) = 0$.
Can you write Python code for it?

x0 = np.array([1.,0,0,1])
xs = [x0]
dt=.05
solve(time_step(dt)(airy_field), xs, stopping=lambda i,x: i>100/dt)

len(xs)

fig = plt.figure(figsize=(15,5))
axs = np.array(xs)
plt.plot(axs[:,-2],axs[:,0])
#plot(axs[1900:,-2],axs[1900:,0],marker='.')


I am working on the file "Demo.ipynb".
The first task is:
and redisplay them if we like:
Can you write Python code for it?

import sys
sys.stdout.write(stdout.read())
sys.stderr.write(stderr.read())


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
If we want a quick man-page style summary, we can call help(spy). We can also help help(spy.<func>) for more information on the function <func>. Since the data that we're looking at is in a different directory, let's specify that now.

Note: This means that you don't need to be in the same directory as your data.
Can you write Python code for it?


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
If we want to use our shiny python scripts, we'll need to import them too.
Can you write Python code for it?

import spinspy as spy
import matpy as mp


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
Perfect, now we have our grid vectors. We can also load in grid information, such as the domain size and limits. This is stored in a class that has a method called `display'. These are illustrated below.
Can you write Python code for it?

data = spy.get_shape()
print(data.Nx,data.Ny,data.Nz)
print('---')
data.display()


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
Plotting

We now have our grid vectors and height field loaded into memory, so it's time to try some plotting! A standard plotting library in python is matplotlib, which replicates many of MATLAB's 2D plotting functions. For example, here we have a pcolor and contour plot.
Can you write Python code for it?

plt.figure(1)
t0 = time.clock()
plt.contour(x,y,q.T) # Transpose for plotting order
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('q');
plt.xlabel('x');
plt.ylabel('y');

plt.figure(2)
t0 = time.clock()
plt.pcolor(x,y,q.transpose());
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('Height Field');
plt.xlabel('x');
plt.ylabel('y');

plt.figure(3)
t0 = time.clock()
plt.pcolormesh(x,y,q.T, cmap='darkjet')
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('Height Field')
plt.xlabel('x')
plt.ylabel('y')
plt.axis('tight')
plt.show()


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
We have q (potential vorticity, this is a QG example). Let's read and plot our initial q field.
Can you write Python code for it?


I am working on the file "Demo/.ipynb_checkpoints/Demo_2d-checkpoint.ipynb".
The first task is:
What the heck did that just do? Let's break it down and look at the inputs.

'h' : This tells the function which variable to load (in this case, h)
0 : This is the index, so we will load h.0.
[0,-1] : The x-range to load. This is shorthand and is equivalent to ':' from MATLAB.
[0,-1] : The y-range

Here we have loaded the entire thing. Alternatively, you could have given a single value or a list of indices. That is, giving 0 would take the first slice, [0,100] takes the first 100 elements, and [0,2,4,6] takes the first, third, fifth, and, seventh elements.
Can you write Python code for it?


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
If we want a quick man-page style summary, we can call help(spy). We can also call help(spy.<func>) for more information on the function <func>. Since the data that we're looking at is in a different directory, let's specify that now.

Note: This means that you don't need to be in the same directory as your data.
Can you write Python code for it?

help(spy.set_path)
print('=======')
print('=======')
print('=======')
help(spy)
spy.set_path('Data/2d')


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
If we want to use our shiny python scripts, we'll need to import them too.
Can you write Python code for it?


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
Note that this is ordered as (Nx,Ny). If we had wanted MATLAB style ordering (Ny,Nx), we could have used the optional argument ordering = 'matlab' (useful for some plotting). The default is ordering = 'natural' (useful because it feels right).

Loading Grids and Simulation Information

To begin, let's load in our grid. Since we aren't dealing with mapped grids, we only need vectors (the default behaviour).
Can you write Python code for it?

x,y = spy.get_grid()
print('The shape of x is {0} and the shape of y is {1}'.format(x.shape,y.shape))


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
Plotting

We now have our grid vectors and potential vorticity loaded into memory, so it's time to try some plotting! A standard plotting library in python is matplotlib, which replicates many of MATLAB's 2D plotting functions. For example, here we have a pcolor and contour plot.
Can you write Python code for it?

q = spy.reader('q', 10)

plt.figure(1)
t0 = time.clock()
plt.contour(x,y,q.T) # Transpose for plotting order
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('q');
plt.xlabel('x');
plt.ylabel('y');

plt.figure(2)
t0 = time.clock()
plt.pcolor(x,y,q.transpose()); # Regrettably, the default colormap is still jet...
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.colorbar()
plt.title('Height Field');
plt.xlabel('x');
plt.ylabel('y');

plt.figure(3)
t0 = time.clock()
plt.pcolormesh(x,y,q.T, cmap='darkjet') # So we made darkjet!
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.colorbar()
plt.title('Height Field')
plt.xlabel('x')
plt.ylabel('y')
plt.axis('tight')
plt.show()


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
Reading Full Files

Now that we have all of our tools, let's start doing! Let's see what we have first.
Can you write Python code for it?


I am working on the file "Demo/Demo_2d.ipynb".
The first task is:
We have q (potential vorticity, this is a QG example). Let's read and plot our initial q field.
Can you write Python code for it?


I am working on the file "Demo/Execute_Algorithms.ipynb".
The first task is:
Compress output files and clean up
Can you write Python code for it?

%%bash

# These files are relatively large and we won't use them to make graphs, so let's delete them.
rm -fv Results_ParamsOptimized/*/Nested_ElapsedTime.tsv
rm -fv Results_ParamsOptimized/*/Nested_Best.tsv
mv Results_ParamsOptimized/diabetes/Nested_Predictions.tsv Results_ParamsOptimized/diabetes/Nested_Predictions.tsv.tmp
rm -fv Results_ParamsOptimized/*/Nested_Predictions.tsv
mv Results_ParamsOptimized/diabetes/Nested_Predictions.tsv.tmp Results_ParamsOptimized/diabetes/Nested_Predictions.tsv
rm -fv Results_FeatureSelection/*/Nested_Predictions.tsv
rm -fv Results_FeatureSelection/*/Nested_*ElapsedTime.tsv
rm -fv Results_FeatureSelection/*/Nested_Best.tsv

rm -rfv Datasets


I am working on the file "Demo/Execute_Algorithms.ipynb".
The first task is:
The Penn Machine Learning Benchmarks (PMLB) repository contains a large number of datasets that can be used to test machine-learning algorithms. We can access this repository using the Python module named pmlb. For demonstration purposes, we will fetch 10 biology-related datasets from PMLB. First, define a list that indicates the unique identifier for each of these datasets.
Can you write Python code for it?

datasets = ['analcatdata_aids',
            'ann-thyroid',
            'breast-cancer',
            'dermatology',
            'diabetes',
            'hepatitis',
            'iris',
            'liver-disorder',
            'molecular-biology_promoters',
            'yeast']


I am working on the file "Demo/download-data.ipynb".
The first task is:
Methods
Can you write Python code for it?

# Methods to pull from google drive

def download_file_from_google_drive(id, destination):
    URL = "https://docs.google.com/uc?export=download"
    session = requests.Session()
    response = session.get(URL, params = { 'id' : id }, stream = True)
    token = get_confirm_token(response)

    if token:
        params = { 'id' : id, 'confirm' : token }
        response = session.get(URL, params = params, stream = True)

    save_response_content(response, destination)

def get_confirm_token(response):
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            return value
    return None

def save_response_content(response, destination):
    CHUNK_SIZE = 32768 #iterate by chunks of data to not loose information
    with open(destination, "wb") as f:
        for chunk in response.iter_content(CHUNK_SIZE):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)


I am working on the file "Demo/download-data.ipynb".
The first task is:
Parameters to pass to Google Drive
Can you write Python code for it?

# download the zip

file_id = '0Bx1nyj-4aTk9RnNnS3JfRDZ6MVU' # the ID to the file on Google Drive
destination = 'public_data.zip'

# pass through the arguements
download_file_from_google_drive(file_id, destination)


I am working on the file "Demo/xbrain-demo-dvid.ipynb".
The first task is:
Display results of new algorithm
Can you write Python code for it?

# show results

print("Vessel Segmentation")
ndp.plot(input_data, vessel_map, slice = 50, alpha = 0.5)

print("Cell Segmentation")
ndp.plot(input_data, cell_map, slice = 50, alpha = 0.5)


I am working on the file "Demo/xbrain-demo-dvid.ipynb".
The first task is:
Display the results of ilastik
Can you write Python code for it?

# pull down the coorisponding matricies
cell_prob_map = probability_maps[:, :, :, 2]
vessel_prob_map = probability_maps[:, :, :, 1]

print("cell_prob_map shape", cell_prob_map.shape)
ndp.plot(cell_prob_map, slice=50, cmap1='jet', alpha=0.5)

print("vessel_prob_map shape", vessel_prob_map.shape)
ndp.plot(vessel_prob_map, slice=50, cmap1='jet')


I am working on the file "Demo/xbrain-demo-dvid.ipynb".
The first task is:
Initialize

Path of files
Parameters for ilastik
Parameters for cell detection
Parameters for blood vessel segmentation
Can you write Python code for it?

# Set folder where data is stored
current_dir = os.path.abspath(os.getcwd())
folder = next(os.walk('.'))[1][0]
#print(os.path.exists(folder)) # testing

# ilastik parameters
classifier_file = folder + '/xbrain_vessel_seg_v7.ilp'
#print(classifier_file)                 # testing
#print(os.path.exists(classifier_file)) # testing

# testing ground truth
image_file = folder + '/V3_imgdata_gt.npy'
#print(image_file)                 # testing
#print(os.path.exists(image_file)) # testing

# Dictate the processing power
ram_size = 4000 # 4000 MB
no_of_threads = 8 # 8 threads

# Cell detection parameters
cell_probability_threshold  = 0.2
stopping_criterion = 0.47
initial_template_size = 18
dilation_size = 8
max_no_cells = 500

# Vessel segmentation parameters
vessel_probability_threshold = .68
dilation_size = 3
minimum_size = 4000


I am working on the file "Demo/xbrain-demo-dvid.ipynb".
The first task is:
Load in the data
Can you write Python code for it?

# load data

dvid = DVIDRemote({ "protocol": "http",
                    "host": "172.19.248.41:8000/",
                  })
chan = dvid.get_channel('a3afee0bf807466c9b7c3b0bbfd1acbd','grayscale')
print(chan)
input_data = dvid.get_cutout(chan,0,[0,2],[0,64],[390,454])#DVID here -- np.load(image_file)

# plot the 50th slice
#ndp.plot(input_data, 50)

# get the shape of the data
print(input_data.shape) # (200, 200, 200)


I am working on the file "Demo/xbrain-demo.ipynb".
The first task is:
Display results of new algorithm
Can you write Python code for it?

# show results

print("Vessel Segmentation")
ndplot(input_data, vessel_map, slice = 50, alpha = 0.5)

print("Cell Segmentation")
ndplot(input_data, cell_map, slice = 50, alpha = 0.5)


I am working on the file "Demo/xbrain-demo.ipynb".
The first task is:
Display the results of ilastik
Can you write Python code for it?

# pull down the coorisponding matricies
cell_prob_map = probability_maps[:, :, :, 2]
vessel_prob_map = probability_maps[:, :, :, 1]

print("cell_prob_map shape", cell_prob_map.shape)
ndplot(cell_prob_map, slice=50, cmap1='jet', alpha=0.5)

print("vessel_prob_map shape", vessel_prob_map.shape)
ndplot(vessel_prob_map, slice=50, cmap1='jet')


I am working on the file "Demo/xbrain-demo.ipynb".
The first task is:
Ingest data and classifer into ilastik
Can you write Python code for it?

# Compute time required for processing
start = time.time()

# Process the data to probability maps
probability_maps = xbrain.classify_pixel(input_data, classifier_file, threads=no_of_threads, ram=ram_size)

end = time.time()
print("\nElapsed time: %f minutes" % ((end - start)/60))


I am working on the file "Demo/xbrain-demo.ipynb".
The first task is:
Initialize

Path of files
Parameters for ilastik
Parameters for cell detection
Parameters for blood vessel segmentation
Can you write Python code for it?


I am working on the file "Demo/xbrain-demo.ipynb".
The first task is:
Load in the data
Can you write Python code for it?

#DVID Data fetch:
dvid = DVIDRemote({
    "protocol": "http",
    "host": "10.194.126.16:8000",
    })

chan = "d108503322e644b7a6853b20285bd811/MaskedImg1"
input_data = dvid.get_cutout(
    dvid.get_channel(chan),0,
    [1400,1600],[1800,2000],[390,590]
    )

# get the shape of the data
print(input_data.shape) # (200, 200, 200)


I am working on the file "Demo_GHQ.ipynb".
The first task is:
Exact integration of polynomials with degree upto 2*deg-1
Can you write Python code for it?

z, w = nppoly.hermite_e.hermegauss(deg=3)
w = w*const
sum(w)

# Let's test on the moments of normal distribution
deg = np.array([2,4,6,8,10,12,14])
moments = [sum(z**2 * w), sum(z**4 * w), sum(z**6 * w), sum(z**8 * w), sum(z**10 * w), sum(z**12 * w), sum(z**14 * w)]
print(moments)

# luckily we know the exact answer: (2*deg-1)!!
scipy.misc.factorial2([1,3,5,7,9,11,13])

# Find out upto which degree integration is correct
deg[np.abs(moments - scipy.misc.factorial2([1,3,5,7,9,11,13])) < 0.1 ]


I am working on the file "Demo_GHQ.ipynb".
The first task is:
Overall GHQ is very accurate for integrating smooth functions
Let's test on Geometric Brownian Motion:
$ S_T = S_0 exp\left(\sigma\sqrt{T} z - \frac12 \sigma^2 T\right)$
Can you write Python code for it?

spot = 100
texp = 2
vol = 0.2

z = np.linspace(-5,5,10)
price = spot * np.exp(vol*np.sqrt(texp)*z - 0.5*vol*vol*texp)
print(price)

# Let's check the expectation of the prices are same as 100 (assuming 0 interest rate)
z, w = nppoly.hermite_e.hermegauss(deg=10)
w = w*const
price = spot * np.exp(vol*np.sqrt(texp)*z - 0.5*vol*vol*texp)
price_mean = sum(price * w)
price_mean - 100

plt.plot(price, w, 'o-')
plt.grid()
plt.show()


I am working on the file "Demo_of_RISE_for_slides_with_Jupyter_notebooks__Julia.ipynb".
The first task is:
And Markdown can include raw HTML
This is a centered span, colored in green.
Iframes are disabled by default, but by using the IPython internals we can include let say a YouTube video:
Can you write Python code for it?


I am working on the file "Demo_of_RISE_for_slides_with_Jupyter_notebooks__Julia.ipynb".
The first task is:
More demo of Markdown code
Lists

Unordered
lists
are easy.

And

and ordered also ! Just
start lines by 1., 2. etc
or simply 1., 1., ...

Images
With a HTML <img/> tag or the ![alt](url) Markdown code:
<img width="100" src="agreg/images/dooku.jpg"/>

Some code
Can you write Python code for it?

# https://gist.github.com/dm-wyncode/55823165c104717ca49863fc526d1354
"""Embed a YouTube video via its embed url into a notebook."""
from functools import partial

from IPython.display import display, IFrame

width, height = (560, 315, )

def _iframe_attrs(embed_url):
    """Get IFrame args."""
    return (
        ('src', 'width', 'height'),
        (embed_url, width, height, ),
    )

def _get_args(embed_url):
    """Get args for type to create a class."""
    iframe = dict(zip(*_iframe_attrs(embed_url)))
    attrs = {
        'display': partial(display, IFrame(**iframe)),
    }
    return ('YouTubeVideo', (object, ), attrs, )

def youtube_video(embed_url):
    """Embed YouTube video into a notebook.

    Place this module into the same directory as the notebook.

    >>> from embed import youtube_video
    >>> youtube_video(url).display()
    """
    YouTubeVideo = type(*_get_args(embed_url)) # make a class
    return YouTubeVideo() # return an object


I am working on the file "Demo_of_RISE_for_slides_with_Jupyter_notebooks__Python.ipynb".
The first task is:
More demo of Markdown code
Lists

Unordered
lists
are easy.

And

and ordered also ! Just
start lines by 1., 2. etc
or simply 1., 1., ...

Images
With a HTML <img/> tag or the ![alt](url) Markdown code:
<img width="100" src="agreg/images/dooku.jpg"/>
Can you write Python code for it?


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
1.2 First goal: pretty print in ASCII text
Our first task will be to implement a small function, or an overload of the numpy.polynomial.Polynomial class to be able to pretty-print a Polynomial object nicely.

Note: more details, along with a documentation, some examples and a fully pylint-compatible code are available on this Bitbucket snippet: bitbucket.org/snippets/lbesson/j6dpz.

The function prettyprintPolynomial defined below implements a natural strategy to print a polynomial: it prints the coefficients, as a_i * X**i, in increasing order.
This function takes care of all the special cases:

it removes a trailing .0 if one coefficient is an integer,
it displays the coefficient between ( and ) if it is negative,
it only displays the non-zero coefficients,
if one coefficient is 1, no need to display it.
Can you write Python code for it?

def prettyprintPolynomial(p):
    """ Small function to print nicely the polynomial p as we write it in maths, in ASCII text."""
    coefs = p.coef  # List of coefficient, sorted by increasing degrees
    res = ""  # The resulting string
    for i, a in enumerate(coefs):
        if int(a) == a:  # Remove the trailing .0
            a = int(a)
        if i == 0:  # First coefficient, no need for X
            if a > 0:
                res += "{a} + ".format(a=a)
            elif a < 0:  # Negative a is printed like (a)
                res += "({a}) + ".format(a=a)
            # a = 0 is not displayed
        elif i == 1:  # Second coefficient, only X and not X**i
            if a == 1:  # a = 1 does not need to be displayed
                res += "X + "
            elif a > 0:
                res += "{a} * X + ".format(a=a)
            elif a < 0:
                res += "({a}) * X + ".format(a=a)
        else:
            if a == 1:
                res += "X**{i} + ".format(i=i)
            elif a > 0:
                res += "{a} * X**{i} + ".format(a=a, i=i)
            elif a < 0:
                res += "({a}) * X**{i} + ".format(a=a, i=i)
    return res[:-3] if res else ""


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
Allright! It starts to look like what we wanted!
Let's try with a bigger polynomial, as we did before:
Can you write Python code for it?


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
But we want the $\LaTeX{}$ code to be pretty-printed by IPython, not just displayed like this.
For this, the internal function Latex from the module IPython.display is required:
Can you write Python code for it?

from IPython.display import Latex

print("X =")
Latex(Polynomial_to_LaTeX(X))

print("Q(X) =")
Latex(Polynomial_to_LaTeX(Q))


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
Once our special printer has been loaded, all polynomials will be represented by their mathematical form instead (as $\LaTeX{}$ code displayed with MathJax):
Can you write Python code for it?


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
Way nicer! Yay!

1.4 A bonus for the end: add this pretty-printer as the default one in IPython:
This manipulation is showed in IPython's documentation.
But we can configure IPython to do this automatically for us as follows.
We hook into the IPython display system and instruct it to use Polynomial_to_LaTeX for the latex mimetype, when encountering objects of the Polynomial type defined in the numpy.polynomial.polynomial module:
Can you write Python code for it?

ip = get_ipython()
latex_formatter = ip.display_formatter.formatters['text/latex']
latex_formatter.for_type_by_name('numpy.polynomial.polynomial',
                                 'Polynomial', Polynomial_to_LaTeX)


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
We can check its behavior on some small polynomials:
Can you write Python code for it?

print("X =", prettyprintPolynomial(X))
print("Q(X) =", prettyprintPolynomial(Q))

Q3 = -1 - 2*X - 17*X**3
print("Q_3(X) =", prettyprintPolynomial(Q3))
print("- Q_3(X) =", prettyprintPolynomial(-Q3))


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
We can create a more complicated polynomial $Q_4(X) = (1 + 2 X + 17 X^3) ^ {12}$ and check that it is also nicely printed:
Can you write Python code for it?

Q4 = (1 + 2*X + 17*X**3) ** 12
print("Q_4(X) =", prettyprintPolynomial(Q4))


I am working on the file "Demonstration of numpy.polynomial.Polynomial and nice display with LaTeX and MathJax (python3).ipynb".
The first task is:
We can quickly try the same examples as before:
Can you write Python code for it?

print("X =", Polynomial_to_LaTeX(X))
print("Q(X) =", Polynomial_to_LaTeX(Q))


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
And for functions:
Can you write Python code for it?

sum_ocaml_1 = %ocaml let sum : (int list -> int) = List.fold_left (+) 0 in sum

print(sum_ocaml_1, type(sum_ocaml_1))

sum_ocaml_1 ([1, 2, 3, 4, 5])  # 15


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Another example:
Can you write Python code for it?

smaller = %ocaml fun (x: int) (y: int) -> x < y
print(smaller)
help(smaller)

smaller_poly = %ocaml fun (x: 'a) (y: 'a) -> x < y
print(smaller_poly)
help(smaller_poly)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
But it does nothing else than opening a sub-process, running ocaml command, feeding it the content of the cell, and then exiting.

Cons: It's not so pretty, as it prints all this startup messages. I thought we could silent them, like python -q (quiet) option... but we can't!
Pros: it shows the value of each line, so it can be used to quickly check something, if needed...

More full %%ocaml cells
Can you write Python code for it?

%%ocaml
let sum : (int list -> int) = List.fold_left (+) 0 in
let a_list (n:int) : int list = Array.to_list (Array.init n (fun i -> i*i+30)) in
for n = 300000 to 300010 do
    Format.printf "\nList of size %2.i had sum = %4.i.%!" n (sum (a_list n));
done;;


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Docstring: <no docstring>
File:      /usr/local/lib/python3.6/dist-packages/ocaml/__init__.py

More remarks
Note that there blog post says that using the opttoploop could be used to compile the OCaml to a faster version (native code), but do not document about this.

Note that with the toploop module, the OCaml code is evaluated by compiling to bytecode which is not optimal, switching to the opttoploop module that generates native code should make it even faster.
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
From standard library
What about Sets, mapped to set?
What about HashTbl, mapped to dict?
And Stack, Queue, etc?
TODO: left as an exercise for the reader.
Map - fail
Can you write Python code for it?

%%ocaml

module IntPairs = struct
    type t = int * int
    let compare (x0,y0) (x1,y1) =
        match Stdlib.compare x0 x1 with
        0 -> Stdlib.compare y0 y1
        | c -> c
end

module PairsMap = Map.Make(IntPairs)

let m = PairsMap.(empty |> add (0,1) "hello" |> add (1,0) "world")

(* not an expression, not usable in %ocaml magic *)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Indeed, in most situations, the tuple form is just not "OCaml"esque, and tedious to use, and does not allow partial application!
Can you write Python code for it?

%%ocaml
(* this is NOT advised *)
let product_curry (xyz : (int * int * int)) : int =
    let x, y, z = xyz in
    x * y * z
in
let x = 1 and y = 2 in
let partial_product = product_curry x y in
let z = 10 in
Format.printf "With x = %i, y = %i, and partial_product applied to z = %i, we got %i." x y z (partial_product z);;


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
It fails, but takes 100% CPU and freezes. But in Python we can do it:
Can you write Python code for it?

list1 = [0]
list2 = [1]
list1.append(list2)
list2.append(list1)
print(list1)
print(list2)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
It fails:
```
SyntaxError: ocaml evaluation error on lines 1:11 to 1:15
Error:

1: let out = (type TODO | DONE);;
Syntax error: operator expected.
```
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
It would be a great project to show to students studying Python and C and OCaml : this tiny IPython magic makes the link between Python and OCaml (one way) using OCaml compiled as a dynamic extension to CPython 😍 !
Reference and missing documentation of ocaml Pypi package
This Jupyter notebook uses the ocaml v0.0.11 package from Pypi.
There is no documentation, I asked the author, who works at JaneStreet and he redirected me to this blog post by JaneStreet.
Apparently, being professional developers doesn't mean they restrain themselves from shipping unfinished and undocumented packages to Pypi. Okay... Why? It seems highly unprofessional! I understand being in a hurry, but then just don't publish to Pypi, and let user install it using pip but from a Git repository:
bash
$ pip install git+https://github.com/<Username>/<Projects>
Their blog post states

Note that this package is not currently very well polished but it should give some ideas of what can be done through this Python-OCaml integration.

But come on, writing a short README or description in the setup.py is the least that they should do...
Iframe showing the https://pypi.org/project/ocaml/ page:
<iframe src="http://pypi.org/project/ocaml/>
I don't do that!  See my Pypi packages on https://pypi.org/user/Naereen/:

lempel-ziv-complexity (Last released on Dec 19, 2019): Fast implementation of the Lempel-Ziv complexity function.
SMPyBandits (Last released on Oct 25, 2019): SMPyBandits: Open-Source Python package for Single- and Multi-Players multi-armed Bandits algorithms.
parcoursup (Last released on Nov 14, 2018): ParcourSup.py : un clone en Python 3 de ParcoursSup, écrit à but didactique.
generatewordcloud (Last released on Oct 16, 2016): A simple Python (2 or 3) script to generate a PNG word-cloud image from a bunch of text files. Based on word_cloud.
ansicolortags (Last released on Jul 2, 2016): ansicolortags brings a simple and powerful way to use colours in a terminal application with Python 2 and 3.

Let's show the current version...
Can you write Python code for it?

%load_ext watermark

%watermark -v -p ocaml


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Just to check the experimental values of $\alpha$ and $\beta$ in my claim above, let's use numpy.polyfit function:
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Not that it has been possible, since a long time, to call an OCaml toplevel like this!
Can you write Python code for it?

%%script ocaml
print_endline "Hello world from OCaml running in Jupyter (from IPython)!";;


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Note that this limitation was explained:

Note that this makes the two OCaml values [Some None] and [None] indistinguishable on the Python side as both are represented using None.

Functions with labels - labels get erased
Can you write Python code for it?

# val fold_left : f:('a -> 'b -> 'a) -> init:'a -> 'b list -> 'a
fold_left = %ocaml ListLabels.fold_left

print(fold_left, type(fold_left))
help(fold_left:)

fold_left(lambda x: lambda y: x + y)(0)([1, 2, 3, 4, 5])


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Now for a "large" array, let's use IPython %timeit magic for very quick benchmarking.

Science is about making hypotheses, designing experiments to check them, and conclude.
My hypothesis here is that the OCaml version will be between 10 to 50 slower than the Python one (and Numpy version is 50-200 faster than Python).
Can you write Python code for it?

sizes = [100, 1000, 10000, 100000, 1000000, 10000000]
print(f"Comparing time of python_sum and ocaml_sun :")
for size in sizes:
    print(f"\n- For size = {size}:")
    X = list(np.random.randn(size))

    print("\tFor python sum: ", end='')
    %timeit python_sum(X)

    assert np.isclose(python_sum(X), ocaml_sum(X))
    print("\tFor OCaml sum: ", end='')
    %timeit ocaml_sum(X)

    assert np.isclose(python_sum(X), numpy_sum(X))
    print("\tFor numpy.sum: ", end='')
    %timeit numpy_sum(X)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Of course it fails!
But it could be translated to Python dictionaries.
Exceptions - fail
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Option type
Can you write Python code for it?

none = %ocaml None
print(none, type(none))

some_int = %ocaml Some 42
print(some_int, type(some_int))

# instinguishable from None, so that's weird!
some_None = %ocaml Some None
print(some_None, type(some_None))


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Or simply
Can you write Python code for it?

sum_ocaml_2 = %ocaml List.fold_left (+) 0

sum_ocaml_2 ([1, 2, 3, 4, 5])  # 15


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Polymorphic function - works!
Can you write Python code for it?

cons = %ocaml fun hd tl -> hd :: tl
print(cons, type(cons))

cons(10)([20, 30])

cons(1.0)([2.0, 30])


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Recursive list?
Can you write Python code for it?

%%ocaml
let rec list1 = 0 :: list2
and list2 = 1 :: list1
in
Format.printf "%i -> %i -> %i -> %i ...%!" (List.hd list1) (List.hd list2) (List.hd (List.tl list2)) (List.hd (List.tl list1));;

# don't run
if False:
    list1, list2 = %ocaml let rec list1 = 0 :: list2 and list2 = 1 :: list1 in (list1, list2)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
So 'a list, 'a array and 'a * 'b * .. heterogeneous tuples get perfectly mapped from OCaml values to Python native values!
But it's not perfect, as for instance OCaml has a char type (similar to the one in C) but Python only has strings, so this'll fail:
ValueError: ocaml error (Failure"unknown type char")
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
So booleans, strings, integers and floats get perfectly mapped from OCaml values to Python native values.
Can you write Python code for it?

l = %ocaml [1, 3, 5]
print(type(l), l)

a = %ocaml [|2; 4; 6|]
print(type(a), a)

t = %ocaml (23, 02, 2021)
print(type(t), t)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
So we can't use it as a classical 3-arguments Python function:
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Stack (or any module using a custom type) - fail
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
That's it for this feature, it's cool and interesting.
Some questions
Can we share variables between two consecutive full cells?
In the %ocaml mode, nothing can be shared from the OCaml side, as it's just an expression, let's check:
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
The only information we have on this function is the OCaml signature, in its docstring:
Can you write Python code for it?


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Using OCaml returned values from Python
This package allows to use dynamically defined OCaml functions from Python, the same way it can be done for other languages lke Julia or R (see this blog post if you never saw these possibilities, or this one).
For instance:
Can you write Python code for it?

b = %ocaml true
print(type(b), b)

s = %ocaml "OK ?"
print(type(s), s)

i = %ocaml 2021
print(type(i), i)

f = %ocaml 2.99792458
print(type(f), f)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Well that was some long explanation, but now comes the magic!
If you use %ocaml to get in Python the values returned from OCaml, then functions are Curryed function!
Can you write Python code for it?

product_curry = %ocaml let product_curry (x:int) (y:int) (z:int) : int = x * y * z in product_curry


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
What about user defined types?
Can you write Python code for it?

%%ocaml
type state = TODO | DONE | Unknown of string;;

let print_state (s:state) =
    match s with
    | TODO -> Format.printf "TODO%!"
    | DONE -> Format.printf "DONE%!"
    | Unknown status -> Format.printf "%s%!" status
;;

print_state TODO;;


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
With the Python standard library, it's possible to use functools.partial to obtain partially evaluated functions, which can be viewed as a limited support of Curryed function.
Can you write Python code for it?

import functools

partial_product = functools.partial(product3values, 1, 2)

z = 10
print(f"With x = {x}, y = {y}, and {partial_product} applied to z = {z}, we got {partial_product(z)}")


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
Woooo, somehow OCaml accepted a polymorphic list at some point?
Can you write Python code for it?

head, tail = %ocaml List.hd, List.tl

a_list = [1, 2, 3]
a_list.append(a_list)
head(a_list), tail(a_list)


I am working on the file "Demonstration_of_Pypi_package_OCaml_for_polyglot_notebook_02-2021.ipynb".
The first task is:
😮 Curryed functions!
Imagine you define this function in math:
$p : (x,y,z) \mapsto x * y * z$, on $\mathbb{N}^3 \to \mathbb{N}$.
Then the Curryed form states that it is equivalent to
$p' : x \mapsto y \mapsto z \mapsto x * y * z$, informally defined on $\mathbb{N} \to \mathbb{N} \to \mathbb{N} \to \mathbb{N}$.
So for instance if $x=1$ and $y=2$, $p'(x)(y)$ is $z \mapsto 1 * 2 * z$, which is also $z \mapsto p(1, 2, z)$.
In Python function, that would be this function:
Can you write Python code for it?

def product3values(x, y, z):
    return x * y * z


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
A more complicated function
and its derivative
Can you write Python code for it?

def g(x, m=np):
    y = x
    for i in range(2):
        # a = m.log(y)
        # b = y ** m.pi
        # c = m.cos(b)
        # y = c * a
        y = m.cos(y**m.pi) * m.log(y)
    return y

gexpr = g(x, m=sympy)
gexpr

sympy.diff(gexpr, x)


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
And sympy can evaluate the formula itself
In Mathematica, this is like adding a //N to the end of an expression
Can you write Python code for it?


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
Example with linear function
Can you write Python code for it?

n = 20
c = np.random.randn(n)
y = np.random.randn(n)

# SRB changing example a bit
# Let h(y) = dot(y,c), then grad h = c
h = lambda y : jnp.vdot(y,c)

print("Gradient via JAX AD is")
print( jax.grad(h)(y) )
# Alternatively,
# jax.grad(jmp.vdot)(y,c) will pass in both y and c to the function
#     but only differentiation with respect to the first input (in this case, y)

print("Gradient worked out by hand is")
print( c )


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
Hand-coding derivatives
The size of these expressions grow exponentially in the number of loop iterations, yet one can write efficient code for computing the derivative by hand.  We use the variational notation
$$ \operatorname{d} f = f'(x) \operatorname{d} x $$
which allows us to break a large computation into simple pieces that we can compute incrementally, instead of trying to build up expressions for complicated functions.  That is, we can differentiate a composition $h(g(f(x)))$ as
\begin{align}
  \operatorname{d} h &= h' \operatorname{d} g \
  \operatorname{d} g &= g' \operatorname{d} f \
  \operatorname{d} f &= f' \operatorname{d} x.
\end{align}
Consider our example above.
Can you write Python code for it?

def gprime(x):
    y = x
    dy = 1
    for i in range(2):
        a = np.log(y)
        da = 1/y * dy
        b = y ** np.pi
        db = np.pi * y ** (np.pi - 1) * dy
        c = np.cos(b)
        dc = -np.sin(b) * db
        y = c * a
        dy = dc * a + c * da
    return y, dy

print('by hand', gprime(1.9))


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
If we use forward mode, we can only compute one direction at a time, effectively
$$ \left(\nabla_{\mathbf x} f\right) \cdot \operatorname d x $$
for one value of the vector $\operatorname d x$ at a time.
We can compute the full gradient by choosing $\operatorname d x$ to be each column of the identity.
Can you write Python code for it?

def dot_x(c, x, dx):
    """Compute derivative in direction dx"""
    n = len(c)
    dsum = 0
    for i in range(n):
        dsum += c[i] * dx[i]
    return dsum

def grad_dot(c, x):
    n = len(c)
    I = np.eye(n)
    grad = np.zeros(n)
    for j in range(n):
        dx = I[:,j]
        grad[j] = dot_x(c, x, dx)
    return grad

grad_dot(c, x)


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
Software

Algorithmic differentiation (AD) software has been around for over 40 years
There are two classical approaches
Source transformation: AD tool emits Fortran (or C, etc.) code, which is compiled by a normal compiler
Operator overleading: each basic operation is overloaded to transform objects holding values + derivatives
Source transformation is usually more efficient, retaining loop structure, etc.
Implementations tend to have poor ergonomics, odd restrictions on use, poor composition.
Vectorization has been poor with most classical tools.
AD implementations have come a long way in the past few years (despite the math being old)
Just-in-time compilation and extensive software engineering
Exemplars:
JAX for Python
Zygote.jl for Julia
AD is great within its domain, but is still intrusive (especially for multi-language projects, languages with poor AD tooling, etc.).  Even in JAX, you'll see various constraints, such as that you can't in-place update an array.
Can you write Python code for it?


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
Symbolic differentiation
We've been differentiating basic mathematical functions, for which there is a formula for the derivative.
Symbolic differentiation is a tool that can compute those expressions (and generate code to evaluate the expressions numerically).
Can you write Python code for it?

import sympy
from sympy.abc import x

f = sympy.cos(x**sympy.pi) * sympy.log(x)
f

sympy.diff(f, x)


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
This code is pretty mechanical to write
It's hard to maintain as you add new features
It's hard to debug
You can test using finite differencing
You can take apart pieces for unit testing and/or debugging
If you know you'll be writing this sort of code, plan ahead!

Variational notation is handy (an example)
We'll differentiate the expression
$$ I = A^{-1} A $$
applying the product rule
$$ 0 = A^{-1} (\operatorname dA) + (\operatorname dA^{-1}) A $$
and collect terms
$$ \operatorname dA^{-1} = - A^{-1} (\operatorname dA) A^{-1}. $$
This expression for the derivative $\operatorname d A^{-1}$ in direction $\operatorname d A$ is useful when differentiating algorithmn that involve linear algebra.
Reverse-mode
What we've done above is called "forward mode", and amounts to placing the parentheses in the chain rule like
$$ \operatorname d h = \frac{dh}{dg} \left(\frac{dg}{df} \left(\frac{df}{dx} \operatorname d x \right) \right) .$$
The expression means the same thing if we rearrange the parentheses,
$$ \operatorname d h = \left( \left( \left( \frac{dh}{dg} \right) \frac{dg}{df} \right) \frac{df}{dx} \right) \operatorname d x .$$
Can you write Python code for it?

def gprime_rev(x):
    # First compute all the values by going through the iteration forwards
    # I'm unrolling two iterations here for clarity ("static single assignment" form)
    # It is possible to write code that keeps the loop structure.
    a1 = np.log(x)
    b1 = x ** np.pi
    c1 = np.cos(b1)
    y1 = c1 * a1
    a2 = np.log(y1)
    b2 = y1 ** np.pi
    c2 = np.cos(b2)
    y = c2 * a2 # Result
    # Now go backwards computing dy/d_ for each variable
    y_ = 1
    y_c2 = y_ * a2
    y_a2 = c2 * y_
    y_b2 = -y_c2 * np.sin(b2) # dy/db2 = dy/dc2 dc2/db2
    y_y1 = y_b2 * np.pi * y1 ** (np.pi - 1) + y_a2 / y1
    y_c1 = y_y1 * a1
    y_a1 = c1 * y_y1
    y_b1 = -y_c1 * np.sin(b1)
    y_x = y_b1 * np.pi * x ** (np.pi - 1) + y_a1 / x
    return y, y_x

print('forward', gprime(1.9))
print('reverse', gprime_rev(1.9))


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
We get the same values in only $O(n)$ work!
The astute reader may recall that we already worked out this case,
$$ \frac{\partial \mathbf c^T \mathbf x}{\partial \mathbf x} = \mathbf c^T .$$

Shape of the gradient (Jacobian)
Suppose we have a vector-valued function of vector-valued input, $\mathbf f(\mathbf x)$ where $\mathbf f$ has length $m$ and $\mathbf x$ has length $n$.
* The gradient (Jacobian) matrix $J = \nabla_{\mathbf x} \mathbf f$ has shape $m\times n$.
* Usually in optimization, $m=1$ because we only have one objective
* If $m\ll n$ then finite differencing and forward-mode differentiation will be much more expensive than reverse-mode differentiation
  * Find a way to use reverse-mode!
* If $m \approx n$ then either is about as efficient, but forward-mode is simpler.
* If $m \gg n$ then forward-mode is the ticket.
* In real computations, there may be expensive stages that have lower dimension inputs or outputs, in which case those can be captured. An example is
$$ \mathbf f(\mathbf x) = \mathbf q \sigma(\mathbf q^T \mathbf x) $$
where $\sigma$ is an expensive nonlinear function.
The Jacobian $J = \nabla_{\mathbf x} \mathbf f$ is a square matrix, but naive forward- and reverse-mode would both require $n$ evaluations of $\sigma$.
Since $\sigma$ is a scalar-valued function of a scalar argument, $\sigma'(\mathbf q^T \mathbf x)$ is just one number, and thus $J = (\sigma') \mathbf q \mathbf q^T$ is readily available (and you know it's rank-1 so don't need to store all $n^2$ entries). Models of this sort show up frequently in physical modeling.
Algorithmic (automatic) differentiation
Next, we'll consider ways to have libraries/compilers generate by-hand code such as we see above.
We'll use the JAX library, which offers differentiation of NumPy computations (and offload to GPUs, which we won't use now).
Uncomment the line below if you need to install jax and jaxlib.
Can you write Python code for it?

# ! pip install jax jaxlib

import jax
import jax.numpy as jnp

# def g_jax(x):
#     """Same function as before, but using jnp in place of np."""
#     y = x
#     for i in range(2):
#         y = jnp.cos(y**jnp.pi) * jnp.log(y)
#     return y

g_jax = lambda x : g(x, m=jnp)

gprime_jax = jax.grad(g_jax)
print(gprime_jax(1.9))
print(gprime(1.9)[1])


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
You can even ask sympy to give you a formula you can evaluate in a coding language
Can you write Python code for it?

sympy.ccode(f, 'y')

sympy.fcode(f, 'y')


I am working on the file "Demos/AutomaticDifferentiation.ipynb".
The first task is:
another example
Here, jax isn't super fast (probably because it has to unroll the loop) but it's more accurate.  The sympy version uses an unstable formula and doesn't have good accuracy (it returns an answer of 0, which is incorrect)
Can you write Python code for it?

m = 30
t = np.random.randn(m)

def f(x):
  y = 1
  for i in range(m):
    y *= x - t[i]
  return y

f(x)

x0 = t[0] + 1e-15

sympy.diff(f(x), x).evalf(25,subs={x: x0})

jax.grad(f)(x0)

%%timeit
sympy.diff(f(x), x).evalf(15,subs={x: x0})

%%timeit
jax.grad(f)(x0)


I am working on the file "Demos/CVX_demo/cvxpy_intro.ipynb".
The first task is:
Example: $\ell_1$-norm minimization
Consider the basis pursuit problem
$$\begin{array}{cc} \text{minimize}& \|x\|_1\\text{subject to} & Ax=y.\end{array}$$
This is a least $\ell_1$-norm problem that will hopefully yield a sparse solution $x$.
We now have an objective, $\|x\|_1$, and an equality constraint $Ax=y$.
Can you write Python code for it?

# make a bogus sparse solution and RHS
m = 200; n = 100;
A = np.random.randn(m,n)
_x = np.zeros((n,1))  # just using this notation to show that this is something we're going to pretend we don't actually have
# _x = np.zeros(n) # better, fewer headaches later, but adjust line 8 too
_k = 10
_I = np.random.permutation(n)[0:_k]
_x[_I] = np.random.randn(_k,1)
y = np.dot(A,_x)  # this is (200,1), as is A.dot(_x)

# Here's an essential step, change from (200,1) to (200,) size
#  if we defined _x=np.zeros((n,1))
y = y.ravel()  # https://www.geeksforgeeks.org/differences-flatten-ravel-numpy/

x = cvx.Variable(n)   # a bit like sympy.  Shape is (n,) not (n,1)

# Even though the cvx.norm function behaves very similarly to
# the np.linalg.norm function, we CANNOT use the np.linalg.norm
# function on CVXPY objects.  If we do, we'll probably get a strange
# error message.
obj = cvx.Minimize(cvx.norm(x,1))

# specify a list of constraints
# constraints = [ A*x == y ]  # A*x is (200,), as is A@x.  This is OK
constraints = [ A@x == y ]

# constraints = [ A.dot(x) == y ] # No, not OK. This is (200,100). CVXPY issue.

# specify and solve the problem
prob = cvx.Problem(obj, constraints)
prob.solve(verbose=True) # let's see the underlying solver's output

print("Problem status: ", prob.status)
print("Optimal value:  ", prob.value)

print("True nonzero inds:      ", sorted(_I))
print("Recovered nonzero inds: ", sorted(np.where(abs(x.value) > 1e-14)[0]))
# Note: we cannot access "x", we need to do "x.value"
#   (also, turn _x to right shape)
err = np.linalg.norm(x.value -_x.ravel())
print(f'Norm of error, ||x-x_est|| is {err:e}')


I am working on the file "Demos/CVX_demo/cvxpy_intro.ipynb".
The first task is:
Example: Relaxation of Boolean LP
Consider the Boolean linear program
$$\begin{array}{cl} \text{minimize} & c^Tx\\text{subject to} & Ax \preceq b\ & x_i \in{0,1}, \quad i=1,...,n.\end{array}$$
Note: the generalized inequality $\preceq$ is just element-wise $\le$ on vectors.
This is not a convex problem, but we can relax it to a linear program and hope that a solution to the relaxed, convex problem is "close" to a solution to the original Boolean LP.  A relaxation of the Boolean LP is the following LP:
$$\begin{array}{cl} \text{minimize} & c^Tx\\text{subject to} & Ax \preceq b\ & \mathbf{0} \preceq x \preceq \mathbf{1}.\end{array}$$
The relaxed solution $x^\text{rlx}$ can be used to guess a Boolean point $\hat{x}$ by rounding based on a threshold $t\in[0,1]$:
$$ \hat{x}_i = \left{\begin{array}{cc} 1 & x_i^\text{rlx} \ge t\0 & \text{otherwise,}\end{array}\right. $$
for $i=1,...,n$.  However, the Boolean point $\hat{x}$ might not satisfy $Ax\preceq b$ (i.e., $\hat{x}$ might be infeasible).
From Boyd and Vandenberghe:

You can think of $x_i$ as a job we either accept or decline, and $-c_i$ as the (positive) revenue we generate if we accept job $i$.  We can think of $Ax\preceq b$ as a set of limits on $m$ resources.  $A_{ij}$, which is positive, is the amount of resource $i$ consumed if we accept job $j$; $b_i$, which is positive, is the amount of recourse $i$ available.
Can you write Python code for it?

m = 300; n = 100;
A = np.random.rand(m,n)
# b = A.dot(np.ones((n,1)))/2.
# c = -np.random.rand(n,1)
b = A.dot(np.ones((n)))/2.
c = -np.random.rand(n)

x_rlx = cvx.Variable(n)
obj = cvx.Minimize(c.T*x_rlx)
constraints = [ A@x_rlx <= b,
                0 <= x_rlx,
                x_rlx <= 1 ]

prob = cvx.Problem(obj, constraints)
prob.solve()

print("Problem status: ", prob.status)
print("Optimal value:  ", prob.value)

plt.hist(x_rlx.value)
plt.xlabel('x_rlx'); plt.ylabel('Count')
plt.title('Histogram of elements of x_rlx')
plt.show()


I am working on the file "Demos/CVX_demo/tutorialSolutions.ipynb".
The first task is:
Problem 2
Can you write Python code for it?

x = cvx.Variable(10)

obj = cvx.Minimize(cvx.norm(x)**2) # cvxpy objects implement the standard python ops
constraints = [cvx.norm(A*x-y) <= 0.1]

prob = cvx.Problem(obj, constraints)
prob.solve(abstol=tol,reltol=tol)

print_status(prob, x)


I am working on the file "Demos/CVX_demo/tutorialSolutions.ipynb".
The first task is:
Problem 4
Can you write Python code for it?

def get_problem1_dual_value():
  obj = cvx.Minimize(cvx.norm(x))
  constraints = [cvx.norm(A*x-y) <= 0.1]

  prob = cvx.Problem(obj, constraints)
  prob.solve(abstol=tol,reltol=tol)

  return constraints[0].dual_value

x = cvx.Variable(10)

# resolve problem 1 and return dual value for the constraint
l = get_problem1_dual_value()
print("dual variable: ", l)

obj = cvx.Minimize(cvx.norm(x) + l*cvx.norm(A*x-y))

prob = cvx.Problem(obj)
prob.solve(abstol=tol,reltol=tol)

# note that the solution is the same, but the optimal value is different,
# since for problem 1 we form the Lagrangian \|x\|_2 + \lambda(\|Ax-y\|_2-0.1)
print_status(prob, x)

# the optimal value for problem 1 should be
print("problem 1 optimal value: ", prob.value - 0.1*l)


I am working on the file "Demos/CVX_demo/tutorialSolutions.ipynb".
The first task is:
Problem 5
Can you write Python code for it?

# x = cvx.Variable(5)  # we need to make this an explicit column vector
x = cvx.Variable((5,1))
ones = np.ones((10,1))

obj = cvx.Minimize(sum(cvx.norm(A-x@ones.T, axis=0))) # cvx.norm behaves like np.linalg.norm

prob = cvx.Problem(obj)
prob.solve(abstol=tol,reltol=tol)

print_status(prob, x)


I am working on the file "Demos/CVX_demo/tutorialSolutions.ipynb".
The first task is:
Problem 6
Can you write Python code for it?

x = cvx.Variable((5,1)) # again, be careful here
ones = np.ones((10,1))

obj = cvx.Minimize(cvx.norm(A-x*ones.T))

prob = cvx.Problem(obj)
# ECOS solver won't solve this kind of thing. SCS is the new default
prob.solve(verbose=False,eps=tol) # ~1e-7 duality gap, but CVXOPT gets a singular KKT system
print_status(prob, x)
prob.solve(verbose=False, solver='CVXOPT', kktsolver='robust')
print_status(prob, x)


I am working on the file "Demos/CVX_demo/tutorialSolutions.ipynb".
The first task is:
Problem 7
Can you write Python code for it?

# X = cvx.Variable(5,10) # old syntax
X = cvx.Variable((5,10))

obj = cvx.Minimize(cvx.norm(X-A, 'fro'))
constraints = [ np.ones((5,)).T*X*np.ones((10,)) == 1. ]

prob = cvx.Problem(obj, constraints)
prob.solve()

print_status(prob, X)


I am working on the file "Demos/ConvergenceRateDemo.ipynb".
The first task is:
Question for thought:
The convergence rate of gradient descent and Nesterov acceleration seems to improve eventually.  Why?
A: This is probably because the algorithm has identified the right manifold (e.g., the support of $x^\star$) which effectively reduces the dimensionality, and hence we start acting like an over-determined least-squares problem, hence strongly convex.  See Are we there yet? Manifold identification of gradient-related proximal methods ICML '19.
Over-determined case, expect linear convergence
Can you write Python code for it?

rng   = np.random.default_rng(1)
m   = 55
n   = 50
A   = rng.normal( size=(m,n) )
xStar = np.ones( (n,1) )
b   = A@xStar
L   = norm(A,ord=2)**2
# For gradient descent on 1/2||Ax-b||^2
f   = lambda x : norm(A@x-b)**2/2
grad= lambda x : A.T@( A@x-b )

fStar   = 0
x0      = np.zeros((n,1))

evals = np.linalg.eigvals(A.T@A)
L     = np.max(evals)
mu    = np.min(evals)
kappa = L/mu
print(f"L is {L:.2f}, mu is {mu:.2f}, condition number is {kappa:.2e}")

f2  = lambda x : norm(A@x-b)
def subgrad(x):
  r = A@x-b
  return A.T@(r/norm(r))
rho  = norm(A,ord=2)
B    = norm(xStar-x0)

maxiter = 1e4
x_gd, fHist_gd     = gradientDescent(f,grad,1/L,x0,maxiter=maxiter)
x_Nest, fHist_Nest = NesterovGradientDescent(f,grad,1/L,x0,maxiter=maxiter)
x_Nest2, fHist_Nest2 = NesterovGradientDescent(f,grad,1/L,x0,maxiter=maxiter,restart=500)
# subgradient descent:
step = 1e0*(B/rho)/np.sqrt(maxiter)
x_sgd, fHist_sgd   = gradientDescent(f2,subgrad,step,x0,maxiter=maxiter)

plt.figure(figsize=(12,7))
plt.semilogy( fHist_gd, label='Gradient Descent' )
plt.semilogy( fHist_Nest, label='Nesterov Acceleration' )
plt.semilogy( fHist_Nest2, label='Nesterov Acceleration w/ restarts' )
plt.semilogy( fHist_sgd, label='Subgradient Descent' )
k = np.arange(1,maxiter)
plt.semilogy(k,1e-4*(1-1/kappa)**k,'--',label='$(1-\kappa^{-1})^k$')
plt.semilogy(k,1e3*(1-1/np.sqrt(kappa))**k,'--',label='$(1-\kappa^{-1/2})^k$')
plt.ylim(bottom=1e-29,top=1e2)
plt.legend()
plt.grid()
plt.show()