I am working on the file "Spike raster demo.ipynb".
The first task is:
Using the neo data format:
Can you write Python code for it?

import neo
st = neo.SpikeTrain(spikes[1], units='sec', t_stop=250.0, unit=1)
#print(st)
#print(st.annotations)

u = neo.Unit()
u.spiketrains = st

from IPython.core.display import HTML
def css_styling():
    styles = open("../Styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "SpikeGenerationUnitTest.py.ipynb".
The first task is:
Be careful in specifying the maximum firing rate. This needs to be equal-to or larger than the maximum value of the conditional intensity
Can you write Python code for it?

testNeuron = {'MaxRate': 5, 'IntensityFunction': intensityFun};

TLength = 10000; # Simulate 10000 seconds (at 20 s per pass, this is 500 passes)
SpikeTimes = GenerateSpikes(testNeuron["IntensityFunction"],testNeuron["MaxRate"], posFun, TLength);


I am working on the file "SpikeGenerationUnitTest.py.ipynb".
The first task is:
Let's make the place field Gaussian so that it will look nice. Note that this function will be called with a position vector and the history of (potential) spike times. These could be used to, e.g., impose a refractory period, but we won't do that here.
Can you write Python code for it?

def intensityFun(x,thistory) :
    return 5*np.exp(-(x - 0.5)**2/(2 * 0.01));


I am working on the file "SpikeGenerationUnitTest.py.ipynb".
The first task is:
Testing a homogeneous Poisson process.
Do we see that the simulated Fano Factor is 1?
For our first test, let's use a constant rate of spiking and a constant rate of running through the maze. We'll set it up so there's just one pass through the maze and a high rate so that we observe many spikes. We'll bin time and then ask about the mean and variance of spikes per bin.
Can you write Python code for it?

def intensityFun(x,tlast) :
    return 40 * np.ones(np.size(x));

testNeuron = {'MaxRate': 40, 'IntensityFunction': intensityFun};

def posFun(t) :
    return t/20

TLength = 1000; # Length of experiment

SpikeTimes = GenerateSpikes(testNeuron["IntensityFunction"],testNeuron["MaxRate"], posFun, TLength);

BinSize = 1; # Seconds, so that intensity function and spike counts match in units!
TimeBins = np.arange(0,TLength+0.1,BinSize); # Need to add a little bit to account for end
BinnedSpikes,SpaceIdx = np.histogram(SpikeTimes, TimeBins);

# Mean and variance should be the same!
(np.mean(BinnedSpikes), np.var(BinnedSpikes))


I am working on the file "SpikeGenerationUnitTest.py.ipynb".
The first task is:
To calculate empirical tuning curve, we will use "binned occupancy". To do this, we need to:
  - bin position as-a-function-of time (giving us total occupancy in seconds per spatial bin)
  - bin spikes as-a-function-of space (giving us total spikes per spatial bin)
Dividing these two will yield a firing rate as a function of position (which is the normal definition of a place field).
Can you write Python code for it?

BehavioralSamplingRate = 100; # Camera frame rate is 100 Hz
tvec = np.arange(0,TLength+1/BehavioralSamplingRate,1/BehavioralSamplingRate);
BehavioralData = posFun(tvec); # This is his position as a function of time sampled at camera rate

# Set up spatial bins, and find spatial occupancy and spike occupancy
BinSize = 0.02; # This will be the bin size for the PSTH.
                # Can't be smaller than camera rate because then those bins are zero
SpaceBins = np.arange(0,1 + BinSize/2,BinSize); # Need to add a little bit to account for end

BinnedBehavior,SpaceIdx = np.histogram(BehavioralData, SpaceBins);
BinnedSpikes,SpaceIdx = np.histogram(posFun(SpikeTimes), SpaceBins);

BinnedBehavioralTime = BinnedBehavior / BehavioralSamplingRate; # Behavior is sampled at 100 Hz, so time is 10ms bins
EmpiricalTuningCurve = BinnedSpikes / BinnedBehavioralTime;

sns.set(rc={'figure.figsize': (12,4)})
sns.set_style('white')
plt.bar(SpaceIdx[0:-1], EmpiricalTuningCurve, BinSize)
plt.plot(SpaceIdx + BinSize/2, intensityFun(SpaceIdx + BinSize/2,0), 'r-'); # Actually extends one extra bin!


I am working on the file "Spiral-Plot.ipynb".
The first task is:
Check list lengths:
Can you write Python code for it?

print len(bar_colors), len(middleBar), len(Opponent), len(half_h)


I am working on the file "Spiral-Plot.ipynb".
The first task is:
Each ray (starting from origin O(0,0)) crosses successive turnings  of the spiral at constant distance points, namely at distance=$2\pi a$.
With our choice a=2, this distance is $4\pi=12.56637$. Hence we set the tallest bar corresponding to
a set score of 7 as having the height 10. The bar height corresponding to any set score in  ${0, 1, 2, \ldots, 7}$  can be read from  the following dictionary:
Can you write Python code for it?

h=7.0
score={0: 0., 1:10./h, 2: 20/h, 3: 30/h, 4: 40/h, 5: 50/h, 6: 60/h, 7: 70/h}
score[6]

import plotly.plotly as py
from plotly.graph_objs import *


I am working on the file "Spiral-Plot.ipynb".
The first task is:
Get data for generating bars and data  to be displayed when hovering the mouse over the plot:
Can you write Python code for it?

a=2.0 # the parameter in spiral equation z(theta)=a*theta exp(-i theta)
theta0=3*PI/2 # the starting point of the spiral
Theta=[]# the list of tournament arc ends
Opponent=[]# the list of opponents in each set of all matches played by halep
rankOp=[]# rank of opponent list
middleBar=[]# theta coordinate for the middle point of each bar base
half_h=[]# the list of  bar heights/2
wb=1.5# bar width along the spiral
rad=wb/(a*theta0)#the  angle in radians corresponding to an arc of length wb,
                 #within the circle of radius a*theta
rank_Halep=[]
Halep_set_sc=[]# list of Halep set scores
Opponent_set_sc=[]# list of opponent set scores
bar_colors=[]# the list of colors assigned to each bar in bar charts

for k, where in enumerate(played_at):
    nr=nmatches[k]# nr is the number of matches played by Halep in the k^th tournament
    Theta.append(theta0)
    for match in range(nr):
        player=jdata[where][3+match].keys()[0]# opponent name in match match

        rankOp.append(int(player.partition('(')[2].partition(')')[0]))#Extract opponent rank:
        set_sc=jdata[where][3+match].values()[0]#set scores in match match
        sets=len(set_sc)
        #set bar colors according to opponent rank
        if rankOp[-1] in range(1,11): col=colors[1]
        elif rankOp[-1] in range(11, 21): col=colors[2]
        elif rankOp[-1] in range(21, 51): col=colors[3]
        elif rankOp[-1] in range(51, 101): col=colors[4]
        else: col=colors[5]

        for s  in range(0, sets, 2):
            middleBar.append(0.5*(2*theta0+rad))#  get the middle of each angular interval
                                                # defining bar base
            rank_Halep+=[jdata[where][0]['rank']]
            Halep_set_sc.append(set_sc[s])

            half_h.append(0.5*score[set_sc[s]])# middle of bar height
            bar_colors.append(colors[0])

            layout['shapes'].append(make_bar(score[set_sc[s]], theta0, colors[0], rad=rad, a=2))
           ...


I am working on the file "Spiral-Plot.ipynb".
The first task is:
The function make_bar returns a Plotly dict that will be used to generate the bar shapes:
Can you write Python code for it?

def make_bar(bar_height, theta0, fill_color, rad=0.2, a=2):
    theta1=theta0+rad
    C=(a*theta1+bar_height)*np.exp(-1j*theta1)
    D=a*theta0*np.exp(-1j*theta0)

    return dict(
            line=Line(color=fill_color, width=0.5
                     ),
            path= make_arc(a, theta0, theta0+rad, 0.0)+str(C.real)+', '+str(C.imag)+' '+\
                  make_arc(a, theta1, theta0, bar_height)[1:]+ str(D.real)+', '+str(D.imag),
            type='path',
            fillcolor=fill_color
        )


I am working on the file "Spiral-Plot.ipynb".
The first task is:
played_at is the list of tournaments Simona Halep participated in:
Can you write Python code for it?

played_at=['Shenzen', 'Australian Open', 'Fed Cup', 'Dubai', 'Indiana Wells', 'Miami',
          'Stuttgart', 'Madrid', 'Rome', 'French Open', 'Birmingham', 'Wimbledon', 'Toronto',
          'Cincinnati', 'US Open', 'Guangzhou',  'Wuhan', 'Beijing', 'WTA Finals' ]

#define a dict giving the number of matches played by Halep in  each tournament k
nmatches={ k: len(jdata[where][3:]) for  (k, where) in enumerate(played_at) }


I am working on the file "SpreadsheetConversion/convert_tables_for_db.ipynb".
The first task is:
Create 'contact_informations'
Create 'projects_categories' Table
Can you write Python code for it?

s = pd.DataFrame(curated["categories"].str.split(',').tolist()).stack()
print(len(s))
s.head()

s.index = s.index.droplevel(-1)

s.name = 'category'

s.head()

category_join = projects_export.join(s)

category_join.head()


I am working on the file "SpreadsheetConversion/convert_tables_for_db.ipynb".
The first task is:
The following columns are ignored for later processing:
* kind, is a category
* area, is geospatial information
* status, has categories
* hashtags, become project tags
* categories, need to be merged with our list
* orgacontact_*, will become part of contact information
* contact_*, will become part of contact information
* programming_languages, are categories
* languages, part of contact details
* organization_type, is a category
* entrydate, automatic DB entry
* software_development_needs, is now a Boolean 'dev_support_needed' combined with categories of platform, etc.
* random_generated_key, should be part of the user information who is entitled to modifying the DB
Create Primary 'projects' Table
Can you write Python code for it?

matching = {
    "url": "web_url",
    "title": "title",
    "description": "description_content",
    "logo": "logo_url",
    "code_repository": "repository_url",
    "organization_name": "organization_name",
    "code_license": "code_license",
    "releasedate": "publishedAt"
}

projects_columns = ["id", "user_id", "title", "description_tech", "description_content",
              "code_license", "data_license", "logo_url", "web_url", "repository_url",
              "organization_name", "dev_support_needed", "general_support_needed",
              "publishedAt", "createddAt", "updatedAt"]

projects_export = curated[list(matching.keys())].copy()
projects_export.columns = [matching[key] for key in projects_export.columns]
projects_export["id"] = projects_export.index + 1

for key in projects_columns:
    if key not in projects_export.columns:
        projects_export[key] = None

projects_export.head()

projects_export.to_csv("projects.csv", index=False, header=False, columns=projects_columns)


I am working on the file "SpreadsheetConversion/convert_tables_for_db.ipynb".
The first task is:
Try to find categories from project table in the existing list. Will probably introduce a lot of missing values. A real string search (using different languages) would yield better results.
Can you write Python code for it?

category_join["category_node_id"] = [title2id.get(category) for category in category_join["category"]]

project_categories_export = category_join[["id", "category_node_id", "category"]].copy()

project_categories_export.rename(columns={"id": "project_id"}, inplace=True)

project_categories_export.dropna(subset=["category_node_id"])


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12.ipynb".
The first task is:
Время работы
Can you write Python code for it?

%timeit binary_search(f, a, b, epsilon)
%timeit golden_search(f, a, b, epsilon)


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12.ipynb".
The first task is:
Метод золотого сечения
Идея:
делить отрезок $[a,b]$ не на две равные насти,
а в пропорции "золотого сечения".
Оценим скорость сходимости аналогично методу дихотомии:
$$
|x_{K+1} - x^*| \leq b_{K+1} - a_{K+1} = \left( \frac{1}{\tau} \right)^{N-1} (b - a) \approx 0.618^K(b-a),
$$
где $\tau = \frac{\sqrt{5} + 1}{2}$.

Константа геометрической прогрессии больше, чем у метода дихотомии
Количество вызовов функции меньше, чем у метода дихотомии
Can you write Python code for it?

def golden_search(f, a, b, tol=1e-5, callback=None):
    tau = (np.sqrt(5) + 1) / 2.0
    y = a + (b - a) / tau**2
    z = a + (b - a) / tau
    while b - a > tol:
        if f(y) <= f(z):
            b = z
            z = y
            y = a + (b - a) / tau**2
        else:
            a = y
            y = z
            z = a + (b - a) / tau
        if callback is not None:
            callback(a, b)
    return (a + b) / 2.0

left_boud_gs = []
right_bound_gs = []
approximation_gs = []

cb_gs = lambda a, b: my_callback(a, b, left_boud_gs, right_bound_gs, approximation_gs)
x_gs = golden_search(f, a, b, epsilon, cb_gs)

print(f(x_opt))
print(f(x_gs))
print(np.abs(x_opt - x_true))


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12.ipynb".
The first task is:
Пример иного поведения методов
$$
f(x) = \sin(\sin(\sin(\sqrt{x}))), \; x \in [2, 60]
$$
Can you write Python code for it?

f = lambda x: np.sin(np.sin(np.sin(np.sqrt(x))))
x_true = (3 * np.pi / 2)**2
a = 2
b = 60
epsilon = 1e-8
plt.plot(np.linspace(a,b), f(np.linspace(a,b)))
plt.xticks(fontsize = 28)
_ = plt.yticks(fontsize = 28)


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12.ipynb".
The first task is:
Сравнение методов одномерной минимизации
Can you write Python code for it?

plt.figure(figsize=(10,6))
plt.semilogy(np.arange(1, len(approximation_bs) + 1), np.abs(x_true - np.array(approximation_bs, dtype=np.float64)), label="Binary search")
plt.semilogy(np.arange(1, len(approximation_gs) + 1), np.abs(x_true - np.array(approximation_gs, dtype=np.float64)), label="Golden search")
plt.xlabel(r"Number of iterations, $k$", fontsize=26)
plt.ylabel("Error rate upper bound", fontsize=26)
plt.legend(loc="best", fontsize=26)
plt.xticks(fontsize = 26)
_ = plt.yticks(fontsize = 26)

%timeit binary_search(f, a, b, epsilon)
%timeit golden_search(f, a, b, epsilon)


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12.ipynb".
The first task is:
Сходимость
Can you write Python code for it?

plt.figure(figsize=(8,6))
plt.semilogy(np.abs(x_true - np.array(approximation_bs, dtype=np.float64)), label="Binary")
plt.semilogy(np.abs(x_true - np.array(approximation_gs, dtype=np.float64)), label="Golden")
plt.legend(fontsize=28)
plt.xticks(fontsize=28)
_ = plt.yticks(fontsize=28)
plt.xlabel(r"Number of iterations, $k$", fontsize=26)
plt.ylabel("Error rate upper bound", fontsize=26)


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12en.ipynb".
The first task is:
Comparison of convergence speed and execution time
Bisection method
Can you write Python code for it?

left_boud_bs = []
right_bound_bs = []
approximation_bs = []

callback_bs = lambda a, b: my_callback(a, b,
            left_boud_bs, right_bound_bs, approximation_bs)

x_opt = binary_search(f, a, b, epsilon, callback_bs)
print(np.abs(x_opt - x_true))


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12en.ipynb".
The first task is:
Execution time
Can you write Python code for it?


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12en.ipynb".
The first task is:
Golden section method
Can you write Python code for it?

left_boud_gs = []
right_bound_gs = []
approximation_gs = []

cb_gs = lambda a, b: my_callback(a, b, left_boud_gs, right_bound_gs, approximation_gs)
x_gs = golden_search(f, a, b, epsilon, cb_gs)

print(np.abs(x_opt - x_true))


I am working on the file "Spring2017-2019/12-NumMethods/Seminar12en.ipynb".
The first task is:
On interpretations of the convergence theorems (B.T. Polyak Introduction to optimization, ch. 1, $\S$ 6)

What answers give convergence theorems?

class of problems for which the method is applicable (it is important to minimize the number of restrictions!)
convexity
smoothness

qualitative behaviour of the method
is initial guess significant for convergence of method?
what convergence can be observed?

estimate of convergence speed
theoretical estimate of the method behavior without experiments
detection of properties that affect convergence (condition number, dimension, etc)
sometimes one can set number of iterations to achieve required accuracy in advance

What answers do not give convergence theorem?

convergence of the method does not mean that it has to be used
convergence estimates depend on unknown constants - only theoretical interest
theorems do not take into account rounding errors and accuracy of solving auxiliary problems

Main point: it is necessary to use common sense and be careful in usage of convergence theorem!
Classes of problems

Uncontrained optimization
Lipschitz objective function
Lipschitz gradient of the objective function

Contrained optimization
polytope
sets with simple structure
general form

Classes of methods

Zero order method: oracle returns only objective function $f(x)$

Fist order method : oracle returns objective function $f(x)$ and its gradient $f'(x)$

Second order method: oracle returns objective function $f(x)$, its gradient $f'(x)$ and its hessian $f''(x)$.

Q: do methods of higher order exist?

One-step methods
$$
x_{k+1} = \Phi(x_k)
$$
Multi-step methods
$$
x_{k+1} = \Phi(x_k, x_{k-1}, ...)
$$

One-dimensional optimization
Definition. Funtion $f(x)$ is unimodal in interval $[a, b]$, if there exists such point $x^ \in [a, b]$, that
- $f(x_1) > f(x_2)$ for any $a \leq x_1 < x_2 < x^$,
and

$f(x_1) < f(x_2)$ for any $x^* < x_1 < x_2 \leq b$.

Q: what geometry of unimodal function?
Bisection method
Idea from the first term CS course: divide given...
Can you write Python code for it?

def binary_search(f, a, b, epsilon, callback=None):
    c = (a + b) / 2.0
    while abs(b - a) > epsilon:
#         Check left subsegment
        y = (a + c) / 2.0
        if f(y) <= f(c):
            b = c
            c = y
        else:
#         Check right subsegment
            z = (b + c) / 2.0
            if f(c) <= f(z):
                a = y
                b = z
            else:
                a = c
                c = z
        if callback is not None:
            callback(a, b)
    return c

def my_callback(a, b, left_bound, right_bound, approximation):
    left_bound.append(a)
    right_bound.append(b)
    approximation.append((a + b) / 2.0)

# %matplotlib inline
import numpy as np
# import matplotlib.pyplot as plt

left_boud_bs = []
right_bound_bs = []
approximation_bs = []

callback_bs = lambda a, b: my_callback(a, b,
            left_boud_bs, right_bound_bs, approximation_bs)

# Target unimodal function on given segment
f = lambda x: (x - 2) * x * (x + 2)**2 # np.power(x+2, 2)
# f = lambda x: -np.sin(x)
x_true = -2
# x_true = np.pi / 2.0
a = -3
b = -1.5
epsilon = 1e-8
x_opt = binary_search(f, a, b, epsilon, callback_bs)
print(np.abs(x_opt - x_true))
plt.figure(figsize=(10,6))
plt.plot(np.linspace(a,b), f(np.linspace(a,b)))
plt.title("Objective function", fontsize=20)
plt.xticks(fontsize=20)
_ = plt.yticks(fontsize=20)


I am working on the file "Spring2017-2019/14-Newton/Seminar14.ipynb".
The first task is:
Реализация метода Ньютона
Can you write Python code for it?

def Newton(f, gradf, hessf, x0, epsilon, num_iter, line_search,
                    disp=False, callback=None, **kwargs):
    x = x0.copy()
    iteration = 0
    opt_arg = {"f": f, "grad_f": gradf}
    for key in kwargs:
        opt_arg[key] = kwargs[key]
    while True:
        gradient = gradf(x)
        hess = hessf(x)
        h = np.linalg.solve(hess, -gradient)
        alpha = line_search(x, h, **opt_arg)
        x = x + alpha * h
        if callback is not None:
            callback(x)
        iteration += 1
        if disp:
            print("Current function val =", f(x))
            print("Current gradient norm = ", np.linalg.norm(gradf(x)))
        if np.linalg.norm(gradf(x)) < epsilon:
            break
        if iteration >= num_iter:
            break
    res = {"x": x, "num_iter": iteration, "tol": np.linalg.norm(gradf(x))}
    return res


I am working on the file "Spring2017-2019/14-Newton/Seminar14.ipynb".
The first task is:
Сравнение времени работы
Can you write Python code for it?

%timeit newton.solve(x0, tol=1e-6, max_iter=50)
%timeit gd.solve(x0, tol=1e-6, max_iter=50)


I am working on the file "Spring2017-2019/14-Newton/Seminar14en.ipynb".
The first task is:
Comparison of running time
Can you write Python code for it?


I am working on the file "Spring2017-2019/14-Newton/Seminar14en.ipynb".
The first task is:
Exact solution with CVXPy
Can you write Python code for it?

import cvxpy as cvx
x = cvx.Variable((n, 1))

obj = cvx.Minimize(cvx.sum(-cvx.log(1 - A.T * x)) -
                   cvx.sum(cvx.log(1 - cvx.square(x))))
prob = cvx.Problem(obj)
prob.solve(solver="SCS", verbose=True, max_iters=1000)
print("Optimal value =", prob.value)


I am working on the file "Spring2017-2019/14-Newton/Seminar14en.ipynb".
The first task is:
Implementation of Newton method
Can you write Python code for it?


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
Время выполнения
Can you write Python code for it?

%timeit scopt.minimize(f, x0, method="CG", tol=tol, jac=grad_f, options={"maxiter": max_iter})
%timeit cg_fr.solve(x0, tol=tol, max_iter=max_iter)
%timeit cg_fr_rest.solve(x0, tol=tol, max_iter=max_iter)
%timeit gd.solve(x0, tol=tol, max_iter=max_iter)


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
График сходимости
Can you write Python code for it?

plt.figure(figsize=(8,6))
plt.semilogy([np.linalg.norm(grad_f(x)) for x in cg_quad.get_convergence()], label=r"$\|f'(x_k)\|^{CG}_2$", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in scopt_cg_array[:50]], label=r"$\|f'(x_k)\|^{CG_{PR}}_2$", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in gd.get_convergence()], label=r"$\|f'(x_k)\|^{G}_2$", linewidth=2)
plt.legend(loc="best", fontsize=20)
plt.xlabel(r"Iteration number, $k$", fontsize=20)
plt.ylabel("Convergence rate", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)

print([np.linalg.norm(grad_f(x)) for x in cg_quad.get_convergence()])

plt.figure(figsize=(8,6))
plt.plot([f(x) for x in cg_quad.get_convergence()], label=r"$f(x^{CG}_k)$", linewidth=2)
plt.plot([f(x) for x in scopt_cg_array], label=r"$f(x^{CG_{PR}}_k)$", linewidth=2)
plt.plot([f(x) for x in gd.get_convergence()], label=r"$f(x^{G}_k)$", linewidth=2)
plt.legend(loc="best", fontsize=20)
plt.xlabel(r"Iteration number, $k$", fontsize=20)
plt.ylabel("Function value", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


import scipy.optimize as scopt
import liboptpy.restarts as restarts

n_restart = 60
tol = 1e-5
max_iter = 600

scopt_cg_array = []
scopt_cg_callback = lambda x: callback(x, scopt_cg_array)
x = scopt.minimize(f, x0, tol=tol, method="CG", jac=grad_f, callback=scopt_cg_callback, options={"maxiter": max_iter})
x = x.x
print("\t CG by Polak-Rebiere")
print("Norm of garient = {}".format(np.linalg.norm(grad_f(x))))
print("Function value = {}".format(f(x)))

print("\t CG by Fletcher-Reeves")
cg_fr = methods.fo.ConjugateGradientFR(f, grad_f, ss.Backtracking("Wolfe", rho=0.9, beta1=0.1, beta2=0.4, init_alpha=1.))
x = cg_fr.solve(x0, tol=tol, max_iter=max_iter, disp=True)

print("\t CG by Fletcher-Reeves with restart n")
cg_fr_rest = methods.fo.ConjugateGradientFR(f, grad_f, ss.Backtracking("Wolfe", rho=0.9, beta1=0.1, beta2=0.4,
                                         init_alpha=1.), restarts.Restart(n // n_restart))
x = cg_fr_rest.solve(x0, tol=tol, max_iter=max_iter, disp=True)

print("\t Gradient Descent")
gd = methods.fo.GradientDescent(f, grad_f, ss.Backtracking("Wolfe", rho=0.9, beta1=0.1, beta2=0.4, init_alpha=1.))
x = gd.solve(x0, max_iter=max_iter, tol=tol, disp=True)

plt.figure(figsize=(8, 6))
plt.semilogy([np.linalg.norm(grad_f(x)) for x in cg_fr.get_convergence()], label=r"$\|f'(x_k)\|^{CG_{FR}}_2$ no restart", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in cg_fr_rest.get_convergence()], label=r"$\|f'(x_k)\|^{CG_{FR}}_2$ restart", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in scopt_cg_array], label=r"$\|f'(x_k)\|^{CG_{PR}}_2$", linewidth=2)

plt.semilogy([np.linalg.norm(grad_f(x)) for x in gd.get_convergence()], label=r"$\|f'(x_k)\|^{G}_2$", linewidth=2)
plt.legend(loc="best", fontsize=16)
plt.xlabel(r"Iteration number, $k$", fontsize=20)
plt.ylabel("Convergence rate", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
Неквадратичная функция
Can you write Python code for it?

import numpy as np
import sklearn.datasets as skldata
import scipy.special as scspec

n = 300
m = 1000

X, y = skldata.make_classification(n_classes=2, n_features=n, n_samples=m, n_informative=n//3)
C = 1
def f(w):
    return np.linalg.norm(w)**2 / 2 +  C * np.mean(np.logaddexp(np.zeros(X.shape[0]), -y * X.dot(w)))

def grad_f(w):
    denom = scspec.expit(-y * X.dot(w))
    return w - C * X.T.dot(y * denom) / X.shape[0]
# f = lambda x: -np.sum(np.log(1 - A.T.dot(x))) - np.sum(np.log(1 - x*x))
# grad_f = lambda x: np.sum(A.dot(np.diagflat(1 / (1 - A.T.dot(x)))), axis=1) + 2 * x / (1 - np.power(x, 2))
x0 = np.zeros(n)
print("Initial function value = {}".format(f(x0)))
print("Initial gradient norm = {}".format(np.linalg.norm(grad_f(x0))))


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
Правильный ответ
Can you write Python code for it?

import scipy.optimize as scopt

def callback(x, array):
    array.append(x)

scopt_cg_array = []
scopt_cg_callback = lambda x: callback(x, scopt_cg_array)
x = scopt.minimize(f, x0, method="CG", jac=grad_f, callback=scopt_cg_callback)
x = x.x
print("||f'(x*)|| =", np.linalg.norm(A.dot(x) - b))
print("f* =", f(x))


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
Распределение собственных значений
Can you write Python code for it?

USE_COLAB = False

%matplotlib inline
import matplotlib.pyplot as plt
if not USE_COLAB:
    plt.rc("text", usetex=True)
    plt.rc("font", family='serif')

if USE_COLAB:
    !pip install git+https://github.com/amkatrutsa/liboptpy

import seaborn as sns
sns.set_context("talk")

eigs = np.linalg.eigvalsh(A)
plt.semilogy(np.unique(eigs))
plt.ylabel("Eigenvalues", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15.ipynb".
The first task is:
Реализация метода сопряжённых градиентов
Can you write Python code for it?

def ConjugateGradientQuadratic(x0, A, b, tol=1e-8, callback=None):
    x = x0
    r = A.dot(x0) - b
    p = -r
    while np.linalg.norm(r) > tol:
        alpha = r.dot(r) / p.dot(A.dot(p))
        x = x + alpha * p
        if callback is not None:
            callback(x)
        r_next = r + alpha * A.dot(p)
        beta = r_next.dot(r_next) / r.dot(r)
        p = -r_next + beta * p
        r = r_next
    return x

import liboptpy.unconstr_solvers as methods
import liboptpy.step_size as ss

print("\t CG quadratic")
cg_quad = methods.fo.ConjugateGradientQuad(A, b)
x_cg = cg_quad.solve(x0, tol=1e-7, disp=True)

print("\t Gradient Descent")
gd = methods.fo.GradientDescent(f, grad_f, ss.ExactLineSearch4Quad(A, b))
x_gd = gd.solve(x0, tol=1e-7, disp=True)

print("Condition number of A =", abs(max(eigs)) / abs(min(eigs)))


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Convergence plot
Can you write Python code for it?


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Eigenvalues distribution
Can you write Python code for it?

USE_COLAB = False

%matplotlib inline
import matplotlib.pyplot as plt

if not USE_COLAB:
    plt.rc("text", usetex=True)
    plt.rc("font", family='serif')

if USE_COLAB:
    !pip install git+https://github.com/amkatrutsa/liboptpy

import seaborn as sns
sns.set_context("talk")

eigs = np.linalg.eigvalsh(A)
plt.semilogy(np.unique(eigs))
plt.ylabel("Eigenvalues", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Exact solution
Can you write Python code for it?


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Implementation of Fletcher-Reeves method
Can you write Python code for it?

def ConjugateGradientFR(f, gradf, x0, num_iter=100, tol=1e-8, callback=None, restart=False):
    x = x0
    grad = gradf(x)
    p = -grad
    it = 0
    while np.linalg.norm(gradf(x)) > tol and it < num_iter:
        alpha = utils.backtracking(x, p, method="Wolfe", beta1=0.1, beta2=0.4, rho=0.5, f=f, grad_f=gradf)
        if alpha < 1e-18:
            break
        x = x + alpha * p
        if callback is not None:
            callback(x)
        grad_next = gradf(x)
        beta = grad_next.dot(grad_next) / grad.dot(grad)
        p = -grad_next + beta * p
        grad = grad_next.copy()
        it += 1
        if restart and it % restart == 0:
            grad = gradf(x)
            p = -grad
    return x


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Implementation of conjugate gradient method
Can you write Python code for it?


I am working on the file "Spring2017-2019/15-ConjGrad/Seminar15en.ipynb".
The first task is:
Running time
Can you write Python code for it?


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18.ipynb".
The first task is:
Выбор начального приближения допустимого по ограничениям и области определения целевой функции
Can you write Python code for it?

x0 = np.zeros(2*n)
x0[:n] = np.random.rand(n)
x0[n:2*n] = b - A.dot(x0[:n])
print(np.linalg.norm(A_lin.dot(x0) - b))
print(np.sum(x0 <= 1e-6))


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18.ipynb".
The first task is:
Комментарии

Было показано, что прямой метод эквивалентен методу Кармаркара
Использует информацию только о прямой задаче
Начальное приближение должно лежать в допустимом множестве - отдельная задача

Сравнение способов решения линейной системы в методе Ньютона

Явно сформированная матрица
Использование блочной структуры и последовательное исключение переменных
Can you write Python code for it?

def generate_linprog_problem(n, m=10):
    x = np.random.rand(n)
    A = np.random.randn(m, n)
    b = A.dot(x)
    c = np.random.randn(n)
    c[c < 0] = 1
    return c, A, b, x

n = 20
c, A, b, x0 = generate_linprog_problem(n)
res = scopt.linprog(c, A_eq=A, b_eq=b, bounds=[(0, None) for i in range(n)])
m = A.shape[0]
# x0 = np.random.rand(n)
# resid = b - A[:, m:].dot(x0[m:])
# x0[:m] = np.linalg.solve(A[:m, :m], resid)
print(np.linalg.norm(A.dot(x0) - b))
mu0 = 1
rho_mu = 0.5

x_bar_simple = BarrierPrimalLinConstr(f, gradf, hessf, A, c, x0, mu0, rho_mu, backtracking,
                                      simple_solver, max_iter=100, tol=1e-8)
x_bar_elim = BarrierPrimalLinConstr(f, gradf, hessf, A, c, x0, mu0, rho_mu, backtracking,
                                    elimination_solver, tol=1e-8)
print(np.linalg.norm(res["x"] - x_bar_simple))
print(np.linalg.norm(res["x"] - x_bar_elim))
print(c.dot(res["x"]))
print(c.dot(x_bar_elim))

n_list = [10*i for i in range(2, 63, 10)]
times_simple = np.zeros(len(n_list))
times_elim = np.zeros(len(n_list))
mu0 = 5
rho_mu = 0.5
for i, n in enumerate(n_list):
    print("Current dimension = {}".format(n))
    c, A, b, x0 = generate_linprog_problem(n)
    time = %timeit -o BarrierPrimalLinConstr(f, gradf, hessf, A, c, x0, mu0, rho_mu, backtracking, simple_solver)
    times_simple[i] = time.average
    time = %timeit -o BarrierPrimalLinConstr(f, gradf, hessf, A, c, x0, mu0, rho_mu, backtracking, elimination_solver)
    times_elim[i] = time.average

plt.semilogy(n_list, times_elim, label="Elimination solver")
plt.semilogy(n_list, times_simple, label="Simple solver")
# plt.semilogy(n_list, 10**(-6)*np.array(n_list)**3 / 3)
plt.legend(fontsize=18)
plt.xlabel("Dimension, $n$", fontsize=18)
plt.ylabel("Computation time, sec.", fontsize=18)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18.ipynb".
The first task is:
Проверим сходимость
Can you write Python code for it?

hist_conv = []
def cl(x):
    hist_conv.append(x)
res = NewtonLinConstraintsFeasible(f, gradf, hessf, A_lin, x0, backtracking, elimination_solver, (c, mu), callback=cl,
                           max_iter=2000, beta1=0.1, rho=0.7)
print("Decrement value = {}".format(res["tol"]))
fstar = f(res["x"], c, mu)
hist_conv_f = [np.abs(fstar - f(descdir_x[1], c, mu)) for descdir_x in hist_conv]
plt.figure(figsize=(12, 5))
plt.subplot(1,2,1)
plt.semilogy(hist_conv_f)
plt.xlabel("Number of iteration, $k$", fontsize=18)
plt.ylabel("$f^* - f_k$", fontsize=18)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)

hist_conv_x = [np.linalg.norm(res["x"] - x[1]) for x in hist_conv]
plt.subplot(1,2,2)
plt.semilogy(hist_conv_x)
plt.xlabel("Number of iteration, $k$", fontsize=18)
plt.ylabel("$\| x_k - x^*\|_2$", fontsize=18)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)
plt.tight_layout()


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18.ipynb".
The first task is:
Реализация барьерного метода
Can you write Python code for it?

def BarrierPrimalLinConstr(f, gradf, hessf, A, c, x0, mu0, rho_mu, linesearch, linsys_solver,
                           tol=1e-8, max_iter=500, disp_conv=False, **kwargs):
    x = x0.copy()
    n = x0.shape[0]
    mu = mu0
    while True:
        res = NewtonLinConstraintsFeasible(f, gradf, hessf, A, x, linesearch, linsys_solver, (c, mu),
                                           disp_conv=disp_conv, max_iter=max_iter, beta1=0.01, rho=0.5)
        x = res["x"].copy()
        if n * mu < tol:
            break
        mu *= rho_mu
    return x

mu0 = 5
rho_mu = 0.5
x = BarrierPrimalLinConstr(f, gradf, hessf, A_lin, c, x0, mu0, rho_mu, backtracking, elimination_solver, max_iter=100)
%timeit BarrierPrimalLinConstr(f, gradf, hessf, A_lin, c, x0, mu0, rho_mu, backtracking, elimination_solver, max_iter=100)
%timeit BarrierPrimalLinConstr(f, gradf, hessf, A_lin, c, x0, mu0, rho_mu, backtracking, simple_solver, max_iter=100)
print(x[:n])


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18.ipynb".
The first task is:
Сравнение времени работы
Can you write Python code for it?

mu0 = 2
rho_mu = 0.5
n_list = range(3, 10)
n_iters = np.zeros(len(n_list))
times_simplex = np.zeros(len(n_list))
times_barrier_simple = np.zeros(len(n_list))
for i, n in enumerate(n_list):
    print("Current dimension = {}".format(n))
    c, A, b, bounds = generate_KleeMinty_test_problem(n)
    res = scopt.linprog(c, A, b, bounds=bounds, options={"maxiter": 2**max(n_list) + 1})
    time = %timeit -o -q scopt.linprog(c, A, b, bounds=bounds, options={"maxiter": 2**max(n_list) + 1})
    times_simplex[i] = time.best
    A_lin = np.zeros((n, n + A.shape[0]))
    A_lin[:n, :n] = A
    A_lin[:n, n:n + A.shape[0]] = np.eye(A.shape[0])
    x0 = np.zeros(2*n)
    x0[:n] = np.random.rand(n)
    x0[n:2*n] = b - A.dot(x0[:n])
    time = %timeit -o -q BarrierPrimalLinConstr(f, gradf, hessf, A_lin, c, x0, mu0, rho_mu, backtracking, simple_solver)
    times_barrier_simple[i] = time.best

plt.figure(figsize=(8, 5))
plt.semilogy(n_list, times_simplex, label="Simplex")
plt.semilogy(n_list, times_barrier_simple, label="Primal barrier")
plt.legend(fontsize=18)
plt.xlabel("Dimension, $n$", fontsize=18)
plt.ylabel("Computation time, sec.", fontsize=18)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18en.ipynb".
The first task is:
Check convergence
Can you write Python code for it?


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18en.ipynb".
The first task is:
Check gradient
Can you write Python code for it?

scopt.check_grad(f, gradf, np.random.rand(n), c, mu)


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18en.ipynb".
The first task is:
Implementation of primal barrier method
Can you write Python code for it?


I am working on the file "Spring2017-2019/18-LinProgPrimalInterior/Seminar18en.ipynb".
The first task is:
Select initial guess vector from feasible set and from the domain of the objective
Can you write Python code for it?


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
Now it works fine for the first set of inputs. Let's try other inputs.
Can you write Python code for it?

# Create a vector of probabilities.
ps = np.arange(5.)
ps /= ps.sum()
ps

entropy(ps)


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
This can be a more convenient way to debug programs and step through the actual execution.
Can you write Python code for it?


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
Using a Debugger
Python comes with a built-in debugger called pdb.  It allows you to step line-by-line through a computation and examine what's happening at each step.  Note that this should probably be your last resort in tracing down a bug.  I've probably used it a dozen times or so in five years of coding.  But it can be a useful tool to have in your toolbelt.
You can use the debugger by inserting the line
python
import pdb; pdb.set_trace()
within your script. To leave the debugger, type "exit()". To see the commands you can use, type "help".
Let's try this out:
Can you write Python code for it?

def entropy(ps):
    items = ps * np.log(ps)
    if np.isnan(items[0]):
      import pdb; pdb.set_trace()
    return -np.sum(items)


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
We get nan, which stands for "Not a Number".  What's going on here?
Often the first thing to try is to simply print things and see what's going on.
Within the file, you can add some print statements in key places:
Can you write Python code for it?

def entropy1(ps):
    print("ps=%s" % str(ps))
    items = ps * np.log(ps)
    if np.isnan(items[0]):
      print(items)
    return -np.sum(items)

entropy1([.1, .2])

np.isnan(np.nan)


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
We should have documented the inputs to the function!
Can you write Python code for it?

def entropy(ps):
    print ps
    items = ps * np.log(ps)
    return -np.sum(items)
ps = [0.1, 0.3, 0.5, 0.7, 0.9]
entropy(ps)


I am working on the file "Spring2018/Debugging-and-Exceptions/Debugging.ipynb".
The first task is:
What's the bug here and how do we resolve?
Can you write Python code for it?

def entropy(ps):
    print ps
    items = ps * np.log(ps)
    return -np.sum(items)
ps = "0.1, 0.3, 0.5, 0.7, 0.9"
entropy(ps)


I am working on the file "Spring2018/Debugging-and-Exceptions/Exceptions.ipynb".
The first task is:
Exceptions
An exception is an event, which occurs during the execution of a program, that disrupts the normal flow of the program's instructions.
You've already seen some exceptions:
- syntax errors
- divide by 0
Many programs want to know about exceptions when they occur. For example, if the input to a program is a file path. If the user inputs an invalid or non-existent path, the program generates an exception. It may be desired to provide a response to the user in this case.
It may also be that programs will generate exceptions. This is a way of indicating that there is an error in the inputs provided. In general, this is the preferred style for dealing with invalid inputs or states inside a python function rather than having an error return.
Catching Exceptions
Python provides a way to detect when an exception occurs. This is done by the use of a block of code surrounded by a "try" and "except" statement.
Can you write Python code for it?

def divide1(numerator, denominator):
    try:
      result = numerator/denominator
      print("result = %f" % result)
    except:
      print("You can't divide by 0!!")

divide1(1.0, 2)

divide1(1.0, 0)

divide1("x", 2)


I am working on the file "Spring2018/Debugging-and-Exceptions/Exceptions.ipynb".
The first task is:
Question: What do you do when you get an exception?
You can get information about exceptions.
Can you write Python code for it?

#1/0

def divide2(numerator, denominator):
    try:
      result = numerator/denominator
      print("result = %f" % result)
    except (ZeroDivisionError, TypeError):
      print("Got an exception")

divide2(1, "x")

# Why doesn't this catch the exception?
# How do we fix it?
divide2("x", 2)

# Exceptions in file handling
def read_safely(path):
  error = None
  try:
    with open(path, "r") as fd:
      lines = fd.readlines()
      print ('\n'.join(lines()))
  except FileNotFoundError as err:
    print("File %s does not exist. Try again." % path)

read_safely("unknown.txt")

# Handle division by 0 by using a small number
SMALL_NUMBER = 1e-3
def divide2(numerator, denominator):
    try:
      result = numerator/denominator
    except ZeroDivisionError:
      result = numerator/SMALL_NUMBER
    print("result = %f" % result)

divide2(1,0)


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Instead we will cycle over the whole training dataset a few times. We will process the dataset in mini-batches and take an SGD step for each such mini-batch.
Can you write Python code for it?

epochs = 5;

batch_size = 32;

N = train_data.shape[0];

hist_loss = [];

for epoch in range(epochs):
	print("Epoch:", epoch)

	for index in range(int(N/(batch_size))):

		batch_x = train_data[index*batch_size:(index+1)*batch_size];
		batch_y = train_labels[index*batch_size:(index+1)*batch_size];

		[vis_loss,a] = sess.run([cross_entropy_loss,train_step], feed_dict={x:batch_x,y:batch_y})
		hist_loss += [vis_loss]

plt.plot(hist_loss)
plt.show()


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Let's see what the model learns:
Can you write Python code for it?

for j in range(5):
    digit = test_data[j].reshape([1,784]);
    actual_label = test_labels[j];

    plt.imshow(digit.reshape([28,28]),cmap='gray')
    plt.show()
    print(actual_label)

    [vis_probs] = sess.run([class_probs], feed_dict={x:digit})
    print(vis_probs)
    input()


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Let's see what they look like:
Can you write Python code for it?

sz = 5;

for i in range(sz*sz):

    plt.subplot(sz, sz, i+1)
    plt.imshow(X_train[i],cmap='gray')
    plt.axis('off')

plt.show()


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Notice how re-running the command above shifts the output of the model towards the actual labels. Of course this instance of the model will be horribly overfit to these few digits. Let's re-initialize the model:
Can you write Python code for it?


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Notice how we did not provide the labels since these are not part of the slice of the computational graph for the class probabilities.
We can also perform an SGD step on this batch by running the session on the training step:
Can you write Python code for it?

for j in range(200):
    [a,vis_probs] = sess.run([train_step,class_probs], feed_dict={x:batch_x,y:batch_y})
print(vis_probs)


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
Pre-processing
We will flatten each digit into a 784-dimensional vector.
Can you write Python code for it?

train_data = X_train.reshape([-1,28*28])
test_data = X_test.reshape([-1,28*28])

print(train_data.shape)
print(test_data.shape)


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
The dataset also includes a label for each digit:
Can you write Python code for it?


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
The pixel value range is 0-255
Can you write Python code for it?

print(X_train[0])

plt.imshow(X_train[0],cmap='gray')
plt.axis('off')
plt.show()


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
To test the model more 'formally' we can compute the accuracy on the test dataset:
Can you write Python code for it?

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(class_probs, 1))

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

train_accuracy = sess.run(accuracy,feed_dict={x:train_data,y:train_labels})
test_accuracy = sess.run(accuracy,feed_dict={x:test_data,y:test_labels})

print("Accuracy on training data:",train_accuracy)
print("Accuracy on test data:",test_accuracy)


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
We also need to define what a training step looks like. The command below tells tensorflow to optimize the loss function using a Stochastic Gradient Descent (SGD) step with a learning rate of 0.5:
Can you write Python code for it?


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
We can use the session to feed the input to the model and get the value of a specific node:
Can you write Python code for it?

batch_x = train_data[:5]
batch_y = train_labels[:5]

vis_probs = sess.run([class_probs], feed_dict={x:batch_x})

print(vis_probs)


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
We have 60000 28x28 images in our training set and 10000 in the test set.
Can you write Python code for it?


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
We need to create a session that will run on the computational graph:
Can you write Python code for it?


I am working on the file "Spring2018/MNIST_classification.ipynb".
The first task is:
We will feed the input into a two dense layers with a tanh() non-linearity. Each layer consists of the weight matrix W and the bias vector b. These have to be declared as variables:
Can you write Python code for it?

h1_sz = 64;

W1 = tf.get_variable("W1", [784,h1_sz])
b1 = tf.Variable(tf.zeros([h1_sz]))
h1 = tf.matmul(x,W1) + b1

h1 = tf.tanh(h1)

W2 = tf.get_variable("W2", [h1_sz,10])
b2 = tf.Variable(tf.zeros([10]))
h2 = tf.matmul(h1,W2) + b2


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
2.2.3 Strings
Can you write Python code for it?

# Can concatenate, substring, find, count, ...

a = "The lazy"
b = "brown fox"
print ("Concatenation: ", a + b)
print ("First three letters: " + a[0:3])
print ("Index of 'z': " + str(a.find('z')))


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
2.3 Tuples
A tuple is an ordered sequence of objects. Tuples cannot be changed; they are immuteable.
Can you write Python code for it?

a_tuple = (1, 'ab', (1,2))
a_tuple

a_tuple[2]


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
2.7 A Shakespearean Detour: "What's in a Name?"
Deep vs. Shallow Copies
A deep copy can be manipulated separately. A shallow copy is a pointer to the same data as the original.
Can you write Python code for it?

# A first name shell game
first_int = 1
second_int = first_int
second_int += 1
second_int

# What is first_int?
first_int

# A second name shell game
a_list = ['a', 'aa', 'aaa']
b_list = a_list
b_list.append('bb')
b_list

# What is a_list?
a_list

# Create a deep copy
import copy
# A second name shell game
a_list = ['a', 'aa', 'aaa']
b_list = copy.deepcopy(a_list)
b_list.append('bb')
print("b_list = %s" % str(b_list))
print("a_list = %s" % str(a_list))


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
Insights on python name resolution
* Names are assigned within a context.
* Context changes with the function and module.
  * Assigning a name in a function creates a new name.
  * Referencing an unassigned name in function uses an existing name.
2.7 Object Essentials
Objects are a "packaging" of data and code. Almost all python entities are objects.
Can you write Python code for it?

# A list and a dict are objects.
# dict has been implemented so that you see its values when you type
# the instance name.
# This is done with many python objects, like list.
a_dict = {'a': [1, 2], 'b': [3, 4, 5]}
a_dict

# You access the data and methods (codes) associated with an object by
# using the "." operator. These are referred to collectively
# as attributes. Methods are followed by parentheses;
# values (properties) are not.
a_dict.keys()

# You can discover the attributes of an object using "dir"
dir(a_dict)


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
Key insight: Deep vs. Shallow Copies
* A deep copy can be manipulated separately from the original.
* A shallow copy cannot.
* Assigning a python immutable creates a deep copy. Non-immutables are shallow copies.
Name Resolution
The most common errors that you'll see in your python codes are:
* NameError
* AttributeError
A common error when using the bash shell is command not found.
Name resolution: Associating a name with code or data.
Resolving a name in the bash shell is done by searching the directories in the PATH environment variable. The first executable with the name is run.
Can you write Python code for it?

# Example 1 of name resolution in python
var = 10
def func(val):
    var = val + 1
    return val

# What is returned?
print("func(2) = %d" % func(2))
# What is var?
print("var = %d" % var)

# Example 2 of name resolution in python
var = 10
def func(val):
    return val + var

# What is returned?
print("func(2) = %d" % func(2))
# What is var?
print("var = %d" % var)


I am working on the file "Spring2019/02_Procedural_Python/Data Basics.ipynb".
The first task is:
Today's Objectives
0. Cloning LectureNotes
1. Opening & Navigating the Jupyter Notebook
2. Data type basics
3. Loading data with pandas
4. Cleaning and Manipulating data with pandas
5. Visualizing data with pandas & matplotlib
0. Cloning Lecture Notes
The course materials are maintained on github. The next lecture will discuss github in detail. Today, you'll get minimal instructions to get access to today's lecture materials.

Open a terminal session
Type 'git clone https://github.com/UWSEDS/LectureNotes.git'
Wait until the download is complete
cd LectureNotes
cd 02_Procedural_Python

1. Opening and Navigating the IPython Notebook
We will start today with the interactive environment that we will be using often through the course: the Jupyter Notebook.
We will walk through the following steps together:

Download miniconda (be sure to get Version 3.6) and install it on your system (hopefully you have done this before coming to class)

Use the conda command-line tool to update your package listing and install the IPython notebook:

Update conda's listing of packages for your system:
   $ conda update conda
Install IPython notebook and all its requirements
   $ conda install jupyter notebook

Navigate to the directory containing the course material. For example:

$ cd LectureNotes/02_Procedural_Python
You should see a number of files in the directory, including these:
```
   $ ls
```

Type jupyter notebook in the terminal to start the notebook

$ jupyter notebook
If everything has worked correctly, it should automatically launch your default browser

Click on Lecture-Python-And-Data.ipynb to open the notebook containing the content for this lecture.

With that, you're set up to use the Jupyter notebook!
2. Data Types Basics
2.1 Data type theory

Components with the same capabilities are of the same type.
For example, the numbers 2 and 200 are both integers.
A type is defined recursively. Some examples.
A list is a collection of objects that can be indexed by position.
A list of integers contains an...
Can you write Python code for it?

# Integer arithematic
1 + 1

# Integer division version floating point division
print (6 // 4, 6/ 4)


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:

Exercise
Objective: Replace the bash magic bits for downloading the Pronto data and uncompressing it with Python code.  Since the download is big, check if the zip file exists first before downloading it again.  Then load it into a pandas dataframe.
Notes:
* The os package has tools for checking if a file exists: os.path.exists
import os
filename = 'pronto.csv'
if os.path.exists(filename):
    print("wahoo!")
* Use the requests package to get the file given a url (got this from the requests docs)
import requests
url = 'https://s3.amazonaws.com/pronto-data/open_data_year_two.zip'
req = requests.get(url)
assert req.status_code == 200 # if the download failed, this line will generate an error
with open(filename, 'wb') as f:
    f.write(req.content)
* Use the zipfile package to decompress the file while reading it into pandas
import pandas as pd
import zipfile
csv_filename = '2016_trip_data.csv'
zf = zipfile.ZipFile(filename)
data = pd.read_csv(zf.open(csv_filename))
Now, use your code from above for the following URLs and filenames
| URL | filename | csv_filename |
|-----|----------|--------------|
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set1.zip?raw=true | open_data_year_two_set1.zip | 2016_trip_data_set1.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set2.zip?raw=true | open_data_year_two_set2.zip | 2016_trip_data_set2.csv |
| https://github.com/UWSEDS/LectureNotes/blob/master/open_data_year_two_set3.zip?raw=true | open_data_year_two_set3.zip | 2016_trip_data_set3.csv |
What pieces of the data structures and flow control that we talked about earlier can you use?

Functions
For loops let you repeat some code for every item in a list.  Functions are similar in that they run the same lines of code for new values of some variable.  They are different in that functions are not limited to looping over items.
Functions are a critical part of writing easy to read, reusable code.
Create a function like:
def function_name (parameters):
    """
   ...
Can you write Python code for it?

def print_string(str):
    """This prints out a string passed as the parameter."""
    print(str)
    return


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
Conditionals can take and and or and not.  E.g.
Can you write Python code for it?

my_favorite = 'pie'

if my_favorite is 'cake' or my_favorite is 'pie':
    print(my_favorite + " : I have a recipe for that!")
else:
    print("Ew!  Who eats that?")


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
For
For loops are the standard loop, though while is also common.  For has the general form:
for items in list:
    do stuff
For loops and collections like tuples, lists and dictionaries are natural friends.
Can you write Python code for it?

for instructor in instructors:
    print(instructor)


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
For loops can be nested
Note: for more on formatting strings, see: https://pyformat.info
Can you write Python code for it?

for i in range(1, 4):
    for j in range(1, 4):
        print('%d * %d = %d' % (i, j, i*j))  # Note string formatting here, %d means an integer


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
If
If statements can be use to execute some lines or block of code if a particular condition is satisfied.  E.g. Let's print something based on the entries in the list.
Can you write Python code for it?

if 'Dorkus the Clown' in instructors:
    print('#fakeinstructor')


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
It is always good practice to handle all cases explicity.  Conditional fall through is a common source of bugs.
Sometimes we wish to test multiple conditions.  Use if, elif, and else.
Can you write Python code for it?

my_favorite = 'pie'

if my_favorite is 'cake':
    print("He likes cake!  I'll start making a double chocolate velvet cake right now!")
elif my_favorite is 'pie':
    print("He likes pie!  I'll start making a cherry pie right now!")
else:
    print("He likes " + my_favorite + ".  I don't know how to make that.")


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
Note: what have you noticed in this session about quotes?  What is the difference between ' and "?
Another simple example:
Can you write Python code for it?

if True is False:
    print("I'm so confused")
else:
    print("Everything is right with the world")


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
Notice that Python (in the newest versions, e.g. 3+) has an object type that is a range.  This saves memory and speeds up calculations vs. an explicit representation of a range as a list - but it can be automagically converted to a list on the fly by Python.  To show the contents as a list we can use the type case like with the tuple above.
Sometimes, in older Python docs, you will see xrange.  This used the range object back in Python 2 and range returned an actual list.  Beware of this!
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
Play around with the above function.
Functions can contain any code that you put anywhere else including:
* if...elif...else
* for...else
* while
* other function calls
Can you write Python code for it?

def print_name_age(first, last, age):
    print_name(first, last)
    print('Your age is %d' % (age))
    if age > 35:
        print('You are really old.')
    return

print_name_age(age=40, last='Beck', first='Dave')


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
Remember earlier with slicing, the syntax :3 meant [0, 1, 2]?  Well, the same upper bound philosophy applies here.
Can you write Python code for it?

for index in range(3):
    instructor = instructors[index]
    if instructor.endswith('Clown'):
        print(instructor + " doesn't sound like a real instructor name!")
    else:
        print(instructor + " is so smart... all those gooey brains!")


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
There is a special do nothing word: pass that skips over some arm of a conditional, e.g.
Can you write Python code for it?

if 'Joe' in instructors:
    print("Congratulations!  Joe is teaching, your class won't stink!")
else:
    pass


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
To call the function, use:
print_string("Dave is awesome!")
Note: The function has to be defined before you can call it!
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Procedural Programming.ipynb".
The first task is:
You can iterate over letters in a string
Can you write Python code for it?

my_string = "DIRECT"
for c in my_string:
    print(c)


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
4. Introduction to Pandas DataFrames
What are the elements of a table?
Can you write Python code for it?

# Pandas DataFrames as table elements
import pandas as pd


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Adjusting the Plot Style
Matplotlib has a number of plot styles you can use. For example, if you like R you might use the ggplot style:
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
By default, the values rather than the index are sorted. Use sort=False to turn this behavior off:
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Converting to numpy Arrays
Can you write Python code for it?

my_array = np.array(my_list)

type(my_array)

my_array.dtype

multi_array.shape

multi_array = np.array([[1, 2, 3], [4, 5, 6]], np.int32)


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Group-by Operation
One of the killer features of the Pandas dataframe is the ability to do group-by operations.
You can visualize the group-by like this (image borrowed from the Python Data Science Handbook)
Can you write Python code for it?

df.head()

df_count = df.groupby(['from_station_id']).count()
df_count.head()

df_mean = df.groupby(['from_station_id']).mean()
df_mean.head()

dfgroup = df.groupby(['from_station_id'])
dfgroup.groups


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
How do we access the 6 element in the second row, third column for native Python list?
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
How do we retrieve a slice of the array, array([[1, 2], [4, 5]])?
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
How do we retrieve the second column of the array?
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Lists in native Python
Let's create a list, a native Python object that we've used earlier today.
Can you write Python code for it?

my_list = [2, 5, 7, 8]
my_list

type(my_list)


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Loading Data into a DataFrame
Because we'll use it so much, we often import under a shortened name using the import ... as ... pattern:
Can you write Python code for it?

import pandas as pd
df = pd.read_csv('pronto.csv')

type(df)

len(df)


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Mathematical operations on columns happen element-wise:
Can you write Python code for it?

trip_duration_hours = df['tripduration']/3600
trip_duration_hours[:2]

trip_duration_hours.head()

df['trip_duration_hours'] = df['tripduration']/3600

del df['trip_duration_hours']

df.head()

df.loc[[0,1],:]

df_long_trips = df[df['tripduration'] >10000]

sel = df['tripduration'] > 10000
df_long_trips = df[sel]

df_long_trips

df[sel].shape

# Make a copy of a slice
df_subset = df[['starttime', 'stoptime']].copy()
df_subset['trip_hours'] = df['tripduration']/3600


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
Other plot types
Pandas supports a range of other plotting types; you can find these by using the <TAB> autocomplete on the plot method:
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
The columns attribute gives us the column names
The index attribute gives us the index names
The dtypes attribute gives the data types of each column:
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
This list is one-dimensional, let's make it multidimensional!
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
We can explore other things as well: day of week, hour of day, etc.
Can you write Python code for it?


I am working on the file "Spring2019/02_Procedural_Python/Sophisticated Data Manipulation.ipynb".
The first task is:
uncomment this to download the data:
Can you write Python code for it?

!curl -o pronto.csv https://data.seattle.gov/api/views/tw7j-dfaw/rows.csv?accessType=DOWNLOAD


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
By printing some of the intermediate items, we see the problem: 0 * np.log(0) is resulting in a NaN. Though mathematically it's true that limx→0[xlog(x)]=0limx→0[xlog(x)]=0, the fact that we're performing the computation numerically means that we don't obtain this result.
Often, inserting a few print statements can be enough to figure out what's going on.
Can you write Python code for it?

def entropy2(p):
    p = np.asarray(p)  # convert p to array if necessary
    print(p)
    items = []
    for val in p:
        item = val * np.log(val)
        if np.isnan(item):
            print("%f makes a nan" % val)
        items.append(item)
    #items = p * np.log(ps)
    return -np.sum(items)

entropy2(p)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
How to find and resolve bugs?
Debugging has the following steps:

Detection of an exception or invalid results.
Isolation of where the program causes the error. This is often the most difficult step.
Resolution of how to change the code to eliminate the error. Mostly, it's not too bad, but sometimes this can cause major revisions in codes.

Detection of Bugs
The detection of bugs is too often done by chance. While running your Python code, you encounter unexpected functionality, exceptions, or syntax errors. While we'll focus on this in today's lecture, you should never leave this up to chance in the future.
Software testing practices allow for thoughtful detection of bugs in software. We'll discuss more in the lecture on testing.
Isolation of Bugs
There are three main methods commonly used for bug isolation:
1. The "thought" method. Think about how your code is structured and so what part of your could would most likely lead to the exception or invalid result.
2. Inserting print statements (or other logging techniques)
3. Using a line-by-line debugger like pdb.
Typically, all three are used in combination, often repeatedly.
Using print statements
Say we're trying to compute the entropy of a set of probabilities.  The
form of the equation is
$$
H = -\sum_i p_i \log(p_i)
$$
We can write the function like this:
Can you write Python code for it?

def entropy(p):
    """
     arg p: list of float
    """
    check1 = []
    for ele in p:
        check1.append((ele <= 1) and (ele >= 0))
    check2 = 1 == sum(p)
    if all(check1) and check2:
        items = p * np.log(p)
        return -np.sum(items)
    else:
        return -1

p = [1.0]
items = p * np.log(p)
print(items)

-sum(items)

print(entropy([0.5, 0.5]))
print(entropy([0.25, 0.25, 0.25, 0.25]))

entropy([1.0, 0])


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
INSTRUCTOR NOTE:

Run as-is. Run. Error. Returns SyntaxError: Missing parentheses in call to print.
Add parentheses. Run. Still an error. Returns SyntaxError: EOL while scanning string literal.
Add closing quotation mark. Run. Should be successful.
Can you write Python code for it?

x = 1; y = 2
b = x == y # Boolean variable that is true when x & y have the same value
b = 1 = 2


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
Note that the print statements significantly reduce legibility of the code. We would like to remove them when we're done debugging.
Can you write Python code for it?

def entropy(p):
    items = p * np.log(p)
    return -np.sum(items)

p = [0.1, 0.3, 0.5, 0.7, 0.9]
entropy(p)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
Now it works fine for the first set of inputs. Let's try other inputs.
We should have documented the inputs to the function!
Can you write Python code for it?

# Create a vector of probabilities.
p = np.arange(start=5., stop=-1., step=-0.5)
p /= np.sum(p)
p

entropy(p)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
Semantic errors
Say we're trying to confirm that a trigonometric identity holds. Let's use the basic relationship between sine and cosine, given by the Pythagorean identity"
$$
\sin^2 \theta + \cos^2 \theta = 1
$$
We can write a function to check this:
Can you write Python code for it?

import math

'''Checks that Pythagorean identity holds for one input, theta'''
def check_pythagorean_identity(theta):
    return math.sin(theta)**2 + math.cos(theta)**2 == 1

check_pythagorean_identity(.4)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
Using Python's debugger, pdb
Python comes with a built-in debugger called pdb.  It allows you to step line-by-line through a computation and examine what's happening at each step.  Note that this should probably be your last resort in tracing down a bug.  I've probably used it a dozen times or so in five years of coding.  But it can be a useful tool to have in your toolbelt.
You can use the debugger by inserting the line
python
import pdb; pdb.set_trace()
within your script. To leave the debugger, type "exit()". To see the commands you can use, type "help".
Let's try this out:
Can you write Python code for it?

def entropy(p):
    items = p * np.log(p)
    import pdb; pdb.set_trace()
    return -np.sum(items)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Debugging.ipynb".
The first task is:
We get nan, which stands for "Not a Number". What's going on here?
Let's add our print statements again, but it only fails later in the range of numbers. We may choose to print only if we find a nan.
Can you write Python code for it?

def entropy1(p):
    print("p=%s" % str(p))
    items = p * np.log(p)
    if [np.isnan(el) for el in items]:
        print(items)
    return -np.sum(items)

entropy1([.1, .2])

entropy1(p)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/Exceptions.ipynb".
The first task is:
Exceptions
An exception is an event, which occurs during the execution of a program, that disrupts the normal flow of the program's instructions.
You've already seen some exceptions in the Debugging lesson.
*
Many programs want to know about exceptions when they occur. For example, if the input to a program is a file path. If the user inputs an invalid or non-existent path, the program generates an exception. It may be desired to provide a response to the user in this case.
It may also be that programs will generate exceptions. This is a way of indicating that there is an error in the inputs provided. In general, this is the preferred style for dealing with invalid inputs or states inside a python function rather than having an error return.
Catching Exceptions
Python provides a way to detect when an exception occurs. This is done by the use of a block of code surrounded by a "try" and "except" statement.
Can you write Python code for it?

def divide(numerator, denominator):
    result = numerator/denominator
    print("result = %f" % result)

divide(1.0, 0)

def divide1(numerator, denominator):
    try:
        result = numerator/denominator
        print("result = %f" % result)
    except:
        print("You can't divide by 0!")

divide1(1.0, 'a')

divide1(1.0, 2)

divide1("x", 2)

def divide2(numerator, denominator):
    try:
        result = numerator / denominator
        print("result = %f" % result)
    except (ZeroDivisionError, TypeError) as err:
        print("Got an exception: %s" % err)

divide2(1, "X")

divide2("x, 2)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
Now let's consider a pattern test. Examining the structure of the calculation of $H$, we consider a situation in which there are $n$ equal probabilities. That is, $p_i = \frac{1}{n}$.
$$
H = -\sum_{i=1}^{n} p_i \log(p_i)
= -\sum_{i=1}^{n} \frac{1}{n} \log(\frac{1}{n})
= n (-\frac{1}{n} \log(\frac{1}{n}) )
= -\log(\frac{1}{n})
$$
For example, entropy([0.5, 0.5]) should be $-log(0.5)$.
Can you write Python code for it?

# Pattern test
def test_equal_probabilities(n):
    prob = 1.0/n
    ps = np.repeat(prob , n)
    if np.isclose(entropy(ps), -np.log(prob)):
        print("Worked!")
    else:
        import pdb; pdb.set_trace()
        print ("Bad result.")

# Run a test
test_equal_probabilities(100000)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
Test Files
Although I presented the elements of unittest in a notebook. your tests should be in a file. If the name of module with the code under test is foo.py, then the name of the test file should be test_foo.py.
The structure of the test file will be very similar to cells above. You will import unittest. You must also import the module with the code under test. Take a look at test_prime.py in this directory to see an example.
Discussion
Question: What tests would you write for a plotting function?
Test Driven Development
Start by writing the tests. Then write the code.
We illustrate this by considering a function geomean that takes a list of numbers as input and produces the geometric mean on output.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntryopy(unittest.TestCase):

    def test_oneshot(self):
        self.assertEqual(geomean([1,1]), 1)

    def test_oneshot2(self):
        self.assertEqual(geomean([3, 3, 3]), 3)

#test_setup(TestGeomean)

#def geomean(argument?):
#    return ?


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
Testing For Exceptions
Edge test cases often involves handling exceptions. One approach is to code this directly.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        try:
            entropy([0.1, 0.5])
            self.assertTrue(False)
        except ValueError:
            self.assertTrue(True)

#test_setup(TestEntropy)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
Unit Tests
Overview and Principles
Testing is the process by which you exercise your code to determine if it performs as expected. The code you are testing is referred to as the code under test.
There are two parts to writing tests.
1. invoking the code under test so that it is exercised in a particular way;
1. evaluating the results of executing code under test to determine if it behaved as expected.
The collection of tests performed are referred to as the test cases. The fraction of the code under test that is executed as a result of running the test cases is referred to as test coverage.
For dynamical languages such as Python, it's extremely important to have a high test coverage. In fact, you should try to get 100% coverage. This is because little checking is done when the source code is read by the Python interpreter. For example, the code under test might contain a line that has a function that is undefined. This would not be detected until that line of code is executed.
Test cases can be of several types. Below are listed some common classifications of test cases.
- Smoke test. This is an invocation of the code under test to see if there is an unexpected exception. It's useful as a starting point, but this doesn't tell you anything about the correctness of the results of a computation.
- One-shot test. In this case, you call the code under test with arguments for which you know the expected result.
- Edge test. The code under test is invoked with arguments that should cause an exception, and you evaluate if the expected exception occurrs.
- Pattern test - Based on your knowledge of the calculation (not implementation) of the code under test, you construct a suite of test cases for which the results are known or there are known patterns in these results that are used to evaluate the results returned.
Another principle of testing is to limit what is done in a single test case. Generally, a test case should focus on one use of one function. Sometimes, this is a challenge since the function being tested...
Can you write Python code for it?

import numpy as np
# Code Under Test
def entropy(ps):
    if any([(p < 0.0) or (p > 1.0) for p in ps]):
        raise ValueError("Bad input.")
    if sum(ps) > 1:
        raise ValueError("Bad input.")
    items = ps * np.log(ps)
    new_items = []
    for item in items:
        if np.isnan(item):
            new_items.append(0)
        else:
            new_items.append(item)
    return np.abs(-np.sum(new_items))

ps = [.8, .2]
ps = [ 1.00000001, 0]
[(p < 0.0) or (p > 1.0) for p in ps]

np.isclose(1.1, 1)

# Smoke test
entropy([0.2, 0.8])

# One shot test
entropy([1, 0, 0, 0])


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
You see that there are many, many cases to test. So far, we've been writing special codes for each test case. We can do better.
Testing Data Producing Codes
Much of your python (or R) codes will be creating and/or transforming dataframes. A dataframe is structured like a table with:

Columns that have values of the same type
Rows that have a value for each column
An index that uniquely identifies a row.
Can you write Python code for it?

def makeProbabilityMatrix(column_names, nrows):
    """
    Makes a dataframe with the specified column names such that each
    cell is a value in [0, 1] and columns sum to 1.
    :param list-str column_names: names of the columns
    :param int nrows: number of rows
    """
    df = pd.DataFrame(np.random.uniform(0, 1, (nrows, len(column_names))))
    df.columns = column_names
    for column in df.columns:
        df[column] = df[column]/df[column].sum()
    return df

# Smoke test
makeProbabilityMatrix(['a', 'b'], 3)

# Test 2: Check columns
COLUMNS = ['a', 'b']
df = makeProbabilityMatrix(COLUMNS, 3)
set(COLUMNS) == set(df.columns)


I am working on the file "Spring2019/04a_Exceptions_and_Testing/unit-tests.ipynb".
The first task is:
unittest provides help with testing exceptions.
Can you write Python code for it?

import unittest

# Define a class in which the tests will run
class TestEntropy(unittest.TestCase):

    def test_invalid_probability(self):
        with self.assertRaises(ValueError):
            entropy([0.1, 0.5])

suite = unittest.TestLoader().loadTestsFromTestCase(TestEntropy)
_ = unittest.TextTestRunner().run(suite)


I am working on the file "Spring2019/06a_Objects/Building Software With Objects.ipynb".
The first task is:
Attributes are data associated with an object (instance) or class. Object attributes (and methods) are specified by using "self". Instance attributes and methods are accessed using the dot "." operator.
Can you write Python code for it?

class Car(object):

    # The following method is called when the class
    # is created or "constructed". The variables "self.x" refers
    # to the variable "x" in a created object.
    def __init__(self, color, car_type, speed):
        self.color = color
        self.car_type = car_type
        self.speed = speed

car1 = Car("blue", "sedan", "very slow")
car2 = Car("red", "sedan", "not so slow")

print(car1.speed, car2.speed)

class Car(object):

    # The following method is called when the class
    # is created or "constructed". The variables "self.x" refers
    # to the variable "x" in a created object.
    def __init__(self, color, car_type, speed, sunroof=True):
        self.color = color
        self.car_type = car_type
        if isinstance(speed, int):
            self.speed = speed
        else:
            raise ValueError("Bad speed value.")
        self.sunroof = sunroof

car = Car("blue", "sedan", 100)

# Creating an object for a class with arguments in the __init__ method
car = Car("Blue", "HatchBack", 100)
car.color

# Creating an object for a class with arguments in the __init__ method
joe_car = Car("Blue", "Sedan", 100)
dave_car = Car("Red", "Sports", 150)
print ("Type of joe_car is %s. Type of dave_car is %s"% (type(joe_car), type(dave_car)))

# Accessed instance attributes
joe_car = Car("Blue", "Sedan", 100)
print ("Type of joe_car has (color, type, speed)=%s." % str((joe_car.color, joe_car.car_type, joe_car.speed)))


I am working on the file "Spring2019/06a_Objects/Building Software With Objects.ipynb".
The first task is:
EXERCISE: Change the constructor for Car to include the attribute "doors".
Instance Methods
Can you write Python code for it?

from IPython.display import Image
Image(filename='InstanceMethods.png')

#Class diagram
from IPython.display import Image
Image(filename='SingleClassDiagram.png', width=200, height=200)


I am working on the file "Spring2019/06a_Objects/Building Software With Objects.ipynb".
The first task is:
EXERCISE: Implement the stop and turn methods. Run the methods.
Inheritance
Inheritance is a common way that classes reuse data and code from other classes. A child class or derived class gets attributes and methods from its parent class.
Programmatically:
- Specify inheritance in the class statement
- Constructor for derived class (class that inherits) have access to the constructor of its parent.
Inheritance is represented in diagrams as an arror from the child class to its parent class.
Can you write Python code for it?

from IPython.display import Image
Image(filename='SimpleClassHierarchy.png', width=400, height=400)

# Code for inheritance
class Sedan(Car):
    # Sedan inherits from car

    def __init__(self, color, speed):
        """
        :param str color:
        :param int speed:
        """
        super().__init__(color, "Sedan", speed)

    def play_cd(self):
        print ("Playing cd in %s sedan" % self.color)

sedan = Sedan("Yellow", 1e6)
sedan.color

sedan.car_type

sedan.car_type

joe_car = Sedan("Blue", 100)
print ("Type of joe_car has (color, type, speed)=%s." % str((joe_car.color, joe_car.car_type, joe_car.speed)))


I am working on the file "Spring2019/06a_Objects/Building Software With Objects.ipynb".
The first task is:
Exercise: Implement SportsCar and create dave_car from SportsCar. Print attributes of dave_car.
Can you write Python code for it?

from IPython.display import Image
Image(filename='ClassInheritance.png', width=400, height=400)


I am working on the file "Spring2019/06a_Objects/Building Software With Objects.ipynb".
The first task is:
Initial concepts

An object is a container of data (attributes) and code (methods)
A class is a template for creating objects

Reuse is provided by:

reusing the same class to create many objects
"inheriting" data and code from other classes
Can you write Python code for it?

# Definiting a Car class
class Car(object):
    pass

car = Car()


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Главное про ускоренный метод Нестерова

Теоретически оптимальная скорость сходимости для выпуклых и сильно выпуклых функций
Необходимо подбирать шаг
Волнообразное поведение может быть подавлено с помощью рестартов
Нелинейные композиции градиентов могут дать более быстрые на практике методы (см самое начало семинара)
Оригинальное доказательство сложно понять на интуитивном уровне, есть гораздо более простые подходы, будут рассказаны позднее

Адаптивный выбор константы $L$

В методе Нестерова шаг у градиента равен $\frac{1}{L}$, где $L$ - константа Липшица градиента
Однако она обычно неизвестна
НО! У нас есть условие на эту константу, а именно

$$
f(y) \leq f(x) + \langle f'(x), y - x \rangle + \frac{L}{2}\|x - y\|_2^2
$$

Аналогия с правилами Армихо и прочими такого типа
Более подробно про этот метод смотрите в книге лектора

```python
def backtracking_L(f, h, grad, x, L0, rho):
L = L0

fx = f(x)

gradx = grad(x)

while True:

    y = x - 1 / L * h

    if f(y) <= fx - 1 / L gradx.dot(h) + 1 / (2 * L) h.dot(h):

        break

    else:

        L = L * rho

return L

```
Эксперименты
Can you write Python code for it?

beta_test = 0.9
methods = {
    "GD Armijo": fo.GradientDescent(f, grad,
            ss.Backtracking("Armijo", rho=0.5, beta=0.1, init_alpha=1.)),
    r"HB, $\beta = {}$".format(beta_test): HeavyBall(f, grad, ss.ConstantStepSize(1 / L), beta=beta_test),
    "Nesterov": fo.AcceleratedGD(f, grad, ss.ConstantStepSize(1 / L)),
    "Nesterov adaptive": fo.AcceleratedGD(f, grad, ss.Backtracking(rule_type="Lipschitz", rho=0.5, init_alpha=1)),
}
x0 = np.zeros(n)
max_iter = 2000
tol = 1e-6

for m in methods:
    _ = methods[m].solve(x0=x0, max_iter=max_iter, tol=tol)

figsize = (10, 8)
fontsize = 26
plt.figure(figsize=figsize)
for m in methods:
    plt.semilogy([np.linalg.norm(grad(x)) for x in methods[m].get_convergence()], label=m)
plt.legend(fontsize=fontsize, loc="best")
plt.xlabel("Number of iteration, $k$", fontsize=fontsize)
plt.ylabel(r"$\| f'(x_k)\|_2$", fontsize=fontsize)
plt.xticks(fontsize=fontsize)
_ = plt.yticks(fontsize=fontsize)

for m in methods:
    print(m)
    %timeit methods[m].solve(x0=x0, max_iter=max_iter, tol=tol)


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
График сходимости
Can you write Python code for it?

plt.figure(figsize=(8,6))
plt.semilogy([np.linalg.norm(grad_f(x)) for x in cg_quad.get_convergence()], label=r"$\|f'(x_k)\|^{CG}_2$", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in scopt_cg_array[:max_iter]], label=r"$\|f'(x_k)\|^{CG_{PR}}_2$", linewidth=2)
plt.semilogy([np.linalg.norm(grad_f(x)) for x in gd.get_convergence()], label=r"$\|f'(x_k)\|^{G}_2$", linewidth=2)
plt.legend(loc="best", fontsize=20)
plt.xlabel(r"Iteration number, $k$", fontsize=20)
plt.ylabel("Convergence rate", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)

print([np.linalg.norm(grad_f(x)) for x in cg_quad.get_convergence()])

plt.figure(figsize=(8,6))
plt.plot([f(x) for x in cg_quad.get_convergence()], label=r"$f(x^{CG}_k)$", linewidth=2)
plt.plot([f(x) for x in scopt_cg_array], label=r"$f(x^{CG_{PR}}_k)$", linewidth=2)
plt.plot([f(x) for x in gd.get_convergence()], label=r"$f(x^{G}_k)$", linewidth=2)
plt.legend(loc="best", fontsize=20)
plt.xlabel(r"Iteration number, $k$", fontsize=20)
plt.ylabel("Function value", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Неквадратичная функция
Can you write Python code for it?

import numpy as np
import sklearn.datasets as skldata
import scipy.special as scspec

n = 300
m = 1000

X, y = skldata.make_classification(n_classes=2, n_features=n, n_samples=m, n_informative=n//3)
C = 1
def f(w):
    return np.linalg.norm(w)**2 / 2 +  C * np.mean(np.logaddexp(np.zeros(X.shape[0]), -y * X.dot(w)))

def grad_f(w):
    denom = scspec.expit(-y * X.dot(w))
    return w - C * X.T.dot(y * denom) / X.shape[0]
# f = lambda x: -np.sum(np.log(1 - A.T.dot(x))) - np.sum(np.log(1 - x*x))
# grad_f = lambda x: np.sum(A.dot(np.diagflat(1 / (1 - A.T.dot(x)))), axis=1) + 2 * x / (1 - np.power(x, 2))
x0 = np.zeros(n)
print("Initial function value = {}".format(f(x0)))
print("Initial gradient norm = {}".format(np.linalg.norm(grad_f(x0))))


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Правильный ответ
Can you write Python code for it?

import scipy.optimize as scopt

def callback(x, array):
    array.append(x)

scopt_cg_array = []
scopt_cg_callback = lambda x: callback(x, scopt_cg_array)
x = scopt.minimize(f, x0, method="CG", jac=grad_f, callback=scopt_cg_callback)
x = x.x
print("||f'(x*)|| =", np.linalg.norm(A.dot(x) - b))
print("f* =", f(x))


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Распределение собственных значений
Can you write Python code for it?

USE_COLAB = False

%matplotlib inline
import matplotlib.pyplot as plt
if not USE_COLAB:
    plt.rc("text", usetex=True)
    plt.rc("font", family='serif')

if USE_COLAB:
    !pip install git+https://github.com/amkatrutsa/liboptpy

import seaborn as sns
sns.set_context("talk")

eigs = np.linalg.eigvalsh(A)
plt.semilogy(np.unique(eigs))
plt.ylabel("Eigenvalues", fontsize=20)
plt.xticks(fontsize=18)
_ = plt.yticks(fontsize=18)


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Реализация метода сопряжённых градиентов
Can you write Python code for it?

def ConjugateGradientQuadratic(x0, A, b, tol=1e-8, callback=None):
    x = x0
    r = A.dot(x0) - b
    p = -r
    while np.linalg.norm(r) > tol:
        alpha = r.dot(r) / p.dot(A.dot(p))
        x = x + alpha * p
        if callback is not None:
            callback(x)
        r_next = r + alpha * A.dot(p)
        beta = r_next.dot(r_next) / r.dot(r)
        p = -r_next + beta * p
        r = r_next
    return x

import liboptpy.unconstr_solvers as methods
import liboptpy.step_size as ss

max_iter = 70
print("\t CG quadratic")
cg_quad = methods.fo.ConjugateGradientQuad(A, b)
x_cg = cg_quad.solve(x0, tol=1e-7, max_iter=max_iter, disp=True)

print("\t Gradient Descent")
gd = methods.fo.GradientDescent(f, grad_f, ss.ExactLineSearch4Quad(A, b))
x_gd = gd.solve(x0, tol=1e-7, max_iter=max_iter, disp=True)

print("Condition number of A =", abs(max(eigs)) / abs(min(eigs)))


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Резюме

Сопряжённые направления
Метод сопряжённых градиентов
Сходимость
Эксперименты

Схема получения оценок снизу на сложность методов и задач

Фиксируем класс функций $\mathcal{F}$
Фиксируем класс методов оптимизации $\mathcal{M}$
Ищем настолько плохую функцию из класса $\mathcal{F}$, что любой метод из класса $\mathcal{M}$ сходится не лучше, чем некоторая оценка
Такая оценка называется оценкой снизу

Пример
Фиксируем класс методов
Рассмотрим такие методы, что
$$
x_{k+1} = x_0 + \texttt{span}{f'(x_0), \ldots, f'(x_k)}
$$

Далее в рамках этого семинара для краткости только такие методы будем называть методами первого порядка
Весь последующий анализ НЕ применим, если

$$
x_{k+1} = x_0 + G(f'(x_0), \ldots, f'(x_k)),
$$
где $G$ - некоторая нелинейная функция

С такими методами мы познакомимся ближе к середине семестра

Фиксируем класс функций
Выпуклые функции с липшицевым градиентом
Теорема. Существует выпуклая функция с Липшицевым градиентом, такая что
$$
f(x_t) - f^ \geq \frac{3L\|x_0 - x^\|_2^2}{32(t+1)^2},
$$
где $x_k = x_0 + \texttt{span}{f'(x_0), \ldots, f'(x_{k-1})}$, $1 \leq k \leq t$
Привести пример такой функции и доказать эту теорему Вам надо в домашнем задании.
Сильно выпуклые функции с липшицевым градиентом
Теорема. Существует сильно выпуклая функция с Липшицевым градиентом, такая что
$$
f(x_t) - f^ \geq \frac{\mu}{2}\left( \frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1} \right)^{2t}\|x_0 - x^\|_2^2,
$$
где $\kappa = \frac{L}{\mu}$ и $x_k = x_0 + \texttt{span}{f'(x_0), \ldots, f'(x_{k-1})}$, $1 \leq k \leq t$
Привести пример такой функции и доказать эту теорему Вам надо в домашнем задании.
Оценки сходимости известных методов
Оценки сходимости для градиентного спуска: напоминание

Пусть

$f(x)$ дифференцируема на $\mathbb{R}^n$
$f(x)$ выпукла
$f'(x)$ удовлетворяет условию Липшица с константой $L$
$\alpha = \dfrac{1}{L}$

Тогда
$$
f(x_k) - f^ \leq \dfrac{2L \| x_0 - x^\|^2_2}{k+4}
$$

Пусть

$f(x)$ дифференцируема на $\mathbb{R}^n$,
градиент $f(x)$ удовлетворяет условию Липшица с константой...
Can you write Python code for it?

USE_COLAB = False
if USE_COLAB:
    !pip install git+https://github.com/amkatrutsa/liboptpy

import liboptpy.base_optimizer as base
import numpy as np
import liboptpy.unconstr_solvers.fo as fo
import liboptpy.step_size as ss
import matplotlib.pyplot as plt
%matplotlib inline

if not USE_COLAB:
    plt.rc("text", usetex=True)

class HeavyBall(base.LineSearchOptimizer):
    def __init__(self, f, grad, step_size, beta, **kwargs):
        super().__init__(f, grad, step_size, **kwargs)
        self._beta = beta

    def get_direction(self, x):
        self._current_grad = self._grad(x)
        return -self._current_grad

    def _f_update_x_next(self, x, alpha, h):
        if len(self.convergence) < 2:
            return x + alpha * h
        else:
            return x + alpha * h + self._beta * (x - self.convergence[-2])

    def get_stepsize(self):
        return self._step_size.get_stepsize(self._grad_mem[-1], self.convergence[-1], len(self.convergence))

np.random.seed(42)
n = 100
A = np.random.randn(n, n)
A = A.T.dot(A)
x_true = np.random.randn(n)
b = A.dot(x_true)
f = lambda x: 0.5 * x.dot(A.dot(x)) - b.dot(x)
grad = lambda x: A.dot(x) - b
A_eigvals = np.linalg.eigvalsh(A)
L = np.max(A_eigvals)
mu = np.min(A_eigvals)
print(L, mu)
print("Condition number = {}".format(L / mu))
alpha_opt = 4 / (np.sqrt(L) + np.sqrt(mu))**2
beta_opt = np.maximum((1 - np.sqrt(alpha_opt * L))**2,
                      (1 - np.sqrt(alpha_opt * mu))**2)
print(alpha_opt, beta_opt)
beta_test = 0.95

methods = {
    "GD fixed": fo.GradientDescent(f, grad, ss.ConstantStepSize(1 / L)),
    "GD Armijo": fo.GradientDescent(f, grad,
            ss.Backtracking("Armijo", rho=0.5, beta=0.1, init_alpha=1.)),
    r"HB, $\beta = {}$".format(beta_test): HeavyBall(f, grad, ss.ConstantStepSize(1 / L), beta=beta_test),
    "HB optimal": HeavyBall(f, grad, ss.ConstantStepSize(alpha_opt), beta = beta_opt),
    "CG": fo.ConjugateGradientQuad(A, b)
}
x0 = np.random.randn(n)
max_iter = 5000
tol = 1e-6

for m in methods:
    _ =...


I am working on the file "Spring2020/acc_grad.ipynb".
The first task is:
Тестовая задача 2
$$
f(x) = - \sum_{i=1}^m \log(1 - a_i^{\top}x) - \sum_{i = 1}^n \log (1 - x^2_i) \to \min_x
$$
Can you write Python code for it?

n = 300
m = 1000

X, y = skldata.make_classification(n_classes=2, n_features=n, n_samples=m, n_informative=n//3)
C = 1
def f(w):
    return np.linalg.norm(w)**2 / 2 +  C * np.mean(np.logaddexp(np.zeros(X.shape[0]), -y * X.dot(w)))

def grad(w):
    denom = scspec.expit(-y * X.dot(w))
    return w - C * X.T.dot(y * denom) / X.shape[0]
# f = lambda x: -np.sum(np.log(1 - A.T.dot(x))) - np.sum(np.log(1 - x*x))
# grad_f = lambda x: np.sum(A.dot(np.diagflat(1 / (1 - A.T.dot(x)))), axis=1) + 2 * x / (1 - np.power(x, 2))
x0 = np.zeros(n)
print("Initial function value = {}".format(f(x0)))
print("Initial gradient norm = {}".format(np.linalg.norm(grad_f(x0))))

alpha_test = 5e-3
beta_test = 0.9

methods = {
    r"GD, $\alpha_k = {}$".format(alpha_test): fo.GradientDescent(f, grad, ss.ConstantStepSize(alpha_test)),
    "GD Armijo": fo.GradientDescent(f, grad,
            ss.Backtracking("Armijo", rho=0.7, beta=0.1, init_alpha=1.)),
    r"HB, $\beta = {}$".format(beta_test): HeavyBall(f, grad, ss.ConstantStepSize(alpha_test), beta=beta_test),

}
# x0 = np.random.rand(n)
x0 = np.zeros(n)
max_iter = 400
tol = 1e-5

for m in methods:
    print(m)
    _ = methods[m].solve(x0=x0, max_iter=max_iter, tol=tol)

scopt_cg_array = []
scopt_cg_callback = lambda x: callback(x, scopt_cg_array)
x = scopt.minimize(f, x0, tol=tol, method="CG", jac=grad, callback=scopt_cg_callback, options={"maxiter": max_iter})
x = x.x

figsize = (10, 8)
fontsize = 26
plt.figure(figsize=figsize)
for m in methods:
    plt.semilogy([np.linalg.norm(grad(x)) for x in methods[m].get_convergence()], label=m)
plt.semilogy([np.linalg.norm(grad(x)) for x in scopt_cg_array], label="CG PR")
plt.legend(fontsize=fontsize, loc="best")
plt.xlabel("Number of iteration, $k$", fontsize=fontsize)
plt.ylabel(r"$\| f'(x_k)\|_2$", fontsize=fontsize)
plt.xticks(fontsize=fontsize)
_ = plt.yticks(fontsize=fontsize)

for m in methods:
    print(m)
    %timeit methods[m].solve(x0=x0, max_iter=max_iter, tol=tol)


I am working on the file "Spring2020/newton_quasi.ipynb".
The first task is:
Вспомогательные функции
Can you write Python code for it?

f = lambda x: -np.sum(np.log(1 - A.T.dot(x))) - np.sum(np.log(1 - x*x))
grad_f = lambda x: np.sum(A.dot(np.diagflat(1 / (1 - A.T.dot(x)))), axis=1) + 2 * x / (1 - np.power(x, 2))
hess_f = lambda x: (A.dot(np.diagflat(1 / (1 - A.T.dot(x))**2))).dot(A.T) + np.diagflat(2 * (1 + x**2) / (1 - x**2)**2)