I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
<div id='func' />

Loss of significance in function evaluation
Back to TOC
Loss of Significance is present too in the representation of functions. A classical example (which you can see in the guide book), is the next function:
$$
\begin{equation}
    f_1(x)= \frac{1 - \cos x}{\sin^{2}x}
\end{equation}
$$
Applying trigonometric identities, we can obtain the 'equivalent' function:
$$
\begin{equation}
    f_2(x)= \frac{1}{1 + \cos x}
\end{equation}
$$
Both of these functions are apparently equals in exact arithmetic. Nevertheless, its graphics say to us another thing when $x$ is equal to zero.
Before we analize the behaviour about $x=0$, let's take a look at them in the range $x\in[-10,10]$.
Can you write Python code for it?

f1 = lambda x: (1.-np.cos(x))/(np.power(np.sin(x),2))
f2 = lambda x: 1./(1+np.cos(x))

x = np.arange(-10,10,0.1)
plt.figure()
plt.plot(x,f1(x),'-')
plt.grid(True)
plt.show()


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
<div id='intro' />

Introduction
Back to TOC
Hello! This notebook is an introduction to how our computers handle the representation of real numbers using double-precision floating-point standard IEEE 754.
To understand the contents of this notebook you should have at least a basic notion of how binary numbers work.
The double-precision floating-point format occupies 64 bits which are divided as follows:

1 bit for the sign
11 bits for the exponent
52 bits for the mantissa

This means that the very next representable number after $1$ is $1 + 2^{-52}$, and their difference, $2^{-52}$, is the called $\epsilon_{mach}$.
Additionally, if you'd like to quickly go from a base-2 integer to a base-10 integer and viceversa, Python has some functions that can help you with that.
Can you write Python code for it?

# This code translate from a binary number using the prefix "0b" to a base 10 number.
int('0b11', 2)

# This code translate from base 10 to base 2.
bin(9)

# Just looking a large binary number
bin(2**53)


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
As you can see, the hypothesis was right. In other words: Floating point numbers are not uniformly distributed across the real numbers, and the distance between them is proportional to their magnitude.
Tiny numbers (~ 0) are closer between each other than larger numbers are.
Moreover, we can conclude that for large values the gap is larger that $1$, which means that there will be integers that will not be stored!!.
<div id='visualization' />

Visualization of floating point numbers
Back to TOC
With the help of bitstring library we could write a function to visualize floating point numbers in its binary representation.
Can you write Python code for it?

# This function shows the bits used for the sign, exponent and mantissa for a 64-bit double presision number.
# fps: Floating Point Standard
# Double: Double precision IEEE 754
def to_fps_double(f):
    b = bs.pack('>d', f)
    b = b.bin
    #show sign + exponent + mantisa
    print(b[0]+' '+b[1:12]+ ' '+b[12:])


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
As you may have expected, the error grows proportionally with the value of $x$.
It is importnant however that you don't get confused with the apparently small difference between the blue and the red lines, actuallu that difference is about 16-order of magnitud!
Just look at the $y$-axis scale, it is a logarithmic scale.
The outcome of this example is that it tells us that the absolute error we are making when we store the value $x$ as a double precision floating point fl $(x)$ is proportional to $x$, this is what it is.
The key point is that we must always remember this.
<div id='loss' />

Loss of significance
Back to TOC
As we mentioned, there's a small gap between 1 and the next representable number, which means that if you want to represent a number between those two, you won't be able to do so; what you would need to do is to round it to a representable number before storing it in memory.
Can you write Python code for it?


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
From teh previous table, we see that the numerator of $f_1(x)$ becomes $0$. What is happening with $1-\cos(x)$ about $x=0$?
In the following code we study numerically what happens with $1-\cos(x)$ as $x$ tends to $0^+$.
Can you write Python code for it?

x = np.logspace(-20,0,40)
plt.figure()
plt.loglog(x,1-np.cos(x),'.',label='$1-\cos(x)$')
plt.loglog(x,0*x+1e-20,'.', label='$10^{-20}$')
plt.grid(True)
plt.xlabel('$x$')
plt.legend(loc='best')
plt.show()

# For this value of x=1e-7 we obtain an outcome greater than 0
to_fps_double(1-np.cos(1.e-7))
# But for x=1e-8 we actually get 0. This explains why in the previous plot the blue dots stop appearing for values less or equal than 1e-8 approximately.
to_fps_double(1-np.cos(1.e-8))


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
I can't stored the Integer $2^{53}+1$! Thus, the first integer not representable is $2^{53}+1$.
<div id='error_bound' />

Understanding the error bound between |fl(x)-x|
Back to TOC
The following code shows a plot of the upper bound of the absolute error we are making when storing the value $x$ as a floating point in double precision, i.e. fl $(x)$.
This means:
$$
    \begin{equation}
        |\text{fl}(x)-x| \leq \frac{1}{2} \epsilon_{\text{mach}} |x|
    \end{equation}.
$$
Can you write Python code for it?

x = np.logspace(-200,800,1000, base=2, dtype=np.dtype(float))
plt.figure()
plt.loglog(x,x,'b',label='fl$(x)$: value stored')
plt.loglog(x,np.power(2.,-52)*np.abs(x)/2,'r',label='Upper bound of $|$fl$(x)-x|$ estimated')
plt.legend(loc='best')
plt.grid(True)
plt.show()


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
In order to prove our hypothesis (that floating point numbers are not uniformly distributed), we will create an array of values: $[2^{-5},...,2^{60}]$ and compute their corresponding gaps.
Can you write Python code for it?

values = np.array([2**i for i in range(-5,60)]).astype(float)

# Corresponding gaps:
# The Numpy function "vectorize" is very useful to be use one wants to apply
# a scalar function to each element of an array.
vgap = np.vectorize(gap)
gaps = vgap(values)


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
In this case we see a different behavior. Is this the correct one?
Yes!
This happens because when $x$ is equal to zero, the first function has an indetermination, but previously, the computer makes a subtraction between numbers that are almost equals. This generates a loss of significance, turning the expression close to this point to be zero.
However, modifying this expression to make the second function, eliminates this substraction, fixing the error in its calculation when $x=0$.
In conclusion, for us, two representations of a function can be equals in exact arithmetic, but for the computer they can be different!
<div id='another' />

Another analysis (example from textbook)
Back to TOC
The following code tries to explain why $f_1(x)$ gives $0$ near $x=0$.
Can you write Python code for it?

# This function corresponds to the numerator of f1(x)
f3 = lambda x: (1.-np.cos(x))
# This function corresponds to the denominator of f2(x)
f4 = lambda x: np.power(np.sin(x),2)
x = np.flip(np.logspace(-19,0,20))
o1 = f1(x)
o2 = f2(x)
o3 = f3(x)
o4 = f4(x)

print("x,            f1(x),        f2(x),        f3(x),                       f4(x)")
for i in np.arange(len(x)):
    print("%1.10f, %1.10f, %1.10f, %1.25f, %1.25f" % (x[i],o1[i],o2[i],o3[i],o4[i]))


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
Let's see some intereseting examples
Can you write Python code for it?

to_fps_double(1.)

int('0b01111111111', 2)

to_fps_double(1.+gap(1.))

to_fps_double(+0.)

to_fps_double(-0.)

to_fps_double(np.inf)

to_fps_double(-np.inf)

to_fps_double(np.nan)

to_fps_double(-np.nan)

to_fps_double(2.**-1074)

print(2.**-1074)

to_fps_double(2.**-1075)

print(2.**-1075)

to_fps_double(9.4)


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
Now, we can scale this difference such that the scaling factor multiplied but $\epsilon_{\text{mach}}$ is one. The factor will be $2^{52}$. This means $2^{52}\,\epsilon_{\text{mach}}=1$. Repeating the same example as before but with the scaling factor we obtain:
Can you write Python code for it?


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
The first plot shows some spikes, are these expected? or is it an artifact?
Notice that we mean that something it is an artifact when it only appears due to a numerical computation but it should not be there theoretically.
Are these spikes real or not?
Can you write Python code for it?

plt.figure()
plt.plot(x,f2(x),'-')
plt.grid(True)
plt.show()


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
This means, the distance now from $2^{52}$ and the following number representable is $1$ !!!! So, what would happend if I can to store $2^{53}+1$?
Can you write Python code for it?


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
We include now a comparison between a linear scale plot and a loglog scale plot. Which one is more useful here?
Can you write Python code for it?

fig = plt.figure(figsize=(10,5))

plt.subplot(121)
plt.plot(values, gaps,'.',markersize=20)
plt.xlabel('float(x)')
plt.ylabel('Gap between next representable number')
plt.title('Linear scale')
plt.grid(True)

plt.subplot(122)
plt.loglog(values, gaps,'.')
plt.xlabel('float(x)')
plt.ylabel('Gap between next representable number')
plt.title('Log scale')
plt.grid(True)

fig.tight_layout()
plt.show()


I am working on the file "SC1v2/02_floating_point_arithmetic.ipynb".
The first task is:
We'll let the students find some representable numbers and some non-representable numbers.
Can you write Python code for it?

num_1 = a
num_2 = b
result = a + b
print("{0:.100}".format(result))


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:

Acknowledgements
Back to TOC

Material created by professor Claudio Torres (ctorres@inf.utfsm.cl) and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. March 2016. v1.1.
Update April 2020 - v1.32 - C.Torres : Re-ordering the notebook.
Update April 2021 - v1.33 - C.Torres : Updating format and re-re-ordering the notebook. Adding 'maxNumberIterations' to bisection, fpi and Newton's method. Adding more explanations.
Update April 2021 - v1.33 - C.Torres : Updating description and solution of 'Proposed classwork'.
Update September 2021 - v1.35 - C.Torres : Updating and commeting code more.
Update September 2021 - v1.36 - C.Torres : Updating the way we show the output tables.
Update September 2021 - v1.37 - C.Torres : Fixing typo suggested by Nicolás Tapia 2021-2. Thanks Nicolás! And removing extra code.

Extra examples
Back to TOC
Propose Classwork

Build a FPI such that given $a$ computes $\displaystyle \frac{1}{a}$. The constraint is that you can't use a division in the 'final' FPI. Write down your solution below or go and see the solution.

Hint: I strongly suggest to use Newton's method.
Can you write Python code for it?

print('Please try to solve it before you see the solution!!!')


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
<div id='bisection' />

Bisection Method
Back to TOC
The bisection method finds the root of a function $f$.
It requires that:
1. $f$ be a continuous function.
2. The interval $[a,b]$, such that $f(a)\cdot f(b) < 0$.
If these 2 conditions are satisfied, it means that there is a value $r$, between $a$ and $b$, for which $f(r) = 0$.
To summarize how this method works, start with the aforementioned interval (checking that there's a root in it), and split it into two smaller intervals $[a,c]$ and $[c,b]$.
Then, check which of the two intervals contains a root.
Keep splitting each "eligible" interval until the algorithm converges or the tolerance is achived.
Can you write Python code for it?

def bisect(f, a, b, tol=1e-5, maxNumberIterations=100):
    # Evaluating the extreme points of the interval provided
    fa = f(a)
    fb = f(b)
    # Iteration counter.
    i = 0
    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None

    # Output table to store the numerical evolution of the algorithm
    output_table = []

    # Main loop: it will iterate until it satisfies one of the two criterias:
    # The tolerance 'tol' is achived or the max number of iterations is reached.
    while ((b-a)/2 > tol) and i<=maxNumberIterations:
        # Obtaining the midpoint of the interval. Quick question: What could happen if a different point is used?
        c = (a+b)/2.
        # Evaluating the mid point
        fc = f(c)
        # Saving the output data
        output_table.append([i, a, c, b, fa, fc, fb, b-a])

        # Did we find the root?
        if fc == 0:
            print('f(c)==0')
            break
        elif np.sign(fa*fc) < 0:
            # This first case consider that the new inetrval is defined by [a,c]
            b = c
            fb = fc
        else:
            # This second case consider that the new interval is defined by [c,b]
            a = c
            fa = fc
        # Increasing the iteration counter
        i += 1

    # Showing final output table
    columns    = ['$i$', '$a_i$', '$c_i$', '$b_i$', '$f(a_i)$', '$f(c_i)$', '$f(b_i)$', '$b_i-a_i$']
    df = pd.DataFrame(data=output_table, columns=columns)
    display(df)

    # Computing the best approximation obtaind for the root, which is the midpoint of the final interval.
    xc = (a+b)/2.
    return xc

# Initial example
f1 = lambda x: x+np.log(x)-3
# A different function, notice that x is multiplied to the exponential now and not added, as before.
f2 = lambda x: x*np.exp(x)-3
# This is the introductory example about Fixed Point Iteration
f3 = lambda x: np.cos(x)-x
bisect(f1,1e-10,3) # Recall to...


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
<div id='fpi-textbook-example' />

FPI - example from textbook
Back to TOC
This example is from the textbook. We are trying to find a root of $f(x)=x^3+x-1$.
Can you write Python code for it?

# These are the three functions proposed.
g1 = lambda x: 1-x**3
g2 = lambda x: (1-x)**(1/3)
g3 = lambda x: (1+2*x**3)/(1+3*x**2)
# Change the input function to evaluate different functions.
# Are the three functions convergent fixed point iterations?
fpi(g3, 0.75, 20, True)

# This is a "hack" to improve the convergence of g2!
a  = -0.6
g4 = lambda x: x+a*(x-g2(x))
fpi(g4, 0.75, 10, True)
# Why does this hack works?


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:

Introduction
Back to TOC
In this document we're going to study how to find roots of 1D equations using numerical methods.
First, let's start with the definition of a root:
Definition: The function $f(x)$ has a root in $x = r$ if $f(r) = 0$.
Example: Let's say we want to solve the equation $r + \log(r) = 3$.
We can re-arrange the equation as follows: $r + \log(r) - 3 = 0$.
Thus, solving the previous equation is equivalent to find the root of $f(x) = x + \log(x) - 3$.
This example shows how we can translate an equation into a root-finding problem!
We will study now several numerical methods to find roots.
We will start by defining a function $f(x)$ using a lambda definition.
Can you write Python code for it?


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
<div id='sol1' />

Solutions
Problem: Build a FPI such that given $a$ computes $\displaystyle \frac{1}{a}$
Solution: Apply Newton's method to the following function: $f(x)=a-\frac{1}{x}$. Make sure you simplify the final expression.
Can you write Python code for it?

# We are finding the 1/a
# Solution code:
a = 2.1
g = lambda x: 2*x-a*x**2
gp = lambda x: 2-2*a*x
r=fpi(g, 0.7, 7, flag_cobweb=False)
print('Reciprocal found :',r)
print('Reciprocal computed explicitly: ', 1/a)
# Are we seeing quadratic convergence?


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
First example, let's compute a root of $\sin(x)$, near $x_0=3.1$.
Can you write Python code for it?

# Example funtion
f = lambda x: np.sin(x)
# The derivative of f
fp = lambda x: np.cos(x)
newton_method(f, fp, 3.1,rel_error=1e-15)


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
However, we prefer convergent fixed-point-iterations! Here is interesting way to make a non-convergent FPI into a convergent one.
Can you write Python code for it?

# This is a "palta" hidden in the code! Think about it. Quick question: what is it doing?
a=-1/(1-(-1.7)) # a = -1 / (1 - g'(r)), where does "a" come from?
G = lambda x: x+a*(x-g(x))
fpi(G, 1, 14, True)


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
It's very important to define a concept called convergence rate.
This rate shows how fast the convergence of a method is at a specified point.
The convergence rate for the bisection is always 0.5 because this method uses the half of the interval for each iteration.
In this particular case we observe $e_{i+1} \approx \dfrac{e_{i}}{2}$, why? where?
<div id='fpi' />

Fixed Point Iteration and Cobweb diagram
Back to TOC
To learn about the Fixed-Point Iteration we will first learn about the concept of a Fixed Point.
A Fixed Point of a function $g$ is a real number $r$, where $g(r) = r$
The Fixed-Point Iteration is based in the Fixed Point concept and works like this to find the root of a function:
\begin{align}
    x_{0} &= initial_guess \
    x_{i+1} &= g(x_{i})
\end{align}
To find an equation's root using this method, we'll have to rearrange the equation to make it of the following form $x = g(x)$.
For example, if we want to obtain the root of $f(r)=0$, one could add a zero convenient this way, $f(r)+r=r$, i.e. we add it $r$ on both sides.
This way we have $g(r)=r+f(r)$ and the fixed point iteration could be performed.
In the following example, we'll find the root of $f(x) = x - \cos(x)$ by iterating over the funcion $g(x) = \cos(x)$.
Can you write Python code for it?

# Just plotting the Cobweb diagram: https://en.wikipedia.org/wiki/Cobweb_plot
def cobweb(x,g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()

# This code performs the fixed point iteration.
def fpi(g, x0, k, flag_cobweb=False):
    # This is where we store all the approximation,
    # this is technically not needed but we store them because we need them for the cobweb diagram at the end.
    x = np.empty(k+1)
    # Just starting the fixed point iteration from the 'initial guess'
    x[0] = x0
    # Initializing the error in NaN
    error_i = np.nan

    # Output table to store the numerical evolution of the algorithm
    output_table = []

    # Main loop
    for i in range(k):
        # Iteration
        x[i+1] = g(x[i])
        # Storing error from previous iteration
        error_iminus1 = error_i
        # Computing error for current iteration.
        # Notice that from the theory we need to compute e_i=|x_i-r|, i.e. we need the root 'r'
        # but we don't have it, so we approximate it by 'x_{i+1}'.
        error_i = abs(x[i]-x[i+1])
        output_table.append([i,x[i],x[i+1],error_i,error_i/error_iminus1,error_i/(error_iminus1**((1+np.sqrt(5))/2.)),error_i/(error_iminus1**2)])

    # Showing final output table
    columns    = ['$i$', '$x_i$', '$x_{i+1}$', '$e_i$', r'$\frac{e_i}{e_{i-1}}$',...


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
Let's quickly explain the Cobweb Diagram we have here. The blue line is the function $y=x$ and the red is the function $y=g(x)$.
The point in which they meet is $r=g(r)$, i.e. the fixed point.
In this particular example, we start at $y = x = 1.5$ (the top right corner) and then we "jump" vertically to $y = \cos(1.5) \approx 0.07$.
After this, we jump horizontally to $x = \cos(1.5) \approx 0.07$.
Then, we jump again vertically to $y = \cos\left(\cos(1.5)\right) \approx 0.997$ and so on.
See the pattern here? We're just iterating over $x = \cos(x)$, getting closer to the center of the diagram where the fixed point resides, in $x \approx 0.739$.
It's very important to mention that the algorithm will converge only if the rate of convergence $S < 1$, where $S = \left| g'(r) \right|$.
If you want to use this method, you'll have to construct $g(x)$ starting from $f(x)$ accordingly.
In this example, $g(x) = \cos(x) \Rightarrow g'(x) = -\sin(x)$ and $|-\sin(0.739)| \approx 0.67$.
Quick question: Do you see the value 0.67 in the previous table?
Another example. Look at this web page to undertand the context: https://divisbyzero.com/2008/12/18/sharkovskys-theorem/amp/?__twitter_impression=true
Can you write Python code for it?

# Consider this funtion
g = lambda x: -(3/2)*x**2+(11/2)*x-2
# Here we compute the derivative of it.
gp = lambda x: -3*x+11/2

# We plot now the funcion itself (red), its derivative (magenta) and the function y=x (blue).
# We also plot the values -1 and 1 with green dashed curves.
# This analyis shows that the fixed point, which is the intersection between teh red and blue curves,
# does not generate a convergent fix-point-iteration since the derivative (magenta curve) has a value
# lower then -1 about the fized point.
x=np.linspace(2,3,100)
plt.figure(figsize=(8,8))
plt.plot(x,g(x),'r-',label=r'$g(x)$')
plt.plot(x,x,'b-',label=r'$x$')
plt.plot(x,gp(x),'m-',label=r'$g\'(x)$')
plt.plot(x,gp(x)*0+1,'g--')
plt.plot(x,gp(x)*0-1,'g--')
plt.grid(True)
plt.legend(loc='best')
plt.show()


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
Notice that we have used the NumPy implementation for the logarithmic function. Quick question: what is the base for this implementation? Is it the natural logarithm or logarithm base 10?
But before we start working on a numerical implementation, we should always consider in solving the problem algebraically.
This can be done with SymPy, i.e. using symbolic computation.
For instance, we will start by defining a symbolic variable:
Can you write Python code for it?

# Definition of symbolic variable
x = sym.Symbol('x')
# Defining 'symbolic' function
fsym = lambda x: x+sym.log(x)-3
# Finding the root 'symbolically' and obtaining the only root
r=sym.solve(sym.Eq(fsym(x), 0), x)[0]
print(textBoldH('Root obtained:'),r)
print(textBoldH('Numerical root:'),r.evalf())


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
Now that we have found the root, let's compute the derivative of each $g(x)$ used previously and understand what exactly was going on.
Can you write Python code for it?


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
Playing with some roots
The following example proposed a particular function $f(x)$.
The idea here is first obtain an initial guess for applying the Newton's method from the plot in semilogy scale.
The plot of $f(x)$ (blue) shows that there seems to be 2 roots in the interval plotted.
Now, the plot of $f'(x)$ (magenta) indicates that the derivative may also have a 0 together with a root, this means that the multiplicity of that root may be higher than 1. Do you see it?
Can you write Python code for it?

f = lambda x: 8*x**4-12*x**3+6*x**2-x
fp = lambda x: 32*x**3-36*x**2+12*x-1

x = np.linspace(-1,1,10000)
plt.figure(figsize=(10,10))

plt.title('What are we seeing with the semiloigy plot? Is this function differentiable?')
plt.semilogy(x,np.abs(f(x)),'b-',label=r'$|f(x)|$')
plt.semilogy(x,np.abs(fp(x)),'m-',label=r'$|fp(x)|$')
plt.grid()
plt.legend()
plt.xlabel(r'$x$',fontsize=16)
plt.show()

r=newton_method(f, fp, 0.4, rel_error=1e-8, m=1)
print([r,f(r)])
# Is this showing quadratic convergence? If not, can you fix it?


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
What is this plot telling us?
This plots shows that, even if we don't know the exact value of $g'(r)$, we can determine if the FPI will convergan by looking at the plot.
In this plot we observe that when plotting $g'(x)$ (magenta), we can determine that the value of $|g'(r)|$ will be less than 1 since it is between the black lines, that are located at $y=-1$ and $y=1$.
Can you write Python code for it?

xx=np.linspace(0.2,0.8,1000)
plt.figure(figsize=(10,10))
plt.plot(xx,g(xx),'r-',label=r'$g(x)$')
plt.plot(xx,gp(xx),'m-',label=r'$gp(x)$')
plt.plot(xx,xx,'b-',label=r'$x$')
plt.plot(xx,0*xx+1,'k--')
plt.plot(xx,0*xx-1,'k--')
plt.legend(loc='best')
plt.grid()
plt.show()


I am working on the file "SC1v2/03_roots_of_1D_equations.ipynb".
The first task is:
What it is interesting about the previous example is that it generates an interesting limit cicle! In the next cell we evaluate the fixed point with initial guess equal to 1. The iteration oscilates generating the following sequence: 1, 2, 3, 1, 2, 3, .... Which is nice!
Can you write Python code for it?

fpi(g, 1, 12, True)

# Suggestion, try the following alternative.
#fpi(g, 2.5, 100, True)


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
<div id='classnotesPaluExample' />

Motivation for PALU in classnotes example
Back to TOC
Can you write Python code for it?

A=np.array([[1e-20, 1],[1,2]])

l,u=lu_decomp(A, show=True)

np.dot(l,u)

p,l,u=palu_decomp(A, show=True)


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:

Examples
Back to TOC

Hilbert Matrix
Back to TOC
Can you write Python code for it?

N=20
F_errors=np.zeros(N+1)
B_errors=np.zeros(N+1)
kappas=np.zeros(N+1)
my_range=np.arange(5,N+1)
for n in my_range:
    A=hilbert(n)
    x_exact=np.ones(n)
    b=np.dot(A,x_exact)
    x=np.linalg.solve(A,b)
    F_errors[n]=np.linalg.norm(x-x_exact)/np.linalg.norm(x_exact)
    kappas[n]=np.linalg.cond(A,2)
    B_errors[n]=np.linalg.norm(b-A @ x)/np.linalg.norm(b)

plt.figure(figsize=(5,10))
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5), sharey = False)

ax1.semilogy(my_range, 10.**(-16+np.log10(kappas[my_range])), marker='o', linestyle='--', color='b',label='Estimated Forward error')
ax1.semilogy(my_range, F_errors[my_range], marker='o', linestyle='--', color='r',label=r'Forward error: $||\mathbf{x}-\mathbf{x}_a||_2/||\mathbf{x}||$')
ax1.semilogy(my_range, B_errors[my_range], marker='o', linestyle='--', color='g',label=r'Backward error: $||\mathbf{b}-A\,\mathbf{x}_a||_2/||\mathbf{b}||$')
ax1.semilogy(my_range, kappas[my_range]*B_errors[my_range], marker='o', linestyle='--', color='m', label=r'$\kappa(A_n)\,||\mathbf{b}-A\,\mathbf{x}_a||_2/||\mathbf{b}||$')
ax1.set_title('Errors')
ax1.grid(True)
ax1.set_xlabel('$n$')
ax1.legend(loc='best')

ax2.semilogy(my_range, kappas[my_range], marker='s', linestyle='--', color='k',label=r'$\kappa(A_n)$')
ax2.set_title(r'$\kappa(A_n)$')
ax2.set_xlabel('$n$')
ax2.grid(True)
plt.show()


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
<div id='gaussseidel' />

Gauss Seidel
Back to TOC
Can you write Python code for it?

def gauss_seidel(A, b, n_iter=50):
    n = A.shape[0]
    #array with solutions
    X = np.empty((n_iter, n))
    #initial guess
    X[0] = np.zeros(n)
    #submatrices
    R = np.tril(A) # R=(L+D)
    U = A-R
    for i in range(1, n_iter):
        #X[i] = solve_triangular(R, b-np.dot(U, X[i-1]), upper=False)
        # v1.11
        X[i] = X[i-1]+solve_triangular(R, b-np.dot(A, X[i-1]), upper=False)
    return X
def gauss_seidel_M(A):
    L = np.tril(A,-1)
    U = np.triu(A,1)
    D = np.diag(np.diag(A))
    M = -np.dot(np.linalg.inv(L+D),U)
    return M


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
<div id='im' />

Iterative Methods
Back to TOC
Can you write Python code for it?

"""
Randomly generates an nxn strictly diagonally
dominant matrix A.
"""
def generate_dd_matrix(n):
    A = np.random.random((n,n))
    deltas = 0.1*np.random.random(n)
    row_sum = A.sum(axis=1)-np.diag(A)
    np.fill_diagonal(A, row_sum+deltas)
    return A

"""
Computes relative error between each row on
X matrix and y vector.
"""
def error(X, y):
    D = X-y
    err = np.linalg.norm(D, axis=1, ord=np.inf)
    return err


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:

Introduction
Back to TOC
In our last Jupyter Notebook we learned how to solve 1D equations. Now, we'll go to the next level and will learn how to solve not just one equation, but a system of linear equations. This is a set of $n$ equations involving $n$ variables wherein all the equations must be satisfied at the same time. You probably know how to solve small 2D systems with methods such as substitution and reduction, but in practical real-life situations it's very likely that you'll find problems of bigger dimensions. As usual, we'll present some useful methods for solving systems of linear equations below.

Direct Methods
Back to TOC
Firstly, we will study direct methods. They compute the analytic solution of the system (from here comes the name direct) limited only by the loss of numerical precision, because of the arithmetic operations performed by the computer. Their counterpart is the iterative methods, which calculate an approximate solution that evolves iteratively converging to the real solution.

LU decomposition
Back to TOC
Given the matrix $A \in \mathbb{R}^{n \times n}$ square and non singular, the main goal of this method involves finding a decomposition like $A = L U$ where $L,U \in  \mathbb{R}^{n \times n}$ are lower and upper triangular matrices respectively.
The algorithm to perform this decomposition is basically a modified version of Gaussian Elimination. It basically iterates through the first $n-1$ columns, making $0$ all the entries below the main diagonal. This is accomplished by performing row operations.
Can you write Python code for it?

def lu_decomp(A, show=False, print_precision=2):
    N,_ = A.shape
    U = np.copy(A)
    L = np.identity(N)
    if show:
        print('Initial matrices')
        print('L = '); print(np.array_str(L, precision=print_precision, suppress_small=True))
        print('U = '); print(np.array_str(U, precision=print_precision, suppress_small=True))
        print('----------------------------------------')
    #iterating through columns
    for j in range(N-1):
        #iterating through rows
        for i in range(j+1,N):
            L[i,j] = U[i,j]/U[j,j]
            U[i] -= L[i,j]*U[j]
            if show:
                print('L = '); print(np.array_str(L, precision=print_precision, suppress_small=True))
                print('U = '); print(np.array_str(U, precision=print_precision, suppress_small=True))
                print('----------------------------------------')
    return L,U


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
<div id='iterativeMethodExample' />

Iterative Methods classnotes example
Back to TOC
Can you write Python code for it?

A=np.array([[5.,4.],[1.,3.]])
b=np.array([6.,-1.])
np_sol = np.linalg.solve(A,b)

jac_sol = jacobi(A, b, n_iter=50)
jac_err = error(jac_sol, np_sol)

gauss_seidel_sol = gauss_seidel(A, b, n_iter=50)
gauss_seidel_err = error(gauss_seidel_sol, np_sol)

sor_sol = sor(A, b, w=1.09310345, n_iter=50)
sor_err = error(sor_sol, np_sol)

plt.figure(figsize=(12,6))
plt.semilogy(it, jac_err, marker='o', linestyle='--', color='b', label='Jacobi')
plt.semilogy(it, gauss_seidel_err, marker='o', linestyle='--', color='r', label='Gauss-Seidel')
plt.semilogy(it, sor_err, marker='o', linestyle='--', color='g', label='SOR')
plt.grid(True)
plt.xlabel('Iterations')
plt.ylabel('Error')
plt.title('Infinity norm error for all methods')
plt.legend(loc=0)
plt.show()

log10_errors=np.zeros((50,3))
log10_errors[:,0]=jac_err
log10_errors[:,1]=gauss_seidel_err
log10_errors[:,2]=sor_err
log10_errors=np.log10(log10_errors[:10,:])
#np.shape(log10_errors)
df=pd.DataFrame(log10_errors, columns=["Jacobi", "Gauss-Seidel","SOR($\omega$)"])
df.style.set_caption("Log10 of error for each method indicated")


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
As before we will create a linear system $A x = b$, with $A$ as a diagonally dominant matrix, which is a sufficient condition for the methods we will study in this section converge
Can you write Python code for it?


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Given a symmetric positive-definite matrix $A \in \mathbb{R}^{n \times n}$, the Cholesky decomposition is of the form $A =R^T R$, with $R$ being an upper triangular matrix. This method takes advantage of the properties of symmetric matrices, reaching approximately twice the efficiency of LU.
Can you write Python code for it?

def cholesky_decomp(A, show=False):
    N,_ = A.shape
    A = np.copy(A)
    R = np.zeros((N,N))
    if show:
        print('Initial matrix')
        print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
        print('R = '); print(np.array_str(R, precision=2, suppress_small=True))
        print('----------------------------------------')
    for i in range(N):
        R[i,i] = np.sqrt(A[i,i])
        u = (1./R[i,i])*A[i,i+1:]
        R[i,i+1:] = u
        A[i+1:,i+1:] -= np.outer(u,u)
        if show:
            print('A = '); print(np.array_str(A, precision=2, suppress_small=True))
            print('R = '); print(np.array_str(R, precision=2, suppress_small=True))
            print('----------------------------------------')
    return R


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Here are some questions about Gauss-Seidel:
- Can you explain what the differences between this and Jacobi method are?
- Why do we use solve_triangular instead of np.linalg.solve or something similar?
Can you write Python code for it?

Mgs = gauss_seidel_M(A)
print(np.linalg.norm(Mgs))

np.max(np.abs(np.linalg.eigvals(Mgs)))


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Here are some questions about SOR:
- Why can averaging the current solution with the Gauss-Seidel solution improve convergence?
- Why do we use $\omega > 1$ and not $\omega < 1$?
- Could you describe a method to find the best value of $\omega$ (the one which optimizes convergence)?
- Would it be a better option to re-compute $\omega$ at each iteration?
<div id='ca' />

Convergence Analysis
Back to TOC
Let's see convergence plots all together
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.semilogy(it, jac_err, marker='o', linestyle='--', color='b', label='Jacobi')
plt.semilogy(it, gauss_seidel_err, marker='o', linestyle='--', color='r', label='Gauss-Seidel')
plt.semilogy(it, sor_err, marker='o', linestyle='--', color='g', label='SOR')
plt.grid(True)
plt.xlabel('Iterations')
plt.ylabel('Error')
plt.title('Infinity norm error for all methods')
plt.legend(loc=0)
plt.show()


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Let's now try our implementations. We begin by creating a random 100$\times$100 linear system:
Can you write Python code for it?

A = np.random.random((3,3))
b = np.ones(3)


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Let's test this new method against the LU and NumPy solvers
Can you write Python code for it?

palu_sol = solve_palu(A, b, show=True, print_precision=4)

np.linalg.norm(palu_sol - lu_sol)

np.linalg.norm(palu_sol - np_sol)

P,L,U = palu_decomp(A)
print('P: ',P)
print('L: ',L)
print('U: ',U)


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Now we test our implementation, comparing time execution with LU and PALU on two different linear systems
Can you write Python code for it?

A = generate_spd_matrix(3)
b = np.ones(3)

b=np.array([4,2,0])
solve_cholesky(A, b, show=True)

A = generate_spd_matrix(100)
b = np.ones(100)

%timeit solve_cholesky(A, b)
%timeit solve_lu(A, b)
%timeit solve_palu(A, b)

A = generate_spd_matrix(1000)
b = np.ones(1000)

%timeit solve_cholesky(A, b)
%timeit solve_lu(A, b)
%timeit solve_palu(A, b)


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Now, we compute the condition number of $A=H_{200}$
Can you write Python code for it?


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Once the decomposition is done, solving a linear system like $A x = b$ is straightforward:
$$A x = b \rightarrow L U x = b \ \ \text{ if we set } \ \  U x = c \rightarrow L c = b \ \ \text{ (solve for c) } \ \rightarrow U x = c$$
and as you might know, solving lower and upper triangular systems can be easily performed by back-substitution and forward-subsitution respectively.
Can you write Python code for it?

"""
Solves a linear system A x = b, where A is a
triangular (upper or lower) matrix
"""
def solve_triangular(A, b, upper=True):
    n = b.shape[0]
    x = np.zeros_like(b)
    if upper==True:
        #perform back-substitution
        x[-1] = (1./A[-1,-1]) * b[-1]
        for i in range(n-2, -1, -1):
            x[i] = (1./A[i,i]) * (b[i] - np.sum(A[i,i+1:] * x[i+1:]))
    else:
        #perform forward-substitution
        x[0] = (1./A[0,0]) * b[0]
        for i in range(1,n):
            x[i] = (1./A[i,i]) * (b[i] - np.sum(A[i,:i] * x[:i]))
    return x

def solve_lu(A, b, show=False, print_precision=2):
    L,U = lu_decomp(A, show, print_precision=print_precision)
    # L.c = b with c = U.x
    c = solve_triangular(L, b, upper=False)
    x = solve_triangular(U, c)
    return x


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
Recall:
$\dfrac{1}{\kappa(A)}\dfrac{\|\mathbf{b}-A\,\mathbf{x}_a\|}{\|\mathbf{b}\|} \leq \dfrac{\|\mathbf{x}-\mathbf{x}_a\|}{\|\mathbf{x}\|} \leq \kappa(A)\,\dfrac{\|\mathbf{b}-A\,\mathbf{x}_a\|}{\|\mathbf{b}\|}$
Let's solve a linear system of equations with $H_{200}$:
Can you write Python code for it?

n = 200
# Generating matrix
A = hilbert(n)
# Defining the 'exact' solution
x_exact = np.ones(n)
# If we know the exact solution, we can compute the RHS just by multiplying 'A' by 'x_exact'
b = A @ x_exact
# Using the NumPy routine to solve the linear system of equations.
x = np.linalg.solve(A,b) # A.x = A.1 = b


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
The procedure to solve the system $Ax=b$ remains almost the same. We have to add the efect of the permutation matrix $P$:
$$A x = b \rightarrow P A x = P b \rightarrow L U x = b' \ \ \text{ if we set } \ \  U x = c \rightarrow L c = b' \ \ \text{ (solve for c) } \ \rightarrow U x = c$$
Can you write Python code for it?

def solve_palu(A, b, show=False, print_precision=2):
    P,L,U = palu_decomp(A, show, print_precision=print_precision)
    # A.x = b -> P.A.x = P.b = b' -> L.U.x = b'
    b = np.dot(P,b)
    # L.c = b' with c = U.x
    c = solve_triangular(L, b, upper=False)
    x = solve_triangular(U, c)
    return x


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
The solve stage remains the same as LU:
Can you write Python code for it?

def solve_cholesky(A, b, show=False):
    R = cholesky_decomp(A, show)
    #R^T.R.x = b -> R^T.c = b with R.x = c
    c = solve_triangular(R.T, b, upper=False)
    x = solve_triangular(R, c)
    return x


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
We compute the Backward error, which is small.
Can you write Python code for it?


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
and find the solution $x$ through np.linalg.solve to use it as the reference solution-
Can you write Python code for it?


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
as you can see, we compute the SOR solution with 5 iterations for each $\omega$ on the given range.
Can you write Python code for it?

plt.figure(figsize=(12,6))
plt.semilogy(w, sor_errors, marker='o', linestyle='--', color='g')
plt.grid(True)
plt.xlabel('w')
plt.ylabel('Error')
plt.title('Infinity norm error after 5 steps of SOR as a function of w')
plt.show()


I am working on the file "SC1v2/04a_linear_systems_of_equations.ipynb".
The first task is:
in order to compare these huge vectors, we use the Euclidean metric as follows:
Can you write Python code for it?


I am working on the file "SC1v2/04b_BONUS_conjugate_gradient_method.ipynb".
The first task is:

Introduction
Welcome to another edition of our Jupyter Notebooks. Here, we'll teach you how to solve $A\,x = b$ with $A$ being a symmetric positive-definite matrix, but the following methods have a key difference with the previous ones: these do not depend on a matrix factorization. The two methods that we'll see are called the Gradient Descent and the Conjugate Gradient Method. On the latter, we'll also see the benefits of preconditioning.

Gradient Descent
This is an iterative method. If you remember the iterative methods in the previous Notebook, to find the next approximate solution $\mathbf{x}{k+1}$ you'd add a vector to the current approximate solution, $\mathbf{x}_k$, that is: $\mathbf{x}{k+1} = \mathbf{x}k + \text{vector}$. In this method, $\text{vector}$ is $\alpha{k}\,\mathbf{r}_k$, where $\mathbf{r}_k$ is the residue ($\mathbf{b} - A\,\mathbf{x}_k$) and $\alpha_k = \cfrac{(\mathbf{r}_k)^T\,\mathbf{r}_k}{(\mathbf{r}_k)^T\,A\,\mathbf{r}_k}$, starting with some initial guess $\mathbf{x}_0$. Let's look at the implementation below:
Can you write Python code for it?

def gradient_descent(A, b, x0, n_iter=10, tol=1e-10):
    n = A.shape[0]
    #array with solutions
    X = np.full((n_iter, n),np.nan)
    X[0] = x0

    for k in range(1, n_iter):
        r = b - np.dot(A, X[k-1])
        if np.linalg.norm(r)<tol: # The algorithm "converged"
            X[k:] = X[k-1]
            return X
            break
        alpha = np.dot(r, r)/np.dot(r, np.dot(A, r))
        X[k] = X[k-1] + alpha*r

    return X


I am working on the file "SC1v2/04b_BONUS_conjugate_gradient_method.ipynb".
The first task is:
Amazing! We started with a huge relative error and reduced it to practically zero in just under 10 iterations (the algorithms all have 100 iterations but we're showing you the first 40).
We can clearly see that the Conjugate Gradient Method converges faster than the Gradient Descent method, even for larger matrices.
We can see that, reached a certain size for the matrix, the amount of iterations needed to reach a small error remains more or less the same. We encourage you to try other kinds of matrices to see how the algorithms behave, and experiment with the code. Now let's move on to profiling.
Of course, you win some, you lose some. Accelerating the convergence of the algorithm means you have to spend more of other resources. We'll use the functions %timeit and %memit to see how the algorithms behave.
Can you write Python code for it?

A = generate_spd_matrix(100)
b = np.ones(100)
x0 = np.random.random(100)

%timeit gradient_descent(A, b, x0, n_iter=100, tol=1e-5)
%timeit conjugate_gradient(A, b, x0, tol=1e-5)

# Commented because it is taking too long, we need to review this!
# %memit gradient_descent(A, b, x0, n_iter=100, tol=1e-5)
# %memit conjugate_gradient(A, b, x0, tol=1e-5)


I am working on the file "SC1v2/04b_BONUS_conjugate_gradient_method.ipynb".
The first task is:
Now let's try our algorithm! But first, let's borrow a function to generate a random symmetric positive-definite matrix, kindly provided by the previous notebook, and another one to calculate the vectorized euclidean metric.
Can you write Python code for it?

"""
Randomly generates an nxn symmetric positive-
definite matrix A.
"""
def generate_spd_matrix(n):
    A = np.random.random((n,n))
    #constructing symmetry
    A += A.T
    #symmetric+diagonally dominant -> symmetric positive-definite
    deltas = 0.1*np.random.random(n)
    row_sum = A.sum(axis=1)-np.diag(A)
    np.fill_diagonal(A, row_sum+deltas)
    return A


I am working on the file "SC1v2/04b_BONUS_conjugate_gradient_method.ipynb".
The first task is:
The science behind this algorithm is in the classnotes and in the textbook (Numerical Analysis, 2nd Edition, Timothy Sauer). Now let's try it!
Here are some questions to think about:
* What are the advantages and disadvantages of each method: gradient_descent and conjugate_gradient?
* In which cases can the Conjugate Gradient Method converge in less than $n$ iterations?
* What will happen if you use the Gradient Descent or Conjugate Gradient Method with non-symmetric, non-positive-definite matrices?
<div id='LP' />

Let's Play: Practical Exercises and Profiling
First of all, define a function to calculate the progress of the relative error for a given method, that is, input the array of approximate solutions X and the real solution provided by Numpy's solver r_sol and return an array with the relative error for each step.
Can you write Python code for it?

def relative_error(X, r_sol):
    n_steps = X.shape[0]
    n_r_sol = np.linalg.norm(r_sol)
    E = np.zeros(n_steps)
    for i in range(n_steps):
        E[i] = np.linalg.norm(X[i] - r_sol) / n_r_sol
    return E


I am working on the file "SC1v2/04b_BONUS_conjugate_gradient_method.ipynb".
The first task is:
Trying the two methods with a small non-symmetric, non-positive-definite matrix and plotting the forward error for all the methods.
Can you write Python code for it?

def show_output_for_non_symmetric_and_npd(np_seed=0):
    np.random.seed(np_seed)
    n = 10
    A = 10 * np.random.random((n,n))
    b = 10 * np.random.random(n)
    x0 = np.zeros(n)

    X1 = gradient_descent(A, b, x0, n)
    X2, D, R, alphas, betas, n_residuals = conjugate_gradient(A, b, x0, True)
    r_sol = np.linalg.solve(A, b)

    E1 = relative_error(X1, r_sol)
    E2 = relative_error(X2, r_sol)
    iterations1 = np.linspace(1, n, n)
    iterations2 = np.linspace(1, X2.shape[0], X2.shape[0])

    plt.figure(figsize=(10,5))
    plt.xlabel('Iteration')
    plt.ylabel('Relative Error')
    plt.title('Evolution of the Relative Forward Error for each method')
    plt.semilogy(iterations1, E1, 'rd', markersize=8, label='GD') # Red diamonds are for Gradient Descent
    plt.semilogy(iterations2, E2, 'b.', markersize=8, label='CG') # Blue dots are for Conjugate Gradient
    plt.grid(True)
    plt.legend(loc='best')
    plt.show()
interact(show_output_for_non_symmetric_and_npd,np_seed=(0,100,1))


I am working on the file "SC1v2/06_BONUS_Interpolation_Splines.ipynb".
The first task is:

Introduction
Previously in our jupyter notebooks, we learn about interpolation. Methods like Newton's Divided Difference, Lagrange, among others. Other alternative for interpolate a set of data points is using  Cubic Splines.
This technique, avoids the Runge's Phenomenon and creates a 3-degree polynomial easily.

Splines
The most common spline is the linear spline. Given a set of points $(x_{1},y_{1}), (x_{2},y_{2}),...,(x_{n},y_{n}) $, this spline connects each point creating a non-smooth curve. However, this polynomial haves a problem. It's no smooth curve! For to avoid this problem, the cubic splines creates a set of 3-degree polynomial (specifically n-1 polynomials)... a much better curve.
Can you write Python code for it?

# Code based on Example from: https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.CubicSpline.html#scipy.interpolate.CubicSpline

# The data
x = np.linspace(0,2*np.pi,12)
y = np.sin(x)*x

# Building interpolation object
cs = CubicSpline(x, y)

# Defining a finer mesh to plot the function
xx = np.linspace(0,2*np.pi,1000)
yyo = np.sin(xx)*xx
yyo1 = np.cos(xx)*xx+np.sin(xx)
yyo2 = -np.sin(xx)*xx+2*np.cos(xx)
yyo3 = -np.cos(xx)*xx-3*np.sin(xx)
yyo4 = np.sin(xx)*xx-4*np.cos(xx)

#Interpolating the date with the spline
yy = cs(xx)
yy1 = cs(xx, 1)
yy2 = cs(xx, 2)
yy3 = cs(xx, 3)
yy4 = cs(xx, 4)

# Plotting the splines and its derivatives
plt.figure(figsize=(M,M))
plt.plot(x,y,'k.',markersize=20,label=r'Data Points')
plt.plot(xx,yy, linewidth=4, label=r'S$(x)$')
plt.plot(xx,yy1, linewidth=4, label=r'$\frac{d}{dx}$S$(x)$')
plt.plot(xx,yy2, linewidth=4, label=r'$\frac{d^2}{dx^2}$S$(x)$')
plt.plot(xx,yy3, linewidth=4, label=r'$\frac{d^3}{dx^3}$S$(x)$')
plt.plot(xx,yy4, linewidth=4, label=r'$\frac{d^4}{dx^4}$S$(x)$')

plt.plot(xx,yyo4,'k--',linewidth=4, label='test', alpha=0.4)

plt.plot(x,y,'k.',markersize=20)

plt.title(r'Cubic Spline is defined as S$(x)$')
plt.axis('tight')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.grid(True)
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.show()

plt.figure()
plt.semilogy(np.abs(yy-yyo))
plt.grid(True)


I am working on the file "SC1v2/06_BONUS_Interpolation_Splines.ipynb".
The first task is:
Example 1 - Hand made interpolation
Can you write Python code for it?


I am working on the file "SC1v2/06_BONUS_Interpolation_Splines.ipynb".
The first task is:
Example 2
Can you write Python code for it?

def show_spline(type_ending='Natural',k1=0, k2=0, p1=0, p2=0):
    x = np.array([1,2,4,5,7,9])
    y = np.array([2,1,4,3,3,4])
    xx=np.linspace(np.min(x),np.max(x),1000)
    S = cubic_spline(x,y,end=type_ending, k1=k1, k2=k2, p1=p1, p2=p2)
    plt.figure(figsize=(M,M))
    plt.plot(xx,cubic_spline_eval(xx,S),'-',linewidth=5,label=r'S$(x)$')
    plt.plot(x,y,'k.',markersize=20,label='Data Points')
    plt.xlabel('$x$')
    plt.ylabel('$y$')
    plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    plt.grid(True)
    plt.show()
widgets.interact(show_spline, type_ending=['Natural','Adjusted','Clamped','Parabolically','NaK'],
                k1=(-20,20,1),k2=(-20,20,1),p1=(-2,2,0.2),p2=(-2,2,0.2))


I am working on the file "SC1v2/07_08_Least_Squares.ipynb".
The first task is:
<div id='ex' />

Examples of QR: Classic vs Modified Gram-Schmidt
Back to TOC
Can you write Python code for it?

# Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Style: DIM, NORMAL, BRIGHT, RESET_ALL
textQRClassic = lambda x: Style.BRIGHT+Fore.RED+x+Style.RESET_ALL
textQRModified = lambda x: Style.BRIGHT+Fore.BLUE+x+Style.RESET_ALL
textAnswer = lambda x: Back.BLACK+Fore.CYAN+x+Style.RESET_ALL

# Example 4.16 from textbook.
d = 1e-10
A = np.array([[1,1,1],[d,0,0],[0,d,0],[0,0,d]])
print(textBold('A: \n'), A)

Qc,Rc = QR(A, type_gram_schmidt = 'classic')
Qm,Rm = QR(A, type_gram_schmidt = 'modified')

################################
print(textBoldH('\n What are the Q\'s?'))
print(textQRClassic('Qc: \n'), Qc)
print(textQRModified('Qm: \n'), Qm)

################################
print(textBoldH('\n Are truly orthogonal the q_i\'s?'))
print(textQRClassic('Qc.T*Qc: \n'),np.dot(Qc.T,Qc))
print(textQRClassic('||Qc.T*Qc-I_3||: '),np.linalg.norm(np.dot(Qc.T,Qc)-np.eye(3))) # Warning: We are just using the transpose since the matrices are real!
print(textAnswer('Not really, since q2^T*q3 \\neq 0.'))

print(textQRModified('\n Qm.T*Qm: \n'),np.dot(Qm.T,Qm))
print(np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textQRModified('||Qm.T*Qm-I_3||: '),np.linalg.norm(np.dot(Qm.T,Qm)-np.eye(3)))
print(textAnswer('This looks much better!'))

################################
print(textBoldH('\n Do we recover A with each algorithm?'))
print(textQRClassic('A-Qc*Rc: \n'),A-np.dot(Qc,Rc))
print(textQRClassic('||A-Qc*Rc||: '),np.linalg.norm(A-np.dot(Qc,Rc)))
print(textAnswer('Yes!'))

print(textQRModified('\n A-Qm*Rm: \n'),A-np.dot(Qm,Rm))
print(textQRModified('||A-Qm*Rm||: '),np.linalg.norm(A-np.dot(Qm,Rm)))
print(textAnswer('Yes!'))


I am working on the file "SC1v2/07_08_Least_Squares.ipynb".
The first task is:

Introduction
Back to TOC
We have learned about square linear system of equations.
However, how can we solve a non-square linear system of equations? i.e. when we have more equations than unknowns! Well, the traditional step will be to find a least squares approximation!

QR Factorization
Gram-Schmidt Orthogonalization
Back to TOC
This algorithm orthonormalize a set of input vectors, returning an orthonormal set that spans the same vector space.
We will only consider now that the input set of vectors are linearly independent.
Let $A=[\mathbf{a}_1\, ...., \mathbf{a}_n]$ a matrix with linearly independent column vectors $\in\mathbb{R}^m$ and $n \le m$.
We know the following for the orthogonal set:
\begin{align}
    \mathbf{q}_i^T\,\mathbf{q}_i & = \|\mathbf{q}_i\|_2^2= 1\
    \mathbf{q}_i^T\,\mathbf{q}_j & = 0, \, \text{ for } i\neq j
\end{align}
Then the classic Gram-Schmidt orthonormalization finds the $\mathbf{q}i$ and $r{ij}$ from the following set of equations and considering the previous constraints:
\begin{align}
    \mathbf{a}1 &= r{11}\,\mathbf{q}1\
    r{11} &= \|\mathbf{a}1\|_2\
    \mathbf{q}_1 &= \dfrac{\mathbf{a}_1}{r{11}}\
    \mathbf{a}2 &= r{12}\,\mathbf{q}1+r{22}\,\mathbf{q}2\
    r{12} &= \mathbf{q}1^T\,\mathbf{a}_2\
    r{22} &= \|\mathbf{a}2-r{12}\,\mathbf{q}1\|_2\
    \mathbf{q}_2 &= \dfrac{\mathbf{a}_2-r{12}\,\mathbf{q}1}{r{22}}\
    \vdots &=  \vdots\
    \mathbf{a}k &= \sum{i=1}^k r_{i,k}\,\mathbf{q}i\
    r{i,k} &= \mathbf{q}i^T\,\mathbf{a}_k, \, \text{ for } i<j\
    r{k,k} &= \left\|\mathbf{a}k-\sum{i=1}^{k-1} r_{i,k}\,\mathbf{q}i\right\|_2\
    \mathbf{q}_k &= \dfrac{\mathbf{a}_k-\sum{i=1}^{k-1} r_{i,k}\,\mathbf{q}i}{r{k,k}}\
    \vdots &=  \vdots\
    \mathbf{a}n &= \sum{i=1}^n r_{in}\,\mathbf{q}i\
    r{in} &= \mathbf{q}i^T\,\mathbf{a}_n, \, \text{ for } i<n\
    r{nn} &= \left\|\mathbf{a}n-\sum{i=1}^{n-1} r_{in}\,\mathbf{q}i\right\|_2\
    \mathbf{q}_n &= \dfrac{\mathbf{a}_n-\sum{i=1}^{n-1} r_{in}\,\mathbf{q}i}{r{nn}}
\end{align}
Thus, we...
Can you write Python code for it?

def QR(A, type_factorization = 'reduced', type_gram_schmidt='classic'):
    A.astype('float')
    m,n = A.shape # m: number of rows, n: number of columns.
    if type_factorization == 'reduced':
        Q = np.zeros((m,n))
        R = np.zeros((n,n))
    elif type_factorization == 'full':
        Q = np.zeros((m,m))
        R = np.zeros((m,n))
    for k in range(n):
        y = A[:,k]
        for i in range(k):
            if type_gram_schmidt == 'classic':
                R[i,k] = np.dot(Q[:,i],A[:,k])
            elif type_gram_schmidt == 'modified':
                R[i,k] = np.dot(Q[:,i],y)
            y=y-R[i,k]*Q[:,i]
        R[k,k] = np.linalg.norm(y)
        Q[:,k] = y/np.linalg.norm(R[k,k])
    # The following lines must be completed by you!
    #if type_factorization == 'full':
        # (1) We need to add 0's to the R matrix so it is of the same shape as the matrix A,
        # fortunately this was already done!
        # (2) We need to add orthogonal vectors to Q so it is square,
        # how do we do this?
    return Q,R


I am working on the file "SC1v2/07_08_Least_Squares.ipynb".
The first task is:

Overdetermined Linear Systems of Equations
Back to TOC
There is cases where the number of equations is greater than variables. Many times, those systems don't have an exact solution (inconsistent system). Then, in this case we needs an approximation closest to the data. Based in orthogonality, the shortest distance from a point to plane. The orthogonal distance represents the error which would be minimum.
\begin{equation}
                 \mathbf{b} - A\,\overline{\mathbf{x}} = \mathbf{r}\
                 \mathbf{b} - A\,\overline{\mathbf{x}} \perp {A\,\mathbf{x}\, |\, \mathbf{x} \in \mathbb{R}^m}
\end{equation}
The idea is that $\mathbf{r}$ would be closest to zero. We need to apply orthogonality to find the vector that satisfied this condition.
\begin{equation}
                 (A\,\mathbf{x})^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\
                 \mathbf{x}^\, A^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=0, \hspace{1cm} \text{for all } \mathbf{x} \in \mathbb{R}^n\
                 A^\,(\mathbf{b}-A\,\overline{\mathbf{x}})=\mathbf{0} \
                 A^\,A\,\overline{\mathbf{x}}= A^\,\mathbf{b} \
\end{equation}
This last equation gives us a new square $n\times n$ matrix, which let us resolve the equation system.
This linear system of equations is known as the Normal Equations.

Initial example:
Back to TOC
In this example we will approximate $m$ points considering a linear relationship.
This means that we will have the data points $(x_i,y_i)$ for $i\in{1,2,\dots,m}$ and consider the relationshop $y=a_0+a_1\,b$.
The error that will be added follows a normal distribution.
Can you write Python code for it?

# Number of points to be used
m = 10
# Relationship considered
fv = np.vectorize(lambda x, a0, a1: a0+a1*x)
# Coefficients considered
a0, a1 = 1, 4

# Error:
np.random.seed(0)
# Standard deviation for the error
sigma = 5e-1
# Error to be added
e = np.random.normal(0,sigma,m)

# Generating data points
x = np.linspace(0,1,m)
y = fv(x,a0,a1)+e
# Challenging question: What if all the xi are the same?
# This means that considering the following definition for x, for instance:
# x = np.linspace(0,1,m)*0+0.5

# Plotting the data
plt.figure(figsize=(10,10))
plt.plot(x,y,'b.',markersize=20)
plt.grid(True)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.show()

# Build the data matrix
A = np.ones((m,2))
A[:,1] = x
# Setting up the right hand side
b = y

# Building and solving the normal equations
# A^T A x_bar = A^T b
x_bar = np.linalg.solve(A.T @ A, A.T @ b)
# Showing solution found, please compare with a0 and a1 defined previously.
# Are the the same? Are they close? What does it happen with them if we increase "m" and decrease "sigma"?
print(x_bar)

# Showing the comparison between the "original function" and the "least-squared reconstructed approximation".
# We added in red a "sample" of possible functions.
# Notice that the colors used follow the description included in the classnotes.
# This means to consider the following analogy:
# blue: data points, this correspond to the right-hand-side vector "b".
# red: this correspond to the sub-space generated by Ax, i.e. the span of the columns of A.
# violet: This correspond to the least-square solution found.
plt.figure(figsize=(10,10))
for i in range(100):
    plt.plot(x,fv(x,x_bar[0]+np.random.normal(0,1),x_bar[1]+np.random.normal(0,1)),'r-',linewidth=1,alpha=0.2)
plt.plot(x,y,'b.',markersize=20)
plt.plot(x,fv(x,x_bar[0],x_bar[1]),'r.',markersize=30)
plt.plot(x,fv(x,a0,a1),'k-',linewidth=8,alpha=0.8)
plt.plot(x,fv(x,x_bar[0],x_bar[1]),'--',color='darkviolet',linewidth=4)
plt.grid(True)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.show()

# Computing the quadratic error...


I am working on the file "SC1v2/07_08_Least_Squares.ipynb".
The first task is:
<div id='sm' />

Examples of models
Back to TOC
Can you write Python code for it?

def least_squares(A,b):
    Q,R = QR(A,type_gram_schmidt='modified')
    return spla.solve_triangular(R,np.dot(Q.T,b))

def solve_model(M):
    A=M['A']
    b=M['b']
    M['x_bar']=least_squares(A,b)
    return M

def create_model(data, type_model='linear'):
    if type_model == 'linear': # f(x)=a0+a1*x
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = data[:,1]
    if type_model == 'parabollic': # f(x)=a0+a1*x+a_2*x^2
        A = np.ones((data.shape[0],3))
        A[:,1] = data[:,0]
        A[:,2] = data[:,0]**2
        b = data[:,1]
    if type_model == 'exponential': #f(x)=a0 \exp(a1*x) = \exp(\log(a0)+a1*x) -> log(f(x))=log(a0)+a1*x = A0+a1+x (it is linear now!)
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = np.log(data[:,1])
    M = {'A':A,
         'b':b,
         'type_model':type_model}
    M=solve_model(M)
    return M

def evaluate_model(M,x):
    x_bar=M['x_bar']
    if M['type_model'] == 'linear':
        return x_bar[0] + x_bar[1]*x
    if M['type_model'] == 'parabollic':
        return x_bar[0] + x_bar[1]*x + x_bar[2]*x**2
    if M['type_model'] == 'exponential':
        return np.exp(x_bar[0]+x_bar[1]*x)

def generate_data(type_of_data='linear'):
    n=40
    np.random.seed(0)
    x = np.linspace(0,10,n)
    y = np.random.rand(n)
    x = np.concatenate((x,x,y),axis=0)
    n = 3*n
    if type_of_data=='linear':
        y = x+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='parabollic':
        y = 4*x**2+0.1*x*np.random.normal(0,1,n)+1.5
    elif type_of_data=='exponential':
        y = np.exp(x+0.1*np.random.normal(0,1,n)+1.5)
    elif type_of_data=='sinusoidal':
        y = np.sin(2*np.pi*x/10)+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='random':
        y = 0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='boston house-prices':
        x,y=datasets.load_boston(return_X_y=True)
        x=x[:,5]
    elif type_of_data=='diabetes':
        x,y=datasets.load_diabetes(return_X_y=True)
        x=x[:,2]
    data =...


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
1D Example
In this example we will solve the logistic equation $\dot{y}(t) = y(t)(1-y(t))$ for several inital conditions $y_0$.
Can you write Python code for it?

# Logistic Equation
def f(t,y):
    return np.array(y*(1-y))

# First Example
def change_N_and_y0_1st_Example(N=20,y0=0.1, ODEsolver=eulerMethod):
    t0 = 0
    T  = 4
    t, y = ODEsolver(t0,T,N,y0,f)
    fig = plt.figure()
    ax  = fig.gca()
    ax.axis([0, 4, -1, 2])
    # Plotting stationary states for this problem.
    t = np.linspace(t0,T,N+1)
    plt.plot(t,t*0+1,'b-',lw=5, alpha=0.5, label='Stationary state y=1')
    plt.plot(t,t*0,'r-',lw=5, alpha=0.5, label='Stationary state y=0')
    # Plotting numerical solution.
    plt.plot(t,y,'.',label=r'$y_i$')

    plt.grid(True)
    ax.legend(loc='lower left', ncol=1, fancybox=True, shadow=True, numpoints=1, bbox_to_anchor=(1,0))
    plt.show()

radio_button_ODEsolvers=RadioButtons(
    options=[('Euler\'s Method',eulerMethod),('RK2',RK2),('RK4',RK4)],
    value=eulerMethod,
    description='ODE solver:',
    disabled=False
)

interact(change_N_and_y0_1st_Example,N=(3,100,1),y0=(-0.5,2,0.1), ODEsolver=radio_button_ODEsolvers)


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:

Boundary Value Problems
Back to TOC
An instance of a BVP corresponds to a ordinary differential equation of the following form:
$$ y''(x) = f(x,y(x),y'(x)),$$
with the following boundary conditions,
\begin{align}
    y(a) &= y_a,\
    y(b) &= y_b.
\end{align}
To undertand them better, let's find a numerical approximation of the following problem:
\begin{align}
    y''(x) &= 4\,y(x),\
    y(0)   &= y_0,\
    y(1)   &= y_n.
\end{align}

Shooting Method
Back to TOC
This method takes advantage that we already know how to handle IVP and uses that knowledge to find a numerical approximation of an BVP.
The main steps are the followings:
- We have a BVP on the form:
    \begin{align}
        y''(x) &= f(x,y(x),y'(x)),\
        y(a)   &= y_a,\
        y(b)   &= y_b.
    \end{align}
- Let's considerer the following steps to translate the BVP to an IVP:
    1. $x \rightarrow t$, this implies that $t_0=a$ and $T=b$.
    2. \begin{align}
            \ddot{y}(t)   &= f(t,y(t),\dot{y}(t)),\
            y(t_0)        &= y_a,\
            \dot{y}(t_0)  &= \alpha.
       \end{align}
       Warning: We actually don't know $\alpha$ but we do need it since we treating the previous problem as a IVP. What we do have is $y(b)=y(T)=y_b$.
    3. So, since we don't have $\alpha$, we need to find it by solving the following 1D root: $F(\alpha) = y_N^{(\alpha)} - y_b$, where $y_N^{(\alpha)}$ denotes the IVP approximation at time $t=T$ considering $\alpha$ for the initial condition of $\dot{y}(t_0)$.
    4. After we find the root for $F(\alpha)$, we can say we have found a numerical approximation of $y(x)$ since we have a numerical approximation that satisfies the ODE and also the boundary condition defined on the interval $[a,b]$. This is wonderful!
Notice however that the function $F(\alpha)$ is the output of the execution of an algorithm, so we only expect it to be continuous.
For instance it will be interesting to apply the Bisection method here!
The main disadvantage of this algorithm is that we...
Can you write Python code for it?

# BVP: y''(x)= 4*y(x), y(0)=1, and y(1)=3, x \in [0,1].
# Dynamical system:
# y1=y  -> y1'=y'=y2
# y2=y' -> y2'=y''=4*y=4*y1.
# y1(0) = 1, (and we know y(1)=3)
# y1(1) = 3.
# y2(0) = alpha, we don't know it.
def f(t,y):
    return np.array([y[1],4*y[0]])

def shooting_method_101(alpha=-1, N=50, ya=1, yb=3, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximation")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_101, alpha=(-2,2,0.1),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers)


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
<div id='DD' />

Finite Differences
Back to TOC
This method opens a new door of algorithms for solving BVP.
The main idea is to approximate each derivative of $y(x)$ by a finite difference approximation and then solve the linear or possible non-linear system of equations associated.
Here we list a few finite difference approximation for the first and one for the second derivative:

Forward Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x)}{h} + O(h)$$

Backward Difference:
$$ y'(x) = \dfrac{y(x) - y(x-h)}{h} + O(h)$$

Central Difference:
$$ y'(x) = \dfrac{y(x+h) - y(x-h)}{2h} + O(h^2)$$

And for the second derivative:
\begin{align}
y''(x) &= \dfrac{\text{Forward Difference } - \text{ Backward Difference}}{h} + O(h^2) \
y''(x) &= \dfrac{\dfrac{y(x+h) - y(x)}{h} - \dfrac{y(x) - y(x-h)}{h}}{h} + O(h^2) \
y''(x) &= \dfrac{y(x+h) - 2\,y(x) + y(x-h)}{h^2} + O(h^2) \
\end{align}
Notice that in all the previous cases we first need to define the function on a grid, it does not have to be regular, but for simplicity we will consider it regular.
Recall that the BVP will be defined on $[a,b]$, thus the regular grid will be defined as follows: $x_i=i\,h$, where $h=\dfrac{b-a}{N}$.
This also help us to define the approximation of the function over the grid, this means the following,
- Traditionally we say $y(x)$ is the function defined on $x\in[a,b]$.
- Now, we will have a grid function $y_i\approx y(x_i)$ for $i\in{0,1,2,\dots,N}$. In particular we would like to point out that the vector $\mathbf{y}$ will be the representation of all the scalar values $y_i$ and the vector $\mathbf{x}$ will store the grid values $x_i$. Thus in this discrete world $\mathbf{y}$ will apporximate the continuous function $y(x)$ on $[a,b]$.
On the other hand, the finite difference operator over the finite grid will be then defined as follows:
- Forward Difference: $ y'(x_i)=\dfrac{y_{i+1} - y_i}{h} + O(h) $.
- Backward Difference: $ y'(x_i) = \dfrac{y_i - y_{i-1}}{h} + O(h) $.
- Central Difference: $ y'(x_i) = \dfrac{y_{i+1} - y_{i-1}}{2h} +...
Can you write Python code for it?

def solve_finite_difference_eq(M=4,ya=1,yb=3):
    # Spatial discretization,
    # 'M' represents the number of intervals to be used.
    h=(1.-0.)/M

    # Building Finite Difference Discretization
    deltas=-(2.+4.*(h**2.))
    A=toeplitz(np.append(np.array([deltas, 1.]), np.zeros(M-3)))
    # Building RHS
    b=np.append(-ya, np.zeros(M-3))
    b=np.append(b,-yb)
    # Solving the linear system of equations
    y=np.linalg.solve(A, b)
    # Adding back the boundary conditions into the solution
    y=np.append([ya], y)
    y=np.append(y,[yb])
    x_FD = np.linspace(0,1,M+1) # xi, i.e. the spatial grid.
    y_FD = y                    # yi, i.e. the approximation of y(x_i).
    return x_FD, y_FD, A, b

def plot_solution_finite_difference_eq(M=4,ya=1,yb=3, FLAG_test=True):

    # Solving with Finite Difference
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    # Plotting
    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    # Plotting the discrete solution
    ax1.grid(True)
    ax1.set_title("FD")
    ax1.plot(x_FD,y_FD,'.',ms=20)
    ax1.grid("on")

    # Plotting the pattern and coefficients of the tri-diagonal matrix
    plot_matrices_with_values(ax2,A,FLAG_test)

interact(plot_solution_finite_difference_eq, M=(3,25,1),
         ya=(-5,5,0.1), yb=(-5,5,0.1),
         FLAG_test=Checkbox(value=True, description='Show numerical values'))


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
<div id='high' />

High order, higher dimensions and dynamical systems
Back to TOC
Why do we study higher order problems?
Because they are so cool!
The Van der Pol oscillator
The equation that model de Van der Pol oscillator is the following:
$$\ddot{y}-\mu\,(1-y^2)\,\dot{y} + y = 0,$$
with $y(0)=2$ and $\dot{y}(0)=0$,
where $\mu$ is a scalar parameter that regulates the non-linearity of the problem and the strength of the damping.
See https://en.wikipedia.org/wiki/Van_der_Pol_oscillator.
So, since it has second derivatives respecto to time, we need to translate it to a dynamical system with the following change of variables:
\begin{align}
    y_1(t) &= y(t),\
    y_2(t) &= \dot{y}(t).
\end{align}
Thus,
\begin{align}
    \dot{y}_1 &= \dot{y} = y_2,\
    \dot{y}_2 &= \ddot{y} = \mu (1-y^2) \dot{y} - y = \mu (1-y_1^2) y_2 - y_1,
\end{align}
with $y_1(0)=2$ and $y_2(0)=0$.
Can you write Python code for it?

def f_vdp(t,y,mu=10):
    y1 = y[0]
    y2 = y[1]
    return np.array([y2, mu*(1-y1**2)*y2-y1])

def f_vdp_eig_jacobian(t,y,mu=10):
    J=[[0,1],[-(2*mu*y[0]*y[1]+1), mu*(1-y[0]**2)]]
    lambs,vs= LA.eig(J)
    return lambs

def plotting_vdp(y01=2, y02=0, N=20000, T=200, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0=0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/2))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)

    ax1.grid(True)
    ax1.set_title("Numerical Approximation")
    ax1.plot(t_times,y_output[:,0],'-', label=r'$y_1(t)$', alpha=0.5)
    ax1.set(xlabel='t')
    ax1.plot(t_times,y_output[:,1],'-r', label=r'$y_2(t)$', alpha=0.5)
    ax1.legend(loc='best')

    ax2.grid(True)
    ax2.set_title("Phase Portrait")
    ax2.plot(y_output[:,0],y_output[:,1],'-')
    ax2.set(xlabel='$y=y_1$', ylabel='$\dot{y}=y_2$')

interact(plotting_vdp,y01=(-3,3,0.1), y02=(-3,3,0.1),
         N=(10,10000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)

def plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_jacobian,h,c1,c2,c3):
    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.plot(t_times,y_output[:,0],'-')
    ax1.plot(t_times,y_output[:,1],'r-')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    for i in range(1,t_times.size):
        k_lambdah=h*f_jacobian(t_times[i],y_output[i,:])
        ax2.plot(np.real(k_lambdah[0]),np.imag(k_lambdah[0]),'.r',ms=10,alpha=.4)
        ax2.plot(np.real(k_lambdah[1]),np.imag(k_lambdah[1]),'sm',ms=10,alpha=.4)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid('on')

    ax3.grid(True)
    ax3.set_title('Phase Portrait')
    ax3.plot(y_output[:,0],y_output[:,1],'-')
    ax3.set(xlabel='$y_1$',...


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
<div id='stab' />

Stability Regions
Back to TOC
To perform a linear stability analysis, we will consider the following ODE $\dot{y} = \lambda y$, with $y(0)=1$ and $\lambda\in\mathbb{C}$, for which we know its exact solution $y(t)=\exp(\lambda\,t)$.
Thus, solving the previos ODE with the Euler method, we have,
\begin{align}
    y_{i+1} &= y_i + \lambda h y_i \
    y_{i+1} &= (1 + \lambda h ) y_i \
    y_{i+1} &= (1 + \lambda h )^{i+1} y_0
\end{align}
Given that the real part of $\lambda$ is negative and $h$ is positive, we require then that $\left|1+\lambda h \right| <1$. We denote this range of values for $h$ the stability region for the Euler Method.
Following a similar approach, we the the following constraints for RK2 and RK4:
- RK2:
$$ \left|1+\lambda h + \dfrac{(\lambda h)^2}{2!} \right| <1 $$

RK4:
$$ \left| 1+\lambda h + \dfrac{(\lambda h)^2}{2!} + \dfrac{(\lambda h)^3}{3!} + \dfrac{(\lambda h)^4}{4!} \right| <1 $$
Can you write Python code for it?

def zplot2(z, ax=plt.gca(), lw=1.5, line_color='k', label=''):
    ax.plot(np.real(z), np.imag(z), line_color, lw=lw, label=label)
def runge_kutta_stability_regions():
    z = np.exp(1j * np.pi * np.arange(201)/100.)
    r = z-1
    d = 1-1./z;
    # Order 1
    W1, W2, W4 = [0], [0], [0]
    for zi in z[1:]:
        W1.append( W1[-1]-(1.+W1[-1]-zi) )
    for zi in z[1:]:
        W2.append( W2[-1]-(1+W2[-1]+.5*W2[-1]**2-zi**2)/(1+W2[-1]) )
    for zi in z[1:]:
        num = (1+W4[-1]+.5*W4[-1]**2+W4[-1]**3/6+W4[-1]**4/24-zi**4)
        den = (1+W4[-1]+W4[-1]**2/2+W4[-1]**3/6.)
        W4.append( W4[-1] - num/den )
    return W1, W2, W4
W1,W2,W4=runge_kutta_stability_regions()

fig = plt.figure(figsize=(L/2,L/2))
ax=fig.gca()
zplot2(W1,ax,line_color='r',label='Euler')
zplot2(W2,ax,line_color='g',label='RK2')
zplot2(W4,ax,line_color='b',label='RK4')
ax.axis("equal")
ax.axis([-5, 2, -3.5, 3.5])
ax.grid("on")
ax.legend(loc='best')
ax.set_title("Stability Regions of some Runge-Kutta ODE solvers")


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
Comparing the Shooting Method with Finite Differences
Can you write Python code for it?

def shooting_method_vs_FD(alpha=-0.420306048726636, N=50, ya=1, yb=3, ODEsolver=eulerMethod, M=4):
    # Shooting Method
    t0 = 0
    T  = 1
    y0 = np.array([ya,alpha])
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    # Finite Differences
    x_FD, y_FD, A, b = solve_finite_difference_eq(M,ya,yb)

    fig = plt.figure(figsize=(L/2,L/2))
    ax = fig.gca()
    plt.grid(True)
    plt.title("Numerical Approximations")
    plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'SM: $y1_i$')
    plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'SM: $y2_i$')

    plt.plot(x_FD,y_FD,'b.',ms=20,label=r'FD: $y_i$')

    plt.plot(0,1,'*k',ms=16,label='Left BC')
    plt.plot(1,3,'*g',ms=16,label='Right BC')
    plt.plot(0,alpha,'r',ms=16,label=r'$\alpha$', marker=r'$\alpha$')

    plt.legend(loc='best')

interact(shooting_method_vs_FD, alpha=(-2,2,0.01),
         N=(5,100,1), ya=(-5,5,0.1), yb=(-5,5,0.1),
         ODEsolver=radio_button_ODEsolvers,
         M=(3,25,1))


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
Here we define the several steps solvers.
These are the version that will be used later on in the code.
Can you write Python code for it?

# This implemantions have been upgraded respect to the ones included in the classnotes to handle
def eulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        y[i+1] = y[i]+f(t[i],y[i])*h
    return t, y
# Challenge: Make backwardEulerMethod work!
'''def backwardEulerMethod(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    y = np.zeros(N+1)
    y[0] = y0
    for i in np.arange(N):
        f_hat= lambda x:  x - y[i]-f(t[i+1],x)*h
        # You must select a solver, "find_root" is just a generic name. The input used are the anonymous function "f_hat" and the initial guess y_i.
        x = find_root(f_hat,y[i])
        y[i+1] = x
    return t, y'''
def RK2(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        y[i+1] = y[i]+f(t[i]+h/2,y[i]+k1*h/2)*h
    return t, y
def RK4(t0,T,N,y0,f):
    t = np.linspace(t0,T,N+1)
    h = (T-t0)/N
    if isinstance(y0,(int,float)):
        y = np.zeros(N+1)
    else:
        y = np.zeros((N+1,len(y0)))
    y[0] = y0
    for i in np.arange(N):
        k1 = f(t[i],y[i])
        k2 = f(t[i]+h/2,y[i]+k1*h/2)
        k3 = f(t[i]+h/2,y[i]+k2*h/2)
        k4 = f(t[i]+h,y[i]+k3*h)
        y[i+1] = y[i]+(k1+2*k2+2*k3+k4)*h/6
    return t, y


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
Let's take a look to $F(\alpha, \dots)$, find the 'BUG'!
Can you write Python code for it?

ODEsolver = eulerMethod

several_alphas = np.linspace(-1,0,100)
Fv = np.vectorize(lambda x: F(x,ODEsolver=ODEsolver))
F_alphas = Fv(several_alphas)

plt.figure(figsize=(10,10))
plt.plot(several_alphas,F_alphas,'.b',label=r'$F(\alpha)$')
plt.plot(several_alphas,F_alphas*0,'-g',alpha=0.8,linewidth=3,label=r'$0$')
plt.grid(True)
plt.plot(alpha,F(alpha,ODEsolver=ODEsolver),'.r',ms=20,label=r'Root: $\alpha$='+str(alpha))
plt.xlabel(r'$\alpha$')
plt.ylabel(r'$F(\alpha)=y_N^{(\alpha)} - y_b$')
plt.legend(loc='best')
plt.show()


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
Lotka-Volterra model
This instance models the evolution of the average population of two species living in a closed system.
The species number 1 is considered that feeds itself with an unlimited amount of food and the species number 2 feeds itself with species number 1.
We will denote the average population of species number 1 as $y_1(t)$ and the average population of species number 2 as $y_2(t)$.
So, if we consider that there is no presence of species number 2 in the system, then species number 1 will grow exponentially, this means $\dot{y}_1=y_1$.
On the other case, if no presence of species number 1 in the system, then species number 2 will decay exponentially as follows $\dot{y}_2=-y_2$.
However, we are not interested in the extreme cases, how can we understand the dynamics when there exists these two species together interacting?
This is modeled by the Lotka-Volterra dynamical system:
\begin{align}
    \dot{y}_1(t)&=(1-y_2(t)/\mu_2)\,y_1(t),\
    \dot{y}_2(t)&=-(1-y_1(t)/\mu_1)\,y_2(t),\
    y_1(0)&=400,\
    y_2(0)&=100,
\end{align}
where $\mu_1$ and $\mu_2$ are normalization constants.
Can you write Python code for it?

mu1=300
mu2=200
def f_predprey(t,y,mu1=mu1,mu2=mu2):
    y1 = y[0]
    y2 = y[1]
    return np.array([(1-y2/mu2)*y1, -(1-y1/mu1)*y2])

def f_predprey_jacobian(t,y,mu1=mu1,mu2=mu2):
    J=[[1-y[1]/mu2,-y[0]/mu2],[y[1]/mu1, -(1-y[0]/mu1)]]
    lambs,vs= LA.eig(J)
    return lambs

def plotting_predprey(y01=70, y02=100, N=1000, T=30, ODEsolver=eulerMethod):
    y0=np.array([y01, y02])
    f= lambda t,y: f_predprey(t,y)
    t0 = 0
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_predprey_jacobian,h,c1,c2,c3)

interact(plotting_predprey,y01=(0,1000,10),y02=(0,1000,10),
         N=(10,1000,10),T=(10,200,10),
         ODEsolver=radio_button_ODEsolvers)


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
Now, let's find $\alpha$ automatically
Can you write Python code for it?

y0 = np.array([1, 1.1])

def F(alpha, ya=1, yb=3, N=100, ODEsolver=eulerMethod):
    t0 = 0
    T  = 1
    y0 = np.zeros(2)
    y0[0] = ya
    y0[1] = alpha
    t_times, y_output = ODEsolver(t0,T,N,y0,f)
    # Notice that we don use absolute value for the error since
    # we want to look at this problem as a rootfinding problem
    # and not as a minimization, that could be another approach.
    return y_output[-1,0]-yb

t0 = 0
T  = 1
ya = 1
yb = 3
N  = 100
ODEsolver = RK4

F_root = lambda alpha: F(alpha, ya=ya, yb=yb, N=N, ODEsolver=ODEsolver)
alpha = root(F_root, 0.).x[0]

y0 = np.array([ya,alpha])
t_times, y_output = ODEsolver(t0,T,N,y0,f)

fig = plt.figure(figsize=(L/2,L/2))
plt.grid(True)
plt.title("Numerical Approximation")
plt.plot(t_times,y_output[:,0],'r.',ms=12,alpha=0.5,label=r'$y1_i$')
plt.plot(t_times,y_output[:,1],'m.',ms=12,alpha=0.5,label=r'$y2_i$')
plt.plot(0,ya,'*k',ms=16,label='Left BC')
plt.plot(1,yb,'*g',ms=16,label='Right BC')
plt.plot(0,alpha,'darkorange',ms=16,label=r'$\alpha$', marker=r'$\alpha$')
plt.legend(loc='best')
print(alpha)


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
So, how do we use the stability region for each method?
The key point here is that we can't define $h$ a priori since we may not know the corresponding $\lambda$, thus, in principle, we can estimate $\lambda$ or just try the solver for different $h$.
Can you write Python code for it?

def plot_log_eq_stability_region(y0=1.2, N=20, T=10, ODEsolver=eulerMethod):

    # How did w get this? Why do I need it?
    def f_1D_prime(y):
        return np.array(1-2*y)

    t0=0
    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    fig = plt.figure(figsize=(L,L/3))
    ax1 = fig.add_subplot(131)
    ax2 = fig.add_subplot(132)
    ax3 = fig.add_subplot(133)

    ax1.axis('equal')
    ax1.grid(True)
    ax1.set_title('Numerical Approximation')
    ax1.axis([0, T, -0.5, 2])
    ax1.plot(t_times,y_output,'k-')
    ax1.plot(t_times,y_output,'b.')

    # The next code is to plot in logscale unstable numerical simulations
    h = (T-t0)/N
    y_all=h*f_1D_prime(y_output)
    y_pos=y_all>0
    y_neg=np.logical_not(y_pos)
    ax3.semilogy(t_times[y_pos],y_all[y_pos],'.r',ms=20)
    ax3.semilogy(t_times[y_neg],-y_all[y_neg],'.b',ms=20)
    ax3.grid(True)
    ax3.set_title(r'$\log10(|y_i|)$')

    zplot2(W1,ax2,line_color=c1)
    zplot2(W2,ax2,line_color=c2)
    zplot2(W4,ax2,line_color=c3)
    k_lambdah=h*f_1D_prime(y_output)
    ax2.plot(np.real(k_lambdah),np.imag(k_lambdah),'.r',ms=20)
    ax2.axis('equal')
    ax2.axis([-5, 2, -3.5, 3.5])
    ax2.grid(True)
    ax2.set_title('Stability Region and $k=\lambda\,h$')

interact(plot_log_eq_stability_region,y0=(-2,5,0.1),N=(2,100,1), T=(1,20,1), ODEsolver=radio_button_ODEsolvers)


I am working on the file "SC1v2/11_ODE.ipynb".
The first task is:
What about the stability region for dynamical systems?
In this case we need to compute the eigenvalues of the Jacobian matrix of $\mathbf{F}(\mathbf{y})$.
So the eigenvalues $\lambda_i$ multiplied by $h$ have to be inside of the stability region of the real part of $\lambda_i$ is negative, for $i\in{1,2,\dots,\text{Dimension of Dynamical System}}$.
Can you write Python code for it?

def plotting_vpd_as_system(y01=2, y02=0, N=1000, T=40, mu=10, ODEsolver=eulerMethod):

    y0=np.array([y01, y02])

    f= lambda t,y: f_vdp(t,y,mu)

    t0 = 0

    t_times, y_output = ODEsolver(t0,T,N,y0,f)

    h = (T-t0)/N

    if ODEsolver==eulerMethod:
        c1,c2,c3='b','k','k'
    elif ODEsolver==RK2:
        c1,c2,c3='k','b','k'
    else:
        c1,c2,c3='k','k','b'

    plot_sol_eig_phase_portrait_2eq(t_times,y_output,f_vdp_eig_jacobian,h,c1,c2,c3)

interact(plotting_vpd_as_system, y01=(-3,3,0.1), y02=(-3,-3,0.1),
         N=(10,4000,10),T=(10,200,10),
         mu=(1,50,0.1),ODEsolver=radio_button_ODEsolvers)


I am working on the file "SC1v2/Bonus - 00 - List Comprehension vs NumPy.ipynb".
The first task is:
<div id='second' />

Second Task: Square the elements of the lists of integers.
Back to TOC
Can you write Python code for it?

# Python base
i1_sq = [i**2 for i in i1]
t_out3 = %timeit -o [i**2 for i in i1]

# Intermediate implementation, notice the effect.
t_out3b = %timeit -o np.power(i1,2)

# NumPy version
i2_sq = np.power(i2,2)
t_out4a = %timeit -o i2**2
t_out4b = %timeit -o np.power(i2,2)


I am working on the file "SC1v2/Bonus - 00 - List Comprehension vs NumPy.ipynb".
The first task is:
In this case we have used two version of the NumPy alternative, in both cases it is way faster than the traditional version of Python.
Can you write Python code for it?

print('NumPy, version a, is', t_out3.average/t_out4a.average, 'times faster than the Python base version!!')
print('NumPy, version b, is', t_out3.average/t_out4b.average, 'times faster than the Python base version!!')

print('The intermediate implementation is OK but not the best, it is only',t_out3.average/t_out3b.average,'times better.')
print('The issue for the intermediate implementation is that NumPy needs to translate i1 to a NumPy array first, \nand that\'s where the execution uses extra time.')


I am working on the file "SC1v2/Bonus - 00 - List Comprehension vs NumPy.ipynb".
The first task is:
We clearly notice (at least in my machine) that version ''a'' is faster than version ''b'', and it could be more readable, but we do encourage to use version ''b'' for advanced tasks to avoid or reduce numerical issues.
For instance, in the Jupyter Notebook ''Bonus - 00 - Playing with Julia.ipynb'' we highlight an issue with adding numbers, which is handled correctly by np.sum, thus, using NumPy is not only for clearness and speed, it is also about correctness of the computation.
<div id='final' />

This and final task: Evaluate the following expression,
Back to TOC
$$
\begin{align}
[x_1,x_2]&= \displaystyle{\mathop{\mathrm{argmin}}{\widehat{x}_1,\widehat{x}_2\in [-1,1]}}\,
\max{x\in [-1,1]} |(x-\widehat{x}_1)\,(x-\widehat{x}_2)|.
\end{align}
$$
This expression appears in the Jupyter Notebook Bonus - 05 - Finding 2 Chebyshev PointsGraphically.ipynb, where we explain why it is important.
Now, we will just evaluate it.
Can you write Python code for it?

# We assume we will work over a discrete grid of points in [-1,1]
# and the number of discrete points will be N. This means that
# the continuous variables "x", "x_1", "x_2", "hat{x}_1" and "hat{x}_2"
# will be discretized in N points.

# Python base version
def find_x1_x2(N):
    list_i = buildListIntegers(N)
    # Here we apply a linear transformation to build the discrete version of "x"
    # which will be denoted by "xi"
    x_discrete = [(2*(i-1)/(N-1))-1 for i in list_i]
    min_value_outer = -1
    x1_hat_min = -1
    x2_hat_min = -1
    for x1_hat in x_discrete:
        for x2_hat in x_discrete:
            max_value_inner=-1
            for xi in x_discrete:
                value_tmp = abs((xi-x1_hat)*(xi-x2_hat))
                # Here we find the max |(x-x1_hat)*(x-x2_hat)|
                if value_tmp>max_value_inner:
                    max_value_inner=value_tmp
            if min_value_outer == -1 or min_value_outer>max_value_inner:
                min_value_outer = max_value_inner
                x1_hat_min = x1_hat
                x2_hat_min = x2_hat
    return x1_hat_min, x2_hat_min

# NumPy version, storing intermediate values.
def find_x1_x2_NumPy(N):
    x=np.linspace(-1,1,N)
    w = lambda x1,x2: np.max(np.abs((x-x1)*(x-x2)))
    wv=np.vectorize(w)
    [X,Y]=np.meshgrid(x,x)
    W=wv(X,Y)
    id_min = np.unravel_index(np.argmin(W, axis=None),W.shape)
    x1_hat = X[id_min]
    x2_hat = Y[id_min]
    return x1_hat, x2_hat

# Debugging the output
# print('found Python base:', find_x1_x2(N))
# print('found Numpy:', find_x1_x2_NumPy(N))
# print(np.sqrt(2)/2)

N = 350
t_out5 = %timeit -o find_x1_x2(N)
t_out6 = %timeit -o find_x1_x2_NumPy(N)

print('The NumPy version is', t_out5.average/t_out6.average, 'times faster than the Python base version!!')


I am working on the file "SC1v2/Bonus - 00 - Solving a linear system of equations.ipynb".
The first task is:
<div id='intro' />

Introduction
Back to TOC
In this jupyter notebook we will analyze two different way for solving a linear system of equations. The main idea is to verify computationally that translating from a theoretical solution into a computational one is not necesarily a straightforward task and we must face some decisions.
The classical problem we will be solving is the following:
$$
\begin{equation}
    A\,\mathbf{x} = \mathbf{b}, \quad A \in \mathbb{R}^{n\times n}, \mathbf{x} \in \mathbb{R}^n, \text{ and } \mathbf{b} \in \mathbb{R}^n,
\end{equation}
$$
where we know $A$ and $\mathbf{b}$, and we are looking for $\mathbf{x}$.
The representation of the solution is the following:
$$
\begin{equation}
    \mathbf{x}=A^{-1}\,\mathbf{b}.
\end{equation}
$$
This way for finding $\mathbf{x}$ can be implemented in two steps:
1. Compute $A^{-1}$,
2. Multiply $A^{-1}$ by $\mathbf{b}$.
This approach has been implemented in the function solve_inverse.
Can you write Python code for it?

# This function solves a linear system of equation
# in a NOT recomended way, we just include it here
# for comparison purposes. Please DON'T do this!
def solve_inverse(A,b):
    B = np.linalg.inv(A) # FORBIDDEN!!
    x = np.dot(B,b)
    return x


I am working on the file "SC1v2/Bonus - 00 - Solving a linear system of equations.ipynb".
The first task is:
Do you notice a difference in the computation time?
To be able to undertand in more detail the tendency we need to test it for several values of n.
Can you write Python code for it?

# This line of code will help us to manipulate the output.
out = %timeit -o -q -n 10 -r 5 np.linalg.solve(A,b)

out.all_runs


I am working on the file "SC1v2/Bonus - 00 - Solving a linear system of equations.ipynb".
The first task is:
The following code compares the solution obtained by each approach for the small problem.
Can you write Python code for it?

x1 = np.linalg.solve(A,b)
print('x1: ',x1)

x2 = solve_inverse(A,b)
print('x2: ',x2)


I am working on the file "SC1v2/Bonus - 00 - The beginning.ipynb".
The first task is:

Introduction
Back to TOC
In this jupyter notebook we will briefly introduce part of ecosystem in Python for Scientific Computing with explanation and examples.
We strongly suggest to follow the explanation step by step, read the suggested documents/links, and play with the jupyter notebook changing it to learn!

NumPy, the core of numerical computing in Python
Back to TOC
We strongly suggests to take a look to the associated paper explaning Arrays in NumPy link.
Now, we will start with some examples provided in Fig. 1 from the previous reference. Also see https://numpy.org.
a) Data structure: An array and a bi-dimensional array.
Can you write Python code for it?

# NumPy array
a1 = np.arange(12)
# NumPy ndarray
a2 = np.array([[1, 2, 3], [4, 5, 6]], float)
a3 = np.reshape(a1,(4,3))

print(textBoldB(' Printing their values '))
print(textBoldH('a1:\n'),a1)
print(textBoldH('a2:\n'),a2)
print(textBoldH('a3:\n'),a3)

print(textBoldB(' Printing their datatypes '))
print(textBoldH('a1:\n'),a1.dtype)
print(textBoldH('a2:\n'),a2.dtype)
print(textBoldH('a3:\n'),a3.dtype)

print(textBoldB(' Printing their type name '))
print(textBoldH('a1:\n'),type(a1).__name__)
print(textBoldH('a2:\n'),type(a2).__name__)
print(textBoldH('a3:\n'),type(a3).__name__)


I am working on the file "SC1v2/Bonus - 00 - The beginning.ipynb".
The first task is:
<div id='matplotlib' />

MatPlotLib, comprehensive 2-D plotting
Back to TOC
In this case, we will go straight to the examples. See https://matplotlib.org and https://github.com/matplotlib/cheatsheets#cheatsheets.
Can you write Python code for it?

def plot_trig_func(n=5):
    x = np.linspace(0,10,n) # Generates a linearly spaced set of n points from 0 to 10
    y1 = np.sin(x) # Vectorization in action!
    y2 = np.cos(x) # Vectorization in action!
    plt.figure(figsize=(5,5)) # Creating the figure
    plt.plot(x,y1,label=r'$\sin(x)$') # Plotting sin
    plt.plot(x,y2,'r.-',label=r'$\cos(x)$') # Plotting cos
    plt.grid(True) # Adding nice grid
    plt.legend(loc='best') # Adding legend
    plt.title('Plotting trigonometric functions') # Adding title
    plt.xlabel(r'$x$') # Adding x-label
    plt.ylabel(r'$y$') # Adding y-label
    plt.show() # Showing the plot
# The following is a iPython widget called interact,
# it allows us to interact with the function that it is being called.
interact(plot_trig_func,n=(3,1000,1)) # Making this 'tunable'!

def plot_funct_2_var(theta=0):
    x = np.linspace(-2,2,20)
    y = x # Recall this is just a 'view'

    X, Y = np.meshgrid(x,y) # It generates two 2D arrays with with cartesian product of 'x' and 'y'.

    # 'lambda' definition help us to organize better the code
    f = lambda x,y: np.sin(x*np.sin(theta)+y*np.cos(theta))*np.exp(-(x**2+y**2))
    Z = f(X,Y) # Evaluation a function of two variables

    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10,5))
    fig.suptitle('Comparison of two ways of plotting $f(x,y)$')

    # This is useful for images.
    ax1.imshow(Z, cmap='hot', interpolation='nearest')
    ax1.set_ylabel('imshow')

    # This is useful for functions in two variables, i.e. z=f(x,y)
    ax2.pcolormesh(X,Y,Z,cmap='hot', shading='auto')
    ax2.axis('equal')
    ax2.set_ylabel('pcolormesh')
    plt.show()
interact(plot_funct_2_var,theta=(0,2*np.pi,2*np.pi/50))

def plot_funct_2_var_3D(theta=0):
    x = np.linspace(-2,2,40)
    y = x # Recall this is just a 'view'

    X, Y = np.meshgrid(x,y) # It generates two 2D arrays with with cartesian product of 'x' and 'y'.

    # 'lambda' definition help us to organize better the code
    f = lambda x,y:...


I am working on the file "SC1v2/Bonus - 00 - The beginning.ipynb".
The first task is:
<div id='scipy' />

SciPy, fundamental library for scientific computing
Back to TOC
In this case, we will go straight to the examples. See https://www.scipy.org.
Can you write Python code for it?

# Numerical Integration
import scipy.integrate as integrate

# Integrating $\int_0^1 \exp(-x^2)\,dx$
result = integrate.quad(lambda x: np.exp(-x**2), 0, 1)
print(result)

# Integrating $\int_0^1 1\,dx=1$
result = integrate.quad(lambda x: 1, 0, 1)
print(result)

# Minimization
from scipy.optimize import minimize
f = lambda x: np.sin(x[0])*(1+np.cos(x[1]))

x0 = np.array([0, 0])
res = minimize(f, x0, method='nelder-mead', options={'xatol': 1e-8, 'disp': True})
print(res.x)

# Plotting the previous function
x = np.linspace(-2*np.pi,2*np.pi)
y = x
X,Y = np.meshgrid(x,y)
f = lambda x,y: np.sin(x)*(1+np.cos(y))
Z = f(X,Y)
plt.figure(figsize=(8,8))
plt.pcolormesh(X,Y,Z,shading='auto')
plt.plot(res.x[0],res.x[1],'r.')
plt.colorbar()
plt.title('The red dot is the minimum found numerically')
plt.show()

# root finding
from scipy.optimize import root
f = lambda x: x - 0.5 * np.exp(x)
sol = root(f, 0.3)
print(sol)


I am working on the file "SC1v2/Bonus - 00 - The beginning.ipynb".
The first task is:
b) Indexing (view)
Can you write Python code for it?

print(textBoldB(' Showing original variable a3 '))
print(textBoldH('a3\n'),a3)

print(textBoldB(' Showing a subset of elements (SLICES) of variable a3 \n the second and third columns '))
print(textBoldH('a3[:,1:]\n'),a3[:,1:])
print(textBoldB(' Storing the outcome in a4 '))
a4 = a3[:,1:]

print(textBoldB(' What does it means it is a "view"? '))
print(textBoldB(' Let\'s change the original a3 and see what happens to a4. '))
a3[0,1]=100
print(textBoldH('a4\n'), a4)
print(textBoldB(' a4 has changed too! So a4 is a "view" of part of a3. '))
print('\n')
print(textBoldB(' So, how do we copy it?'))
print(textBoldB(' Let\'s use np.copy to copy the "view" obtained before,'+
                '\n then modify the original variable and see what had happened'))
a5 = np.copy(a3[:,1:])
print(textBoldH('a3\n'), a3)
print(textBoldH('a4=a3[:,1:])\n'), a4)
print(textBoldH('a5=np.copy(a3[:,1:])\n'), a5)
a3[0,1]=1
print(textBoldB(' Changing a3 as "a3[0,1]=1"'))
print(textBoldH('a3\n'), a3)
print(textBoldH('a4\n'), a4)
print(textBoldH('a5\n'), a5)
print(textBoldB(' Thus, chaning a3 have changed a4 since it is a "view" and not changed a5 since it is a copy. '))

print(textBoldB(' Another way to extract slices is with steps: '))
a6 = a3[:,::2]
print(textBoldH('a3\n'), a3)
print(textBoldH('a6 = a3[:,::2]\n'), a6)
print(textBoldB(' In this case we obtained the first and third columns, \n '+
               ' the pattern consider ini:end:step, where "ini" is the first index considered,\n'+
               ' "end" defines that the pattern will be considere until the index "end-1",\n'+
               ' and "step" indicates the step for selecting the next index, it is 1 by default.'))


I am working on the file "SC1v2/Bonus - 00 - The beginning.ipynb".
The first task is:
d) Vectorization (THE HEART OF THE HEART OF NUMERICAL COMPUTING!!)
Can you write Python code for it?

print(textBoldH('a3\n'), a3)
print(textBoldB(' We will define a new ndarray n1 = np.ones((4,3)) and add it to the array a3.'))
n1 = np.ones((4,3))

print(textBoldB(' The first implementation one may think of is the following:'))
def sum_two_arrays(a3,n1):
    out = np.zeros_like(a3) # What does this function do?
    for i in range(4):
        for j in range(3):
            out[i,j] = a3[i,j]+n1[i,j]
    return out
n2 = sum_two_arrays(a3,n1)

# To quantify the computation time it takes let's measure it:
%timeit n2 = sum_two_arrays(a3,n1)

# The output it shows in the computer we run the experiments was:
# 33.9 µs ± 1.68 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
# This means it took about 34 microseconds.

# Now, let's try the NumPy way! i.e. the vectorized approach!
%timeit n2 = a3+n1

# The output it shows in the computer we run the experiments was:
# 1.05 µs ± 12.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
# This means it was about 34 times faster!!
# and also it only took "one" line of code where in the previous version took "7" lines of code!!
print(textBoldR(' Given the previous argument, it is crucial you get familir with VECTORIZATION computing with NumPy!! '))