I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
While we cannot load the light curves to the database in parallel, using the concept of MapReduce we can use the iPython "cluster" that we created to map the I/O process (typically lots of overhead) to multiple machines, and then we can reduce their output to a single object to be loaded into the database.
Of course, before you parallelize, you profile.
Problem 3d
Earlier it was claimed that bulk loading the data is faster than loading it line by line. Prove this assertion, use %%timeit to "profile" the two different options (bulk load with executemany and loading one photometric measurement at a time via for loop).
Compare both serial solutions, to load the data in parallel.
Hint - to avoid corruption of your current working database, MiniBroker.db, create temporary databases for the pupose of running this test. Note - the creation and connection to the database should happen outside your %%timeit magic function. This means your solution will have (at least) 4 separate cells.
Can you write Python code for it?

tmp_conn1 = sqlite3.connect("tmp1.db")
tmp_cur1 = tmp_conn1.cursor()

tmp_cur1.execute("""drop table if exists rawPhot""") # drop the table if it already exists
tmp_cur1.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

%%timeit
# time how long it takes to load every observation individually

for filename in filenames:
    var = Variable(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    for obs in var._lc_df.values:
        tmp_cur1.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(obs)))

tmp_conn2 = sqlite3.connect("tmp2.db")
tmp_cur2 = tmp_conn2.cursor()

tmp_cur2.execute("""drop table if exists rawPhot""") # drop the table if it already exists
tmp_cur2.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

%%timeit
# time how long it takes to load all observations of a single source at once

for filename in filenames:
    var = Variable(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    data = [(objId,) + tuple(x) for x in var._lc_df.values] # array of tuples

    tmp_cur2.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
You shold have confirmed that loading a full light curve is faster than loading individual observations (not by a ton, but faster is faster and microseconds count when you are dealing with 37 billion sources).
Now we will parallelize this operation in order to perform the I/O in parallel, followed by a single bulk load of the data. To do so, we need the @require decorator, which will ensure the necessary packages and definitions are available on each computer within our cluster.
Can you write Python code for it?

# execute this
@require(Source, Variable, 'pandas as pd', 'numpy as np')
def load_data(filename):
    var = Variable(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    data = [(objId,) + tuple(x) for x in var._lc_df.values] # array of tuples
    return data


I am working on the file "Sessions/Session08/Day4/Algorithmic Complexity Notes.ipynb".
The first task is:
Plot the results...
Can you write Python code for it?

plt.figure()
plt.loglog(nn,times,'x', label='data')
plt.loglog(nnn,times[0]*(nnn/nnn[0])**2, label=r'$O(n^2)$')
plt.legend();plt.xlabel('data size');plt.ylabel('run time (s)')


I am working on the file "Sessions/Session08/Day4/Algorithmic Complexity Notes.ipynb".
The first task is:
Several Common Asymptotics
Can you write Python code for it?

plt.figure()
plt.loglog(nn,times,'x', label='data')
plt.loglog(nnn,t1.average*(nnn/nnn[0])**3, label=r'$O(n^3)$')
plt.loglog(nnn,times[0]*(nnn/nnn[0])**2, label=r'$O(n^2)$')
plt.loglog(nnn,times[0]*(nnn/nnn[0]), label=r'$O(n)$')
plt.loglog(nnn,t1.average*(nnn/nnn[0])*np.log(nnn/nnn[0]), label=r'$O(n \log(n))$')

plt.legend()
plt.xlabel('data size')
plt.ylabel('run time (s)')


I am working on the file "Sessions/Session08/Day4/Algorithmic Complexity Notes.ipynb".
The first task is:
The answer
Each time the function is called, we have
python
n = len(x)
mini = x[0]
that's two instructions (again, ignoring how much goes into the len call).
Then, the for loop body requires either one or two instructions. You always have the comparison x[i] < mini, but you may or may not have the assignment mini = x[i].
Exercise
Compute the number  number of instructions for this input data
python
x = [4, 3, 2, 1]
and
python
y = [1, 3, 2, 4]
$N_{inst}(x) = 9$
$N_{inst}(y) = 7$
As usual, pessimism is the most useful view
The answer to "how long does this take" is...it depends on the input.
Since we would like to know how long an algorithm will take before we run it, let's examine the worst case scenario.
This allows us to looking for from $T(d)$ to $f(n)$, where $n \equiv \mathrm{size}(d)$ is the size of the dataset.
For our simple problem,
$$f(n) = 2 + 4n$$
Asymptotics
Let's look at a pair of cubic polynomials,
$$ f(n) = f_0 + f_1 n + f_2 n^2 + f_3 n^3 $$
$$ g(n) = g_0 + g_1 n + g_2 n^2 + g_3 n^3 $$
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
2b) Use the %timeit function to determine whether the Numba version of the hyptonenuse function is better than the original Python implementation.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
All that's left is to create a list of random particles with assigned masses:
Can you write Python code for it?

n = 1000
particles = [Particle(m = 1 / n) for i in range(n)]


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Hold on - our new pre-compiled function is running even slower than the original python version! What's going on here?
Problem 2 - A Better Numba Function
(This problem borrowed from seibert's 2018 gtc numba tutorial.)
As we saw in the first example, Numba isn't going to speed up everything. Generally, Numba will help you most in circumstances where python's line-by-line interperability and lack of type casting is slowing it down. We can use a slightly more complicated function to demonstrate this. The following is a function to calculate the hypotenuse of two numbers, that has been carefully designed to compensate for the computer's finite precision representation of numbers (check out https://en.wikipedia.org/wiki/Hypot for more info).
2a) Use the @jit decorator to generate a Numba version of this function.
Can you write Python code for it?

# add code here
def hypotenuse(x, y):
  x = abs(x);
  y = abs(y);
  t = min(x, y);
  x = max(x, y);
  t = t / x;
  return x * math.sqrt(1+t*t)


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
More commonly, you will use jit as a decorator, by adding @jit to the line above your function definition, but the above version shows you that at heart, @jit is just a python function that takes other functions as its argument!
1c)  By default, a Numba function saves the original python version of the function in the variable py_func. Check that the original python version gives you the same answer as the Numba version.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Now, test the function, first with two scalar integers:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Numpy Universal Functions
You'll notice a couple of new things here. In the first set of brackets, we have specified both the argument types of the function (those are inside the parentheses), as well as the return type of the function. This is just making explicit what Numba was previously inferring on our behalf. In second set of brackets you'll see that we have specified a 'target' architechture for the function. The default is cpu, which means that Numba is optimizing the function to your specific machine. Other options include parallel, which allows you to take advantage of multicore processors, and cuda, which we'll be discussing more tomorrow. You'll also notice that we called this a 'ufunc', which is short for Universal Function. In brief, universal functions are numpy functions that operate on ndarrays in element by element fashion. So if we pass two vectors to our add_func, they will be added together and a return vector of the same shape will be returned.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Okay, so our function can add things. Now, let's use Numba's jit function to create a Numba version of our addition function:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Problem 3 - Fun with Fractals
Now that we've got the basics of the Numba jit decorator down, let's have a little fun. A classic example problem in parallel programming is the calculation of a fractal, because a large fraction of the work can be done in parallel. Below is some code that calculates whether a number is a member of the Julia set, and then computes the set on a discrete domain to calculate a fractal.
3a) Modify the code below to use Numba and test how much faster it is than the original python implementation.
Can you write Python code for it?

# add code here
def julia(x, y, max_iters):
    """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Julia
    set given a fixed number of iterations.
    """
    i = 0
    c = complex(-0.8, 0.156)
    a = complex(x,y)
    for i in range(max_iters):
        a = a*a + c
        if (a.real*a.real + a.imag*a.imag) > 1000:
            return 0
    return 255

# add code here
def create_fractal(min_x, max_x, min_y, max_y, image, iters):
    height = image.shape[0]
    width = image.shape[1]

    pixel_size_x = (max_x - min_x) / width
    pixel_size_y = (max_y - min_y) / height
    for x in range(width):
        real = min_x + x * pixel_size_x
        for y in range(height):
            imag = min_y + y * pixel_size_y
            color = julia(real, imag, iters)
            image[y, x] = color

    return image

image = np.zeros((500, 750), dtype=np.uint8)
%timeit create_fractal(-2.0, 2.0, -1.0, 1.0, image, 200)


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Problem 5 - Direct Summation
(This problem borrowed in its entirety from gforsyth's scipy17 numba tutorial.)
Many physical problems require the evaluation of all pairwise interactions of a large number of particles, so-called N-body problems. These problems arise in molecular dynamics, astrodynamics and electromagnetics among others.
Their pairwise interactions can be expressed as:
\begin{equation}
f_i = \sum_{j=1}^n{P \left(\boldsymbol{x}_i, \boldsymbol{x}_j \right)w_j} \ \ \ \text{for } i=1,2,...,n
\end{equation}

where subscripts $i$,  $j$ respectively denote target and source
$f_i$ can be a potential (or force) at target point $i$
$w_j$ is the source weight
$\boldsymbol{x}_i, \boldsymbol{x}_j$ are the spatial positions of particles
$P \left(\boldsymbol{x}_i, \boldsymbol{x}_j \right)$ is the interaction kernel.

In order to evalute the potential $f_i$ at a target point $i$, we have to loop over each source particle $j$. Since there are $n$ target points $i$, this 'brute-force' approach costs $\mathcal{O} \left(n^2 \right)$ operations.
One possible approach in this kind of problem is to define a few classes, say Point and Particle and then loop over the objects and perform the necessary point-to-point calculations.
Can you write Python code for it?

class Point():
    """
    Arguments:
        domain: the domain of random generated coordinates x,y,z,
                default=1.0

    Attributes:
        x, y, z: coordinates of the point
    """
    def __init__(self, domain=1.0):
        self.x = domain * np.random.random()
        self.y = domain * np.random.random()
        self.z = domain * np.random.random()

    def distance(self, other):
        return ((self.x - other.x)**2 +
                (self.y - other.y)**2 +
                (self.z - other.z)**2)**.5

class Particle(Point):
    """
    Attributes:
        m: mass of the particle
        phi: the potential of the particle
    """

    def __init__(self, domain=1.0, m=1.0):
        Point.__init__(self, domain)
        self.m = m
        self.phi = 0.


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
So far we have been using what Numba refers to as "lazy" (or "call-time") decoration. Basically, we've been letting Numba do the work of figuring out how we're using the function and inferring the types for us. Alternatively, if we know how we are going to use a given function, we can use "eager" (or "compile-time") decoration. To do this, we make use of the vectorize decorator. For example, if we want to make an integer-only version of our addition function, we could write:
Can you write Python code for it?

@vectorize(['int64(int64, int64)'], target='cpu')
def add_ufunc(x, y):
  return x + y


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
So, scalars are faster than arrays (makes sense), and python's addition function is better than ours (seems reasonable). Now, let's see how fast our pre-compiled Numba addition function is.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
This tells us the array contains integers, and each integer has been assigned 64bits in memory (or equivalently, 8 bytes). Most python functions are defined to work on arbitrary types, so that if you use the + operator, for example, you can add integers, floats, complex numbers, or even strings! However, this flexibility comes at a cost, performance-wise. Numba, on the other hand, compiles each function based on the types of its arguments, and infers the type of the result. You can see this if you run the inspect_types function on a numba function:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
Want to see what you made? Run the following code to plot the image. Feel free to pick your favorite matplotlib color map :)
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
We don't have a distance method anymore, so we need to write a function to take care of that.
5c) Write a jit function distance to calculate the distance between two particles of dtype particle_dtype.
Can you write Python code for it?

# add code here

distance(particles[0], particles[1])

%%timeit
distance(particles[0], particles[1])


I am working on the file "Sessions/Session08/Day4/numba_intro.ipynb".
The first task is:
What is Numba?
Numba is a just-in-time, type-specializing, function compiler for accelerating numerically-focused Python. That's a long list, so let's break down those terms:

function compiler: Numba compiles Python functions, not entire applications, and not parts of functions. Numba does not replace your Python interpreter, but is just another Python module that can turn a function into a (usually) faster function.
type-specializing: Numba speeds up your function by generating a specialized implementation for the specific data types you are using. Python functions are designed to operate on generic data types, which makes them very flexible, but also very slow. In practice, you only will call a function with a small number of argument types, so Numba will generate a fast implementation for each set of types.
just-in-time: Numba translates functions when they are first called. This ensures the compiler knows what argument types you will be using. This also allows Numba to be used interactively in a Jupyter notebook just as easily as a traditional application.
numerically-focused: Currently, Numba is focused on numerical data types, like int, float, and complex. There is very limited string processing support, and many string use cases are not going to work well on the GPU. To get best results with Numba, you will likely be using NumPy arrays.

Problem 1 - A First Numba Function
1a) To start our exploration of Numba's features, let's write a python function to add two numbers. We'll creatively name it add:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
1b) With Numpy, we can use our function to add not just scalars, but vectors as well. Using your favorite array creation routine, create two integer arrays with ten elements each, called a and b, and use your add function to add them.
Can you write Python code for it?

a = np.arange(0,10) # add code here
b = np.arange(1,11) # add code here

add(a, b) # add code here


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
2b) Use the %timeit function to determine whether the Numba version of the hyptonenuse function is better than the original Python implementation.
Can you write Python code for it?

%timeit hypotenuse(3,4) # add code here
%timeit hypotenuse.py_func(3,4) # add code here


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
2c) Numba functions can call other functions, provided they are also Numba functions. Below is a function that loops through two numpy arrays and puts their sum into an output array. Modify the following function to calculate the hypotenuse instead.
Can you write Python code for it?

@njit # this is an alias for @jit(nopython=True)
def ex_func(x, y, out):
  for i in range(x.shape[0]):
    out[i] = hypotenuse(x[i], y[i]) # change this line

in1 = np.arange(10, dtype=np.float64)
in2 = 2 * in1 + 1
out = np.empty_like(in1)

print('in1:', in1)
print('in2:', in2)

ex_func(in1, in2, out)

print('out:', out)

# This test will fail until you fix the ex1 function
np.testing.assert_almost_equal(out, np.hypot(in1, in2))


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
3b) There is more than one type of fractal in the world, however! Below is a function that determines membership in the Mandelbrot set. Modify the function using to take advantage of Numba, then modify the code above to produce a new pretty picture.
Can you write Python code for it?

@njit #add code here
def mandel(x, y, max_iters):
    """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Mandelbrot
    set given a fixed number of iterations.
    """
    i = 0
    c = complex(x,y)
    z = 0.0j
    for i in range(max_iters):
        z = z*z + c
        if (z.real*z.real + z.imag*z.imag) >= 4:
            return i
    return 255

# add code here
@njit
def create_fractal(min_x, max_x, min_y, max_y, image, iters):
    height = image.shape[0]
    width = image.shape[1]

    pixel_size_x = (max_x - min_x) / width
    pixel_size_y = (max_y - min_y) / height
    for x in range(width):
        real = min_x + x * pixel_size_x
        for y in range(height):
            imag = min_y + y * pixel_size_y
            color = mandel(real, imag, iters)
            image[y, x] = color

    return image

image = np.zeros((500, 750), dtype=np.uint8)
create_fractal(-2.0, 2.0, -1.0, 1.0, image, 200)
plt.imshow(image)
plt.viridis()
plt.show()


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
4a) Numba has inferred the types for this function based on how we've used it. Try out your numba_add function with two floating point numbers, then re-inspect the types of the Numba function. Are they the same?
Can you write Python code for it?

#Add code here
numba_add(3.,4.)
numba_add.inspect_types()


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
5a) Run the direct summation code and determine how long it takes with 10, 100, 1000 particles. Is there a relationship?
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
5d) Modify the direct_sum function above to instead work a NumPy array of particles.  Loop over each element in the array and calculate its total potential. Time the result and compare it to your previous version of this function.
Can you write Python code for it?

@njit
def direct_sum(particles):
    for i, target in enumerate(particles):
        for j, source in enumerate(particles):
            if i != j:
                r = distance(target, source)
                target.phi += source.m / r

    return particles

%timeit direct_sum(particles)


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Because this is an $\mathcal{O} \left(n^2 \right)$ function, each addition of 10x particles costs 100x more in computational time. Given how expensive it is, let's see if we can improve things a bit with Numba. (This also happens to be a highly parallelizable problem, but we'll get to that tomorrow.)
How do we use Numba on this problem? There is a subtle issue here - Numba doesn't support jitting native Python classes.  There is a jit_class structure in Numba but it's still in early development. But we'd like to have attributes for readable programming. The solution is to build NumPy custom dtypes.
Can you write Python code for it?

particle_dtype = np.dtype({'names':['x','y','z','m','phi'],
                             'formats':[np.double,
                                        np.double,
                                        np.double,
                                        np.double,
                                        np.double]})

myarray = np.ones(3, dtype=particle_dtype)

myarray


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Next, we define a function to calculate the particle interaction via direct summation:
Can you write Python code for it?

def direct_sum(particles):
    """
    Calculate the potential at each particle
    using direct summation method.

    Arguments:
        particles: the list of particles

    """
    for i, target in enumerate(particles):
        for source in (particles[:i] + particles[i+1:]):
            r = target.distance(source)
            target.phi += source.m / r


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Now we'll create our array of particles using the new function.
Can you write Python code for it?

particles = create_n_random_particles(1000, .001, 1)

particles[:3]


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Numpy Universal Functions
You'll notice a couple of new things here. In the first set of brackets, we have specified both the argument types of the function (those are inside the parentheses), as well as the return type of the function. This is just making explicit what Numba was previously inferring on our behalf. In second set of brackets you'll see that we have specified a 'target' architechture for the function. The default is cpu, which means that Numba is optimizing the function to your specific machine. Other options include parallel, which allows you to take advantage of multicore processors, and cuda, which we'll be discussing more tomorrow. You'll also notice that we called this a 'ufunc', which is short for Universal Function. In brief, universal functions are numpy functions that operate on ndarrays in element by element fashion. So if we pass two vectors to our add_func, they will be added together and a return vector of the same shape will be returned.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Problem 3 - Fun with Fractals
Now that we've got the basics of the Numba jit decorator down, let's have a little fun. A classic example problem in parallel programming is the calculation of a fractal, because a large fraction of the work can be done in parallel. Below is some code that calculates whether a number is a member of the Julia set, and then computes the set on a discrete domain to calculate a fractal.
3a) Modify the code below to use Numba and test how much faster it is than the original python implementation.
Can you write Python code for it?

@njit # add code here
def julia(x, y, max_iters):
    """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Julia
    set given a fixed number of iterations.
    """
    i = 0
    c = complex(-0.8, 0.156)
    a = complex(x,y)
    for i in range(max_iters):
        a = a*a + c
        if (a.real*a.real + a.imag*a.imag) > 1000:
            return 0
    return 255

@njit # add code here
def create_fractal(min_x, max_x, min_y, max_y, image, iters):
    height = image.shape[0]
    width = image.shape[1]

    pixel_size_x = (max_x - min_x) / width
    pixel_size_y = (max_y - min_y) / height
    for x in range(width):
        real = min_x + x * pixel_size_x
        for y in range(height):
            imag = min_y + y * pixel_size_y
            color = julia(real, imag, iters)
            image[y, x] = color

    return image

image = np.zeros((500, 750), dtype=np.uint8)
%timeit create_fractal(-2.0, 2.0, -1.0, 1.0, image, 200)


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Problem 5 - Direct Summation
(This problem borrowed in its entirety from gforsyth's scipy17 numba tutorial.)
Many physical problems require the evaluation of all pairwise interactions of a large number of particles, so-called N-body problems. These problems arise in molecular dynamics, astrodynamics and electromagnetics among others.
Their pairwise interactions can be expressed as:
\begin{equation}
f_i = \sum_{j=1}^n{P \left(\boldsymbol{x}_i, \boldsymbol{x}_j \right)w_j} \ \ \ \text{for } i=1,2,...,n
\end{equation}

where subscripts $i$,  $j$ respectively denote target and source
$f_i$ can be a potential (or force) at target point $i$
$w_j$ is the source weight
$\boldsymbol{x}_i, \boldsymbol{x}_j$ are the spatial positions of particles
$P \left(\boldsymbol{x}_i, \boldsymbol{x}_j \right)$ is the interaction kernel.

In order to evalute the potential $f_i$ at a target point $i$, we have to loop over each source particle $j$. Since there are $n$ target points $i$, this 'brute-force' approach costs $\mathcal{O} \left(n^2 \right)$ operations.
One possible approach in this kind of problem is to define a few classes, say Point and Particle and then loop over the objects and perform the necessary point-to-point calculations.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
This tells us the array contains integers, and each integer has been assigned 64bits in memory (or equivalently, 8 bytes). Most python functions are defined to work on arbitrary types, so that if you use the + operator, for example, you can add integers, floats, complex numbers, or even strings! However, this flexibility comes at a cost, performance-wise. Numba, on the other hand, compiles each function based on the types of its arguments, and infers the type of the result. You can see this if you run the inspect_types function on a numba function:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
Want to see what you made? Run the following code to plot the image. Feel free to pick your favorite matplotlib color map :)
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
We don't have a distance method anymore, so we need to write a function to take care of that.
5c) Write a jit function distance to calculate the distance between two particles of dtype particle_dtype.
Can you write Python code for it?

@njit
def distance(part1, part2):
    '''calculate the distance between two particles'''
    return ((part1.x - part2.x)**2 +
            (part1.y - part2.y)**2 +
            (part1.z - part2.z)**2)**.5

distance(particles[0], particles[1])

%%timeit
distance(particles[0], particles[1])


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
What is Numba?
Numba is a just-in-time, type-specializing, function compiler for accelerating numerically-focused Python. That's a long list, so let's break down those terms:

function compiler: Numba compiles Python functions, not entire applications, and not parts of functions. Numba does not replace your Python interpreter, but is just another Python module that can turn a function into a (usually) faster function.
type-specializing: Numba speeds up your function by generating a specialized implementation for the specific data types you are using. Python functions are designed to operate on generic data types, which makes them very flexible, but also very slow. In practice, you only will call a function with a small number of argument types, so Numba will generate a fast implementation for each set of types.
just-in-time: Numba translates functions when they are first called. This ensures the compiler knows what argument types you will be using. This also allows Numba to be used interactively in a Jupyter notebook just as easily as a traditional application.
numerically-focused: Currently, Numba is focused on numerical data types, like int, float, and complex. There is very limited string processing support, and many string use cases are not going to work well on the GPU. To get best results with Numba, you will likely be using NumPy arrays.

Problem 1 - A First Numba Function
1a) To start our exploration of Numba's features, let's write a python function to add two numbers. We'll creatively name it add:
Can you write Python code for it?

def add(x, y):
  return x + y # add code here


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
What's going on here? %timeit is running our function many times, and then reporting the average time it takes to run. This is generally a better approach than timing a single function execution, because it accounts for random events that may cause any given run to perform poorly.
1d) Compare the time it takes to run your function with scalar vs array arguments, then your function vs python's add function (the standard ''+'' operator).
Can you write Python code for it?

%timeit add(1,2) # add code here
%timeit add(a, b) # add code here
%timeit a + b # add code here


I am working on the file "Sessions/Session08/Day4/numba_intro_solutions.ipynb".
The first task is:
You can access an individual "attribute" like this:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
2c) Remake a new version of the addition ufunc with 32bit floats that targets the cpu. Compare the resulting time to execute with the gpu version you just timed.
Can you write Python code for it?

# add code here
def add_ufunc_cpu(x, y):
    return x + y
%timeit add_ufunc_cpu(x, y)


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
2d) Now go back and try the two functions (gpu and cpu) with even larger arrays. When does the GPU start to win? Does the execution time on the GPU scale with the number of array elements the same way that the CPU version does?
If your result is like mine, you may have seen a slight timing advantage in the GPU version with a million array elements, but it was close (and that wasn't even counting the data transfer time). That's because we're still not giving the GPU enough work to keep all those cores busy all the time! By the time we hit 10 million, the GPU was clearly winning. The time it took the CPU function continued to increase linearly with array size, but the GPU function time increased much more slowly.
2e) Let's practice some more memory management. Given the following ufuncs:
Can you write Python code for it?

@vectorize(['float32(float32, float32)'], target='cuda')
def add_ufunc(x, y):
    return x + y

@vectorize(['float32(float32, float32, float32)'], target='cuda')
def make_pulses(i, period, amplitude):
    return max(math.sin(i / period) - 0.3, 0.0) * amplitude


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
3b) Let's practice writing a function that requires an atomic operation - a histogramming kernel. This will take an array of input data, a range and a number of bins, and count how many of the input data elements land in each bin. Below is an example CPU implementation of histogramming:
Can you write Python code for it?

def cpu_histogram(x, xmin, xmax, histogram_out):
    '''Increment bin counts in histogram_out, given histogram range [xmin, xmax).'''
    # Note that we don't have to pass in nbins explicitly, because the size of histogram_out determines it
    nbins = histogram_out.shape[0]
    bin_width = (xmax - xmin) / nbins

    # This is a very slow way to do this with NumPy, but looks similar to what you will do on the GPU
    for element in x:
        bin_number = np.int32((element - xmin)/bin_width)
        if bin_number >= 0 and bin_number < histogram_out.shape[0]:
            # only increment if in range
            histogram_out[bin_number] += 1

x = np.random.normal(size=10000, loc=0, scale=1).astype(np.float32)
xmin = np.float32(-4.0)
xmax = np.float32(4.0)
histogram_out = np.zeros(shape=10, dtype=np.int32)

cpu_histogram(x, xmin, xmax, histogram_out)

histogram_out


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
Convert the following code to use device allocations so that there are only host<->device copies at the beginning and end. Then benchmark the performance change.
Hint #1: You may want to keep the following code in place for reference, and create the version with GPU arrays in the additional cells below.
Hint #2: Determine how many arrays you will need on the device.
Can you write Python code for it?

n = 100000
noise = (np.random.normal(size=n) * 3).astype(np.float32)
t = np.arange(n, dtype=np.float32)
period = n / 23

pulses = make_pulses(t, period, 100.0)
waveform = add_ufunc(pulses, noise)

plt.plot(waveform)

# add code here

# add code here


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
Cool, it worked! But what actually just happened? Well, a lot of things. Numba automatically:
+ Compiled a CUDA kernel to execute the ufunc operation in parallel over all the input elements.
+ Allocated GPU memory for the inputs and the output.
+ Copied the input data to the GPU.
+ Executed the CUDA kernel with the correct kernel dimensions given the input sizes.
+ Copied the result back from the GPU to the CPU.
+ Returned the result as a NumPy array on the host.
1a) Determine how fast the CUDA addition function is. Compare that to a function compiled for the CPU. How does the GPU do?
You'll probably want to write two functions with separate names to compare them.
Can you write Python code for it?

# add code here

# add code here
def add_ufunc_cpu(x, y):
    return x + y

# add code here


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
In order to turn this into a GPU implementation, we'd like to have a kernel function (create_fractal) call another function (julia) on the device. Numba has a way of specifying functions that will be called from within a kernel by passing the cuda.jit decorator an argument:
Can you write Python code for it?

@cuda.jit(device=True)
def julia(x, y, max_iters):
    """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Julia
    set given a fixed number of iterations.
    """
    i = 0
    c = complex(-0.8, 0.156)
    a = complex(x,y)
    for i in range(max_iters):
        a = a*a + c
        if (a.real*a.real + a.imag*a.imag) > 1000:
            return 0
    return 255


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
Multi-dimensional grids
For some problems, it makes sense to define a two- or three-dimensional grid of thread blocks. That way, when you're indexing a single thread, you can map it to, say, the pixel position in an image. Multi-dimensional grids are created by passing tuples to the kernel function. You can ensure that you launch a big enough grid by calculating the size of each dimension as a function of the array size and number of threads per block:
Can you write Python code for it?

threadsperblock = 16
xblocks = (image.shape[1] + (threadsperblock - 1)) // threadsperblock
yblocks = (image.shape[0] + (threadsperblock - 1)) // threadsperblock


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
Problem 0 - Accessing the GPU
0a) In order to run Numba functions using the GPU, we have to do a couple of things. First, go to the Runtime menu, click on 'Change Runtime Type', and in the pop-up box, under 'Hardware Accelerator', select 'GPU'. Save the Runtime.
0b) Ideally, that's all we should have to do. But in practice, even though the CUDA libararies are installed, for some reason Colab usually can't find them. So, we'll figure out where they are, and then point Colab to them.
Can you write Python code for it?

!find / -iname 'libdevice'
!find / -iname 'libnvvm.so'


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
Problem 3 - Writing Cuda Kernels
While targeting ufuncs with the cuda syntax is the most straightforward way to access the GPU with Numba, it may not be flexible enough for your needs. If you want to write a more detailed GPU program, at some point you are probably going to need to write CUDA kernels.
As discussed in the lecture, the CUDA programming model allows you to abstract the GPU hardware into a software model composed of a grid containing blocks of threads. These threads are the smallest individual unit in the programming model, and they execute together in groups (traditionally called warps, consisting of 32 thread each). Determiming the best size for your grid of thread blocks is a complicated problem that often depends on the specific algorithm and hardware you're using, but here a few good rules of thumb:
+ the size of a block should be a multiple of 32 threads, with typical block sizes between 128 and 512 threads per block.
+ the size of the grid should ensure the full GPU is utilized where possible. Launching a grid where the number of blocks is 2x-4x the number of "multiprocessors" on the GPU is a good starting place. Something in the range of 20 - 100 blocks is usually a good starting point.
+ The CUDA kernel launch overhead does depend on the number of blocks, so it may not be best to launch a grid where the number of threads equals the number of input elements when the input size is very big. We'll show a pattern for dealing with large inputs below.
As a first example, let's return to our vector addition function, but this time, we'll target it with the cuda.jit decorator:
Can you write Python code for it?

@cuda.jit
def add_kernel(x, y, out):
    tidx = cuda.threadIdx.x # this is the unique thread ID within a 1D block
    bidx = cuda.blockIdx.x  # Similarly, this is the unique block ID within the 1D grid

    block_dimx = cuda.blockDim.x  # number of threads per block
    grid_dimx = cuda.gridDim.x    # number of blocks in the grid

    start = tidx + bidx * block_dimx
    stride = block_dimx * grid_dimx

    # assuming x and y inputs are same length
    for i in range(start, x.shape[0], stride):
        out[i] = x[i] + y[i]


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
The calling syntax is designed to mimic the way CUDA kernels are launched in C, where the number of blocks per grid and threads per block are specified in the square brackets, and the arguments to the function are specified afterwards in parentheses.
Note that, unlike the ufunc, the arguments are passed to the kernel as full NumPy arrays. A thread within the kernel can access any element in the array it wants, regardless of its position in the thread grid. This is why CUDA kernels are significantly more powerful than ufuncs. (But with great power, comes a greater amount of typing...)
Numba has created some helper functions to cut down on the typing. We can write the previous kernel much more simply as:
Can you write Python code for it?

@cuda.jit
def add_kernel(x, y, out):
    start = cuda.grid(1)      # the 1 argument means a one dimensional thread grid, this returns a single value
    stride = cuda.gridsize(1) # ditto

    # assuming x and y inputs are same length
    for i in range(start, x.shape[0], stride):
        out[i] = x[i] + y[i]


I am working on the file "Sessions/Session08/Day5/gpu_intro.ipynb".
The first task is:
x_device and y_device are now Numba "device arrays" that are in many ways equivalent to Numpy ndarrays except that they live in the GPU's global memory, rather than on the CPU. These device arrays can be passed to Numba cuda functions just the way Numpy arrays can, but without the memory copying overhead.
2b) Try out your function using host vs device arrays. How does the time compare?
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
2c) Remake a new version of the addition ufunc with 32bit floats that targets the cpu. Compare the resulting time to execute with the gpu version you just timed.
Can you write Python code for it?

@vectorize(['float32(float32, float32)'], target='cpu') # add code here
def add_ufunc_cpu(x, y):
    return x + y
%timeit add_ufunc_cpu(x, y)


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
2d) Now go back and try the two functions (gpu and cpu) with even larger arrays. When does the GPU start to win? Does the execution time on the GPU scale with the number of array elements the same way that the CPU version does?
If your result is like mine, you may have seen a slight timing advantage in the GPU version with a million array elements, but it was close (and that wasn't even counting the data transfer time). That's because we're still not giving the GPU enough work to keep all those cores busy all the time! By the time we hit 10 million, the GPU was clearly winning. The time it took the CPU function continued to increase linearly with array size, but the GPU function time increased much more slowly.
2e) Let's practice some more memory management. Given the following ufuncs:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
3b) Let's practice writing a function that requires an atomic operation - a histogramming kernel. This will take an array of input data, a range and a number of bins, and count how many of the input data elements land in each bin. Below is an example CPU implementation of histogramming:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
And then we can use a special out keyword argument to the ufunc to specify the output buffer:
Can you write Python code for it?

%timeit add_ufunc(x_device, y_device, out=out_device)


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
As before, using NumPy arrays forces Numba to allocate GPU memory, copy the arguments to the GPU, run the kernel, then copy the argument arrays back to the host. This not very efficient, so you will often want to allocate device arrays.
3a) Allocate device arrays for x, y, and the output, then try out your new Cuda kernel using the pre-copied device arrays. Compare the time to a version without moving the data first.
Can you write Python code for it?

# add code here
x_device = cuda.to_device(x)
y_device = cuda.to_device(y)
out_device = cuda.device_array_like(x)

# add code here
%timeit add_kernel[blocks_per_grid, threads_per_block](x, y, out)

%timeit add_kernel[blocks_per_grid, threads_per_block](x_device, y_device, out_device); out = out_device.copy_to_host()


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
As we saw in the last problem, copying the data to and from the GPU for every function is not necessarily the most efficient way to use the GPU. To address this, Numba provides the to_device function in the cuda module to allocate and copy arrays to the GPU:
Can you write Python code for it?

x_device = cuda.to_device(x)
y_device = cuda.to_device(y)

print(x_device)
print(x_device.shape)
print(x_device.dtype)


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Atomic Operations and avoiding Race Conditions
CUDA, like many general purpose parallel execution frameworks, makes it possible to have race conditions in your code. A race condition in CUDA arises when threads read or write a memory location that might be modified by another independent thread. Generally speaking, you need to worry about:
+ read-after-write hazards: One thread is reading a memory location at the same time another thread might be writing to it.
+ write-after-write hazards: Two threads are writing to the same memory location, and only one write will be visible when the kernel is complete.
A common strategy to avoid both of these hazards is to organize your CUDA kernel algorithm such that each thread has exclusive responsibility for unique subsets of output array elements, and/or to never use the same array for both input and output in a single kernel call. (Iterative algorithms can use a double-buffering strategy if needed, and switch input and output arrays on each iteration.)
However, there are many cases where different threads need to combine results. Consider something very simple, like: "every thread increments a global counter." Implementing this in your kernel requires each thread to:
1. Read the current value of a global counter.
2. Compute counter + 1.
3. Write that value back to global memory.
However, there is no guarantee that another thread has not changed the global counter between steps 1 and 3. To resolve this problem, CUDA provides "atomic operations" which will read, modify and update a memory location in one, indivisible step. Numba supports several of these functions, described here.
As an example, let's make a thread counter kernel:
Can you write Python code for it?

@cuda.jit
def thread_counter_race_condition(global_counter):
    global_counter[0] += 1  # This is bad

@cuda.jit
def thread_counter_safe(global_counter):
    cuda.atomic.add(global_counter, 0, 1)  # Safely add 1 to offset 0 in global_counter array

# This gets the wrong answer
global_counter = cuda.to_device(np.array([0], dtype=np.int32))
thread_counter_race_condition[64, 64](global_counter)

print('Should be %d:' % (64*64), global_counter.copy_to_host())

# This works correctly
global_counter = cuda.to_device(np.array([0], dtype=np.int32))
thread_counter_safe[64, 64](global_counter)

print('Should be %d:' % (64*64), global_counter.copy_to_host())


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Convert the following code to use device allocations so that there are only host<->device copies at the beginning and end. Then benchmark the performance change.
Hint: how many arrays will you need on the device?
Can you write Python code for it?

n = 100000
noise = (np.random.normal(size=n) * 3).astype(np.float32)
t = np.arange(n, dtype=np.float32)
period = n / 23

pulses = make_pulses(t, period, 100.0)
waveform = add_ufunc(pulses, noise)

plt.plot(waveform)

t_device = cuda.to_device(t)
noise_device = cuda.to_device(noise)
pulses_device = cuda.device_array(shape=(n,), dtype=np.float32)
waveform_device = cuda.device_array(shape=(n,), dtype=np.float32)
%timeit make_pulses(t_device, period, 100.0, out=pulses_device)
%timeit add_ufunc(pulses_device, noise_device, out=waveform_device)
waveform = waveform_device.copy_to_host()

plt.plot(waveform)


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Cool, it worked! But what actually just happened? Well, a lot of things. Numba automatically:
+ Compiled a CUDA kernel to execute the ufunc operation in parallel over all the input elements.
+ Allocated GPU memory for the inputs and the output.
+ Copied the input data to the GPU.
+ Executed the CUDA kernel with the correct kernel dimensions given the input sizes.
+ Copied the result back from the GPU to the CPU.
+ Returned the result as a NumPy array on the host.
1a) Determine how fast the CUDA addition function is. Compare that to a function compiled for the CPU. How does the GPU do?
You'll probably want to write two functions with separate names to compare them.
Can you write Python code for it?

%timeit add_ufunc(x, y) # add code here

@vectorize(['int64(int64, int64)'], target='cpu') # add code here
def add_ufunc_cpu(x, y):
    return x + y
%timeit add_ufunc_cpu(x, y)


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
In the space below, create a cuda version of this kernel, then run it to check that you get the same answer as the CPU version.
You can use much of the same syntax that we used in the cuda addition kernel.
Can you write Python code for it?

@cuda.jit
def cuda_histogram(x, xmin, xmax, histogram_out):
    '''Increment bin counts in histogram_out, given histogram range [xmin, xmax).'''

    # add code here

    start = cuda.grid(1)
    stride = cuda.gridsize(1)

    nbins = histogram_out.shape[0]
    bin_width = (xmax - xmin) / nbins

    for i in range(start, x.shape[0], stride):
      bin_number = np.int32((x[i] - xmin)/bin_width)
      if bin_number >= 0 and bin_number < histogram_out.shape[0]:
        cuda.atomic.add(histogram_out, bin_number, 1)

threads_per_block = 128
blocks_per_grid = 30
histogram_out = np.zeros(shape=10, dtype=np.int32)
cuda_histogram[blocks_per_grid, threads_per_block](x, xmin, xmax, histogram_out)

histogram_out


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Paste the location of the libraries into the following code box (if it's different, otherwise you can just run the code):
Can you write Python code for it?

import os
os.environ['NUMBAPRO_LIBDEVICE'] = "/usr/local/cuda-10.0/nvvm/libdevice"
os.environ['NUMBAPRO_NVVM'] = "/usr/local/cuda-10.0/nvvm/lib64/libnvvm.so"


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Problem 0 - Accessing the GPU
0a) In order to run Numba functions using the GPU, we have to do a couple of things. First, go to the Runtime menu, click on 'Change Runtime Type', and in the pop-up box, under 'Hardware Accelerator', select 'GPU'. Save the Runtime.
0b) Ideally, that's all we should have to do. But in practice, even though the CUDA libararies are installed, for some reason Colab usually can't find them. So, we'll figure out where they are, and then point Colab to them.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Problem 3 - Writing Cuda Kernels
While targeting ufuncs with the cuda syntax is the most straightforward way to access the GPU with Numba, it may not be flexible enough for your needs. If you want to write a more detailed GPU program, at some point you are probably going to need to write CUDA kernels.
As discussed in the lecture, the CUDA programming model allows you to abstract the GPU hardware into a software model composed of a grid containing blocks of threads. These threads are the smallest individual unit in the programming model, and they execute together in groups (traditionally called warps, consisting of 32 thread each). Determiming the best size for your grid of thread blocks is a complicated problem that often depends on the specific algorithm and hardware you're using, but here a few good rules of thumb:
+ the size of a block should be a multiple of 32 threads, with typical block sizes between 128 and 512 threads per block.
+ the size of the grid should ensure the full GPU is utilized where possible. Launching a grid where the number of blocks is 2x-4x the number of "multiprocessors" on the GPU is a good starting place. Something in the range of 20 - 100 blocks is usually a good starting point.
+ The CUDA kernel launch overhead does depend on the number of blocks, so it may not be best to launch a grid where the number of threads equals the number of input elements when the input size is very big. We'll show a pattern for dealing with large inputs below.
As a first example, let's return to our vector addition function, but this time, we'll target it with the cuda.jit decorator:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
Problem 4 - Return to the Fractals!
Yesterday we defined two functions to create an instance of the Julia set:
Can you write Python code for it?

def julia(x, y, max_iters):
    """
    Given the real and imaginary parts of a complex number,
    determine if it is a candidate for membership in the Julia
    set given a fixed number of iterations.
    """
    i = 0
    c = complex(-0.8, 0.156)
    a = complex(x,y)
    for i in range(max_iters):
        a = a*a + c
        if (a.real*a.real + a.imag*a.imag) > 1000:
            return 0
    return 255

def create_fractal(min_x, max_x, min_y, max_y, image, iters):
    height = image.shape[0]
    width = image.shape[1]

    pixel_size_x = (max_x - min_x) / width
    pixel_size_y = (max_y - min_y) / height
    for x in range(width):
        real = min_x + x * pixel_size_x
        for y in range(height):
            imag = min_y + y * pixel_size_y
            color = julia(real, imag, iters)
            image[y, x] = color

    return image

image = np.zeros((500, 750), dtype=np.uint8)
create_fractal(-2.0, 2.0, -1.0, 1.0, image, 200)
plt.imshow(image)
plt.viridis()
plt.show()


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
That's a lot more typing than our ufunc example, and it is much more limited: it only works on 1D arrays, it doesn't verify input sizes match, etc. Most of the function is spent figuring out how to turn the block and grid indices and dimensions into unique offsets in the input arrays. The pattern of computing a starting index and a stride is a common way to ensure that your grid size is independent of the input size. The striding will maximize bandwidth by ensuring that threads with consecuitive indices are accessing consecutive memory locations as much as possible. Thread indices beyond the length of the input (x.shape[0], since x is a NumPy array) automatically skip over the for loop.
Let's call the function now on some data:
Can you write Python code for it?

n = 100000
x = np.arange(n).astype(np.float32)
y = 2 * x
out = np.empty_like(x)

threads_per_block = 128
blocks_per_grid = 30

add_kernel[blocks_per_grid, threads_per_block](x, y, out)
print(out[:10])


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
The calling syntax is designed to mimic the way CUDA kernels are launched in C, where the number of blocks per grid and threads per block are specified in the square brackets, and the arguments to the function are specified afterwards in parentheses.
Note that, unlike the ufunc, the arguments are passed to the kernel as full NumPy arrays. A thread within the kernel can access any element in the array it wants, regardless of its position in the thread grid. This is why CUDA kernels are significantly more powerful than ufuncs. (But with great power, comes a greater amount of typing...)
Numba has created some helper functions to cut down on the typing. We can write the previous kernel much more simply as:
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
You should see a big performance improvement already, but we are still allocating a device array for the output of the ufunc and copying it back to the host. We can create an output buffer on the GPU with the numba.cuda.device_array() function:
Can you write Python code for it?

out_device = cuda.device_array(shape=(n,), dtype=np.float32)  # does not initialize the contents, much like np.empty()


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
You should see an even bigger improvement. Once we've finished all of our calculations on the GPU, we can copy the array back from the device using the copy_to_host method:
Can you write Python code for it?

out_host = out_device.copy_to_host()
print(out_host[:10])


I am working on the file "Sessions/Session08/Day5/gpu_intro_solutions.ipynb".
The first task is:
x_device and y_device are now Numba "device arrays" that are in many ways equivalent to Numpy ndarrays except that they live in the GPU's global memory, rather than on the CPU. These device arrays can be passed to Numba cuda functions just the way Numpy arrays can, but without the memory copying overhead.
2b) Try out your function using host vs device arrays. How does the time compare?
Can you write Python code for it?

%timeit add_ufunc(x, y) # add code here

%timeit add_ufunc(x_device, y_device) # add code here


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Note a couple changes from the previous helper function –– we have added a grid to the plot (this will be useful for visualizing the entropy), and we have also normalized the brightness measurements from 0 to 1.
Problem 2) The Shannon entropy
As noted above, to calculate the Shannon entropy we need to sum the data over partitions in the normalized $(\phi, m)$ plane.
This is straightforward using histogram2d from numpy.
Problem 2a
Write a function shannon_entropy to calculate the Shannon entropy, $H_0$, for a timeseries, $m(t_i)$, at a given period, p.
Hint - use histogram2d and a 10 x 10 grid (as plotted above).
Can you write Python code for it?

def shannon_entropy(m, t, p):
    '''Calculate the Shannon entropy

    Parameters
    ----------
    m : array-like
        brightness measurements of the time-series data

    t : array-like (default=1)
        timestamps corresponding to the brightness measurements

    p : float
        period of the periodic signal

    Returns
    -------
    H0 : float
        Shannon entropy for m(t) at period p
    '''

    m_norm = # complete
    phases = # complete
    H, _, _ = np.histogram2d( # complete

    occupied = np.where(H > 0)
    H0 = # complete

    return H0


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 1c
Generate a signal with $A = 2$, $p = \pi$, and Gaussian noise with variance = 0.01 over a regular grid between 0 and 10. Plot the phase-folded results (and make sure the results behave as you would expect).
Hint - your simulated signal should have at least 100 data points.
Can you write Python code for it?

x = np.linspace( # complete
y = # complete

# complete plot


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 2d
Plot the Shannon entropy periodogram, and return the best-fit period from the periodogram.
Hint - recall what we learned about frequency grids earlier today.
Can you write Python code for it?

f_grid = # complete
se_p = # complete

fig,ax = plt.subplots()
# complete
# complete
# complete

print("The best fit period is: {:.4f}".format( # complete


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 3) The Conditional Entropy
The CE is very similar to the Shannon entropy, though we need to condition the calculation on the occupation probability of the partitions in phase.
Problem 3a
Write a function conditional_entropy to calculate the CE, $H_c$, for a timeseries, $m(t_i)$, at a given period, p.
Hint - if you use histogram2d be sure to sum along the correct axes
Hint 2 - recall from session 8 that we want to avoid for loops, try to vectorize your calculation.
Can you write Python code for it?

def conditional_entropy(m, t, p):
    '''Calculate the conditional entropy

    Parameters
    ----------
    m : array-like
        brightness measurements of the time-series data

    t : array-like
        timestamps corresponding to the brightness measurements

    p : float
        period of the periodic signal

    Returns
    -------
    Hc : float
        Conditional entropy for m(t) at period p
    '''

    m_norm = # complete
    phases = # complete
    # complete
    # complete
    # complete
    Hc = # complete

    return Hc


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 3b
What is the conditional entropy for the simulated signal at periods = 1, $\pi$-0.05, and $\pi$?
Do these results make sense given your understanding of CE?
Can you write Python code for it?

print('For p = 1, \t\tH_c = {:.5f}'.format( # complete
print('For p = pi - 0.05, \tH_c = {:.5f}'.format( # complete
print('For p = pi, \t\tH_c = {:.5f}'.format( # complete


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 3c
Write a function, ce_periodogram, to calculate the conditional entropy for observations $m$, $t$ over a frequency grid f_grid.
Can you write Python code for it?

def ce_periodogram(m, t, f_grid):
    '''Calculate the conditional entropy at every freq in f_grid

    Parameters
    ----------
    m : array-like
        brightness measurements of the time-series data

    t : array-like
        timestamps corresponding to the brightness measurements

    f_grid : array-like
        trial periods for the periodic signal

    Returns
    -------
    ce_p : array-like
        conditional entropy for m(t) at every trial freq
    '''

    # complete
    for # complete in # complete
        # complete

    return ce_p


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 4d
Plot the light curve phase-folded on the best-fit period, as measured by the Shannon entropy periodogram.
Does this look reasonable? Why or why not?
Hint - it may be helpful to zoom in on the periodogram.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 4e
Using the conditional entropy, determine the best period for this light curve.
Can you write Python code for it?

ce_p = # complete

print("The best fit period is: {:.9f}".format( # complete


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 4f
Plot the CE periodogram.
Can you write Python code for it?

fig, ax = plt.subplots()
# complete
# complete
# complete


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
Problem 4g
Plot the light curve phase-folded on the best-fit period, as measured by the CE periodogram.
Does this look reasonable? If not - can you make it look better?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
The Shannon entropy and CE return nearly identical results for a simulated sinusoidal signal. Now we will examine how each performs with actual astronomical observations.
Problem 4) SE vs. CE for real observations
Problem 4a
Load the data from our favorite eclipsing binary from this morning's LS exercise. Plot the light curve.
Hint - if you haven't already, download the example light curve.
Can you write Python code for it?

data = pd.read_csv("example_asas_lc.dat")

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel('HJD (d)')
ax.set_ylabel('V (mag)')
ax.set_ylim(ax.get_ylim()[::-1])
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ConditionalEntropy.ipynb".
The first task is:
We know the correct period of the simulated data is $\pi$, so it makes sense that this period minimizes the Shannon entropy.
Problem 2c
Write a function, se_periodogram to calculate the Shannon entropy for observations $m$, $t$ over a frequency grid f_grid.
Can you write Python code for it?

def se_periodogram(m, t, f_grid):
    '''Calculate the Shannon entropy at every freq in f_grid

    Parameters
    ----------
    m : array-like
        brightness measurements of the time-series data

    t : array-like
        timestamps corresponding to the brightness measurements

    f_grid : array-like
        trial periods for the periodic signal

    Returns
    -------
    se_p : array-like
        Shannon entropy for m(t) at every trial freq
    '''

    # complete
    for # complete in # complete
        # complete

    return se_p


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
An important, but necessary, aside ––
the slightly unusual scheduling for this week means that the lectures will not unfold in the most natural order. An in depth examination of Fourier methods will occur on Wednesday and Thursday, and ideally this would happen before any discussion of the Lomb-Scargle periodogram.
However, Cohort 2 has already seen Lomb-Scargle and Gaussian Processes, so we will start with that and then cover Fourier methods. I'll proceed with a brief review of Fourier analysis, if you haven't seen this before it may be worth revisiting this notebook later in the week.
(Either way - the main takeaways from Lomb-Scargle will be clear by the end of this notebook)
Problem 2) A Brief Review of Fourier Analysis
In astronomical time series, we crave the detection of periodic signals because they can often provide fundamental insight into the sources we are studying (e.g., masses in a binary, classifications for pulsating stars, etc).
The standard$^\dagger$ choice for most astronomers to identify such signals is the Lomb-Scargle (LS) periodogram (Lomb 1976; Scargle 1982).
$^\dagger$Standard does not mean best, fastest, or even correct depending on your specific application.
At the heart of understanding any periodic signals is Fourier analysis. Thus, to understand how to interpret the LS periodogram, we first need to consider Fourier transforms.
Note - here we aim to jog your memory regarding Fourier analysis. Detailed mathematical calculations can be found elsewhere.
Given a continuous signal, $g(t)$ the Fourier transform of that signal is defined as:
$$\hat{\mathrm{g}}(f) = \int_{-\infty}^{\infty} g(t) \,e^{-2\pi i f t} \,dt,$$
where $i$ is an imaginary number. The inverse of this equation is defined as:
$$ g(t) = \int_{-\infty}^{\infty} \hat{\mathrm{g}}(f) \,e^{2\pi i f t} \,df.$$
For convenience, we will use the Fourier transform operator $\mathcal{F}$, from which the above equations reduce to:
$$\mathcal{F}(g) = \hat g$$
$$\mathcal{F}^{-1}(\hat{g}) = g$$
There are many useful properties of...
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Herein lies a fundamental issue regarding the LS periodogram: the model does not search for "periodicity." The LS model asks if the data support a sinusoidal signal. As astronomers we typically assume this question is good enough, but as we can see in the example of this eclipsing binary that is not the case [and this is not limited to eclipsing binaries].
We can see why LS is not sufficient for an EB by comparing the model to the phase-folded light curve:
Can you write Python code for it?

fig, ax = plt.subplots()

ax.errorbar(data['hjd']/ls_period % 1, data['mag'], data['mag_unc'], fmt='o', zorder=-1)
ax.plot(np.linspace(0,1,1000),
        LombScargle(data['hjd'],data['mag'], data['mag_unc']).model(np.linspace(0,1,1000)*ls_period, 1/ls_period) )
ax.set_xlabel('Phase')
ax.set_ylabel('V (mag)')
ax.set_ylim(ax.get_ylim()[::-1])
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Interestingly, there are very strong peaks in the data at $P \approx 3\,\mathrm{d} \;\&\; 365\,\mathrm{d}$.
What is this telling us? Essentially that observations are likely to be repeated at intervals of 3 or 365 days (shorter period spikes are aliases of the 3 d peak).
This is important to understand, however, because this same power will be present in the periodogram where we search for the periodic signal.
Problem 4f
Calculate the periodogram for the data and compare it to the periodogram for the window function.
Can you write Python code for it?

ls = LombScargle( # complete

frequency, power = # complete

fig, (ax,ax2) = plt.subplots(2,1, sharex=True)
ax.plot( # complete
ax.set_ylabel("Power")
ax.set_ylim(0,1)

ax2.plot( # complete
ax2.set_ylabel("Power")
ax2.set_xlabel("Period (d)")
ax2.set_xlim(0,10)
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
One way to combat this very specific issue is to include more Fourier terms at the harmonic of the best fit period. This is easy to implement in LombScargle with the nterms keyword. [Though always be weary of adding degrees of freedom to a model, especially at the large pipeline level of analysis.]
Problem 5d
Calculate the LS periodogram for the eclipsing binary, with nterms = 1, 2, 3, 4, 5. Report the best-fit period for each of these models.
Hint - we have good reason to believe that the best fit frequency is < 3 in this case, so set maximum_frequency = 3.
Can you write Python code for it?

for i in np.arange(1,6):
    frequency, power = # complete
    # complete
    print('For {:d} harmonics, P_LS = {:.8f}'.format( # complete


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Periodic Signals & The Lomb-Scargle Periodogram
Version 0.1

By AA Miller (CIERA/Northwestern & Adler)
This notebook discusses the detection of periodic signals in noisy, irregular data (the standard for ground-based astronomical surveys). The discussion below is strongly influenced by Understanding the Lomb-Scarge Periodogram, by Jake VanderPlas, a former DSFP lecturer (VanderPlas 2017). Beyond that, the original papers by Lomb 1976 and Scargle 1982 are also worth a read.
There are many, many papers on the use of the Lomb-Scargle periodogram (and other period detection methods). Former DSFP lecturer, Matthew Graham and colleagues conducted a systematic analysis of many of the most popular tools used to search for periodic signals on actual astronomical data (Graham et al. 2013). [Somewhat to my (our?) dismay, they found that none of the solutions work really well across all use cases.]
Problem 1) Helper Functions
Throughout this lecture, there are a number of operations that we will be performing again and again. Thus, we'll create a few helper functions to minimize repetitive commands (e.g., plotting a phase folded light curve).
Problem 1a
Create a function, gen_periodic_data, that creates simulated data (including noise) over a grid of user supplied positions:
$$ y = A\,cos\left(\frac{2{\pi}x}{P} - \phi\right) + \sigma_y$$
where $A, P, \phi$ are inputs to the function. gen_periodic_data should include Gaussian noise, $\sigma_y$, for each output $y_i$.
Can you write Python code for it?

def gen_periodic_data(x, period=1, amplitude=1, phase=0, noise=0):
    '''Generate periodic data given the function inputs

    y = A*cos(x/p - phase) + noise

    Parameters
    ----------
    x : array-like
        input values to evaluate the array

    period : float (default=1)
        period of the periodic signal

    amplitude : float (default=1)
        amplitude of the periodic signal

    phase : float (default=0)
        phase offset of the periodic signal

    noise : float (default=0)
        variance of the noise term added to the periodic signal

    Returns
    -------
    y : array-like
        Periodic signal evaluated at all points x
    '''

    y = amplitude*np.sin(2*np.pi*x/(period) - phase) + np.random.normal(0, np.sqrt(noise), size=len(x))
    return y


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 1b
Generate a noise-free signal with $A = 2$ and $p = \pi$ over a regular grid between 0 and 10. Plot the results (and make sure gen_periodic_data behaves as you would expect).
Can you write Python code for it?

x = # complete
y = gen_periodic_data( # complete

fig, ax = plt.subplots()
ax.scatter( # complete
# complete
# complete
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 1d
Plot the phase folded data generated in 1b.
Does you plot match your expectations?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3b
Write a function to minimize the $\chi^2$ given everything but $A_f$ and $\phi_f$.
Hint - you may find minimize within the scipy package useful.
Can you write Python code for it?

from scipy.optimize import minimize

def min_chi2( # complete
    # complete
    # complete


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3c
Write a function, ls_periodogram, to calculate the LS periodogram for observations $y$, $\sigma_y$, $t$ over a frequency grid f_grid.
Can you write Python code for it?

def ls_periodogram( # complete
    psd = np.empty_like(f_grid)
    chi2_0 = # complete
    for f_num, f in enumerate(f_grid):
        psd[f_num] = # complete

    return psd


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3d
Generate a periodic signal with 100 observations taken over a time period of 10 days. Use an input period of 5.25, amplitude of 7.4, and variance of the noise = 0.8. Then compute and plot the periodogram for the simulated data. Do you recover the simulated period?
Hint - set the minimum frequency in the grid to $1/T$ where $T$ is the duration of the observations. Set the maximum frequnecy to 10, and use an equally spaced grid with 50 points.
Can you write Python code for it?

np.random.seed(23)
# calculate the periodogram

x = # complete
y = # complete
y_unc = # complete

f_grid = # complete

psd_ls = # complete

# plot the periodogram
fig, ax = plt.subplots()
ax.plot( # complete
# complete
# complete
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3e
For the same data, include 1000 points in f_grid and calculate and plot the periodogram.
Now do you recover the correct period?
Can you write Python code for it?

# calculate the periodogram
f_grid = # complete

psd_ls = # complete

# plot the periodogram
fig,ax = plt.subplots()

ax.plot(# complete
# complete
# complete
fig.tight_layout()

print("The best fit period is: {:.4f}".format( # complete


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3f
Plot the phase-folded data at the newly found "best" fit period.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 4) Other Considerations and Faster Implementations
While ls_periodogram functions well, it would take a long time to evaluate $\sim4\times 10^5$ frequencies for $\sim2\times 10^7$ variable LSST sources. Fortunately, there are significantly faster implementations, including (as you may have guessed) one in astropy.
Problem 4a
LombScargle in astropy.stats is fast. Run it below to compare to ls_periodogram.
Can you write Python code for it?

from astropy.stats import LombScargle
frequency, power = LombScargle(x, y, y_unc).autopower()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 5b
Use LombScargle to measure the periodogram. Then plot the periodogram and the phase folded light curve at the best-fit period.
Hint - search periods longer than 2 hr.
Can you write Python code for it?

frequency, power = # complete

# complete

fig,ax = plt.subplots()

ax.plot(# complete
ax.set_ylabel("Power")
ax.set_xlabel("Period (d)")
ax.set_xlim(0, 800)

axins = plt.axes([.25, .55, .6, .3])
axins.plot( # complete
axins.set_xlim(0,5)
fig.tight_layout()

# plot the phase folded light curve
phase_plot( # complete


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 5c
Now plot the light curve folded at twice the best LS period.
Which of these 2 is better?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Sampling a signal directly at the Nyquist frequency results in a lack of any variability. But does this just mean that $f_\mathrm{Ny}$ is special? What happens at $f > f_\mathrm{Ny}$?
Problem 2b
As above, generate and plot a periodic signal with $f = 0.7$ on an even grid from 0 to 10. Overplot the underlying signal in addition to the observations.
Can you write Python code for it?

x = # complete
y = # complete
x_signal = # complete
y_signal = # complete
fig, ax = plt.subplots()
ax.scatter( # complete
ax.plot( # complete
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
The observations are identical! Here is what you need to remember about the Nyquist frequency:

If you are going to obtain observations at regular intervals, and there is a specific signal you wish to detect, then be sure to sample the data such that $f_\mathrm{Ny} > f_\mathrm{signal}$.
For all $f > f_\mathrm{Ny}$, $f$ will be aliased with $f \pm 2n f_\mathrm{Ny}$ signals, where $n$ is an integer. Practically speaking, this means it does not make sense to search for signals with $f > f_\mathrm{Ny}$.
Finally, (and this is something that is often wrong in the literature) there is no Nyquist limit for unevenly sampled data (see VanderPlas 2017 for further details). Thus, for (virtually all) ground-based observing one need not worry about the Nyquist limit.

Staying on the topic of non-continuous observations, I present without derivation the discrete Fourier transform:
$$ \hat g_\mathrm{obs}(f) = \sum_{n = 0}^N g_n\,e^{-2\pi i f n\Delta t}$$
where $g_n = g(n\Delta t)$, and $\Delta t$ is the sampling interval. Our discussion of the Nyquist frequency tells us that we cannot detect frequencies $f > 1/2\Delta T$. Thus, the relevant frequencies to search for power given $\Delta t$ are between 0 and $f_\mathrm{Ny}$, which we can sample on a grid $\Delta f = 1/(N \Delta t)$. From there:
$$\hat g_k = \sum_{n = 0}^N g_n\,e^{-2\pi i f knN}$$
where $\hat g_k = \hat g_\mathrm{obs} (k\Delta f)$. This is the discrete Fourier transform.
I said a full derivation will not be provided, and that is true. To understand how we went from a continuous integral to the summation above, recall that regular observations of a continous signal over a finite interval is equivalent to multiplying the continuous signal by a Dirac comb function and a window function. The delta functions from the Dirac comb function collapse the integral to a sum, while the window function limits that sum from $0$ to $N$.
From the discrete Fourier transform we can then calculate the periodogram (an estimator of the power spectrum):
$$\mathcal{P}(f) =...
Can you write Python code for it?

def chi2( # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
This example also shows why it is somewhat strange to provide an uncertainty with a LS best-fit period. Errors tend to be catastrophic, and not some small fractional percentage, with the LS periodogram.
In the case of the above EB, the "best-fit" period was off by a factor 2. This is not isolated to EBs, however, LS periodograms frequently identify a correct harmonic of the true period, but not the actual period of variability.
Conclusions
The Lomb-Scargle periodogram is a useful tool to search for sinusoidal signals in noisy, irregular data.
However, as highlighted throughout, there are many ways in which the methodology can run awry.
In closing, I will summarize some practical considerations from VanderPlas (2017):

Choose an appropriate frequency grid (defaults in LombScargle are not sufficient)
Calculate the LS periodogram for the observation times to search for dominant signals (e.g., 1 day in astro)
Compute LS periodogram for data (avoid multi-Fourier models if signal unknown)
Plot periodogram and various FAP levels (do not over-interpret FAP)
If window function shows strong aliasing, plot phased light curve at each peak (now add more Fourier terms if necessary)
If looking for a particular signal (e.g., detatched EBs), consider different methods that better match expected signal
Inject fake signals into data to understand systematics if using LS in a survey pipeline

Finally, Finally
As a very last note: know that there are many different ways to search for periodicity in astronomical data. Depending on your application (and computational resources), LS may be a poor choice (even though this is often the default choice by all astronomers!) Graham et al. (2013) provides a summary of several methods using actual astronomical data. The results of that study show that no single method is best. However, they also show that no single method performs particularly well: the detection efficiences in Graham et al. (2013) are disappointing given the importance of periodicity in astronomical signals.
Period...
Can you write Python code for it?

def fourier_pairs_plot():

    fig, ax = plt.subplots(4, 2, figsize=(10, 6))
    fig.subplots_adjust(left=0.04, right=0.98, bottom=0.02, top=0.95,
                        hspace=0.3, wspace=0.2)

    x = np.linspace(-5, 5, 1000)

    for axi in ax.flat:
        axi.xaxis.set_major_formatter(plt.NullFormatter())
        axi.yaxis.set_major_formatter(plt.NullFormatter())

        # draw center line
        axi.axvline(0, linestyle='dotted', color='gray')
        axi.axhline(0, linestyle='dotted', color='gray')

    style_re = dict(linestyle='solid', color='k', linewidth=2)
    style_im = dict(linestyle='solid', color='gray', linewidth=2)
    text_style = dict(size=14, color='gray')

    # sine -> delta
    ax[0, 0].plot(x, np.cos(x),**style_re)
    ax[0, 0].set(xlim=(-5, 5), ylim=(-1.2, 1.2))
    ax[0, 0].annotate('', (-np.pi, 0), (np.pi, 0),
                      arrowprops=dict(arrowstyle='|-|', color='gray'))
    ax[0, 0].text(0, 0, '$1/f_0$', ha='center', va='bottom', **text_style)
    ax[0, 0].set_title('Sinusoid')

    ax[0, 1].plot([-5, 2, 2, 2, 5], [0, 0, 1, 0, 0], **style_re)
    ax[0, 1].plot([-5, -2, -2, -2, 5], [0, 0, 1, 0, 0], **style_re)
    ax[0, 1].set(xlim=(-5, 5), ylim=(-0.2, 1.2))
    ax[0, 1].annotate('', (0, 0.4), (2, 0.4), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[0, 1].annotate('', (0, 0.4), (-2, 0.4), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[0, 1].text(1, 0.45, '$+f_0$', ha='center', va='bottom', **text_style)
    ax[0, 1].text(-1, 0.45, '$-f_0$', ha='center', va='bottom', **text_style)
    ax[0, 1].set_title('Delta Functions')

    # gaussian -> gaussian
    ax[1, 0].plot(x, np.exp(-(2 * x) ** 2), **style_re)
    ax[1, 0].set(xlim=(-5, 5), ylim=(-0.2, 1.2))
    ax[1, 0].annotate('', (0, 0.35), (0.6, 0.35), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[1, 0].text(0, 0.4, '$\sigma$', ha='center', va='bottom', **text_style)
    ax[1, 0].set_title('Gaussian')

    ax[1, 1].plot(x, np.exp(-(x / 2) ** 2), **style_re)
    ax[1, 1].set(xlim=(-5, 5),...


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Uncertainty on the best-fit period
How do we report uncertainties on the best-fit period from LS? For example, for the previously simulated LSST light curve we would want to report something like $P = 102 \pm 4\,\mathrm{d}$. However, the uncertainty from LS periodograms cannot be determined in this way.
Naively, one could report the width of the peak in the periodogram as the uncertainty in the fit. However, we previously saw that the peak width $\propto 1/T$ (the peak width does not decrease as the number of observations or their S/N increases; see Vander Plas 2017). Reporting such an uncertainty is particularly ridiculous for long duration surveys, whereby the peaks become very very narrow.
An alternative approach is to report the False Alarm Probability (FAP), which estimates the probability that a dataset with no periodic signal could produce a peak of similar magnitude, due to random gaussian fluctuations, as the data.
There are a few different methods to calculate the FAP. Perhaps the most useful, however, is the bootstrap method. To obtain a bootstrap estimate of the LS FAP one leaves the observation times fixed, and then draws new observation values with replacement from the actual set of observations. This procedure is then repeated many times to determine the FAP.
One nice advantage of this procedure is that any effects due to the window function will be imprinted in each iteration of the bootstrap resampling.
The major disadvantage is that many many periodograms must be calculated. The rule of thumb is that to acieve a FAP $= p_\mathrm{false}$, one must run $n_\mathrm{boot} \approx 10/p_\mathrm{false}$ bootstrap periodogram calculations. Thus, an FAP $\approx 0.1\%$ requires an increase of 1000 in computational time.
LombScargle provides the false_alarm_probability method, including a bootstrap option. We skip that for now in the interest of time.
As a final note of caution - be weary of over-interpreting the FAP. The specific question answered by the FAP is, what is the probability that...
Can you write Python code for it?

data = # complete

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel('HJD (d)')
ax.set_ylabel('V (mag)')
ax.set_ylim(ax.get_ylim()[::-1])
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Unlike ls_periodogram, LombScargle effectively takes no time to run on the simulated data.
Problem 4b
Plot the periodogram for the simulated data.
Can you write Python code for it?

fig, ax = plt.subplots()

ax.plot( # complete
# complete
# complete
# complete
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
We can see that the best fit model doesn't match the signal in the case where we do not allow a floating mean.
Can you write Python code for it?

fit_mean_model = LombScargle(x[bright], y[bright], y_unc[bright],
                             fit_mean=True).model(np.linspace(0,10,1000),
                                                  freq_fit_mean[np.argmax(power_fit_mean)])
no_mean_model = LombScargle(x[bright], y[bright], y_unc[bright],
                             fit_mean=False).model(np.linspace(0,10,1000),
                                                  freq_no_mean[np.argmax(power_no_mean)])

fig, ax = plt.subplots()
ax.errorbar(x[bright], y[bright], y_unc[bright], fmt='o', label='data')
ax.plot(np.linspace(0,10,1000), fit_mean_model, label='fit mean')
ax.plot(np.linspace(0,10,1000), no_mean_model, label='no mean')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend()
fig.tight_layout()


I am working on the file "Sessions/Session09/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Window Functions
Recall that the convolution theorem tells us that:
$$\mathcal{F}[f\cdot g] = \mathcal{F}(f) \ast \mathcal{F}(g)$$
Telescope observations are effectively the product of a continous signal with several delta functions (corresponding to the times of observations). As a result, the convolution that produces the periodogram will retain signal from both the source and the observational cadence.
To illustrate this effect, let us simulate "realistic" observations for a 10 year telescope survey. We do this by assuming that a source is observed every 3 nights (the LSST cadence) within $\pm 4\,\mathrm{hr}$ of the same time, and that $\sim 30\%$ of the observations did not occur due to bad weather. We further assume that the source cannot be observed for 40% of the year because it is behind the sun.
Simulate a periodic signal with this cadence, a period = 220 days (typical for Miras), amplitude = 12.4, and noise = 1. Plot the simulated light curve.
Problem 4d
Simulate a periodic signal with 3 day cadence (and the observing conditions described above), a period = 220 days (typical for Miras), amplitude = 12.4, and variance of the noise = 1. Plot the simulated light curve.
Can you write Python code for it?

# set up simulated observations
t_obs = np.arange(0, 10*365, 3)      # 3d cadence
# complete
# complete
# complete

y = gen_periodic_data( # complete
# complete

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel("Time (d)")

ax.set_ylabel("Flux (arbitrary units)")


I am working on the file "Sessions/Session09/Day1/gps/02-Inference.ipynb".
The first task is:

Exercise 2

The following dataset was generated from a zero-mean Gaussian Process with a Squared Exponential Kernel of unity amplitude and unknown timescale. Compute the marginal log likelihood of the data over a range of reasonable values of $l$ and find the maximum. Plot the likelihood (not log likelihood) versus $l$; it should be pretty Gaussian. How well are you able to constrain the timescale of the GP?
Can you write Python code for it?

import matplotlib.pyplot as plt
t, y, sigma = np.loadtxt("data/sample_data.txt", unpack=True)
plt.plot(t, y, "k.", alpha=0.5, ms=3)
plt.xlabel("time")
plt.ylabel("data");


I am working on the file "Sessions/Session09/Day1/gps/02-Inference.ipynb".
The first task is:

Exercise 3a

The timeseries below was generated by a linear function of time, $y(t)= mt + b$. In addition to observational uncertainty $\sigma$ (white noise), there is a fair bit of correlated (red) noise, which we will assume is well described
by the squared exponential covariance with a certain (unknown) amplitude $A$ and timescale $l$.
Your task is to estimate the values of $m$ and $b$, the slope and intercept of the line, respectively. In this part of the exercise, assume there is no correlated noise. Your model for the $n^\mathrm{th}$ datapoint is thus
$$
\begin{align}
    y_n \sim \mathcal{N}(m t_n + b, \sigma_n\mathbf{I})
\end{align}
$$
and the probability of the data given the model can be computed by calling your GP likelihood function:
python
def lnprob(params):
    m, b = params
    model = m * t + b
    return ln_gp_likelihood(t, y - model, sigma, A=0, l=1)
Note, importantly, that we are passing the residual vector, $y - (mt + b)$, to the GP, since above we coded up a zero-mean Gaussian process. We are therefore using the GP to model the residuals of the data after applying our physical model (the equation of the line).
To estimate the values of $m$ and $b$ we could generate a fine grid in those two parameters and compute the likelihood at every point. But since we'll soon be fitting for four parameters (in the next part), we might as well upgrade our inference scheme and use the emcee package to do Markov Chain Monte Carlo (MCMC). If you haven't used emcee before, check out the first few tutorials on the documentation page. The basic setup for the problem is this:
```python
import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob)
initial = [4.0, 15.0]
p0 = initial + 1e-3 * np.random.randn(nwalkers, ndim)
print("Running burn-in...")
p0, _, _ = sampler.run_mcmc(p0, nburn)   # nburn = 500 should do
sampler.reset()
print("Running...
Can you write Python code for it?

t, y, sigma = np.loadtxt("data/sample_data_line.txt", unpack=True)
m_true, b_true, A_true, l_true = np.loadtxt("data/sample_data_line_truths.txt", unpack=True)
plt.errorbar(t, y, yerr=sigma, fmt="k.", label="observed")
plt.plot(t, m_true * t + b_true, color="C0", label="truth")
plt.legend(fontsize=12)
plt.xlabel("time")
plt.ylabel("data");


I am working on the file "Sessions/Session09/Day1/gps/02-Inference.ipynb".
The first task is:
Here are the functions we wrote in the previous tutorial to compute and draw from a GP:
Can you write Python code for it?

import numpy as np
from scipy.linalg import cho_factor

def ExpSquaredKernel(t1, t2=None, A=1.0, l=1.0):
    """
    Return the ``N x M`` exponential squared
    covariance matrix between time vectors `t1`
    and `t2`. The kernel has amplitude `A` and
    lengthscale `l`.

    """
    if t2 is None:
        t2 = t1
    T2, T1 = np.meshgrid(t2, t1)
    return A ** 2 * np.exp(-0.5 * (T1 - T2) ** 2 / l ** 2)

def draw_from_gaussian(mu, S, ndraws=1, eps=1e-12):
    """
    Generate samples from a multivariate gaussian
    specified by covariance ``S`` and mean ``mu``.

    (We derived these equations in Day 1, Notebook 01, Exercise 7.)
    """
    npts = S.shape[0]
    L, _ = cho_factor(S + eps * np.eye(npts), lower=True)
    L = np.tril(L)
    u = np.random.randn(npts, ndraws)
    x = np.dot(L, u) + mu[:, None]
    return x.T

def compute_gp(t_train, y_train, t_test, sigma=0, A=1.0, l=1.0):
    """
    Compute the mean vector and covariance matrix of a GP
    at times `t_test` given training points `y_train(t_train)`.
    The training points have uncertainty `sigma` and the
    kernel is assumed to be an Exponential Squared Kernel
    with amplitude `A` and lengthscale `l`.

    """
    # Compute the required matrices
    kernel = ExpSquaredKernel
    Stt = kernel(t_train, A=1.0, l=1.0)
    Stt += sigma ** 2 * np.eye(Stt.shape[0])
    Spp = kernel(t_test, A=1.0, l=1.0)
    Spt = kernel(t_test, t_train, A=1.0, l=1.0)

    # Compute the mean and covariance of the GP
    mu = np.dot(Spt, np.linalg.solve(Stt, y_train))
    S = Spp - np.dot(Spt, np.linalg.solve(Stt, Spt.T))

    return mu, S


I am working on the file "Sessions/Session09/Day1/gps/02-Inference.ipynb".
The first task is:
The Marginal Likelihood
In the previous notebook, we learned how to construct and sample from a simple GP. This is useful for making predictions, i.e., interpolating or extrapolating based on the data you measured. But the true power of GPs comes from their application to regression and inference: given a dataset $D$ and a model $M(\theta)$, what are the values of the model parameters $\theta$ that are consistent with $D$? The parameters $\theta$ can be the hyperparameters of the GP (the amplitude and time scale), the parameters of some parametric model, or all of the above.
A very common use of GPs is to model things you don't have an explicit physical model for, so quite often they are used to model "nuisances" in the dataset. But just because you don't care about these nuisances doesn't mean they don't affect your inference: in fact, unmodelled correlated noise can often lead to strong biases in the parameter values you infer. In this notebook, we'll learn how to compute likelihoods of Gaussian Processes so that we can marginalize over the nuisance parameters (given suitable priors) and obtain unbiased estimates for the physical parameters we care about.
Given a set of measurements $y$ distributed according to
$$
\begin{align}
    y \sim \mathcal{N}(\mathbf{\mu}(\theta), \mathbf{\Sigma}(\alpha))
\end{align}
$$
where $\theta$ are the parameters of the mean model $\mu$ and $\alpha$ are the hyperparameters of the covariance model $\mathbf{\Sigma}$, the marginal likelihood of $y$ is
$$
\begin{align}
    \ln P(y | \theta, \alpha) = -\frac{1}{2}(y-\mu)^\top \mathbf{\Sigma}^{-1} (y-\mu) - \frac{1}{2}\ln |\mathbf{\Sigma}| - \frac{N}{2} \ln 2\pi
\end{align}
$$
where $||$ denotes the determinant and $N$ is the number of measurements. The term marginal refers to the fact that this expression implicitly integrates over all possible values of the Gaussian Process; this is not the likelihood of the data given one particular draw from the GP, but given the ensemble of all possible draws from $\mathbf{\Sigma}$.
<div...
Can you write Python code for it?

def ln_gp_likelihood(t, y, sigma=0, A=1.0, l=1.0):
    """

    """
    # do stuff in here
    pass


I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1b
Plot an example RR Lyrae light curve.
Can you write Python code for it?

# RRL example
t, m, m_unc = read_lc("./training_lcs/011815-3912.8")
lc_plot(t, m, m_unc, period=0.510918)


I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1c
Plot an example detatched eclipsing binary (EB) light curve.
Can you write Python code for it?

# dEB example
t, m, m_unc = read_lc("./training_lcs/153835-6727.8")
lc_plot(t, m, m_unc, period=2*1.107174)


I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1f
Plot an example Cepheid light curve.
Can you write Python code for it?

# Cepheid example
t, m, m_unc = read_lc("./training_lcs/065640+0011.4")
lc_plot(t, m, m_unc, period=4.022837)


I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1g
Plot an example R Cor Bor star light curve.
Can you write Python code for it?

# R Cor Bor example
t, m, m_unc = read_lc("./training_lcs/163242-5315.6")
lc_plot(t, m, m_unc, period=0.0)