I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Exercise 7
Using your previous functions, check that all perfect numbers $n < 10,000$ can be written as $2^{k-1} \times (2^k - 1)$, where $2^k-1$ is a Mersenne prime.
Solution
In fact we did this above already:

$6 = 2^{2-1} \times (2^2 - 1)$. 2 is the first number on our Mersenne list.
$28 = 2^{3-1} \times (2^3 - 1)$. 3 is the second number on our Mersenne list.
$496 = 2^{5-1} \times (2^5 - 1)$. 5 is the third number on our Mersenne list.
$8128 = 2^{7-1} \times (2^7 - 1)$. 7 is the fourth number on our Mersenne list.

Exercise 8 (bonus)
Investigate the timeit function in python or IPython. Use this to measure how long your function takes to check that, if $k$ on the Mersenne list then $n = 2^{k-1} \times (2^k - 1)$ is a perfect number, using your functions. Stop increasing $k$ when the time takes too long!
Note
You could waste considerable time on this, and on optimizing the functions above to work efficiently. It is not worth it, other than to show how rapidly the computation time can grow!
Solution
Can you write Python code for it?


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Exercise 8
The Wallis formula for $\pi$ is
\begin{equation}
  \pi = 2 \prod_{n=1}^{\infty} \frac{ (2 n)^2 }{(2 n - 1) (2 n + 1)}.
\end{equation}
We can define a partial product $\pi_N$ as
\begin{equation}
  \pi_N = 2 \prod_{n=1}^{N} \frac{ (2 n)^2 }{(2 n - 1) (2 n + 1)},
\end{equation}
each of which are rational numbers.
Construct a list of the first 20 rational number approximations to $\pi$ and print them out. Print the sorted list to show that the approximations are always increasing. Then convert them to floating point numbers, construct a numpy array, and subtract this array from $\pi$ to see how accurate they are.
Solution
Can you write Python code for it?

def wallis_rational(N):
    """
    The partial product approximation to pi using the first N terms of Wallis' formula.

    Parameters
    ----------

    N : int
        Number of terms in product

    Returns
    -------

    partial : Rational
        A rational number approximation to pi
    """

    partial = Rational(2,1)
    for n in range(1, N+1):
        partial = partial * Rational((2*n)**2, (2*n-1)*(2*n+1))
    return partial

pi_list = [wallis_rational(n) for n in range(1, 21)]
print(pi_list)
print(sorted(pi_list))

import numpy
print(numpy.pi-numpy.array(list(map(float, pi_list))))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Finally, loop through the sums and check to see if it matches any possible term on the RHS:
Can you write Python code for it?

for i, lhs in enumerate(lhs_sums):
    if lhs in range_to_power:
        rhs_primitive = int(lhs**(0.2))
        lhs_primitive = (numpy.array(lhs_combinations[i])**(0.2)).astype(int)
        print("The LHS terms are {}.".format(lhs_primitive))
        print("The RHS term is {}.".format(rhs_primitive))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Floating point numbers
Exercise 1
Computers cannot, in principle, represent real numbers perfectly. This can lead to problems of accuracy. For example, if
\begin{equation}
  x = 1, \qquad y = 1 + 10^{-14} \sqrt{3}
\end{equation}
then it should be true that
\begin{equation}
  10^{14} (y - x) = \sqrt{3}.
\end{equation}
Check how accurately this equation holds in Python and see what this implies about the accuracy of subtracting two numbers that are close together.
Solution
Can you write Python code for it?

from math import sqrt

x = 1.0
y = 1.0 + 1e-14 * sqrt(3.0)
print("The calculation gives {}".format(1e14*(y-x)))
print("The result should be {}".format(sqrt(3.0)))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
For those that have checked Wikipedia, you'll see Brun's theorem which suggests a specific scaling, that $\pi_N$ is bounded by $C N / \log(N)^2$. Checking this numerically on this data:
Can you write Python code for it?

pyplot.semilogx(N, twin_prime_fraction * numpy.log(N)**2)
pyplot.xlabel(r"$N$")
pyplot.ylabel(r"$\pi_N \times \log(N)^2 / N$")
pyplot.show()


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
I am too lazy to work back through the definitions and flip all the signs; it should be clear how to do this!
Exercise 3
Use these generators to write another generator that produces a basis of $\mathbb{P^3} \times \mathbb{P^4}$.
Solution
Hopefully by now you'll be aware of how useful itertools is!
Can you write Python code for it?

from itertools import product

def basis_product():
    """
    Basis of the product space
    """
    yield from product(basis_pN(3), basis_pN_variant(4))

for p1, p2 in basis_product():
    print("Basis element is ({}) X ({}).".format(p1, p2))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
I've cheated here as I haven't introduced the yield from syntax (which returns an iterator from a generator). We could write this out instead as
Can you write Python code for it?

def basis_product_long_form():
    """
    Basis of the product space (without using yield_from)
    """
    prod = product(basis_pN(3), basis_pN_variant(4))
    yield next(prod)

for p1, p2 in basis_product():
    print("Basis element is ({}) X ({}).".format(p1, p2))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
In later cases, after having covered exceptions, I would suggest raising a NotImplementedError for negative edge lengths.
Exercise 2
Write a function to compute the time (in seconds) taken for an object to fall from a height $H$ (in metres) to the ground, using the formula
\begin{equation}
  h(t) = \frac{1}{2} g t^2.
\end{equation}
Use the value of the acceleration due to gravity $g$ from scipy.constants.g. Test your code on sample values such as

$H = 1$m (result should be $\approx 0.452$s);
$H = 10$m (result should be $\approx 1.428$s);
$H = 0$m (result should be $0$s);
$H = -1$m (what do you think the result should be?).

Solution
Can you write Python code for it?

def fall_time(H):
    """
    Give the time in seconds for an object to fall to the ground
    from H metres.

    Parameters
    ----------

    H : float
        Starting height (metres)

    Returns
    -------

    T : float
        Fall time (seconds)
    """

    from math import sqrt
    from scipy.constants import g

    if (H < 0):
        print("Negative height makes no sense!")
        return 0

    return sqrt(2.0*H/g)

print(fall_time(1))
print(fall_time(10))
print(fall_time(0))
print(fall_time(-1))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
It's worth thinking about the operation counts of the various functions implemented here. The implementations are inefficient, but even in the best case you see how the number of operations (and hence computing time required) rapidly increases.
Logistic map
Partly taken from Newman's book, p 120.
The logistic map builds a sequence of numbers ${ x_n }$ using the relation
\begin{equation}
  x_{n+1} = r x_n \left( 1 - x_n \right),
\end{equation}
where $0 \le x_0 \le 1$.
Exercise 1
Write a program that calculates the first $N$ members of the sequence, given as input $x_0$ and $r$ (and, of course, $N$).
Solution
Can you write Python code for it?

def logistic(x0, r, N = 1000):
    sequence = [x0]
    xn = x0
    for n in range(N):
        xnew = r*xn*(1.0-xn)
        sequence.append(xnew)
        xn = xnew
    return sequence


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Lorenz attractor
The Lorenz system is a set of ordinary differential equations which can be written
\begin{equation}
  \frac{\text{d} \vec{v}}{\text{d} \vec{t}} = \vec{f}(\vec{v})
\end{equation}
where the variables in the state vector $\vec{v}$ are
\begin{equation}
  \vec{v} = \begin{pmatrix} x(t) \ y(t) \ z(t) \end{pmatrix}
\end{equation}
and the function defining the ODE is
\begin{equation}
  \vec{f} = \begin{pmatrix} \sigma \left( y(t) - x(t) \right) \ x(t) \left( \rho - z(t) \right) - y(t) \ x(t) y(t) - \beta z(t) \end{pmatrix}.
\end{equation}
The parameters $\sigma, \rho, \beta$ are all real numbers.
Exercise 1
Write a function dvdt(v, t, params) that returns $\vec{f}$ given $\vec{v}, t$ and the parameters $\sigma, \rho, \beta$.
Solution
Can you write Python code for it?

def dvdt(v, t, sigma, rho, beta):
    """
    Define the Lorenz system.

    Parameters
    ----------

    v : list
        State vector
    t : float
        Time
    sigma : float
        Parameter
    rho : float
        Parameter
    beta : float
        Parameter

    Returns
    -------

    dvdt : list
        RHS defining the Lorenz system
    """

    x, y, z = v

    return [sigma*(y-x), x*(rho-z)-y, x*y-beta*z]


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Now check by finding the twin primes with $N<20$:
Can you write Python code for it?

for tp in twin_primes(20):
    print(tp)


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Now we can define the first basis:
Can you write Python code for it?

def basis_pN(N):
    """
    A generator for the simplest basis of P^N.
    """

    for n in range(N+1):
        yield Monomial(n*[0])


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Now we can generate pairs using the pairwise recipe:
Can you write Python code for it?

from itertools import tee

def pair_primes(N):
    "Generate consecutive prime pairs, using the itertools recipe"
    a, b = tee(all_primes(N))
    next(b, None)
    return zip(a, b)


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
The combinations function returns all the combinations (ignoring order) of r elements from a given list. For example, take a list of length 6, [1, 2, 3, 4, 5, 6] and compute all the combinations of length 4:
Can you write Python code for it?

input_list = numpy.arange(1, 7)
combinations = list(itertools.combinations(input_list, 4))
print(combinations)


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
The shortest published Mathematical paper
A candidate for the shortest mathematical paper ever shows the following result:
\begin{equation}
  27^5 + 84^5 + 110^5 + 133^5 = 144^5.
\end{equation}
This is interesting as

This is a counterexample to a conjecture by Euler ... that at least $n$ $n$th powers are required to sum to an $n$th power, $n > 2$.

Exercise 1
Using python, check the equation above is true.
Solution
Can you write Python code for it?


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Then calculate the sums:
Can you write Python code for it?

lhs_sums = []
for lhs_terms in lhs_combinations:
    lhs_sums.append(numpy.sum(numpy.array(lhs_terms)))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Then test it on $\mathbb{P}^N$:
Can you write Python code for it?

for poly in basis_pN(3):
    print(poly)


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Then use the filter function to define another generator:
Can you write Python code for it?

def twin_primes(N):
    """
    Return all twin primes
    """
    return filter(check_twin, pair_primes(N))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
There is a difference in the fifth significant figure in both solutions in the first case, which gets to the third (arguably the second) significant figure in the second case. Comparing to the limiting solutions above, we see that the larger root is definitely more accurately captured with the first formula than the second (as the result should be bigger than $10^{-2n}$).
In the second case we have divided by a very small number to get the big number, which loses accuracy.
Exercise 5
The standard definition of the derivative of a function is
\begin{equation}
  \left. \frac{\text{d} f}{\text{d} x} \right|{x=X} = \lim{\delta \to 0} \frac{f(X + \delta) - f(X)}{\delta}.
\end{equation}
We can approximate this by computing the result for a finite value of $\delta$:
\begin{equation}
  g(x, \delta) = \frac{f(x + \delta) - f(x)}{\delta}.
\end{equation}
Write a function that takes as inputs a function of one variable, $f(x)$, a location $X$, and a step length $\delta$, and returns the approximation to the derivative given by $g$.
Solution
Can you write Python code for it?

def g(f, X, delta):
    """
    Approximate the derivative of a given function at a point.

    Parameters
    ----------

    f : function
        Function to be differentiated
    X : real
        Point at which the derivative is evaluated
    delta : real
        Step length

    Returns
    -------

    g : real
        Approximation to the derivative
    """

    return (f(X+delta) - f(X)) / delta


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
This looks horrible, but is correct. To really make this look good, we need to improve the output. If we use
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""
    explanation = "I am a monomial"

    def __init__(self, roots):
        Polynomial.__init__(self, roots, 1)

    def __repr__(self):
        if len(self.roots):
            string = ""
            n_zero_roots = len(self.roots) - numpy.count_nonzero(self.roots)
            if n_zero_roots == 1:
                string = "x"
            elif n_zero_roots > 1:
                string = "x^{}".format(n_zero_roots)
        else: # Monomial degree 0.
            string = "1"
        for root in self.roots:
            if root > 0:
                string = string + "(x - {})".format(root)
            elif root < 0:
                string = string + "(x + {})".format(-root)
        return string


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
This shows the sensitive dependence on initial conditions that is characteristic of chaotic behaviour.
Systematic ODE solving with sympy
We are interested in the solution of
\begin{equation}
  \frac{\text{d} y}{\text{d} t} = e^{-t} - y^n, \qquad y(0) = 1,
\end{equation}
where $n > 1$ is an integer. The "minor" change from the above examples mean that sympy can only give the solution as a power series.
Exercise 1
Compute the general solution as a power series for $n = 2$.
Solution
Can you write Python code for it?

import sympy
sympy.init_printing()

y, t = sympy.symbols('y, t')

sympy.dsolve(sympy.diff(y(t), t) + y(t)**2 - sympy.exp(-t), y(t))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
Twin primes
A twin prime is a pair $(p_1, p_2)$ such that both $p_1$ and $p_2$ are prime and $p_2 = p_1 + 2$.
Exercise 1
Write a generator that returns twin primes. You can use the generators above, and may want to look at the itertools module together with its recipes, particularly the pairwise recipe.
Solution
Note: we need to first pull in the generators introduced in that notebook
Can you write Python code for it?

def all_primes(N):
    """
    Return all primes less than or equal to N.

    Parameters
    ----------

    N : int
        Maximum number

    Returns
    -------

    prime : generator
        Prime numbers
    """

    primes = []
    for n in range(2, N+1):
        is_n_prime = True
        for p in primes:
            if n%p == 0:
                is_n_prime = False
                break
        if is_n_prime:
            primes.append(n)
            yield n


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
We can already see that the number of terms to consider is large.
Note that we have used the list function to explicitly get a list of the combinations. The combinations function returns a generator, which can be used in a loop as if it were a list, without storing all elements of the list.
How fast does the number of combinations grow? The standard formula says that for a list of length $n$ there are
\begin{equation}
  \begin{pmatrix} n \ k \end{pmatrix} = \frac{n!}{k! (n-k)!}
\end{equation}
combinations of length $k$. For $k=4$ as needed here we will have $n (n-1) (n-2) (n-3) / 24$ combinations. For $n=144$ we therefore have
Can you write Python code for it?

n_combinations = 144*143*142*141/24
print("Number of combinations of 4 objects from 144 is {}".format(n_combinations))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
We could examine the results of the two primes directly. But an efficient solution is to use python's filter function. To do this, first define a function checking if the pair are twin primes:
Can you write Python code for it?

def check_twin(pair):
    """
    Take in a pair of integers, check if they differ by 2.
    """
    p1, p2 = pair
    return p2-p1 == 2


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
We have a combination of floating point inaccuracies: in the numerator we have two terms that are nearly equal, leading to a very small number. We then divide two very small numbers. This is inherently inaccurate.
This does not mean that you can't calculate derivatives to high accuracy, but alternative approaches are definitely recommended.
Prime numbers
Exercise 1
Write a function that tests if a number is prime. Test it by writing out all prime numbers less than 50.
Solution
This is a "simple" solution, but not efficient.
Can you write Python code for it?

def isprime(n):
    """
    Checks to see if an integer is prime.

    Parameters
    ----------

    n : integer
        Number to check

    Returns
    -------

    isprime : Boolean
        If n is prime
    """

    # No number less than 2 can be prime
    if n < 2:
        return False

    # We only need to check for divisors up to sqrt(n)
    for m in range(2, int(n**0.5)+1):
        if n%m == 0:
            return False

    # If we've got this far, there are no divisors.
    return True

for n in range(50):
    if isprime(n):
        print("Function says that {} is prime.".format(n))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
We see that the first three digits are correct. This isn't too surprising: we expect 16 digits of accuracy for a floating point number, but $x$ and $y$ are identical for the first 14 digits.
Exercise 2
The standard quadratic formula gives the solutions to
\begin{equation}
  a x^2 + b x + c = 0
\end{equation}
as
\begin{equation}
  x = \frac{-b \pm \sqrt{b^2 - 4 a c}}{2 a}.
\end{equation}
Show that, if $a = 10^{-n} = c$ and $b = 10^n$ then
\begin{equation}
  x = \frac{10^{2 n}}{2} \left( -1 \pm \sqrt{1 - 4 \times 10^{-4n}} \right).
\end{equation}
Using the expansion (from Taylor's theorem)
\begin{equation}
  \sqrt{1 - 10^{-4 n}} \simeq 1 - 2 \times 10^{-4 n} + \dots, \qquad n \gg 1,
\end{equation}
show that
\begin{equation}
  x \simeq -10^{2 n} + 10^{-2 n} \quad \text{and} \quad -10^{-2n}, \qquad n \gg 1.
\end{equation}
Solution
This is pen-and-paper work; each step should be re-arranging.
Exercise 3
By multiplying and dividing by $-b \mp \sqrt{b^2 - 4 a c}$, check that we can also write the solutions to the quadratic equation as
\begin{equation}
  x = \frac{2 c}{-b \mp \sqrt{b^2 - 4 a c}}.
\end{equation}
Solution
Using the difference of two squares we get
\begin{equation}
  x = \frac{b^2 - \left( b^2 - 4 a c \right)}{2a \left( -b \mp \sqrt{b^2 - 4 a c} \right)}
\end{equation}
which re-arranges to give the required solution.
Exercise 4
Using Python, calculate both solutions to the quadratic equation
\begin{equation}
  10^{-n} x^2 + 10^n x + 10^{-n} = 0
\end{equation}
for $n = 3$ and $n = 4$ using both formulas. What do you see? How has floating point accuracy caused problems here?
Solution
Can you write Python code for it?

a = 1e-3
b = 1e3
c = a
formula1_n3_plus = (-b + sqrt(b**2 - 4.0*a*c))/(2.0*a)
formula1_n3_minus = (-b - sqrt(b**2 - 4.0*a*c))/(2.0*a)
formula2_n3_plus = (2.0*c)/(-b + sqrt(b**2 - 4.0*a*c))
formula2_n3_minus = (2.0*c)/(-b - sqrt(b**2 - 4.0*a*c))
print("For n=3, first formula, solutions are {} and {}.".format(formula1_n3_plus,
                                                                formula1_n3_minus))
print("For n=3, second formula, solutions are {} and {}.".format(formula2_n3_plus,
                                                                 formula2_n3_minus))

a = 1e-4
b = 1e4
c = a
formula1_n4_plus = (-b + sqrt(b**2 - 4.0*a*c))/(2.0*a)
formula1_n4_minus = (-b - sqrt(b**2 - 4.0*a*c))/(2.0*a)
formula2_n4_plus = (2.0*c)/(-b + sqrt(b**2 - 4.0*a*c))
formula2_n4_minus = (2.0*c)/(-b - sqrt(b**2 - 4.0*a*c))
print("For n=4, first formula, solutions are {} and {}.".format(formula1_n4_plus,
                                                                formula1_n4_minus))
print("For n=4, second formula, solutions are {} and {}.".format(formula2_n4_plus,
                                                                 formula2_n4_minus))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
We see that the relative error decreases, whilst the absolute error grows (significantly).
Basic functions
Exercise 1
Write a function to calculate the volume of a cuboid with edge lengths $a, b, c$. Test your code on sample values such as

$a=1, b=1, c=1$ (result should be $1$);
$a=1, b=2, c=3.5$ (result should be $7.0$);
$a=0, b=1, c=1$ (result should be $0$);
$a=2, b=-1, c=1$ (what do you think the result should be?).

Solution
Can you write Python code for it?

def cuboid_volume(a, b, c):
    """
    Compute the volume of a cuboid with edge lengths a, b, c.
    Volume is abc. Only makes sense if all are non-negative.

    Parameters
    ----------

    a : float
        Edge length 1
    b : float
        Edge length 2
    c : float
        Edge length 3

    Returns
    -------

    volume : float
        The volume a*b*c
    """

    if (a < 0.0) or (b < 0.0) or (c < 0.0):
        print("Negative edge length makes no sense!")
        return 0

    return a*b*c

print(cuboid_volume(1,1,1))
print(cuboid_volume(1,2,3.5))
print(cuboid_volume(0,1,1))
print(cuboid_volume(2,-1,1))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
With 17 million combinations to work with, we'll need to be a little careful how we compute.
One thing we could try is to loop through each possible "smallest instance" (the term on the right hand side) in increasing order. We then check all possible combinations of left hand sides.
This is computationally very expensive as we repeat a lot of calculations. We repeatedly recalculate combinations (a bad idea). We repeatedly recalculate the powers of the same number.
Instead, let us try creating the list of all combinations of powers once.
Exercise 2b

Construct a numpy array containing all integers in $1, \dots, 144$ to the fifth power.
Construct a list of all combinations of four elements from this array.
Construct a list of sums of all these combinations.
Loop over one list and check if the entry appears in the other list (ie, use the in keyword).

Solution
Can you write Python code for it?

nmax=145
range_to_power = numpy.arange(1, nmax)**5
lhs_combinations = list(itertools.combinations(range_to_power, 4))


I am working on the file "ExercisesSolutions.ipynb".
The first task is:
then we can deal with the uglier cases, and re-running the test we get
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:

Table of Contents

Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Cleaning Data
Making a DataFrame easier to work with
Difficulty: easy/medium
It happens all the time: someone gives you data containing malformed strings, Python, lists and missing data. How do you tidy it up so you can get on with the analysis?
Can you write Python code for it?

df = pd.DataFrame({'From_To': ['LoNDon_paris', 'MAdrid_miLAN', 'londON_StockhOlm',
                               'Budapest_PaRis', 'Brussels_londOn'],
              'FlightNumber': [10045, np.nan, 10065, np.nan, 10085],
              'RecentDelays': [[23, 47], [], [24, 43, 87], [13], [67, 32]],
                   'Airline': ['KLM(!)', '<Air France> (12)', '(British Airways. )',
                               '12. Air France', '"Swiss Air"']})


I am working on the file "Exercises_part_A.ipynb".
The first task is:
DataFrame basics
Difficulty: easy
A few of the fundamental routines for selecting, sorting, adding and aggregating data in DataFrames
Consider the following Python dictionary data and Python list labels:
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
DataFrames: harder problems
These might require a bit of thinking outside the box...
...but all are solvable using just the usual pandas/NumPy methods (and so avoid using explicit for loops).
Difficulty: hard
Task: Consider a DataFrame consisting of purely numerical data. Create a list of the row-column indices of the 3 largest values.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Calculate the mean age for each different animal in df.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Calculate the sum of all visits (the total number of visits).
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Change the age in row 'f' to 1.5.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Create a DataFrame df from this dictionary data which has the index labels.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Delete the From_To column from df and attach the temporary DataFrame from the previous questions.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Delete the columns named frequency and rank
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Display a summary of the basic information about this DataFrame and its data.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Given a DataFrame with a column of group IDs, 'groups', and a column of corresponding integer values, 'vals', replace any negative values in 'vals' with the group mean.
Can you write Python code for it?

import pandas as pd

df = pd.DataFrame([[1,1],[1,-1],[2,1],[2,2]], columns=["groups", "vals"])
df

# Write your answer here


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Reshape the DataFrame so that you can compare the values of Rank and Abundance from the three sample categories by placing them next to each other in one row per taxon. In other words, reshape in a way that you get one row per taxon, placing Rank and Abundance values from the three sample categories next to each other (from "long" to "wide" format).
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Return the first 3 rows of the DataFrame df.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Select the rows where the age is missing, i.e. is NaN.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Select the rows where the animal is a cat and the age is less than 3.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Select the top 2 most abundant taxa per sample category. Write a function that expects a DataFrame, a column-name, and top-n (an Integer indicating the top n most abundant things within the given column-name).
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Some values in the the FlightNumber column are missing. These numbers are meant to increase by 10 with each row. Therefore the numbers 10055 and 10075 need to replace the missing values. Fill in these missing numbers and make the column an integer column (instead of a float column).
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: Sort df first by the values in the 'age' in decending order, then by the value in the 'visit' column in ascending order.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task: The 'priority' column contains the values 'yes' and 'no'. Replace this column with a column of boolean values: 'yes' should be True and 'no' should be False.
Can you write Python code for it?


I am working on the file "Exercises_part_A.ipynb".
The first task is:
Task:. In the RecentDelays column, the values have been entered into the DataFrame as a list. We would like each first value in its own column, each second value in its own column, and so on. If there isn't an Nth value, the value should be NaN.
Expand the Series of lists into a DataFrame named delays, rename the columns delay_1, delay_2, etc. and replace the unwanted RecentDelays column in df with delays.
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:

Table of Contents

Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Apply
Task: The From_To column would be better as two separate columns! Split each string on the underscore delimiter _ to make two new columns with the correct values. Assign the correct column names to a new temporary DataFrame called temp.
Can you write Python code for it?

# Answer
# temp = df.From_To.str.split('_', expand=True)
# temp.columns = ['From', 'To']

# or

temp = pd.DataFrame()
temp["From"], temp["To"] = zip(*df["From_To"].apply(lambda s: s.split("_")))


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Cleaning Data
Making a DataFrame easier to work with
Difficulty: easy/medium
It happens all the time: someone gives you data containing malformed strings, Python, lists and missing data. How do you tidy it up so you can get on with the analysis?
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
DataFrame basics
Difficulty: easy
A few of the fundamental routines for selecting, sorting, adding and aggregating data in DataFrames
Consider the following Python dictionary data and Python list labels:
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Append a new row 'k' to df with your choice of values for each column. Then delete that row to return the original DataFrame.
Can you write Python code for it?

# Answer
df.loc['k'] = [5.5, 'dog', 'no', 2]

# and then deleting the new row...

df = df.drop('k')


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Calculate the mean age for each different animal in df.
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Calculate the sum of all visits (the total number of visits).
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Change the age in row 'f' to 1.5.
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Create a column named Rank that ranks the taxa by their abundance within each SampleCategory in descending order (most abundant taxa get the lowest rank).
Can you write Python code for it?

# Answer:
df["Rank"] = df.groupby("SampleCategory")["Abundance"].rank(ascending=False)
df


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Delete the From_To column from df and attach the temporary DataFrame from the previous questions.
Can you write Python code for it?

# Answer
df = df.drop('From_To', axis=1)
df = df.join(temp)


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Delete the columns named frequency and rank
Can you write Python code for it?

# Answer:
df = df.drop(["rank", "frequency"], axis=1)
df


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: For each animal type and each number of visits, find the mean age. In other words, each row is an animal, each column is a number of visits and the values are the mean ages (hint: use a pivot table).
DataFrames: beyond the basics
Slightly trickier: you may need to combine two or more methods to get the right answer
Difficulty: medium
The previous section was tour through some basic but essential DataFrame operations. Below are some ways that you might need to cut your data, but for which there is no single "out of the box" method.
Task:  How do you count how many unique rows a DataFrame has (i.e. ignore all rows that are duplicates)?
Can you write Python code for it?

df = pd.DataFrame({'A': [1, 2, 2, 3, 4, 5, 5, 5, 6, 7, 7]})

# Answer
len(df) - df.duplicated(keep=False).sum()

# or perhaps more simply...

len(df.drop_duplicates(keep=False))


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Given a DataFrame with a column of group IDs, 'groups', and a column of corresponding integer values, 'vals', replace any negative values in 'vals' with the group mean.
Can you write Python code for it?

import pandas as pd

df = pd.DataFrame([[1,1],[1,-1],[2,1],[2,2]], columns=["groups", "vals"])
df

# Answer
def replace_within_group(group):
    mask = group < 0
    # Select those values where it is < 0, and replace
    # them with the mean of the values which are not < 0.
    group[mask] = group[~mask].mean() # "~" is the "invert" or "complement" operation
    return group

df.groupby(['groups'])['vals'].transform(replace_within_group)
# http://stackoverflow.com/questions/14760757/replacing-values-with-groupby-means/
# credit: unutbu


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Reshape the DataFrame so that you can compare the values of Rank and Abundance from the three sample categories by placing them next to each other in one row per taxon. In other words, reshape in a way that you get one row per taxon, placing Rank and Abundance values from the three sample categories next to each other (from "long" to "wide" format).
Can you write Python code for it?

# Answer:
abu = df.pivot_table(values='Abundance', index='TaxName', columns='SampleCategory')
rank = df.pivot_table(values='Rank', index='TaxName', columns='SampleCategory')
dfp = pd.concat([rank, abu], axis=1).reset_index()
dfp


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Return the first 3 rows of the DataFrame df.
Can you write Python code for it?

# Answer
df.iloc[:3]

# or equivalently

df.head(3)


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Select just the 'animal' and 'age' columns from the DataFrame df.
Can you write Python code for it?

# Answer
df.loc[:, ['animal', 'age']]

# or

df[['animal', 'age']]


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Select only the rows where the number of visits is greater than 2.
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Select the rows where the age is missing, i.e. is NaN.
Can you write Python code for it?


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Select the rows where the animal is a cat and the age is less than 3.
Can you write Python code for it?

# Answer
df[(df['animal'] == 'cat') & (df['age'] < 3)]


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Some values in the the FlightNumber column are missing. These numbers are meant to increase by 10 with each row. Therefore the numbers 10055 and 10075 need to replace the missing values. Fill in these missing numbers and make the column an integer column (instead of a float column).
Can you write Python code for it?

# Answer
df['FlightNumber'] = df['FlightNumber'].interpolate().astype(int)


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
Task: Sort df first by the values in the 'age' in decending order, then by the value in the 'visit' column in ascending order.
Can you write Python code for it?

# Answer
df.sort_values(by=['age', 'visits'], ascending=[False, True])


I am working on the file "Exercises_part_A_with_Solutions.ipynb".
The first task is:
The DataFrame should look much better now.
Rename, delete, rank and pivot
Can you write Python code for it?

fn = "data/Saliva.txt"
df = pd.read_csv(fn, sep='\t')
df


I am working on the file "Exercises_part_B.ipynb".
The first task is:

Table of Contents

Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Combine DataFrames
Task: In the data/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each. Write code that imports each of the data spreadsheets and combines them into a single DataFrame, adding the identifying information from the metadata spreadsheet as columns in the combined DataFrame.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Concatenate, Merge & Join
Task: Import the first dataset cars1 and cars2. Assign each to a to a variable called cars1 and cars2.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Consider the following DataFrames for the next exercises
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
GroupBy Titanic data
Load the dataset in titanic.xls. It contains data on all the passengers that travelled on the Titanic.
Task:
Women and children first?

Use the groupby method to calculate the proportion of passengers that survived by sex.
Calculate the same proportion, but by class and sex.
Create age categories: children (under 14 years), adolescents (14-20), adult (21-64), and senior(65+), and calculate survival proportions by age category, class and sex. Additionally, count the number of passengers per group.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Groupby
Task: Which continent drinks most beer on average?
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Let's consider another data set to do some more Merge, Join & Concatenate exerciseses
Can you write Python code for it?

raw_data_1 = {
        'subject_id': ['1', '2', '3', '4', '5'],
        'first_name': ['Alex', 'Amy', 'Allen', 'Alice', 'Ayoung'],
        'last_name': ['Anderson', 'Ackerman', 'Ali', 'Aoni', 'Atiches']}

raw_data_2 = {
        'subject_id': ['4', '5', '6', '7', '8', '9', '10'],
        'first_name': ['Alice', 'Ayoung', 'Bran', 'Bryce', 'Betty', 'Jane', np.nan],
        'last_name': ['Aoni', 'Atiches', 'Balwner', 'Brice', 'Btisan', np.nan, 'Doe']}

raw_data_3 = {
        'subject_id': ['1', '2', '3', '4', '5', '7', '8', '9', '10', '11'],
        'test_id': [51, 15, 15, 61, 16, 14, 15, 1, 61, 16]}
data1 = pd.DataFrame(raw_data_1, columns = ['subject_id', 'first_name', 'last_name'])
data2 = pd.DataFrame(raw_data_2, columns = ['subject_id', 'first_name', 'last_name'])
data3 = pd.DataFrame(raw_data_3, columns = ['subject_id','test_id'])


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Let's plot the number of survivors grouped by sex and passenger class.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Pivot
Task: Let's reshape this small example DataFrame of ICD10 codes. Each person has different code-associations. Only positive associations are listed. Transform (reshape) the DataFrame to a wide format (one column per code) that lists positive and negative (missing) associations as Booleans.
Can you write Python code for it?

df = pd.DataFrame({"Person": ["a", "a", "a", "b", "c", "c"], "Code": ["D99", "E32", "A41", "D99", "D99", "A41"]}, columns=["Person", "Code"])
df

# Write your answer here


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Another way of comparing the groups is to look at the survival rate, by adjusting for the number of people in each group.
Create a stacked, horizontal Barplot of the adjusted death counts
1. Sum the death_counts per passenger-class and sex, and convert to data type float (for Python 2.x division purposes).
2. Compute the adjusted survival rate by dividing the death_counts by the result from 1.
3. Plot a stacked, horizontal Barplot from the result of 3.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: For each continent print "the" statistics (summary stats using "df.describe()") for wine consumption.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: It seems our first dataset has some unnamed blank columns, fix cars1.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Join the two dataframes, data1 and data2, along columns and assing to all_data_col.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Join the two dataframes, data1 and data2, along rows and assign all_data. Make sure that the row index is unique.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Let's also look at the deaths (and not only at the survivors) within the groups and create a stacked Barplot of survivers vs. deaths grouped by sex and passenger-class (as before).
1. Convert the "survived" column to boolean values
2. Compute the cross tabulation (a.k.a. contingency table) of passenger-class and sex vs. survived. Assign the result to the variable name "death_counts" --> checkout pd.crosstab()
3. Create a stacked barplot of the computed death_counts
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: List all unique continents.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Merge all_data and data3 along the subject_id value.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Print the mean, min and max values for spirit consumption.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Print the median alcoohol consumption per continent for every column
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: See the first 10 entries
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: We would expect the result to now have mean 0 and standard deviation 1 within each group, which we can easily check. Calculate the mean and standard deviation within each group.
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Which countries have missing values in the continent column?
Can you write Python code for it?


I am working on the file "Exercises_part_B.ipynb".
The first task is:
Task: Which country has the highest alcohol consumption (total litres of pure alcohol)?
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:

Table of Contents

Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Apply (interspersed)
Task: Create function that returns the first word of the string in the "car" column, the manufacturer name. Use the "apply" method to create a new column in the DataFrame.
Can you write Python code for it?

# Answer
def grep_firt_word(string_):
    return string_.split()[0]

cars["vendor"] = cars["car"].apply(grep_firt_word)

# or

cars["vendor"] = cars["car"].apply(lambda s: s.split()[0])


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Concatenate, Merge & Join
Task: Import the first dataset cars1 and cars2. Assign each to a to a variable called cars1 and cars2.
Can you write Python code for it?

# Answer:
cars1 = pd.read_csv("data/cars1.csv")
cars2 = pd.read_csv("data/cars2.csv")


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Consider the following DataFrames for the next exercises
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Groupby
Task: Which continent drinks most beer on average?
Can you write Python code for it?

# Answer
# df.groupby("continent").beer_servings.mean()
# or
df.groupby("continent")["beer_servings"].mean()


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Let's plot the number of survivors grouped by sex and passenger class.
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Another way of comparing the groups is to look at the survival rate, by adjusting for the number of people in each group.
Create a stacked, horizontal Barplot of the adjusted death counts
1. Sum the death_counts per passenger-class and sex, and convert to data type float (for Python 2.x division purposes).
2. Compute the adjusted survival rate by dividing the death_counts by the result from 1.
3. Plot a stacked, horizontal Barplot from the result of 3.
Can you write Python code for it?

# Answer

# 1.
sum_death_counts = death_counts.sum(axis=1).astype(float)

# 2.
adjusted_death_counts = death_counts.div(sum_death_counts, axis=0)

# 3.
adjusted_death_counts.plot(kind='barh', stacked=True)

# or in short:
# death_counts.div(death_counts.sum(1).astype(float), axis=0).plot(kind='barh', stacked=True)


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Join cars1 and cars2 into a single DataFrame called cars
Can you write Python code for it?

# Answer:
cars = cars1.append(cars2)

# or
cars_ = pd.concat([cars1, cars2], axis=0)

# check for equality
cars_.equals(cars)


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Join the two dataframes, data1 and data2, along columns and assing to all_data_col.
Can you write Python code for it?

# Answer:
all_data_col = pd.concat([data1, data2], axis = 1)
all_data_col


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Join the two dataframes, data1 and data2, along rows and assign all_data. Make sure that the row index is unique.
Can you write Python code for it?

# Answer:
all_data = pd.concat([data1, data2], ignore_index=True)
assert len(set(all_data.index)) == all_data.shape[0]

# otherwise you might get unexpected behaviour:
all_data = pd.concat([data1, data2], ignore_index=False)
all_data.loc[0, :]


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Let's also look at the deaths (and not only at the survivors) within the groups and create a stacked Barplot of survivers vs. deaths grouped by sex and passenger-class (as before).
1. Convert the "survived" column to boolean values
2. Compute the cross tabulation (a.k.a. contingency table) of passenger-class and sex vs. survived. Assign the result to the variable name "death_counts" --> checkout pd.crosstab()
3. Create a stacked barplot of the computed death_counts
Can you write Python code for it?

# Answer
# 1.
titanic.survived = titanic.survived.astype(bool)

# 2.
death_counts = pd.crosstab([titanic.pclass, titanic.sex], titanic.survived)

# 3.
death_counts.plot(kind='bar', stacked=True)


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Merge all_data and data3 along the subject_id value.
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Merge only the data that has the same 'subject_id' in both data1 and data2.
Can you write Python code for it?

# Answer:
pd.merge(data1, data2, how='inner')


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Print the median alcoohol consumption per continent for every column
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: See the first 10 entries
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Set "the" missing continent with a name of your choice.
Can you write Python code for it?

# Answer:
df.loc[df["continent"].isnull(), "continent"] = "NAs"


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Use groupby in conjunction with transform across multiple columns: We want to group by one to n columns and apply a function on these groups across two columns.
    1. Calculate the sum of a and b and assign it to a column named e.
    2. Group by 'c' and d, and calculate the sum of e
Can you write Python code for it?


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: We would expect the result to now have mean 0 and standard deviation 1 within each group, which we can easily check. Calculate the mean and standard deviation within each group.
Can you write Python code for it?

# Answer:
grouped = transformed.groupby(key)
grouped.mean()
grouped.std()


I am working on the file "Exercises_part_B_with_Solutions.ipynb".
The first task is:
Task: Which countries have missing values in the continent column?
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
As an example, Spitzer data is expected to be store in the directory structure:
$HOME/PLANET_DIRECTORY/PLANETNAME/data/raw/AORDIR/CHANNEL/bcd/
EXAMPLE:

On a Linux machine
With user tempuser,
And all Spitzer data is store in Research/Planets
The planet named Happy-5b
Observed during AOR r11235813
In CH2 (4.5 microns)

The loadfitsdir should read as: /home/tempuser/Research/Planets/HAPPY5/data/raw/r11235813/ch2/bcd/
Can you write Python code for it?

from os import environ

planetName      = 'GJ1214'
planetDirectory = '/Research/Planets/'

channel = 'ch2'
# channel = 'ch2/'

dataSub = 'bcd/'

dataDir     = environ['HOME'] + planetDirectory + planetName + '/data/raw/' + channel + '/big/'

AORs = []
for dirNow in glob(dataDir + '/*'):
    AORs.append(dirNow.split('/')[-1])

fileExt = '*bcd.fits'
uncsExt = '*bunc.fits'

print(dataDir)

len(AORs)

iAOR        = 0
AORNow      = AORs[iAOR]
loadfitsdir = dataDir + AORNow + '/' + channel + '/' + dataSub
print(loadfitsdir)

nCores = cpu_count()

fitsFilenames = glob(loadfitsdir + fileExt);print(len(fitsFilenames))
uncsFilenames = glob(loadfitsdir + uncsExt);print(len(uncsFilenames))

header_test = fits.getheader(fitsFilenames[0])
print('AORLABEL:\t{}\nNum Fits Files:\t{}\nNum Unc Files:\t{}'.format\
          (header_test['AORLABEL'], len(fitsFilenames), len(uncsFilenames)))


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
Double check for NaNs
Can you write Python code for it?

example_wanderer_median.imageCube[np.where(isnan(example_wanderer_median.imageCube))] = \
                                                    np.nanmedian(example_wanderer_median.imageCube)


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
Identifier Strong Outliers
Can you write Python code for it?

print('Find, flag, and NaN the "Bad Pixels" Outliers' + '\n')
example_wanderer_median.find_bad_pixels()

print('Fit for All Centers: Flux Weighted, Gaussian Fitting, Gaussian Moments, Least Asymmetry' + '\n')
# example_wanderer_median.fit_gaussian_centering()
example_wanderer_median.fit_flux_weighted_centering()
# example_wanderer_median.fit_least_asymmetry_centering()
# example_wanderer_median.fit_all_centering() # calling this calls least_asymmetry, which does not work :(

# start = time()
# example_wanderer_median.mp_fit_gaussian_centering(subArray=6)
# print('Operation took {} seconds with {} cores'.format(time()-start, example_wanderer_median.nCores))

# centering_df_copy = example_wanderer_median.centering_df
# example_wanderer_median.centering_df = {}

start = time()
example_wanderer_median.mp_lmfit_gaussian_centering(subArraySize=6, recheckMethod=None, median_crop=False)
print('Operation took {} seconds with {} cores'.format(time()-start, example_wanderer_median.nCores))

nSig       = 10.1
medY       = median(example_wanderer_median.centering_GaussianFit.T[y])
medX       = median(example_wanderer_median.centering_GaussianFit.T[x])
stdY       = std(example_wanderer_median.centering_GaussianFit.T[y])
stdX       = std(example_wanderer_median.centering_GaussianFit.T[x])

ySig = 4
xSig = 4
outliers   = (((example_wanderer_median.centering_GaussianFit.T[y] - medY)/(ySig*stdY))**2 + \
              ((example_wanderer_median.centering_GaussianFit.T[x] - medX)/(xSig*stdX))**2) > 1

ax = figure().add_subplot(111)
cx, cy = example_wanderer_median.centering_GaussianFit.T[x],example_wanderer_median.centering_GaussianFit.T[y]
ax.plot(cx,cy,'.',ms=1)
ax.plot(cx[outliers],cy[outliers],'.',ms=1)
# ax.plot(median(cx), median(cy),'ro',ms=1)
ax.set_xlim(medX-nSig*stdX,medX+nSig*stdX)
ax.set_ylim(medY-nSig*stdY,medY+nSig*stdY)

from sklearn.cluster import DBSCAN

dbs     = DBSCAN(n_jobs=-1, eps=0.2, leaf_size=10)
dbsPred = dbs.fit_predict(example_wanderer_median.centering_GaussianFit)

dbs_options = [k for k in...


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
Load Stored Instance from Save Files
If this is not your time, then you can load a saved file in the loadfitsdir directory that has the filename:
PLANETNAME_Wanderer_METHOD_DATE.pickle.save
```python
method = 'mean'
example_wanderer_mean = wanderer(loadfitsdir=loadfitsdir, filetype=filetype,
                                            yguess=yguess, xguess=xguess, method=method)
example_wanderer_mean.load_data_from_save_files(savefiledir='./SaveFiles/',
                        saveFileNameHeader=planetName+'Wanderer_Mean', saveFileType='.pickle.save')
```
```python
method = 'median'
example_wanderer_median = wanderer(loadfitsdir=loadfitsdir_ModA, filetype=filetype,
                                            yguess=yguess, xguess=xguess, method=method)
example_wanderer_median.load_data_from_save_files(savefiledir='./SaveFiles/', saveFileNameHeader='Example_Wanderer_Median_', saveFileType='.pickle.save')
```
Start a New Instance with Median for the Metric
Can you write Python code for it?

method = 'median'

print('Initialize an instance of `wanderer` as `example_wanderer_median`\n')
example_wanderer_median = wanderer(fitsFileDir=loadfitsdir, filetype=filetype, telescope='Spitzer',
                                            yguess=yguess, xguess=xguess, method=method, nCores=nCores)

example_wanderer_median.AOR        = AORNow
example_wanderer_median.planetName = planetName
example_wanderer_median.channel    = channel

print('Load Data From Fits Files in ' + loadfitsdir + '\n')
# exarymple_wanderer_median.load_data_from_fits_files()
example_wanderer_median.spitzer_load_fits_file(outputUnits='electrons')#(outputUnits='muJ_per_Pixel')

print('Skipping Load Data From Save Files in ' + loadfitsdir + '\n')
# example_wanderer_median.load_data_from_save_files(savefiledir='./SaveFiles/', \
# saveFileNameHeader='Example_Wanderer_Median_', saveFileType='.pickle.save')


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
Load Wanderer Class
Necessary Constants Spitzer
Can you write Python code for it?

ppm             = 1e6
y,x             = 0,1

yguess, xguess  = 15., 15.   # Specific to Spitzer circa 2010 and beyond
filetype        = 'bcd.fits' # Specific to Spitzer Basic Calibrated Data


I am working on the file "Exoplanet_TSO_-_Photometric_Extraction_Pipeline.ipynb".
The first task is:
Master Class for Exoplanet Time Series Observation Photometry
Can you write Python code for it?

from wanderer import wanderer

def clipOutlier2D(arr2D, nSig=10):
    arr2D     = arr2D.copy()
    medArr2D  = median(arr2D,axis=0)
    sclArr2D  = np.sqrt(((scale.mad(arr2D)**2.).sum()))
    outliers  = abs(arr2D - medArr2D) >  nSig*sclArr2D
    inliers   = abs(arr2D - medArr2D) <= nSig*sclArr2D
    arr2D[outliers] = median(arr2D[inliers],axis=0)
    return arr2D

rcParams['figure.dpi'] = 150
rcParams['image.interpolation'] = 'None'
rcParams['image.cmap']          = 'Blues_r'
rcParams['axes.grid']           = False


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
As an example, Spitzer data is expected to be store in the directory structure:
$HOME/PLANET_DIRECTORY/PLANETNAME/data/raw/AORDIR/CHANNEL/bcd/
EXAMPLE:

On a Linux machine
With user tempuser,
And all Spitzer data is store in Research/Planets
The planet named Happy-5b
Observed during AOR r11235813
In CH2 (4.5 microns)

The loadfitsdir should read as: /home/tempuser/Research/Planets/HATP26/data/raw/r11235813/ch2/bcd/
Can you write Python code for it?

from os import environ

planetName      = 'HATP26'
planetDirectory = '/Research/Planets/'

channel = 'ch2'
# channel = 'ch2/'

dataSub = 'bcd/'

dataDir     = environ['HOME'] + planetDirectory + planetName + '/data/raw/' + channel + '/big/'

AORs = []
for dirNow in glob(dataDir + '/*'):
    AORs.append(dirNow.split('/')[-1])

fileExt = '*bcd.fits'
uncsExt = '*bunc.fits'

print(dataDir)

len(AORs)

iAOR        = 0 # Change this to 0,1,2,3,4,5,6,7 to "cycle" over the 8 AORs in the base directory /home/tempuser/Research/Planets/HATP26/data/raw/
AORNow      = AORs[iAOR] # This will flag an error if no AORs were found in the directory `dataDir`
loadfitsdir = dataDir + AORNow + '/' + channel + '/' + dataSub
print(loadfitsdir)

nCores = cpu_count()

fitsFilenames = glob(loadfitsdir + fileExt);print(len(fitsFilenames))
uncsFilenames = glob(loadfitsdir + uncsExt);print(len(uncsFilenames))

header_test = fits.getheader(fitsFilenames[0])
print('AORLABEL:\t{}\nNum Fits Files:\t{}\nNum Unc Files:\t{}'.format\
          (header_test['AORLABEL'], len(fitsFilenames), len(uncsFilenames)))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Check that the majority of data is an inlier
Can you write Python code for it?

inlier_master = array(list(example_wanderer_median.inliers_Phots.values())).mean(axis=0) == 1.0

((~inlier_master).sum() / inlier_master.size)*100


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Compute the PLD components -- normalized and store the PLD vectors
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Compute the RMS in the raw data as a function of the apeture radius
Can you write Python code for it?

color_cycle = rcParams['axes.prop_cycle'].by_key()['color']

ax = figure().add_subplot(111)
for key in example_wanderer_median.flux_TSO_df.keys():
    aperRad = float(key.split('_')[-2])
    ax.scatter(aperRad, scale.mad(np.diff(example_wanderer_median.flux_TSO_df[key])), color=color_cycle[0])

ax.set_xlabel('Aperture Radius')
ax.set_ylabel('MAD( Diff ( Flux ) )')

print('Entire Pipeline took {} seconds'.format(time() - startFull))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Compute the sigma-clipping outliers for plotting purpose
Can you write Python code for it?

nSig       = 10.1
medY       = median(example_wanderer_median.centering_GaussianFit.T[y])
medX       = median(example_wanderer_median.centering_GaussianFit.T[x])
stdY       = std(example_wanderer_median.centering_GaussianFit.T[y])
stdX       = std(example_wanderer_median.centering_GaussianFit.T[x])

ySig = 4
xSig = 4
outliers   = (((example_wanderer_median.centering_GaussianFit.T[y] - medY)/(ySig*stdY))**2 + \
              ((example_wanderer_median.centering_GaussianFit.T[x] - medX)/(xSig*stdX))**2) > 1


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Compute the time series with static aperture radii only
Can you write Python code for it?

print('Iterating over Background Techniques, Centering Techniques, Aperture Radii' + '\n')
centering_choices  = ['Gaussian_Fit']#, 'Gaussian_Mom', 'FluxWeighted']#, 'LeastAsymmetry']
background_choices = ['AnnularMask']#example_wanderer_median.background_df.columns
staticRads         = np.arange(1, 6,0.5)#[1.0 ]# aperRads = np.arange(1, 6,0.5)

for staticRad in tqdm_notebook(staticRads, total=len(staticRads), desc='Static'):
    example_wanderer_median.mp_compute_flux_over_time_varRad(staticRad, varRad=None, centering_choices[0], background_choices[0], useTheForce=True)


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Convert Wanderer output to Skywalker input
I made up this loop and did not test it
Keys for skywalker input
Can you write Python code for it?

flux_key = 'phots'
time_key = 'times'
flux_err_key = 'noise'
eff_width_key = 'npix'
pld_coeff_key = 'pld'
ycenter_key = 'ycenters'
xcenter_key = 'xcenters'
ywidth_key = 'ywidths'
xwidth_key = 'xwidths'


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Create Beta Variable Radius
Can you write Python code for it?

example_wanderer_median.mp_compute_flux_over_time_betaRad()

print('Entire Pipeline took {} seconds'.format(time() - startFull))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Double check for NaNs
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Flux Weighted Centroiding -- just to say we tried it
Can you write Python code for it?

print('Fit for All Centers: Flux Weighted, Gaussian Fitting, Gaussian Moments, Least Asymmetry' + '\n')
example_wanderer_median.fit_flux_weighted_centering()


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Identifier Strong Outliers
Can you write Python code for it?

print('Find, flag, and NaN the "Bad Pixels" Outliers' + '\n')
example_wanderer_median.find_bad_pixels()


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Load Wanderer Class
Necessary Constants Spitzer
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Make sure that there are only a handful (<< 1%) of outliers
Can you write Python code for it?

npix = 3

stillOutliers = np.where(abs(example_wanderer_median.centering_GaussianFit - medGaussCenters) > 4*sclGaussCenterAvg)[0]
print(len(stillOutliers))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Master Class for Exoplanet Time Series Observation Photometry
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Plot the background to make sure that the (to be subtracted) flux is stable overtime
Can you write Python code for it?

fig = figure(figsize=(20,10))
ax  = fig.add_subplot(111)
ax.plot(example_wanderer_median.timeCube, example_wanderer_median.background_CircleMask,'.',alpha=0.2)
ax.plot(example_wanderer_median.timeCube, example_wanderer_median.background_Annulus,'.',alpha=0.2)
ax.plot(example_wanderer_median.timeCube, example_wanderer_median.background_MedianMask,'.',alpha=0.2)
ax.plot(example_wanderer_median.timeCube, example_wanderer_median.background_KDEUniv,'.',alpha=0.2)
ax.axvline(example_wanderer_median.timeCube.min()-.01+0.02)
ax.set_ylim(-25,100)


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Plot the full extent of the data to show that DBSCAN was able to identify the inliers correctly
Can you write Python code for it?

fig = figure(figsize=(6,6))
ax  = fig.add_subplot(111)

medGaussCenters   = median(example_wanderer_median.centering_GaussianFit,axis=0)
sclGaussCenters   = scale.mad(example_wanderer_median.centering_GaussianFit)
sclGaussCenterAvg = np.sqrt(((sclGaussCenters**2.).sum()))

yctrs = example_wanderer_median.centering_GaussianFit.T[y]
xctrs = example_wanderer_median.centering_GaussianFit.T[x]

nSigmas         = 5
for nSig in linspace(1,10,10):
    CircularAperture(medGaussCenters[::-1],nSig*sclGaussCenterAvg).plot(ax=ax)

for dbsOpt in dbs_options:
    ax.plot(xctrs[dbsPred==dbsOpt], yctrs[dbsPred==dbsOpt],'.',zorder=0, ms=1)


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Plot the inliers (blue) vs outliers (not blue)
Can you write Python code for it?

ax = figure().add_subplot(111)
cx, cy = example_wanderer_median.centering_GaussianFit.T[x],example_wanderer_median.centering_GaussianFit.T[y]
ax.plot(cx,cy,'.',ms=1)
ax.plot(cx[outliers],cy[outliers],'.',ms=1)
# ax.plot(median(cx), median(cy),'ro',ms=1)
ax.set_xlim(medX-nSig*stdX,medX+nSig*stdX)
ax.set_ylim(medY-nSig*stdY,medY+nSig*stdY)


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Select the "class" dbsClean == 0 for the inliers
Can you write Python code for it?

dbsClean  = 0
dbsKeep   = (dbsPred == dbsClean)

nCores = example_wanderer_median.nCores
start = time()
example_wanderer_median.mp_measure_background_annular_mask()
print('AnnularBG took {} seconds with {} cores'.format(time() - start, nCores))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
The following code is a copy/paste from a different notebook of mine.
This is the code I used to make the for loop above
If the for loop does not work, try / check this
Can you write Python code for it?

timeCube = example_wanderer_median.timeCube
phots_array = example_wanderer_median.flux_TSO_df.values
PLDFeatures = example_wanderer_median.PLD_components.T

try:
    inliers_Phots = example_wanderer_median.inliers_Phots.values()
except:
    inliers_Phots = np.ones(photsLocal.shape)

try:
    inliers_PLD = example_wanderer_median.inliers_PLD.values()
except:
    inliers_PLD = np.ones(PLDFeatureLocal.shape)

# Gaussian_Fit_AnnularMask_rad_2.5_0.0

staticRad = '2.5' # Need to cycle over all possible values here: [1.0, 1.5, 2.0, ..., 5.5]
varRad = '0.0'
key_flux_now = 'Gaussian_Fit_AnnularMask_rad_'+staticRad+'_'+varRad
phot_select = np.where(example_wanderer_median.flux_TSO_df.keys() == key_flux_now)[0][0]

inliersMaster = array(list(inliers_Phots)).all(axis=0) # Need to Switch `axis=0` for Qatar-2
inliersMaster = inliersMaster * inliers_PLD.all(axis=1)

nSig = 6 # vary this as desired

if inliersMaster.all():
    # If inliersMaster keeps ALL values, then double check with 3D inlier flagging
    print('Working on AOR {}'.format(AORNow))
    cy_now, cx_now        = example_wanderer_median.centering_GaussianFit.T
    phots_now             = phots_array[:,phot_select]

    phots_clipped         = clipOutlier2D(phots_now, nSig=nSig)
    cy_clipped, cx_clipped= clipOutlier2D(transpose([cy_now, cx_now]),nSig=nSig).T
    arr2D_clipped         = transpose([phots_clipped, cy_clipped, cx_clipped])

    # 3D inlier selection
    inliersMaster = (phots_clipped == phots_now)*(cy_clipped==cy_now)*(cx_clipped==cx_now)
else:
    print("this box is just to double check -- keeping all inlier flags from above"

ypos, xpos = clipOutlier2D(transpose([example_wanderer_median.centering_GaussianFit.T[y][inliersMaster], \
                                           example_wanderer_median.centering_GaussianFit.T[x][inliersMaster]])).T

npix = sqrt(example_wanderer_median.effective_widths[inliersMaster])

flux_c = phots_array[:, phot_select][inliersMaster]

# noise_c = np.sqrt(flux_c) # Photon limit
noise_c =...


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Things that DO change with respect to aperture radii
Can you write Python code for it?

for phot_select, key_flux_now in tqdm(enumerate(example_wanderer_median.flux_TSO_df.keys())):
    if key_flux_now[-3:] == 0.0: # only do static radii
        flux_c = example_wanderer_median.flux_TSO_df[key_flux_now].values[inliersMaster]
        noise_c = example_wanderer_median.noise_TSO_df[key_flux_now].values[inliersMaster]

        output_dict = {time_key: time_c,
                       flux_key: flux_c,
                       flux_err_key: noise_c,
                       eff_width_key: npix_c,
                       xcenter_key: xpos_c,
                       ycenter_key: ypos_c,
                       xwidth_key: xwidth_c,
                       ywidth_key: ywidth_c,
                       pld_coeff_key: pld_comp_c}

        # This creates 1 joblib output file for one static aperture radius -- need to be cycled from above: change `staticRad = '2.5'` to new radius
        joblib.dump(output_dict, '{}_full_output_for_skywalker_pipeline_{}_{}_{}.joblib.save'.format(planet_dir_name, channel, staticRad, varRad))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Things that DON'T change with respect to aperture radii
Can you write Python code for it?

timeCube = example_wanderer_median.timeCube
phots_array = example_wanderer_median.flux_TSO_df.values
PLDFeatures = example_wanderer_median.PLD_components.T

try:
    inliers_Phots = example_wanderer_median.inliers_Phots.values()
except:
    inliers_Phots = np.ones(photsLocal.shape)

try:
    inliers_PLD = example_wanderer_median.inliers_PLD.values()
except:
    inliers_PLD = np.ones(PLDFeatureLocal.shape)

inliersMaster = array(list(inliers_Phots)).all(axis=0) # Need to Switch `axis=0` for Qatar-2
inliersMaster = inliersMaster * inliers_PLD.all(axis=1)

nSig = 6 # vary this as desired for 3D sigma clipping double check

ypos, xpos = clipOutlier2D(transpose([example_wanderer_median.centering_GaussianFit.T[y][inliersMaster], \
                                           example_wanderer_median.centering_GaussianFit.T[x][inliersMaster]])).T

npix = sqrt(example_wanderer_median.effective_widths[inliersMaster])
time_c = timeCube[inliersMaster]
ywidths_c, xwidths_c = example_wanderer_median.widths_GaussianFit[inliersMaster].T
pld_comp_c = example_wanderer_median.PLD_components.T # this is new to Carlos's notebook instance
pld_output_c = np.array(list([time_c]) + list(pld_comp_c))


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Use Advanced clustering algorithms DBSCAN to compute the outliers of the PLD distributions.
Can you write Python code for it?


I am working on the file "Exoplanet_TSO_Pipeline_Median.ipynb".
The first task is:
Use Advanced clustering algorithms DBSCAN to compute the outliers of the flux distribution. This is sensitive the structure in the data (i.e. transit vs outlier), which is not always true with sigma-clipping.
Can you write Python code for it?


I am working on the file "Expected_Error_LLN.ipynb".
The first task is:
Central Limit Theorem - Non identical distributions
Sum of dice rolls and coin tosses
Can you write Python code for it?

def toss_coin(n):
    l = []
    for i in range(n):
        d = random.randint(0,1)
        l += [d]
    return l

n = 100
h = []
for i in range(1000):
    sum = reduce(lambda x, y: x+y, toss_coin(100) + roll_dice(100))
    h.append(sum/100.0)

plt.hist(h)
plt.title("Gaussian Histogram")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.show()


I am working on the file "Expected_Error_LLN.ipynb".
The first task is:
It can be shown that the curve is $\frac{\sigma(X)}{\sqrt(n)}$<br/>
Var(X) = 91/6 for dice rolling
Can you write Python code for it?

# Check whether error^2 approaches var(x)/n for large n
n = 1000
print sim_error(n)
print np.sqrt(2.9167)/np.sqrt(n)


I am working on the file "Experiment.ipynb".
The first task is:
And more...
Can you write Python code for it?


I am working on the file "Experiment.ipynb".
The first task is:
Now, Begin!
when Count > 10
Can you write Python code for it?

X1_train = experts_count[:80000][experts_count.Count>10]

X1_title=tfv.fit_transform(list(X1_train['Title']) \
                         +list(experts_count[80000:]['Title']))

print (type(X1_title))
print (X1_title.shape)

X_train = X1_title[:X1_train.shape[0]]
Y_train = experts_count['Label'][:80000][experts_count.Count>10]
X_test = X1_title[X1_train.shape[0]:]
Y_test = experts_count['Label'][80000:]

# for C=2
lr = LogisticRegression(C=2)
lr.fit(X_train, Y_train)
y = lr.predict(X_test)
accuracy_score(Y_test, y)


I am working on the file "Experiment.ipynb".
The first task is:
Title
start from here
Can you write Python code for it?

Y = experts_count.Label
X_title = experts_count.Title
print (type(Y),type(X_title))

X_title = tfv.fit_transform(list(X_title))

X_title

# for fun
from sklearn.cross_validation import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_title, Y, test_size=.10)

lr = LogisticRegression(C=2)
lr.fit(X_train, Y_train)
y = lr.predict(X_test)
accuracy_score(Y_test, y)


I am working on the file "ExperimentEval.ipynb".
The first task is:
<footer id="attribution" style="float:right; color:#999; background:#fff;">
Sitola seminar 2019</footer>

Variance decomposition

in ANOVA we study different factors and their interactions
single factor means marginalize out all other dependancies

Bias - variance tradeoff

increasing model complexity to remove systematic residuals
uncertainties of parameters rise fast due to correlations

Design of experiments
full and partial "designs"
see for example
https://newonlinecourses.science.psu.edu/stat503/
Simple 4th order polynom
we try to model data with set of polynomes of type $\sum_{i=0}^n a_i x^i$ with increasing degree $n$
true values correspond to 4-th order polynom (quadratic term dominates)
Can you write Python code for it?

x=np.r_[-3:3:20j]
sigy=3.
tres=[0.5,0.2,7,-0.5,0] #skutecne parametry
ytrue=np.polyval(tres,x)
pl.plot(x,ytrue,'k')
y=ytrue+np.random.normal(0,sigy,size=x.shape)
pl.plot(x,y,'*')


I am working on the file "ExperimentEval.ipynb".
The first task is:
Interactive histograms
Can you write Python code for it?

import nbinteract as nbi
def empirical_props(num_plants):
    return none
opts = {'title': 'Distribution of sample proportions',
    'xlabel': 'Sample Proportion','ylabel': 'Percent per unit',
    'xlim': (0.64, 0.84),'ylim': (0, 25), 'bins': 20,}
nbi.hist(empirical_props, options=opts,
         num_plants=widgets.ToggleButtons(options=[100, 200, 400, 800]))


I am working on the file "ExperimentEval.ipynb".
The first task is:
Principal component analysis

"ultimate correlation tool" (C.R.Jenkins) - transforming into space of orthogonal directions

first component $a_1 X$ to maximize variance in chosen direction $a_1$

next
Can you write Python code for it?

%matplotlib inline
from matplotlib import pyplot as pl
import numpy as np
x=np.r_[0:8:100j]
fun=[np.ones_like(x),0.5*x]#0.05*x**2]
fun.append(0.3*np.sin(x*2.))
fun.append(np.exp(-(x-5)**2/2.))
fun.append(np.exp(-(x-2)**2))
nmeas=20
[pl.plot(x,f) for f in fun]
pl.legend(["constant","linear","periodic","low peak","high peak"])
allpars=np.array([np.random.normal(a,0.4*abs(a),size=nmeas) for a in [3,-1,1.5,2,3.1]]) #sada ruznych koefientu jednotlivych komponent


I am working on the file "ExperimentEval.ipynb".
The first task is:
Statistical significance of parameters
t-values for null-hypothesis of individual parameters (polynomial coefficients)
Can you write Python code for it?

errs=[[round(p,5) for p in np.sqrt(r[1].diagonal()[::-1])/sigy] for r in res]
text=["   ".join(["%6.3f"%p for p in abs(res[i][0][::-1]/np.array(errs[i]))]) for i in range(len(res))]
for i in range(1,len(text)+1):
    print("order %i:"%i,text[i-1])


I am working on the file "ExperimentEval.ipynb".
The first task is:
we have 20 sets of parameters, generate combination of basic functions and add some gaussian noise
Can you write Python code for it?

testfunc=allpars.T.dot(fun)
for i in range(len(testfunc)):
    testfunc[i]+=np.random.normal(0,0.3,size=len(x))
pl.plot(x,testfunc[3])
pl.plot(x,testfunc[1]);

renfunc=testfunc-testfunc.mean(1)[:,np.newaxis]
renfunc/=np.sqrt((renfunc**2).sum(1))[:,np.newaxis]
cormat10=renfunc[:10].dot(renfunc[:10].T)
eig10,vecs10=np.linalg.eig(cormat10)
#eig10
prinfunc=vecs10.T.dot(renfunc[:10])
[pl.plot(p) for p in prinfunc[:4]];


I am working on the file "Experiment_1.ipynb".
The first task is:
By trial and error, I ended up picking index 4341 where the image matches the left turn nicely.
Can you write Python code for it?

import os
from PIL import Image

def get_record_and_image(index):
    record = df.iloc[index]
    path = os.path.join('data', record.center)
    return record, Image.open(path)

left_record, left_image = get_record_and_image(4341)
print('Steering angle {}'.format(left_record.steering))
plt.imshow(left_image)


I am working on the file "Experiment_1.ipynb".
The first task is:
Let's do the same with the hard right turn (steering angle > 0.5).
Can you write Python code for it?


I am working on the file "Experiment_1.ipynb".
The first task is:
Now I need to massage the images and corresponding steering angles to form that is usable in model training.
Can you write Python code for it?

X_train = [np.array(image) for image in [left_image, right_image, straight_image]]
X_min = np.min(X_train)
X_max = np.max(X_train)
X_normalized = (X_train - X_min) / (X_max - X_min) - 0.5
y_train = np.array([record['steering'] for record in [left_record, right_record, straight_record]])

from random import randrange

def generator():
    while 1:
        i = randrange(3)
        # Create a one item batch by taking a slice
        yield X_normalized[i:i+1], y_train[i:i+1]

model.compile('adam', 'mse')
history = model.fit_generator(generator(), samples_per_epoch=1000, validation_data=(X_normalized, y_train), nb_epoch=10, verbose=2)


I am working on the file "Experiment_1.ipynb".
The first task is:
Now I need to pick an record for driving straight. There should be plenty of choices to pick from, so some random exploration of the choices is probably the best way to find one that looks ok.
Can you write Python code for it?

# I used this code to pick random images until I found one I liked
#index = df[(df['steering'] > -0.1) & (df['steering'] < 0.1)].sample(n=1).iloc[0].name
#print('Index', index)

straight_record, straight_image = get_record_and_image(796)
plt.imshow(straight_image)


I am working on the file "Experiment_1.ipynb".
The first task is:
Seems that we are mostly steering straight here.
Step 2: Picking the training images
Now I need to pick three images from the sample driving data that correspond to steering left, right and straight. This set of images should be enough to see that the model is able to learn the differences between the images if it can predict the different steering angles correctly.
Let's first get the indices
of records where steering angle is hard left (< -0.5).
Can you write Python code for it?


I am working on the file "Experiment_1.ipynb".
The first task is:
Training hits zero validation loss after epoch 5, i.e., it should have learned the data perfectly. Lets see how well the model predicts.
Can you write Python code for it?

for X, y in zip(X_expand, y_train):
    print('Actual steering angle {} model prediction {}'.format(y, model.predict(X)[0][0]))


I am working on the file "Experiment_2.ipynb".
The first task is:
A bit better but even after 20 epochs not that much of an improvement. I begin to suspect that I need to increase the model's complexity quite a bit. At this point I will try to replicate the architecture from the NVidia paper (http://images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf) and see what kind of difference it makes.
Can you write Python code for it?

model_4 = Sequential()
model_4.add(Convolution2D(24, 5, 5, border_mode='valid', subsample=(2, 2), input_shape=(80, 160, 3)))
model_4.add(Activation('relu'))
model_4.add(Convolution2D(36, 5, 5, border_mode='valid', subsample=(2, 2)))
model_4.add(Activation('relu'))
model_4.add(Convolution2D(48, 5, 5, border_mode='valid', subsample=(2, 2)))
model_4.add(Activation('relu'))
model_4.add(Convolution2D(64, 3, 3, border_mode='valid'))
model_4.add(Activation('relu'))
model_4.add(Convolution2D(64, 3, 3, border_mode='valid'))
model_4.add(Activation('relu'))
model_4.add(Flatten())
model_4.add(Dense(100))
model_4.add(Activation('relu'))
model_4.add(Dense(50))
model_4.add(Activation('relu'))
model_4.add(Dense(10))
model_4.add(Activation('relu'))
model_4.add(Dense(1))
model_4.add(Activation('tanh'))

layer_info(model_4)

train(model_4)
sample_predictions(model_4)

model_4 = Sequential()
model_4.add(Convolution2D(24, 5, 5, border_mode='valid', subsample=(2, 2), input_shape=(80, 160, 3)))
model_4.add(Activation('relu'))
model_4.add(Dropout(0.5))
model_4.add(Convolution2D(36, 5, 5, border_mode='valid', subsample=(2, 2)))
model_4.add(Activation('relu'))
model_4.add(Dropout(0.5))
model_4.add(Convolution2D(48, 5, 5, border_mode='valid', subsample=(2, 2)))
model_4.add(Activation('relu'))
model_4.add(Dropout(0.5))
model_4.add(Convolution2D(64, 3, 3, border_mode='valid'))
model_4.add(Activation('relu'))
model_4.add(Dropout(0.5))
model_4.add(Convolution2D(64, 3, 3, border_mode='valid'))
model_4.add(Activation('relu'))
model_4.add(Dropout(0.5))
model_4.add(Flatten())
model_4.add(Dense(100))
model_4.add(Activation('relu'))
model_4.add(Dense(50))
model_4.add(Activation('relu'))
model_4.add(Dense(10))
model_4.add(Activation('relu'))
model_4.add(Dense(1))
model_4.add(Activation('tanh'))

layer_info(model_4)

train(model_4)
sample_predictions(model_4)

model_4 = Sequential()
model_4.add(Convolution2D(24, 5, 5, border_mode='valid', subsample=(2, 2), input_shape=(80, 160, 3)))
model_4.add(Activation('relu'))
model_4.add(Convolution2D(36, 5,...


I am working on the file "Experiment_2.ipynb".
The first task is:
Now I need to create the actual training data, X_train and y_train. I will just read all the images and store them as NumPy arrays to X_train. Similary, I read the corresponding steering angles and store them to y_train.
Note: I ended up scaling the images down to half size to conserve memory and speed up training. This was also mentioned in the project cheat sheet (https://carnd-forums.udacity.com/questions/26214464/behavioral-cloning-cheatsheet).
Can you write Python code for it?

from tqdm import tqdm

X_train = []
y_train = []
for i in tqdm(range(len(df))):
    record, image = get_record_and_image(i)
    image = image.resize((image.width // 2, image.height // 2))
    X_train.append(np.array(image))
    image.close()
    y_train.append(record['steering'])


I am working on the file "Experiment_2.ipynb".
The first task is:
Step 2: Create training set
Can you write Python code for it?

import numpy as np
import pandas as pd

df = pd.read_csv('data/driving_log.csv')


I am working on the file "Experiment_2.ipynb".
The first task is:
Step 3: Train the model
Here I use all the data from the sample training data, 8036 images and their steering angles. Instead of using the training data generator as in the experiment 1, I just give the whole training set to model.fit and let it split it to training and validation sets. After training, I save the model so it can be loaded to the simulator for testing if the training seems to proceed well.
Can you write Python code for it?

import keras.optimizers

def train(model, nb_epoch=10, learning_rate=0.001):
    adam = keras.optimizers.Adam(lr=learning_rate)
    model.compile(loss='mse', optimizer=adam)
    model.fit(X_normalized, y_train, validation_split=0.2, nb_epoch=nb_epoch, verbose=2)
    model.save('model.h5')

train(model)


I am working on the file "Experiment_2.ipynb".
The first task is:
The performace is even poorer now so the model is probably not complex enough to learn the given data set. I could increase the layer dimensions directly, but there is another way: remove the pooling layers. Pooling is analogous to downsampling and it reduces the amount of weights in the model. Let's strip the pooling layers and see what happens.
Can you write Python code for it?

model_3 = Sequential()
model_3.add(Convolution2D(6, 5, 5, border_mode='valid', subsample=(5, 5), input_shape=(80, 160, 3)))
model_3.add(Dropout(0.5))
model_3.add(Activation('relu'))
#model_3.add(MaxPooling2D(pool_size=(2, 2)))
model_3.add(Convolution2D(16, 5, 5, border_mode='valid'))
model_3.add(Dropout(0.5))
model_3.add(Activation('relu'))
#model_3.add(MaxPooling2D(pool_size=(2, 2)))
model_3.add(Flatten())
model_3.add(Dense(120))
model_3.add(Activation('relu'))
model_3.add(Dense(84))
model_3.add(Activation('relu'))
model_3.add(Dense(1))
model_3.add(Activation('tanh'))

layer_info(model_3)

train(model_3, 20)
sample_predictions(model_3)


I am working on the file "Experiment_2.ipynb".
The first task is:
The sample predictions do not look very good. Some tweaks to the model are in place.
Step 4: Tweaking the model
So what could be done to the model to improve it? Basically there are three different approaches for changing the model:

Keep the model as it is, but try to improve its generalization ability
Keep the current architecture, but increase the amount of weights
Do some changes to the model's architecture

Before going for options 2 or 3, let's consider option 1 as it is more conservative than the other. A simple way to try to increase the generalization ability is add dropout layers, which force the model to learn redundant connections. Let's try that.
Can you write Python code for it?

from keras.layers import Dropout

model_2 = Sequential()
model_2.add(Convolution2D(6, 5, 5, border_mode='valid', subsample=(5, 5), input_shape=(80, 160, 3)))
model_2.add(Dropout(0.5))
model_2.add(Activation('relu'))
model_2.add(MaxPooling2D(pool_size=(2, 2)))
model_2.add(Convolution2D(16, 5, 5, border_mode='valid', subsample=(2, 2)))
model_2.add(Dropout(0.5))
model_2.add(Activation('relu'))
model_2.add(MaxPooling2D(pool_size=(2, 2)))
model_2.add(Flatten())
model_2.add(Dense(120))
model_2.add(Activation('relu'))
model_2.add(Dense(84))
model_2.add(Activation('relu'))
model_2.add(Dense(1))
model_2.add(Activation('tanh'))

layer_info(model_2)

train(model_2)
sample_predictions(model_2)


I am working on the file "Experiment_2.ipynb".
The first task is:
The validation error does not get much lower after epoch 4 or so, whereas the training error keeps falling. This indicates overtraining and poor generalization ability.
Lets do a bit of random sampling of the predicted steering angles to get a feeling how they match with the actual angles.
Can you write Python code for it?

from random import randrange

def sample_predictions(model):
    for i in range(10):
        index = randrange(len(df))
        X = np.expand_dims(X_normalized[index], axis=0)
        y = y_train[index]
        print('Actual steering angle {} model prediction {}'.format(y, model.predict(X)[0][0]))

sample_predictions(model)


I am working on the file "Experimental Games on Networks/show.ipynb".
The first task is:
Blue networkの場合の選択を図示する。
グラフ上部のアルファベットはプレイヤーの位置を示している。
今回の分析で用いるグラフの共通事項として、x軸にラウンド数、y軸にACTIVEを選択した割合を用いている。
Can you write Python code for it?

x1.show('A')
x1.show('B')
x1.show('C')
x1.show('D')
x1.show('E')


I am working on the file "Experimental Games on Networks/show.ipynb".
The first task is:
p=0.2の場合を表示する。グラフ上部の数字は隣人の数である。
Can you write Python code for it?


I am working on the file "Experimental Games on Networks/show.ipynb".
The first task is:
p=0.8の場合を表示する。グラフ上部の数字は隣人の数である。
Can you write Python code for it?


I am working on the file "Experimental Games on Networks/show.ipynb".
The first task is:
以下、Blue netework, Red network, Brown networkをプレイヤーの位置ごとに比較する。
三つのグラフは上から順にBlue netework, Red network, Brown networkの、
グラフタイトルで示される場所にいた時の選択を示している。
Can you write Python code for it?

x1.show('A')
x2.show('A')
x3.show('A')

x1.show('B')
x2.show('B')
x3.show('B')

x1.show('C')
x2.show('C')
x3.show('C')

x1.show('D')
x2.show('D')
x3.show('D')

x1.show('E')
x2.show('E')
x3.show('E')

data = pd.read_csv('Network2 (accessed 2015-09-14).csv')
data_group = data[['Subsession.round_number','Group.networktype','Player.role','Player.decision', 'Player.nghb','Player.num_active','Player.payoff']]
network_data = data_group.dropna()
network_data.columns = ["round_number", "networktype", "role","decision", "nghb", "num_active","payoff"]
network_data

network_data_np2 = np.asarray(network_data)
y = AnalysisII(network_data_np2)

data = pd.read_csv('Network8 (accessed 2015-09-14).csv')
data_group = data[['Subsession.round_number','Group.networktype','Player.role','Player.decision', 'Player.nghb','Player.num_active','Player.payoff']]
network_data = data_group.dropna()
network_data.columns = ["round_number", "networktype", "role","decision", "nghb", "num_active","payoff"]
network_data_np3 = np.asarray(network_data)
z = AnalysisII(network_data_np3)


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
2. Build the TensorFlow Model
2.1 Define Metadata
Can you write Python code for it?

HEADER = ['userId', 'movieId', 'rating', 'timestamp']
HEADER_DEFAULTS = [0, 0, 0.0, 0]
TARGET_NAME = 'rating'
num_users = ratings_data.userId.max()
num_movies = movies_data.movieId.max()


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
2.2 Define Data Input Function
Can you write Python code for it?

def make_input_fn(file_pattern, batch_size, num_epochs,
                  mode=tf.estimator.ModeKeys.EVAL):

    def _input_fn():
        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=True,
            num_epochs=num_epochs,
            shuffle= (mode==tf.estimator.ModeKeys.TRAIN)
        )
        return dataset

    return _input_fn


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
2.3 Create Feature Columns
Can you write Python code for it?

def create_feature_columns(embedding_size):

    feature_columns = []

    feature_columns.append(
        tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_identity(
                'userId', num_buckets=num_users + 1),
            embedding_size
        )
    )

    feature_columns.append(
        tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_identity(
                'movieId', num_buckets=num_movies + 1),
            embedding_size
        )
    )

    return feature_columns


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
2.5 Create Estimator
Can you write Python code for it?

def create_estimator(params, run_config):

    estimator = tf.estimator.Estimator(
        model_fn,
        params=params,
        config=run_config
    )

    return estimator


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
2.7 Run Experiment with Parameters
Can you write Python code for it?

MODELS_LOCATION = 'models/movieles'
MODEL_NAME = 'recommender_01'
model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)

params  = tf.contrib.training.HParams(
    batch_size=265,
    traning_steps=1000,
    learning_rate=0.1,
    embedding_size=16,
    eval_throttle_secs=0,
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=10000,
    keep_checkpoint_max=3,
    model_dir=model_dir,
)

estimator = train_and_evaluate_experiment(params, run_config)


I am working on the file "Experimental/Movielens Recommendation.ipynb".
The first task is:
5. Export the Model
This needed to receive a userId and produce the embedding for the user.
Can you write Python code for it?

def make_serving_input_receiver_fn():
    return tf.estimator.export.build_raw_serving_input_receiver_fn(
        {'userId': tf.placeholder(shape=[None], dtype=tf.int32)}
    )

export_dir = os.path.join(model_dir, 'export')

if tf.gfile.Exists(export_dir):
    tf.gfile.DeleteRecursively(export_dir)

estimator.export_savedmodel(
    export_dir_base=export_dir,
    serving_input_receiver_fn=make_serving_input_receiver_fn()
)

import os

export_dir = os.path.join(model_dir, "export")
saved_model_dir = os.path.join(
    export_dir, [f for f in os.listdir(export_dir) if f.isdigit()][0])

print(saved_model_dir)

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
)

output = predictor_fn({'userId': [1]})
print(output)


I am working on the file "Experimental/SVMDocs.ipynb".
The first task is:
As you can imagine the generated CSV has the same format as the input CSV with the only exception being the appended prediction column added to the end.
Can you write Python code for it?

df = pd.io.parsers.read_csv(
    'Data/svmoutput.csv',

)

print(df.shape)
print('\n')
print(df.head(10))


I am working on the file "Experimental/SVMDocs.ipynb".
The first task is:
Implementing an SVM from scratch would be a tedious and tricky process. Luckily Scikit-Learn has already done so by creating a python wrapper for the C++ library LibSVM. LibSVM is a very efficient library for running SVM related tasks.
Can you write Python code for it?

from sklearn.svm import SVC
clf = SVC(C=1.0, cache_size=200, class_weight='balanced', coef0=0.0,
decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',
max_iter=-1, probability=True, random_state=None, shrinking=True,
tol=0.001, verbose=False)

clf.fit(X_train,y_train)


I am working on the file "Experimental/SVMDocs.ipynb".
The first task is:
Of the input parameters above, the most important are C, class_weight, gamma and kernel. The purpose of C is to decide the trade off between fitting the model to the training set and maintaining a smooth hyperplane. class_weight simply denotes the structure of the training data provided relative to its classes. Our data is balanced hence we have used that here. gamma corresponds to how much influence a single training point has over the fitting of the hyperplane. In this example we have let sklearn select gamma automatically. Finally the kernel is the function that is responsible for finding the mathematical relationship between the independent feature vectors and corresponding classes. In our case we have selected 'rbf' or 'radial based field'. This kernel allows fitting of a non linear hyperplane to the data. This is useful as the relationship between our features (i.e. avgtemp, avghumid etc) and our classes (Fire, No fire) may not necessarily be linear.
The above code equates to training the model. Predictions can now be made with the following code snippet
Can you write Python code for it?


I am working on the file "Experimental/SVMDocs.ipynb".
The first task is:
Our code in practice
The ideas and code snippets above are the basis for our code. We have opted to use an object orientated approach as this lends us several advantages such as clarity and generalisability. Below is an example of a script that utilises our code to process, standardise, train then test a model. It outputs a prediction for every value in the test dataset along with its probability (calculated through Platt scaling) and the correct value. Finally it outputs the overall accuracy the model achieved when making predictions upon the dataset.
Can you write Python code for it?

'''
Author: Flinn Dolman

@License: MIT

An example script that leverages our code to train a model and make predictions based upon it. Predictions
are printed to stdout and then the model used to make the predictions is saved.
'''
from SVM import SVM
from Standardiser import Standardiser

def Main():
    forecast_loc = 'Data/svminput.csv'
    standard_data = Standardiser()
    standard_data.initialise()
    clf = SVM()
    clf.initialise(standard_data.get_std_X_train(),standard_data.get_std_X_test(),standard_data.get_y_train(),standard_data.get_y_test())
    print('\nThese are the predictions: {}\n'.format(clf.predictions()))
    predictions, probs = clf.predictions()
    y_test = standard_data.get_y_test()

    for i in range(0,len(predictions)-1):
        print('Prediction: {}, with probability: {}, correct value: {}'.format(predictions[i],probs[i], y_test[i]))

    print('Accuracy is: {}%'.format(clf.accuracy()*100))

    fore_Pred, fore_Prob = clf.forecast_Pred(standard_data.loadForecast(forecast_loc))

    standard_data.make_CSV(fore_Pred,fore_Prob,'Data/svmoutputnew.csv')

    clf.saveModel()

if __name__ =="__main__":

    Main()


I am working on the file "Experimental/SVMDocs.ipynb".
The first task is:
This accuracy can be tweaked by changing hyper-parameters used to train the model as well as altering the data that is trained upon by means of changing the seed when splitting the data. Our final model obtains an accuracy of x%.
What about probability though?
When designing a model for a scenario such as a wildfire it would be far more useful to have probability values associated with the classe outcomes predicted. Unfortunately this is not an out of the box functionality of SVM's. Luckily however, it can be achieved by applying a procedure known as Platt scaling (https://en.wikipedia.org/wiki/Platt_scaling). Essentially this approach applies a probability distribution to predicted classes by means of the sigmoid function. In doing so it allows us to associate probabilities of data points being in certain classes. LibSVM, thus by proxy Scikit-Learn has an efficient implementation of this procedure which means it is only a single line to do so:
Can you write Python code for it?


I am working on the file "Experimental/distribution/multi-node/mnist-keras-2/Keras_MNIST_local.ipynb".
The first task is:
Create the directories on GCS
Can you write Python code for it?

import tensorflow as tf
import os

try:
  ROOT_DIR = 'gs://{}'.format(BUCKET_NAME)
except NameError:
  ROOT_DIR = './tutorial'

DATA_DIR = '{}/data'.format(ROOT_DIR)
MODEL_DIR = '{}/{}'.format(ROOT_DIR, MODEL_NAME)
EXPORT_DIR = '{}/{}'.format(ROOT_DIR, MODEL_NAME)
CHECKPOINT_PATH = '{}/checkpoint'.format(MODEL_DIR)

os.environ['BUCKET_NAME']=BUCKET_NAME
os.environ['MODEL_NAME']=MODEL_NAME

# Remove CHECKPOINT_DIR if needed
if tf.gfile.IsDirectory(MODEL_DIR):
  tf.logging.info('delete {}'.format(MODEL_DIR))
  tf.gfile.DeleteRecursively(MODEL_DIR)