I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
QUESTION pour Guillaume:
a quoi correspondent les '' dans la colonne "Pvalue_MMAP_V2_sans_intron_and_Intergenic" (danger)?
Ansewer: cnv for which we have no dangerosity information
Can you write Python code for it?

danger_not_empty = dangers != ''
danger_scores = dangers[danger_not_empty]
danger_scores = np.asarray([danger_score(pstr, pH1_with_apriori) for pstr in danger_scores])

print("We have {:d} empty score !".format(EXPECTED_LINES -1 - len(danger_scores)))

danger_scores.max()

h = plt.hist(danger_scores[danger_scores > sst.scoreatpercentile(danger_scores, 99)], bins=100)


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
TEST score2proba
Can you write Python code for it?

score2prob = create_score2prob(p_cnv)
scores = np.arange(15,50,1)
probs = [score2prob(sc) for sc in scores]
plt.plot(scores, probs)


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Testing dangerosity
Can you write Python code for it?

for k in range(1,30, 30):
    print(cnv[cnv.keys()[k]], ' yields ', dangerosity(cnv[cnv.keys()[k]]))

test_dangerosity_input = [[(1., .5), (1., .5), (1., .5), (1., .5)],
                          [(2., 1.)],
                          [(10000., 0.)]]
test_dangerosity_output = [2., 2., 0]

#print( [dangerosity(icnv) for icnv in test_dangerosity_input]) # == test_dangerosity_output
assert( [dangerosity(icnv) for icnv in test_dangerosity_input] == test_dangerosity_output)


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
Testing the function danger_score
Can you write Python code for it?

def test_danger_score(tests, verbose=True):
    """
    tests: the list of strings that look like "danger"
    results: expected results
    """
    for idx, test in enumerate(tests):
        pstring = test[0]
        func = test[1]
        arg = test[2]
        result = test[3]
        status = True

        try:
            dger = danger_score(pstring, func, arg)
            if verbose: print('dger:',dger)
            if abs(dger - result) < np.finfo('float').eps:
                pass
            else:
                if verbose: print(test, " is failing", dger, results[idx])
                status = False
        except:
            if result == "Failed":
                pass
            else:
                if verbose: print(test, " is failing", results[idx])
                status = False

    return(status)

print(danger_score('1., 0., .5', pH1_with_apriori))
print(danger_score('1.', pH1_with_apriori))
print(danger_score('1.', pH1_with_apriori, {'pi':0.5, 'beta':0.8}))

one_beta_pi = (1. - defaultarg['beta'])*defaultarg['pi']
pH1_res_1 = one_beta_pi / (one_beta_pi + 1. - defaultarg['pi'])
assert abs(pH1_res_1 - pH1_with_apriori(1.)) < np.finfo('float').eps

tests = [('1., 0., .5', pH1_simple, defaultarg,          1.5),
         ('.2, -.01',    pH1_simple, defaultarg,         'Failed'),
         ('0.1, 0.1, .8',pH1_simple, defaultarg,         2.),
         ('.0',          pH1_simple, defaultarg,         1.),
         ('.0',          pH1_with_apriori, defaultarg,   1.),
         ('1.',          pH1_with_apriori, defaultarg,   pH1_res_1)]

assert test_danger_score(tests, verbose=False)


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
To be or not to be a CNV
Can you write Python code for it?

#get the scores
scores = np.asarray([line.split('\t')[i_score] for line in arr[1:]])
assert len(scores) == expected_nb_values

# show first 42 values
scores[:42]

len(np.unique(scores))

# comma to point for floats
tmp = [str2floats(s, comma2point=True)[0] for s in scores]

a = [print(scores[i], tmp[i]) for i in range(1400,1410)]

#h = plt.hist(tmp, bins=100)
tmp = np.asarray(tmp)
print(tmp.shape)
assert tmp.shape[0] == EXPECTED_LINES - 1
# h = plt.hist(tmp[tmp < sst.scoreatpercentile(tmp, 95)], bins=100)

h = plt.hist(tmp[tmp > sst.scoreatpercentile(tmp, 99)], bins=100)

h = plt.hist(tmp[tmp < 50], bins=100)

print("# CNV with score == 0.: ", (tmp==0.).sum())
print("# CNV with score >=15 < 17.5 : ", np.logical_and(tmp >= 15., tmp < 17.5).sum())
tmp.max()


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
transforming the "Pvalue_MMAP_V2_..." into danger score
Can you write Python code for it?

#print(len(scores[danger_not_empty]))
#f,b = plt.hist(dangers[danger_not_empty].astype('float'), bin=50)

def str2floats(value, sep=',', comma2point=False):
    """
    takes a string with one or several p-values
    returns:
    combined score dangerosite
    """
    if comma2point:
        value = value.replace(',', '.')
    lst = value.split(sep)
    lst = [l.rstrip().lstrip() for l in lst if (l != '')]
    try:
        return [float(f) for f in lst]
    except:
        print("value: ", value, "value.split(sep): ", lst)
        raise ValueError

def pH1_with_apriori(alpha, pi=0.5, beta=.8):
    """
    Returns the probability of H1 if we know power and a priori
    """
    return (1 - beta) * pi / ((1 - beta) * pi + alpha * (1 - pi))

def pH1_simple(alpha, pi=None, beta=None):
    """
    pi and beta unsused:
    """
    return (1. - alpha)

defaultarg = {'pi':0.5, 'beta':0.8}

def danger_score(pvalues, pval2score=pH1_simple, argvals=defaultarg):
    """
    take a series of pvalue (one line), return a dangerosity score
    """

    pvalues = np.asarray(str2floats(pvalues))

    assert np.all( np.logical_and(pvalues <= 1., pvalues >=0)), \
            [pvalues, np.logical_and((pvalues <= 1.), (pvalues >=0))]

    """ how to combine the p-values?
    I want a score that scales with the number of gene affected. Per gene, if the p is zero,
    I want my score to be 1. and if the p value is close to 1, the score should be 0.
    To start with, I'm setting the score to be 1-p:

        res = (1. - pvalues).sum()

    The problem with this approach: I would like something closer to the probability of
    CNV to be dangerous, which requires an apriori on the probablility that the CNV is dangerous,
    and the power of the test (leading to the p value that we are using).

        p(CNV dangerous) =  (1 - beta) * pi / ((1 - beta) * pi + alpha * (1 - pi))

    """
    res = np.asarray([pval2score(p, **argvals) for p in pvalues]).sum()
    return res


I am working on the file "CNV_guillaume_huguet.ipynb".
The first task is:
transforms the scores into P(cnv is real)
Can you write Python code for it?


I am working on the file "CO2 Emission.ipynb".
The first task is:
# Joint plot for CO2 emissions from solid fuel consumption in Iran and Turkey
 As there were no values for the gaseous column before 1978, I preferred to present my data using solid fuel emission values instead.
Can you write Python code for it?

SOLID_FUEL_COLUMN_INDEX = 2
a = sns.jointplot(iran_csv.ix[:,SOLID_FUEL_COLUMN_INDEX],
              turkey_csv.ix[:,SOLID_FUEL_COLUMN_INDEX]).set_axis_labels(
    "IRAN: " + iran_csv.columns[SOLID_FUEL_COLUMN_INDEX],
    "TURKEY: " + turkey_csv.columns[SOLID_FUEL_COLUMN_INDEX])
a.savefig("output.png")


I am working on the file "CO2 Emission.ipynb".
The first task is:
As I wanted the emission types be my coloumns and the years be the rows, I used transpose() function. Some data was missing for the last three years which I substituded their value by zero.
Can you write Python code for it?

iran_csv = iran_csv.transpose()
iran_csv = iran_csv.fillna(0)
iran_csv.columns = iran_csv.ix[0,:]
iran_csv = iran_csv.ix[1:,:]
iran_csv.astype(np.float64)
iran_csv.head(5)

turkey_csv = turkey_csv.transpose()
turkey_csv = turkey_csv.fillna(0)
turkey_csv.columns = turkey_csv.ix[0,:]
turkey_csv = turkey_csv.ix[1:,:]
turkey_csv.astype(np.float64)
turkey_csv.head(5)


I am working on the file "CO2 Emission.ipynb".
The first task is:
Distribution plot for CO2 emssions from liquid fuel consumption
Can you write Python code for it?

#Iran (Blue)
sns.distplot(iran_csv.ix[:,1])

#Turkey (Green)
sns.distplot(turkey_csv.ix[:,1])


I am working on the file "CO2 Emission.ipynb".
The first task is:
Importing the CSV files
This CSV file is available on Irans dataset in World bank](https://www.worldbank.org/en/country/iran) and [Turkeys dataset in the same website . I cleaned this data using LibreOffice and kept the important rows to prevent complexity in my code.
Can you write Python code for it?

# Importing Iran`s dataset
IRAN_SOURCE_FILE = 'iran_emission_dataset.csv'
iran_csv = pd.read_csv(IRAN_SOURCE_FILE)
iran_csv.head(5)

# Importing Turkey`s dataset
TURKEY_SOURCE_FILE = 'turkey_emission_dataset.csv'
turkey_csv = pd.read_csv(TURKEY_SOURCE_FILE)
turkey_csv.head(5)


I am working on the file "CO2 temperature analysis.ipynb".
The first task is:
Calculation of temperature sensitivity
From equation (2) we then directly calculate:
Can you write Python code for it?

from math import log
deltaT = dT * log(2) / log(1.0*N/N0)
print("∆T = ", deltaT, "C")


I am working on the file "CO2 temperature analysis.ipynb".
The first task is:
Let's fetch the raw data of $CO_2$ measurements at Mauna Loa from the noaa.gov website:
Can you write Python code for it?

# Only execute this if you want to regenerate the downloaded file
open("data/co2_mm_mlo.txt", "wb").write(urllib.request.urlopen("ftp://ftp.cmdl.noaa.gov/ccg/co2/trends/co2_mm_mlo.txt").read())

D = loadtxt("data/co2_mm_mlo.txt")
years = D[:, 2]
average = D[:, 3]
interpolated = D[:, 4]
trend = D[:, 5]


I am working on the file "CO2 temperature analysis.ipynb".
The first task is:
Let's get numbers for $N$ (year 2010) and $N_0$ (year 1980) by reading it off the $y$-axis:
Can you write Python code for it?

idx1980 = sum(years < 1980)
idx2010 = sum(years < 2010)
N0 = trend[idx1980]
N  = trend[idx2010]
print("N0 = %.2f ppm  (year %.3f)" % (N0, years[idx1980]))
print("N  = %.2f ppm  (year %.3f)" % (N,  years[idx2010]))


I am working on the file "CO2 temperature analysis.ipynb".
The first task is:
Warming over the past 30 years using satellite measuremenets (see a separate notebook for this calculation):
Can you write Python code for it?


I am working on the file "CO2 temperature analysis.ipynb".
The first task is:
We'll use the satellite measurements, as arguably they have less systematic errors.
Temperature difference:
Can you write Python code for it?


I am working on the file "COGSTools/COGS 2201 Spring 2017 -- PS#3.ipynb".
The first task is:
After you run the cell above, test the cohort model with a fragment of English:
Can you write Python code for it?

EnglishWords = ['cathedral', 'cat','dog','catheter']
context      = "I went to London and saw St. Paul's"
word         = 'cathedral'
testRun      = cohortModel(word, EnglishWords)


I am working on the file "COGSTools/COGS 2201 Spring 2017 -- PS#3.ipynb".
The first task is:
Now show what the timeCourse of the word recognition process is for our testRun. It is  less critical to understand how this next chunk works. It is just printing the output (and part of the input) in an interpretable fashion.
Can you write Python code for it?

for letter, timeSlice in zip(word, testRun):
    print("After letter {}, the activated words are".format(letter),timeSlice)


I am working on the file "COGSTools/COGS 2201 Spring 2017 -- PS#3.ipynb".
The first task is:
Now, answer some questions about the model!
 Question 1: Describe the above implementation of the cohort model in terms of Marr's three levels. One or two sentences per level will suffice.
Question 2: How does the run-time of the model depend on the size of the lexicon, EnglishWords, that it is given? In other words, how does the length of time the computer takes to run cohortModel depend on the number of words in EnglishWords? You can answer this in an informal way -- no need to get mathematically precise. You can answer this just by analyzing the code, but you could also test your analysis by running the model with lexicons of different sizes. If you pursue this path, you may find what I wrote below helpful...
Can you write Python code for it?

import time

def squares_to_n(n):
    squares = []
    for x in range(n):
        squares.append(x)
    return squares

start_time = time.time()
my_squares = squares_to_n(10**7)
duration = time.time() - start_time

print(duration)


I am working on the file "COGSTools/COGS 2201 Spring 2017 -- PS#3.ipynb".
The first task is:
Test the model. Of course, if you programmed it right, this next cell should output 'cat'.
Can you write Python code for it?

test_object = ['meows','small','plays-with-string']
dog_or_cat(test_object)


I am working on the file "COGSTools/COGS 2201 Spring 2017 -- PS#3.ipynb".
The first task is:
Try more test-cases. Again, if your function was written appropriately, then running the next chunk should not generate any errors!
Can you write Python code for it?

assert dog_or_cat(['meows','small','plays-with-string']) == 'cat'
assert dog_or_cat(['barks','big',  'plays-with-bone'])   == 'dog'
assert dog_or_cat(['meows','small','plays-with-bone'])   == 'cat'


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
(In case it isn't clear # introduces a 'comment'. It's basically a way of telling the computer not to interpret anything after it as Python code. Instead, you (the programmer) write this or that comment to help explain the program, so someone reading it later can understand it.)
int is integer. A float (short for 'floating point' number) is not really simply a number with a decimal point, but it will suffice for now to think of them that way.
The next data type we are already familiar with from our logic lectures!
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
(This is not a terribly important concept in programming or Python -- I just mention it to preempt possible confusion about the notebook's behavior.)
Fundamental types
Here, we'll just think of 'types' as the 'kinds of things' a programming language has. Python's fundamental types (some of which we've seen):
Can you write Python code for it?

# integer
x = 1
type(x)

# floating-point
x = 1.0
type(x)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
...with this...:
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
A slight tangent:
If it isn't clear, the reason the loop gives computing systems such power is that it lets the system process a large amount of information with a small amount of code/instructions if the process is repetitive in some way.
This is practically important for the "real world" (e.g., automating boring clerical work), but we could argue that it also has implications for cognitive science. That is, much of our cognition involves applying the same process to a sequence of elements. The problem set will exemplify this a little bit.
Questions??
Fin
There is, without a doubt, much, much more to Python. I haven't even covered all of what I would consider "the basics". But using just what I've given you, you really can go quite far. Certainly it will suffice for the problem set.
If you are interested in learning more programming, you could talk to me or Garrett. And/or, you might consider taking CSE classes! Many professors, graduate students, researchers, etc. elsewhere in the cognitive science community also use programming for cognitive science applications, so they might provide good learning experiences, as well.
Questions about anything we covered today?
Like much of the material in this course, to solidify your understanding of and facility with the above material, you probably need to practice. So, you might try some...
Optional exercises
For each exercise, there is a cell where you should try to fill in the function (delete the ...). The following cell acts as a test, run it to see if your function is correct.
Each exercise can be solved with just a few lines, using almost exclusively the Python functionality we covered above (when you perhaps need more stuff, I tell you what it is).
Write a function distance that takes two tuples, each containing two numbers (x, y). Compute the distance between them using the formula
$$
d = \sqrt{(x_1 - x_2)^2 + (y_1 - y_2)^2}
$$
As a hint I've imported the sqrt function from the math module.
Can you write Python code for it?

from math import sqrt
sqrt(9)

def distance(a, b):
    ...

assert distance((0, 0), (0, 0)) == 0
assert distance((0, 0), (1.5, 0)) == 1.5
assert distance((0, 0), (1, 1)) == sqrt(2)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
But this does:
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
But you actually can't use indexing to change a value of a string! (For reasons we won't go into.)
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Define a function is_palindrome that returns True for inputs that are the same backwards and forwards.
Might find the following trick helpful:
Can you write Python code for it?

my_string = 'palindrome'
my_string[::-1]

def is_palindrome(word):
    ...

assert is_palindrome('radar')
assert is_palindrome('x')
assert is_palindrome('xx')
assert is_palindrome('')
assert not is_palindrome('abc')


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Define a function max_of_three that takes three numbers and returns the largest of them. One way to solve this is by using if/else. But probably easier: remember the min() function? Given that that function exists, what other function probably exists?
Can you write Python code for it?

def max_of_three(a, b, c):
    ...

assert max_of_three(1, 2, 3) == 3
assert max_of_three(3, 2, 1) == 3
assert max_of_three(1, -1, 1) == 1
assert max_of_three(100.25, 0, -1.5) == 100.25


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
For list objects:
Can you write Python code for it?

the_numbers.index(23)

the_numbers.reverse()
the_numbers

the_numbers.sort()
the_numbers


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Having different data types is important because they let you do different things. That is, different types afford different actions, or operations (which we'll cover in a minute). (Quickly, int and float let you do math-y things, bool logic-y things, and str text-y/language-y thing.)
Questions so far? We do have time to field some questions throughout!
Converting types (a.k.a. type casting)
What happens if we try to add the integer 1, and the string '1'? Before we execute this next line, any guesses what the output will be?
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Hopefully this reminds you of statement logic! We could also just define atomic letters with truth-values and compute the truth-value of complex propositions like this:
Can you write Python code for it?

A = True
B = False
C = False

(A or B) and not (C)

#A->B
(not A) or B

#A<>B
(A and B) or (not A and not B)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
I'll go over more of the things you can do with strings later, in the "Objects" section below.
Questions?
Above we covered data types like bool, int, float, and str. Sometimes, you want to put a whole bunch of instances of such types into a single container. This is where we could use:
Lists
They are exactly what they sound like! A list is an ordered sequence of any combination of data types.
Lists are defined by square brackets [], with individual elements inside separated by commas.
Can you write Python code for it?

the_numbers = [4, 8, 15, 16, 23, 42]
type(the_numbers)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
If you screw up the indentation, Python will let you know.
Can you write Python code for it?

x = 1
    print(x)

if x == 1:
print('yep')


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Logical operators
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
More generally, any type of data structure that we can "iterate" over can be turned into a list. (More on the notion of 'iteration' in a minute.)
Can you write Python code for it?

r = 'russell'
print(len(r))
r = 'richie'
print(len(r))

list(name)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Note that a function like char_freq already exists in Python, as collections.Counter, but see if you can recreate it.
Can you write Python code for it?

def char_freq(sentence):
    ...

from collections import Counter

sentence = 'The quick brown fox jumps over the lazy dog.'.lower()
assert char_freq(sentence) == Counter(sentence)

sentence = 'aksjdhjsdbvjbfajsdhbajfgsodig'
assert char_freq(sentence) == Counter(sentence)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Note that the type is associated with the value, not the name.
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Note: indexing works similarly for strings.
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Objects
We've seen the word 'object' pop up here and there. In Python, pretty much everything is an object. Every instance of some broader class is an object. That includes literals (like 1 or 'a' or ['a',2]), variables, and more. But for the purposes of this class, don't fret terribly much about the precise definition of 'object'...
What's important is this -- objects have values and functions (technically called methods) attached to them that are accessed with a . (dot). We already saw a couple instances of these functions with .append and .extend. Some other important ones:
For string objects:
Can you write Python code for it?

name.endswith('ie')

' '.join(['Russell','Richie'])

name.upper()


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
So 0-based indexing means an index of n gets you the n+1th element.
If 0-based indexing seems weird, think of it like this:

Think of the indexes as 'fence posts' between the elements of the list.
Rest assured, this does become more intuitive as you use it.
Use [#:#] to get a range of elements.
Can you write Python code for it?

the_numbers[2:5] # get everything between the 2 and 5 fenceposts

the_numbers[:2] # get everything up to the 2 fencepost

the_numbers[2:] # get everything after the 2 fencepost


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
The range function is useful for creating simple lists. Although it behaves like a list, it's technically not a list unless you convert it to one explicitly.
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
There are no logical operators/connectives for the conditional or biconditional in base (ordinary) Python, but do you really need them? Why or why not?
String operators
You can concatenate strings:
Can you write Python code for it?

first_name = 'Russell'
last_name  = 'Richie'
full_name  = first_name + ' ' + last_name # ' '  is a string with a space!
print(full_name)


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Trying to understand your error messages is key to understanding why your program isn't working as expected!!
Googling your error message will usually lead to helpful information. In fact, if you can google, you can learn to program. Any question you have about programming has been asked and answered online 100 times before!
Real quick: in case you haven't noticed...if the last line of code in a cell is simply an expression -- and not an assignment -- jupyter notebook prints the result of that expression to the screen. Contrast this...:
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
Write a function factorial that computes the factorial of a number.
Factorial is the ! operation. For example $4! = 4 \cdot 3 \cdot 2 \cdot 1 = 24$.
Include the special case that $0! = 1$.
Can you write Python code for it?

def factorial(x):
    ...

assert factorial(0) == 1
assert factorial(1) == 1
assert factorial(2) == 2
assert factorial(3) == 6
assert factorial(4) == 24
assert factorial(5) == 120


I am working on the file "COGSTools/Cognitive Computing 1 and 2 -- Python programming.ipynb".
The first task is:
You can build an empty dictionary -- which you will fill in later in your program -- with:
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
Almost 70,000 words now...
Can you write Python code for it?

start_time     = time.time()
categorization = d_or_t_final(250, preceding_context='scra', english_words=my_words)
new_duration   = time.time() - start_time

print(new_duration)
print(new_duration/duration)


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
But if my_words was a very long list.....again, don't worry about how exactly I'm reading in this file of words...
Can you write Python code for it?

with open('wordlist.txt') as f:
    my_words = [x.strip() for x in f.readlines()]
print(my_words[:10])
print(len(my_words))


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
But this line fails.
Can you write Python code for it?


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
If our function is behaving properly, the above cell shouldn't raise any errors.
Let's now think of the time complexity of d_or_t_final. Does anyone have any guesses about which input to the function could get bigger and bigger and bigger, and hence slow down the model? (As a reminder, the inputs are preceding_vowel_length, preceding_context, and english_words.) Why would size of that particular input influence the speed of the model? (This very well might be hard to guess!)
.
.
.
.
.
The size of english_words will influence how long the model takes to run. This is because in the first two if statements, the program has to check the words in english_words one by one and check if the there's a match to the two possible completions in it. The longer the list is, the longer the search takes.
This can be deduced from reasoning about the code, but we can also simply show it with testing. We'll just record how long it takes the function to run with english_words of two different sizes!
(Don't worry about some of the new functionality in this next cell like import time.)
Can you write Python code for it?

import time

start_time = time.time()
my_words = ['read','bead','beat','cat']
categorization = d_or_t_final(250, preceding_context='scra',english_words = my_words)
duration   = time.time() - start_time

print(duration)


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
So that is a computational model of cognition because it is a computationally-explicit theory of the process by which we accomplish some cognitive function. Put another way, it specifies an algorithm by which we do this particular function.
Awesome! We solved speech recognition!
Well, not really. Let's also consider the following: people speaker at different speeds, and correspondingly, people vary in their typical boundaries between consonants like 'd' and 't'.
Okay, let's try to account for that:
Can you write Python code for it?

def d_or_t_extended(preceding_vowel_length, speaker):

    if speaker == 'Russell': # imagining if I was an atypically fast speaker
        boundary = 150
    else:
        boundary = 200

    if preceding_vowel_length > boundary:
        return 'd'
    else:
        return 't'

print( d_or_t_extended( preceding_vowel_length=175, speaker='Russell' ) )
print( d_or_t_extended( preceding_vowel_length=175, speaker='Garrett' ) )


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
Suppose the d/t sound comes after 'rea?' -- the '?' is representing the d/t sound that needs to be categorized. As far as I know, 'read' is a word, but 'reat' is not. So, a categorization function could theoretically rely on that information and be less likely to categorize a sound as 't' after 'rea'.
But both of our current d_or_t models are not sensitive to that information. We could say that they are modularized with respect to word context of the d/t sound.
So of course, the addition of the variable preceding does not change the operation of d_or_t in this next cell.
Can you write Python code for it?

preceding = 'rea'
whole_word = preceding + d_or_t(150)
print(whole_word)


I am working on the file "COGSTools/Cognitive Computing 3 -- Cognition illuminated.ipynb".
The first task is:
That shows how the function d_or_t is encapsulated from certain information outside the function. Modularity also often includes the reverse relation: that the function doesn't expose all of its information to the outside. In modularity of mind, we'll learn that that is called cognitive impenetrability.
Our extended function illustrates this. Inside the extended function, a variable called boundary is defined. However, that information only exists inside the function. If we try to access that variable outside the function, we fail.
Can you write Python code for it?

d_or_t_extended( preceding_vowel_length=175, speaker='Russell' )
print(boundary)


I am working on the file "COHERENT.ipynb".
The first task is:
COHERENT event rate
Calculate number of CEvNS signal events at COHERENT (in bins of 2 PE)
Can you write Python code for it?

#Nuclear properties for Cs and I
A_Cs = 133.0
Z_Cs = 55.0

A_I = 127.0
Z_I = 53.0

#Mass fractions
f_Cs = A_Cs/(A_Cs + A_I)
f_I = A_I/(A_Cs + A_I)

mass = 14.6 #target mass in kg
time = 308.1 #exposure time in days

PEperkeV = 1.17 #Number of PE per keV

#Get the differential rate function from the CEvNS module
#Note that this function allows for an extra vector mediator,
#but the default coupling is zero, so we'll forget about it
diffRate_CEvNS = CEvNS.differentialRate_CEvNS

#Differential rates (times efficiency) for the two target nuclei, per PE
dRdPE_Cs = lambda x: (1.0/PEperkeV)*efficiency(x)*mass*time*f_Cs*diffRate_CEvNS(x/PEperkeV, A_Cs, Z_Cs)
dRdPE_I = lambda x: (1.0/PEperkeV)*efficiency(x)*mass*time*f_I*diffRate_CEvNS(x/PEperkeV, A_I, Z_I)

#Calculate number of signal events in each bin in the Standard Model (SM)
PE_bins = np.linspace(0, 50, 26)

N_SM_Cs = np.zeros(25)
N_SM_I = np.zeros(25)
N_SM_tot = np.zeros(25)

for i in tqdm(range(25)):
    N_SM_Cs[i] = quad(dRdPE_Cs, PE_bins[i], PE_bins[i+1], epsabs = 0.01)[0]
    N_SM_I[i] = quad(dRdPE_I, PE_bins[i], PE_bins[i+1], epsabs = 0.01)[0]
    N_SM_tot[i] = N_SM_Cs[i] + N_SM_I[i]

print("Total CEvNS events expected: ", np.sum(N_SM_tot))


I am working on the file "COHERENT.ipynb".
The first task is:
Calculate delta-chisquared over a grid and save to file
Can you write Python code for it?

Ngrid = 101

ulist = np.linspace(-1.0, 1.0, Ngrid)
dlist = np.linspace(-1.0, 1.0, Ngrid)
UL, DL = np.meshgrid(ulist, dlist)
delta_chi2_grid_emu = 0.0*UL

#Not very elegant loop
for i in tqdm(range(Ngrid)):
    for j in range(Ngrid):
        delta_chi2_grid_emu[i,j] = deltachisq_NSI_emu(UL[i,j], DL[i,j])

#Find best-fit point
ind_BF = np.argmin(delta_chi2_grid_emu)
BF = [UL.flatten()[ind_BF], DL.flatten()[ind_BF]]

print("Best fit point: ", BF)
np.savetxt("results/COHERENT_NSI_deltachi2_emu.txt", delta_chi2_grid_emu, header="101x101 grid, corresponding to (uV, dV) values between -1 and 1.")

pl.figure(figsize=(6,6))
#pl.contourf(DL, UL, delta_chi2_grid, levels=[0,1,2,3,4,5,6,7,8,9,10],cmap="Blues")
pl.contourf(DL, UL, delta_chi2_grid_emu, levels=[0,4.6],cmap="Blues")
#levels=[0,4.60]
#pl.colorbar()

pl.plot(0.0, 0.0,'k+', markersize=12.0, label="Standard Model")

pl.plot(BF[1], BF[0], 'ro', label="Best fit")
#pl.plot(-0.25, 0.5, 'ro')

pl.ylabel(r"$\epsilon_{e\mu}^{uV}$", fontsize=22.0)
pl.xlabel(r"$\epsilon_{e\mu}^{dV}$" ,fontsize=22.0)

pl.title(r"$90\%$ CL allowed regions", fontsize=16.0)

pl.legend(frameon=False, fontsize=12, numpoints=1)

pl.savefig("plots/COHERENT_NSI_emu.pdf", bbox_inches="tight")
pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Calculate simplified (single bin) chi-squared (see chi-squared expression around p.32 in COHERENT paper):
Can you write Python code for it?

def deltachisq_NSI_ee(eps_uV, eps_dV):
    #NB: bins_I and bins_Cs are calculated further up in the script (they are the SM signal prediction)

    #Signal events from Iodine (with NSI correction only applying to electron neutrino events)
    N_sig_I = (N_SM_I + (NSI_corr(eps_uV, eps_dV, A_I, Z_I) - 1.0)*bins_I_e)
    #Now signal events from Caesium
    N_sig_Cs = (N_SM_Cs + (NSI_corr(eps_uV, eps_dV, A_Cs, Z_Cs) - 1.0)*bins_Cs_e)

    #Number of signal events
    N_NSI = np.sum(N_sig_I + N_sig_Cs)

    return deltachi2_Nsig(N_NSI)


I am working on the file "COHERENT.ipynb".
The first task is:
Comparing with the COHERENT results
First, let's load in the observed data and calculated spectrum (digitized from arXiv:1708.01294).
Can you write Python code for it?

COHERENT_data = np.loadtxt("DataFiles/COHERENT_data.txt", usecols=(1,))
COHERENT_upper = np.loadtxt("DataFiles/COHERENT_upper.txt", usecols=(1,)) - COHERENT_data
COHERENT_lower = COHERENT_data - np.loadtxt("DataFiles/COHERENT_lower.txt", usecols=(1,))
COHERENT_spect = np.loadtxt("DataFiles/COHERENT_spectrum.txt", usecols=(1,))
COHERENT_bins = np.arange(1,50,2)


I am working on the file "COHERENT.ipynb".
The first task is:
Do some plotting:
Can you write Python code for it?

pl.figure(figsize=(6,6))
pl.semilogx(maglist, deltachi2_list_mag, linewidth=2.0)
#pl.ylim(0, 25)

pl.axhline(2.706, linestyle='--', color='k')
pl.axvline(upper_limit, linestyle=':', color='k')

pl.text(1e-11, 3, "90% CL")

pl.ylabel(r"$\Delta \chi^2$")
pl.xlabel(r"Neutrino magnetic moment, $\mu_{\nu} / \mu_B$")

pl.savefig("plots/COHERENT_magnetic.pdf", bbox_inches="tight")

pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Fit to signal strength
Very simple fit to the number of CEvNS signal events, using only a 1-bin likelihood.
We start by defining the $\chi^2$, as given in arXiv:1708.01294. We use a generic form, so that we don't have to recalculate the number of signal events all the time...
Can you write Python code for it?

def chisq_generic(N_sig, alpha, beta):

    #Beam-on backgrounds
    N_BG = 6.0
    #Number of measured events
    N_meas = 142.0
    #Statistical uncertainty
    sig_stat = np.sqrt(N_meas + 2*405 + N_BG)

    #Uncertainties
    unc = (alpha/0.28)**2 + (beta/0.25)**2

    return ((N_meas - N_sig*(1.0+alpha) - N_BG*(1.0+beta))**2)/sig_stat**2 + unc

#Calculate minimum chi-squared as a function of (alpha, beta) nuisance parameters
def minchisq_Nsig(Nsig):

    minres = minimize(lambda x: chisq_generic(Nsig, x[0], x[1]), (0.0,0.0))
    return minres.fun

Nsiglist= np.linspace(0, 1000,1001)
chi2list = [minchisq_Nsig(Ns) for Ns in Nsiglist]

delta_chi2 = (chi2list - np.min(chi2list))

pl.figure(figsize=(6,6))
pl.plot(Nsiglist, delta_chi2, linewidth=2.0)
pl.ylim(0, 25)

pl.axvline(np.sum(N_SM_tot), linestyle='--', color='k')
pl.text(172, 20, "SM prediction")

pl.ylabel(r"$\Delta \chi^2$")
pl.xlabel(r"CE$\nu$NS counts")

pl.savefig("plots/COHERENT_likelihood.pdf", bbox_inches="tight")

pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Flavour-changing NSI ($e\mu$)
Now the correction to the CEvNS rate from flavor-changing NSI ($e\mu$-type):
Can you write Python code for it?

def NSI_corr_changing(eps_uV, eps_dV, A, Z):
    SIN2THETAW = 0.2387

    #Calculate standard weak nuclear charge (squared)
    Qsq = 4.0*((A - Z)*(-0.5) + Z*(0.5 - 2*SIN2THETAW))**2

    #Calculate the modified nuclear charge from NSI
    Qsq_NSI = Qsq + 4.0*((A-Z)*(eps_uV + 2.0*eps_dV) + Z*(2.0*eps_uV + eps_dV))**2

    return Qsq_NSI/Qsq

def deltachisq_NSI_emu(eps_uV, eps_dV):
    #NB: bins_I and bins_Cs are calculated further up in the script (they are the SM signal prediction)

    N_sig_I = (N_SM_I)*NSI_corr_changing(eps_uV, eps_dV, A_I, Z_I)
    #Now signal events from Caesium
    N_sig_Cs = (N_SM_Cs)*NSI_corr_changing(eps_uV, eps_dV, A_Cs, Z_Cs)

    #Number of signal events
    N_NSI = np.sum(N_sig_I + N_sig_Cs)

    return deltachi2_Nsig(N_NSI)


I am working on the file "COHERENT.ipynb".
The first task is:
Flavour-changing NSI ($e\tau$)
Finally, allowed regions for Flavour-changing NSI ($e\tau$-type)
Can you write Python code for it?

def deltachisq_NSI_etau(eps_uV, eps_dV):
    #NB: bins_I and bins_Cs are calculated further up in the script (they are the SM signal prediction)

    #Signal events from Iodine (with NSI correction only applying to electron neutrino events)
    N_sig_I = (N_SM_I + (NSI_corr_changing(eps_uV, eps_dV, A_I, Z_I) - 1.0)*bins_I_e)
    #Now signal events from Caesium
    N_sig_Cs = (N_SM_Cs + (NSI_corr_changing(eps_uV, eps_dV, A_Cs, Z_Cs) - 1.0)*bins_Cs_e)

    #Number of signal events
    N_NSI = np.sum(N_sig_I + N_sig_Cs)

    return deltachi2_Nsig(N_NSI)

Ngrid = 101

ulist = np.linspace(-1.0, 1.0, Ngrid)
dlist = np.linspace(-1.0, 1.0, Ngrid)
UL, DL = np.meshgrid(ulist, dlist)
delta_chi2_grid_etau = 0.0*UL

#Not very elegant loop
for i in tqdm(range(Ngrid)):
    for j in range(Ngrid):
        delta_chi2_grid_etau[i,j] = deltachisq_NSI_etau(UL[i,j], DL[i,j])

#Find best-fit point
ind_BF = np.argmin(delta_chi2_grid_etau)
BF = [UL.flatten()[ind_BF], DL.flatten()[ind_BF]]

print("Best fit point: ", BF)
np.savetxt("results/COHERENT_NSI_deltachi2_etau.txt", delta_chi2_grid_etau, header="101x101 grid, corresponding to (uV, dV) values between -1 and 1.")

pl.figure(figsize=(6,6))
#pl.contourf(DL, UL, delta_chi2_grid, levels=[0,1,2,3,4,5,6,7,8,9,10],cmap="Blues")
pl.contourf(DL, UL, delta_chi2_grid_etau, levels=[0,4.6],cmap="Blues")
#levels=[0,4.60]
#pl.colorbar()

pl.plot(0.0, 0.0,'k+', markersize=12.0, label="Standard Model")

pl.plot(BF[1], BF[0], 'ro', label="Best fit")
#pl.plot(-0.25, 0.5, 'ro')

pl.ylabel(r"$\epsilon_{e\tau}^{uV}$", fontsize=22.0)
pl.xlabel(r"$\epsilon_{e\tau}^{dV}$" ,fontsize=22.0)

pl.title(r"$90\%$ CL allowed regions", fontsize=16.0)

pl.legend(frameon=False, fontsize=12, numpoints=1)

pl.savefig("plots/COHERENT_NSI_etau.pdf", bbox_inches="tight")
pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Flavour-conserving NSI
Now, let's calculate the correction to the CEvNS rate from flavor-conserving NSI:
Can you write Python code for it?

def NSI_corr(eps_uV, eps_dV, A, Z):
    SIN2THETAW = 0.2387

    #Calculate standard weak nuclear charge (squared)
    Qsq = 4.0*((A - Z)*(-0.5) + Z*(0.5 - 2*SIN2THETAW))**2

    #Calculate the modified nuclear charge from NSI
    Qsq_NSI = 4.0*((A - Z)*(-0.5 + eps_uV + 2.0*eps_dV) + Z*(0.5 - 2*SIN2THETAW + 2*eps_uV + eps_dV))**2

    return Qsq_NSI/Qsq


I am working on the file "COHERENT.ipynb".
The first task is:
Import the CEvNS module (for calculating the signal spectrum and loading the neutrino fluxes)
Can you write Python code for it?


I am working on the file "COHERENT.ipynb".
The first task is:
Limits on a new scalar mediator
Finally, let's look at limits on the couplings of a new scalar mediator $\phi$. We start by calculating the contribution to the number of signal events for a given mediator mass (this can be rescaled by the coupling $g_\phi^4$ later):
Can you write Python code for it?

def calc_Nsig_scalar(m_med):
    scalar_rate = lambda x: (1.0/PEperkeV)*efficiency(x)*mass*time*(f_Cs*CEvNS.differentialRate_scalar(x/PEperkeV, A_Cs, Z_Cs,1,m_med)\
                                                              + f_I*CEvNS.differentialRate_scalar(x/PEperkeV, A_I, Z_I, 1,m_med))

    xlist = np.linspace(4,50,100)
    integ_vals =  np.vectorize(scalar_rate)(xlist)

    return np.trapz(integ_vals, xlist)

    #return quad(scalar_rate, PE_min, PE_max)[0]


I am working on the file "COHERENT.ipynb".
The first task is:
Limits on new vector mediators
First, let's calculate the total number of signal events at a given mediator mass and coupling...
It takes a while to recalculate the number of signal events for each mediator mass and coupling, so we'll do some rescaling and interpolation trickery:
Can you write Python code for it?

def tabulate_rate( m_med):
    vector_rate = lambda x, gsq: (1.0/PEperkeV)*efficiency(x)*mass*time*(f_Cs*CEvNS.differentialRate_CEvNS(x/PEperkeV, A_Cs, Z_Cs,gsq,m_med)\
                                                              + f_I*CEvNS.differentialRate_CEvNS(x/PEperkeV, A_I, Z_I, gsq,m_med))
    alpha = 1.0

    PE_min = 4.0
    PE_max = 50.0
    Nvals = 500
    PEvals = np.logspace(np.log10(PE_min), np.log10(PE_max),Nvals)

    Rvals_A = [np.sqrt(vector_rate(PEvals[i], 0)) for i in range(Nvals)]
    Rvals_B = [(1.0/(4.0*alpha*Rvals_A[i]))*(vector_rate(PEvals[i], alpha) - vector_rate(PEvals[i], -alpha)) for i in range(Nvals)]

    tabrate_A = InterpolatedUnivariateSpline(PEvals, Rvals_A, k = 1)
    tabrate_B = InterpolatedUnivariateSpline(PEvals, Rvals_B, k = 1)

    return tabrate_A, tabrate_B

def N_sig_vector(gsq, m_med):
    integrand = lambda x: (1.0/PEperkeV)*efficiency(x)*mass*time*(f_Cs*CEvNS.differentialRate_CEvNS(x/PEperkeV, A_Cs, Z_Cs,gsq,m_med)\
                                                              + f_I*CEvNS.differentialRate_CEvNS(x/PEperkeV, A_I, Z_I, gsq,m_med))

    xlist = np.linspace(4,50,100)
    integ_vals =  np.vectorize(integrand)(xlist)

    return np.trapz(integ_vals, xlist)

def N_sig_vector_tab(gsq, tabrate_A, tabrate_B):
    integrand = lambda x: (tabrate_A(x) + tabrate_B(x)*gsq)**2.0

    xlist = np.linspace(4,50,100)
    integ_vals =  np.vectorize(integrand)(xlist)

    return np.trapz(integ_vals, xlist)

    #return quad(integrand, 4.0, 50, epsabs=0.01)[0]

def tabulate_Nsig(tabrate_A, tabrate_B):
    N_A = N_sig_vector_tab(0, tabrate_A, tabrate_B)
    N_C = 0.5*(N_sig_vector_tab(1.0, tabrate_A, tabrate_B) + N_sig_vector_tab(-1.0, tabrate_A, tabrate_B))- N_A
    N_B = N_sig_vector_tab(1.0, tabrate_A, tabrate_B) - N_A - N_C
    return N_A, N_B, N_C

def N_sig_fulltab(gsq, Nsig_A, Nsig_B, Nsig_C):
    return Nsig_A + gsq*Nsig_B + gsq**2*Nsig_C

#Calculate the number of signal events for a 1000 MeV Z', with coupling 1e-4 by...


I am working on the file "COHERENT.ipynb".
The first task is:
Limits on the neutrino magnetic moment
Now let's calculate a limit on the neutrino magnetic moment (again, from a crude single-bin $\chi^2$).
Can you write Python code for it?

#Calculate the number of neutrino magnetic moment scattering events
#assuming a universal magnetic moment (in units of 1e-12 mu_B)

diffRate_mag = np.vectorize(CEvNS.differentialRate_magnetic)

dRdPE_mag = lambda x: (1.0/PEperkeV)*efficiency(x)*mass*time*(f_Cs*diffRate_mag(x/PEperkeV, A_Cs, Z_Cs, 1e-12)\
                                                              + f_I*diffRate_mag(x/PEperkeV, A_I, Z_I, 1e-12))
N_mag = quad(dRdPE_mag, 0, 50)[0]
print("Number of magnetic moment signal events (for mu_nu = 1e-12 mu_B):", N_mag)

def deltachisq_mag(mu_nu):

    #Signal events is sum of standard CEvNS + magnetic moment events
    N_sig = np.sum(N_SM_tot) + N_mag*(mu_nu/1e-12)**2

    return deltachi2_Nsig(N_sig)


I am working on the file "COHERENT.ipynb".
The first task is:
Now grid-scan to get the $\Delta \chi^2$:
Can you write Python code for it?

m_list = np.logspace(-3, 7,50)
gsq_list = np.logspace(0, 4, 50)

#Again, need to search in a specific range of coupling values to find the limit...
g_upper = 1e-10*(50**2+m_list**2)
g_lower = 1e-14*(50**2+m_list**2)

deltachi2_scal_grid = np.zeros((len(m_list), len(gsq_list)))

for i in tqdm(range(len(m_list))):
    Nsig_scalar = calc_Nsig_scalar(m_list[i])
    for j in range(len(gsq_list)):
        deltachi2_scal_grid[i,j] = deltachi2_Nsig(np.sum(N_SM_tot) + Nsig_scalar*(gsq_list[j]*g_lower[i])**2)

mgrid, ggrid = np.meshgrid(m_list, gsq_list, indexing='ij')
ggrid *= 1e-14*(50**2+mgrid**2)

np.savetxt("results/COHERENT_scalar.txt", np.c_[mgrid.flatten(), ggrid.flatten(), deltachi2_scal_grid.flatten()])

pl.figure(figsize=(6,6))
pl.loglog(m_list, g_upper, 'k--')
pl.loglog(m_list, g_lower, 'k--')
pl.contourf(mgrid, ggrid, deltachi2_scal_grid, levels=[2.7,1e10],cmap="Blues")
#pl.colorbar()
pl.xlabel(r"$m_{\phi}$ [MeV]")
pl.ylabel(r"$g_{\phi}^2$")
pl.title("Blue region (and above) is excluded...", fontsize=12)
pl.savefig("plots/COHERENT_scalar.pdf")
pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Now plot the results:
Can you write Python code for it?

pl.figure(figsize=(10,6))

pl.step(PE_bins, np.append(N_SM_tot,0), 'g', linestyle="-", where = "post", label="CEvNS signal (this work)",linewidth=1.5)
pl.step(PE_bins, np.append(COHERENT_spect,0), 'g', linestyle="--", where = "post", label="CEvNS signal (1708.01294)",linewidth=1.5)

pl.axhline(0, linestyle='--', color = 'gray')

pl.errorbar(COHERENT_bins, COHERENT_data, fmt='ko', \
            yerr = [COHERENT_lower, COHERENT_upper], label="COHERENT data",\
            capsize=0.0)

pl.xlabel("Number of photoelectrons (PE)")
pl.ylabel("Res. counts / 2 PE")

pl.legend( fontsize=14)

pl.xlim(0, 50)
pl.ylim(-15, 35)

pl.savefig("plots/COHERENT_data.pdf", bbox_inches="tight")
pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Now we scan over a grid in $g^2$ and $m_V$ to calculate the $\chi^2$ at each point:
Can you write Python code for it?

gsq_list = np.append(np.logspace(0, 2, 100),1e20)
m_list = np.sort(np.append(np.logspace(-2, 4,49), [1e-6,1e8]))

#Need to search for the limit in a narrow band of coupling values
g_upper = 1e-11*(50**2+m_list**2)
g_lower = 1e-13*(50**2+m_list**2)

deltachi2_vec_grid = np.zeros((51, 101))

for i in tqdm(range(len(m_list))):
    rate_A, rate_B = tabulate_rate(m_list[i])
    N_A, N_B,N_C = tabulate_Nsig(rate_A, rate_B)
    for j, gsq in enumerate(gsq_list):
        N_sig = N_sig_fulltab(gsq*g_lower[i], N_A, N_B, N_C)
        deltachi2_vec_grid[i, j] = deltachi2_Nsig(N_sig)

mgrid, ggrid = np.meshgrid(m_list, gsq_list, indexing='ij')
ggrid *= 1e-13*(50**2 + mgrid**2)

np.savetxt("results/COHERENT_Zprime.txt", np.c_[mgrid.flatten(), ggrid.flatten(), deltachi2_vec_grid.flatten()])

pl.figure(figsize=(6,6))
pl.loglog(m_list, g_upper, 'k--')
pl.loglog(m_list, g_lower, 'k--')
pl.contourf(mgrid, ggrid, deltachi2_vec_grid, levels=[2.7,1e10],cmap="Blues")

pl.ylim(1e-10, 1e5)
#pl.colorbar()
pl.xlabel(r"$m_{Z'}$ [MeV]")
pl.ylabel(r"$g_{Z'}^2$")
pl.title("Blue region (and above) is excluded...", fontsize=12)
pl.savefig("plots/COHERENT_Zprime.pdf")
pl.show()


I am working on the file "COHERENT.ipynb".
The first task is:
Scan over a grid:
Can you write Python code for it?

Ngrid = 501

maglist = np.logspace(-12, -6, Ngrid)

deltachi2_list_mag = 0.0*maglist
#Not very elegant loop
for i in tqdm(range(Ngrid)):
    deltachi2_list_mag[i] = deltachisq_mag(maglist[i])

upper_limit = maglist[deltachi2_list_mag > 2.706][0]

print("90% upper limit: ", upper_limit)


I am working on the file "COHERENT.ipynb".
The first task is:
To speed things up later (so we don't have to do the minimization every time), we'll tabulate and interpolate the chi-squared as a function of the number of signal events. This works because we're using a simple chi-squared which depends only on the number of signal events:
Can you write Python code for it?

deltachi2_Nsig = interp1d(Nsiglist, delta_chi2, bounds_error=False, fill_value=delta_chi2[-1])


I am working on the file "COLOP in SCI mode.ipynb".
The first task is:
Define the input and output files and input parameters for operator estimation
Can you write Python code for it?

#Input spectrum files
seisfile="F2_01_seismic_amplitude_spectrum.dat"
wellfile="F2_01_well_AI_amplitude_spectrum.dat"

#Shape parameter for Kaiser window
beta=70

#Normalized amplitude treshold, smaller amplitudes will be skipped during computation
treshold=0.2

#Operator phase spectrum depending on polarity of the seismic data
#SEG normal(AI increase = through): phase=0; SEG reverse (AI increase = Peak): phase=-180
phase=-90

#Number of samples of the operator
num=100

#Ouput operator file
operatorfile="F2_01_colour_operator.dat"


I am working on the file "COLOP in SCI mode.ipynb".
The first task is:
Load spectrums from input files, and calculate linear regression based on well impedance spectrum on log-log scale
Can you write Python code for it?

#Load exported spectrums from OpendTect
freqseis, ampseis=np.loadtxt(seisfile, unpack=True)
freqwell, ampwell=np.loadtxt(wellfile, unpack=True)

# dB to amplitude conversion
ampwell=np.power(10,ampwell/20)
ampseis=np.power(10,ampseis/20)

#Normalize seismic spectrum
normseis = ampseis / np.max(ampseis)

#Calculate logarithmic well spectrum
logfreq= np.log10(freqwell)
logamp=np.log10(ampwell)

#Linear regression on logarithmic well spectrum
slope, intercept, rvalue, pvalue, stderr = linregress(logfreq,logamp)
print ('Regression results:')
print ("Intercept:", intercept)
print ("Slope    :", slope)
print ("R-value  :", rvalue)

#Plot well based AI spectrum with regression line
lintrend=intercept+slope*logfreq
plt.figure(0)
plt.title('Well Impedance spectrum')
plt.scatter(logfreq,logamp, label="AI impedance spectrum")
plt.xlabel("log10(frequency)")
plt.ylabel("log10(amplitude)")
plt.plot(logfreq,lintrend, label="Trend line", linewidth=3, color='red')
plt.xlim(np.min(logfreq),np.max(logfreq))
plt.legend()
plt.grid()
plt.show()


I am working on the file "COLOP in SCI mode.ipynb".
The first task is:
Precalculation for inverse dft, to transform raw operator to time domain.
First setup a complex amplitude spectrum with the given phase, then calculate ifft and reorder the result.
Can you write Python code for it?

#Calculate dt
dt=1/(2*np.max(freqseis))

#Setup complex amplitude spectrum for ifft with phase assumption
cspectrum_poz=ResidualSpectrum*(np.cos(np.radians(phase))+1j*np.sin(np.radians(phase)))
cspectrum_neg=ResidualSpectrum*(np.cos(-1*np.radians(phase))+1j*np.sin(-1*np.radians(phase)))
rev_cspectrum_neg=np.fliplr([cspectrum_neg])[0]
input_cspectrum=np.append(cspectrum_poz,rev_cspectrum_neg)

#Calculate ifft and reorder arrays
t_op=np.fft.ifft(input_cspectrum)
start_t=(-1/2)*dt*(len(input_cspectrum))+dt
t_shift=np.linspace(start_t,-1*start_t,len(t_op))-dt/2
t_op_shift=np.fft.ifftshift(t_op)

#Tapering of the time domain operator using a Kaiser window, and calculation of the operator triming indexes, and plot the final operator

#Tapering
window_kaiser=np.kaiser(len(t_shift),beta)

t_op_final=t_op_shift*window_kaiser

#Operator trimming indexes
start_i=(int(len(t_shift)/2))-int(num/2)
stop_i=(int(len(t_shift)/2))+int(num/2)

#Plot final time domain operator
plt.figure(2)
plt.title('Colour inversion operator')
plt.plot(t_shift,t_op_final.real, label='Time domain operator')
#plt.fill_between(t_shift, t_op_final.real,0, t_op_final > 0.0, interpolate=False, hold=True, color='blue', alpha = 0.5)
plt.xlim(t_shift[start_i],t_shift[stop_i])
plt.ylim(-0.07,0.09)
plt.xlabel('Time [s]')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()

#Save final operator
np.savetxt(operatorfile,t_op_final[start_i:stop_i].real)

#QC operator by transform tapered operator to frequency domain
bt_op=np.fft.fft(t_op_final)
backfreq=np.fft.fftfreq(len(t_op_final),dt)

#Keeping only the positive frequencies
backfreq_poz=backfreq[:int(len(backfreq)/2)]
bt_op_poz=bt_op[:int(len(bt_op)/2)]

output_spectrum=bt_op_poz*normseis

plt.figure(3)
plt.title('QC tapering')
plt.plot(backfreq_poz,abs(bt_op_poz), label='Tapered backtransformed operator', color='red')
plt.plot(freqseis,ResidualSpectrum, label='Frequency domain raw operator')
#plt.fill_between(backfreq_poz, abs(bt_op_poz),0, abs(bt_op_poz) > 0.0, interpolate=False, hold=True,...


I am working on the file "COLOP in SSB mode.ipynb".
The first task is:
Calculate raw operator, based on the ratio of trend spectrum from wells and normalized seismic amplitude spectrum
Can you write Python code for it?

#Reflectivity trend well spectrum
WelltrendSpectrum = np.power(freqseis,(1+slope))
NormWellTrendSpectrum = WelltrendSpectrum / max(WelltrendSpectrum)

#Calculate residual spectrum
ResidualSpectrum=np.zeros(len(normseis))
for i in range(len(normseis)):
    if normseis[i]>treshold:
        ResidualSpectrum[i]= WelltrendSpectrum[i] / normseis[i]

#Normalize residual spectrum
ResidualSpectrum=ResidualSpectrum / np.max(ResidualSpectrum)

#Plot normalized seismic spectrum with well trend spectrum
plt.figure(1)
thold=np.ones(len(freqseis))
thold=treshold*thold
plt.title('Seismic spectrums')
plt.plot(freqseis,normseis, label='Normalized seismic amplitude spectrum')
plt.plot(freqseis,NormWellTrendSpectrum, label='Regression based reflectivity spectrum')
plt.plot(freqseis,ResidualSpectrum, label='Frequency domain raw operator', color='red')
plt.plot(freqseis,thold, label='Amplitude treshold', color='grey')
plt.xlabel('Frequency [Hz]')
plt.ylabel('Normalized Amplitude')
plt.ylim(0,1.5)
plt.xlim(0,100)
plt.legend()
plt.grid()
plt.show()


I am working on the file "COLOP in SSB mode.ipynb".
The first task is:
Define the input and output files and input parameters for operator estimation
Can you write Python code for it?

#Input spectrum files
seisfile="F2_01_seismic_amplitude_spectrum.dat"
wellfile="F2_01_well_AI_amplitude_spectrum.dat"

#Shape parameter for Kaiser window
beta=150

#Normalized amplitude treshold, smaller amplitudes will be skipped during computation
treshold=0.2

#Operator phase spectrum depending on polarity of the seismic data
#SEG normal(AI increase = through): phase=0; SEG reverse (AI increase = Peak): phase=180
phase=0

#Number of samples of the operator
num=80

#Ouput operator file
operatorfile="F2_01_ssb_operator.dat"


I am working on the file "COLOP in SSB mode.ipynb".
The first task is:
Load spectrums from input files, and calculate linear regression based on well impedance spectrum on log-log scale
Can you write Python code for it?

#Load exported spectrums from OpendTect
freqseis, ampseis=np.loadtxt(seisfile, unpack=True)
freqwell, ampwell=np.loadtxt(wellfile, unpack=True)

# dB to amplitude conversion
ampwell=np.power(10,ampwell/20)
ampseis=np.power(10,ampseis/20)

#Normalize seismic spectrum
normseis = ampseis / np.max(ampseis)

#Calculate logarithmic well spectrum
logfreq= np.log10(freqwell)
logamp=np.log10(ampwell)

#Linear regression on logarithmic well spectrum
slope, intercept, rvalue, pvalue, stderr = linregress(logfreq,logamp)
print ('Regression results:')
print ("Intercept:", intercept)
print ("Slope    :", slope)
print ("R-value  :", rvalue)

#Plot well based AI spectrum with regression line
lintrend=intercept+(slope)*logfreq
plt.figure(0)
plt.title('Well Impedance spectrum')
plt.scatter(logfreq,logamp, label="AI impedance spectrum")
plt.xlabel("log10(frequency)")
plt.ylabel("log10(amplitude)")
plt.plot(logfreq,lintrend, label="Trend line", linewidth=3, color='red')
plt.xlim(np.min(logfreq),np.max(logfreq))
plt.legend()
plt.grid()
plt.show()


I am working on the file "COLOP in SSB mode.ipynb".
The first task is:
Precalculation for inverse dft, to transform raw operator to time domain.
First setup a complex amplitude spectrum with the given phase, then calculate ifft and reorder the result.
Can you write Python code for it?

#Calculate dt
dt=1/(2*np.max(freqseis))

#Setup complex amplitude spectrum for ifft with phase assumption
cspectrum_poz=ResidualSpectrum*(np.cos(np.radians(phase))+1j*np.sin(np.radians(phase)))
cspectrum_neg=ResidualSpectrum*(np.cos(-1*np.radians(phase))+1j*np.sin(-1*np.radians(phase)))
rev_cspectrum_neg=np.fliplr([cspectrum_neg])[0]
input_cspectrum=np.append(cspectrum_poz,rev_cspectrum_neg)

#Calculate ifft and reorder arrays
t_op=np.fft.ifft(input_cspectrum)
start_t=(-1/2)*dt*(len(input_cspectrum))+dt
t_shift=np.linspace(start_t,-1*start_t,len(t_op))-dt/2
t_op_shift=np.fft.ifftshift(t_op)

#Tapering of the time domain operator using a Kaiser window, and calculation of the operator triming indexes, and plot the final operator

#Tapering
window_kaiser=np.kaiser(len(t_shift),beta)

t_op_final=t_op_shift*window_kaiser

#Operator trimming indexes
start_i=((int(len(t_shift)/2))-int(num/2))
stop_i=(int(len(t_shift)/2))+int(num/2)+1

#Plot final time domain operator
plt.figure(2)
plt.title('Spectral blueing operator')
plt.plot(t_shift,t_op_final.real, label='Time domain operator')
#plt.fill_between(t_shift, t_op_final.real,0, t_op_final > 0.0, interpolate=False, hold=True, color='blue', alpha = 0.5)
plt.xlim(t_shift[start_i],t_shift[stop_i])
plt.ylim(-0.07,0.25)
plt.xlabel('Time [s]')
plt.ylabel('Amplitude')
plt.legend()
plt.grid()
plt.show()

#Save final operator
np.savetxt(operatorfile,t_op_final[start_i:stop_i].real)

#QC operator by transform tapered operator to frequency domain
bt_op=np.fft.fft(t_op_final)
backfreq=np.fft.fftfreq(len(t_op_final),dt)

#Keeping only the positive frequencies
backfreq_poz=backfreq[:int(len(backfreq)/2)]
bt_op_poz=bt_op[:int(len(bt_op)/2)]

output_spectrum=bt_op_poz*normseis

plt.figure(3)
plt.title('QC tapering')
plt.plot(backfreq_poz,abs(bt_op_poz), label='Tapered backtransformed operator', color='red')
plt.plot(freqseis,ResidualSpectrum, label='Frequency domain raw operator')
#plt.fill_between(backfreq_poz, abs(bt_op_poz),0, abs(bt_op_poz) > 0.0, interpolate=False, hold=True,...


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
<h2> Analyzing taxicab demand </h2>

Let's pull the number of trips for each day in the 2015 dataset using Standard SQL.
Can you write Python code for it?

%bq query
SELECT
  EXTRACT (DAYOFYEAR from pickup_datetime) AS daynumber
FROM `bigquery-public-data.new_york.tlc_yellow_trips_2015`
LIMIT 5


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
<h3> Exploratory analysis </h3>

Is there a relationship between maximum temperature and the number of trips?
Can you write Python code for it?

j = data.plot(kind='scatter', x='maxtemp', y='numtrips')


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
<h3> Merge datasets </h3>

Let's use Pandas to merge (combine) the taxi cab and weather datasets day-by-day.
Can you write Python code for it?

data = pd.merge(weather, trips, on='daynumber')
data[:5]


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Analyzing taxicab demand
Let's pull the number of trips for each day in the 2015 dataset using Standard SQL.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Benchmark
Often, a reasonable estimate of something is its historical average. We can therefore benchmark our machine learning model against the historical average.
Can you write Python code for it?

avg = np.mean(trips['numtrips'])
print('Just using average={0} has RMSE of {1}'.format(avg, np.sqrt(np.mean((trips['numtrips'] - avg)**2))))


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Hurrah, we seem to have found a predictor. It appears that people use taxis more later in the week. Perhaps New Yorkers make weekly resolutions to walk more and then lose their determination later in the week, or maybe it reflects tourism dynamics in New York City.
Perhaps if we took out the confounding effect of the day of the week, maximum temperature will start to have an effect. Let's see if that's the case:
Can you write Python code for it?

j = data[data['dayofweek'] == 7].plot(kind='scatter', x='maxtemp', y='numtrips')


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Let's update our benchmark based on the 80-20 split and the larger dataset.
Can you write Python code for it?

trainsize = int(len(shuffled['numtrips']) * 0.8)
avg = np.mean(shuffled['numtrips'][:trainsize])
rmse = np.sqrt(np.mean((targets[trainsize:] - avg)**2))
print('Just using average={0} has RMSE of {1}'.format(avg, rmse))


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Linear regression with tf.contrib.learn
We scale the number of taxicab rides by 400,000 so that the model can keep its predicted values in the [0-1] range. The optimization goes a lot faster when the weights are small numbers.  We save the weights into ./trained_model_linear and display the root mean square error on the test dataset.
Can you write Python code for it?

SCALE_NUM_TRIPS = 600000.0
trainsize = int(len(shuffled['numtrips']) * 0.8)
testsize = len(shuffled['numtrips']) - trainsize
npredictors = len(predictors.columns)
noutputs = 1
tf.logging.set_verbosity(tf.logging.WARN) # change to INFO to get output every 100 steps ...
shutil.rmtree('./trained_model_linear', ignore_errors=True) # so that we don't load weights from previous runs
estimator = tf.contrib.learn.LinearRegressor(model_dir='./trained_model_linear',
                                             feature_columns=tf.contrib.learn.infer_real_valued_columns_from_input(predictors.values))

print("starting to train ... this will take a while ... use verbosity=INFO to get more verbose output")
def input_fn(features, targets):
  return tf.constant(features.values), tf.constant(targets.values.reshape(len(targets), noutputs)/SCALE_NUM_TRIPS)
estimator.fit(input_fn=lambda: input_fn(predictors[:trainsize], targets[:trainsize]), steps=10000)

pred = np.multiply(list(estimator.predict(predictors[trainsize:].values)), SCALE_NUM_TRIPS )
rmse = np.sqrt(np.mean(np.power((targets[trainsize:].values - pred), 2)))
print('LinearRegression has RMSE of {0}'.format(rmse))


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Merge datasets
Let's use Pandas to merge (combine) the taxi cab and weather datasets day-by-day.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Removing the confounding factor does seem to reflect an underlying trend around temperature. But ... the data are a little sparse, don't you think?  This is something that you have to keep in mind -- the more predictors you start to consider (here we are using two: day of week and maximum temperature), the more rows you will need so as to avoid <em> overfitting </em> the model.
<h3> Adding 2014 and 2016 data </h3>

Let's add in 2014 and 2016 data to the Pandas dataframe.  Note how useful it was for us to modularize our queries around the YEAR.
Can you write Python code for it?

data2 = data # 2015 data
for year in [2014, 2016]:
    query_parameters = [
      {
        'name': 'YEAR',
        'parameterType': {'type': 'STRING'},
        'parameterValue': {'value': year}
      }
    ]
    weather = wxquery.execute(query_params=query_parameters).result().to_dataframe()
    trips = taxiquery.execute(query_params=query_parameters).result().to_dataframe()
    data_for_year = pd.merge(weather, trips, on='daynumber')
    data2 = pd.concat([data2, data_for_year])
data2.describe()

j = data2[data2['dayofweek'] == 7].plot(kind='scatter', x='maxtemp', y='numtrips')


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Removing the confounding factor does seem to reflect an underlying trend around temperature. But ... the data are a little sparse, don't you think?  This is something that you have to keep in mind -- the more predictors you start to consider (here we are using two: day of week and maximum temperature), the more rows you will need so as to avoid <em> overfitting </em> the model.
Adding 2014 and 2016 data
Let's add in 2014 and 2016 data to the Pandas dataframe.  Note how useful it was for us to modularize our queries around the YEAR.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
The RMSE here (57K) is lower than the benchmark (62K) indicates that we are doing about 10% better with the machine learning model than we would be if we were to just use the historical average (our benchmark).
Neural network with tf.contrib.learn
Let's make a more complex model with a few hidden nodes.
Can you write Python code for it?

SCALE_NUM_TRIPS = 600000.0
trainsize = int(len(shuffled['numtrips']) * 0.8)
testsize = len(shuffled['numtrips']) - trainsize
npredictors = len(predictors.columns)
noutputs = 1
tf.logging.set_verbosity(tf.logging.WARN) # change to INFO to get output every 100 steps ...
shutil.rmtree('./trained_model', ignore_errors=True) # so that we don't load weights from previous runs
estimator = tf.contrib.learn.DNNRegressor(model_dir='./trained_model',
                                          hidden_units=[5, 5],
                                          feature_columns=tf.contrib.learn.infer_real_valued_columns_from_input(predictors.values))

print("starting to train ... this will take a while ... use verbosity=INFO to get more verbose output")
def input_fn(features, targets):
  return tf.constant(features.values), tf.constant(targets.values.reshape(len(targets), noutputs)/SCALE_NUM_TRIPS)
estimator.fit(input_fn=lambda: input_fn(predictors[:trainsize], targets[:trainsize]), steps=10000)

pred = np.multiply(list(estimator.predict(predictors[trainsize:].values)), SCALE_NUM_TRIPS )
rmse = np.sqrt(np.mean((targets[trainsize:].values - pred)**2))
print('Neural Network Regression has RMSE of {0}'.format(rmse))


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
The data do seem a bit more robust.  If we had even more data, it would be better of course. But in this case, we only have 2014-2016 data for taxi trips, so that's what we will go with.
<h2> Machine Learning with Tensorflow </h2>

We'll use 80% of our dataset for training and 20% of the data for testing the model we have trained. Let's shuffle the rows of the Pandas dataframe so that this division is random.  The predictor (or input) columns will be every column in the database other than the number-of-trips (which is our target, or what we want to predict).
The machine learning models that we will use -- linear regression and neural networks -- both require that the input variables are numeric in nature.
The day of the week, however, is a categorical variable (i.e. Tuesday is not really greater than Monday). So, we should create separate columns for whether it is a Monday (with values 0 or 1), Tuesday, etc.
Against that, we do have limited data (remember: the more columns you use as input features, the more rows you need to have in your training dataset), and it appears that there is a clear linear trend by day of the week. So, we will opt for simplicity here and use the data as-is.  Try uncommenting the code that creates separate columns for the days of the week and re-run the notebook if you are curious about the impact of this simplification.
Can you write Python code for it?

import tensorflow as tf
shuffled = data2.sample(frac=1, random_state=13)
# It would be a good idea, if we had more data, to treat the days as categorical variables
# with the small amount of data, we have though, the model tends to overfit
#predictors = shuffled.iloc[:,2:5]
#for day in range(1,8):
#  matching = shuffled['dayofweek'] == day
#  key = 'day_' + str(day)
#  predictors[key] = pd.Series(matching, index=predictors.index, dtype=float)
predictors = shuffled.iloc[:,1:5]
predictors[:5]

shuffled[:5]

targets = shuffled.iloc[:,5]
targets[:5]


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
The mean here is about 400,000 and the root-mean-square-error (RMSE) in this case is about 52,000. In other words, if we were to estimate that there are 400,000 taxi trips on any given day, that estimate is will be off on average by about 52,000 in either direction.
Let's see if we can do better than this -- our goal is to make predictions of taxicab demand whose RMSE is lower than 52,000.
What kinds of things affect people's use of taxicabs?
<h2> Weather data </h2>

We suspect that weather influences how often people use a taxi. Perhaps someone who'd normally walk to work would take a taxi if it is very cold or rainy.
One of the advantages of using a global data warehouse like BigQuery is that you get to mash up unrelated datasets quite easily.
Can you write Python code for it?

%bq query
SELECT * FROM `bigquery-public-data.noaa_gsod.stations`
WHERE state = 'NY' AND wban != '99999' AND name LIKE '%LA GUARDIA%'


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
The mean here is about 400,000 and the root-mean-square-error (RMSE) in this case is about 52,000. In other words, if we were to estimate that there are 400,000 taxi trips on any given day, that estimate is will be off on average by about 52,000 in either direction.
Let's see if we can do better than this -- our goal is to make predictions of taxicab demand whose RMSE is lower than 52,000.
What kinds of things affect people's use of taxicabs?
Weather data
We suspect that weather influences how often people use a taxi. Perhaps someone who'd normally walk to work would take a taxi if it is very cold or rainy.
One of the advantages of using a global data warehouse like BigQuery is that you get to mash up unrelated datasets quite easily.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
The scatterplot above doesn't look very promising. There appears to be a weak downward trend, but it's also quite noisy.
Is there a relationship between the day of the week and the number of trips?
Can you write Python code for it?

j = data.plot(kind='scatter', x='dayofweek', y='numtrips')


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Using a neural network results in similar performance to the linear model when I ran it -- it might be because there isn't enough data for the NN to do much better. (NN training is a non-convex optimization, and you will get different results each time you run the above code).
<h2> Running a trained model </h2>

So, we have trained a model, and saved it to a file. Let's use this model to predict taxicab demand given the expected weather for three days.
Here we make a Dataframe out of those inputs, load up the saved model (note that we have to know the model equation -- it's not saved in the model file) and use it to predict the taxicab demand.
Can you write Python code for it?

input = pd.DataFrame.from_dict(data =
                               {'dayofweek' : [4, 5, 6],
                                'mintemp' : [60, 40, 50],
                                'maxtemp' : [70, 90, 60],
                                'rain' : [0, 0.5, 0]})
# read trained model from ./trained_model
estimator = tf.contrib.learn.LinearRegressor(model_dir='./trained_model_linear',
                                          feature_columns=tf.contrib.learn.infer_real_valued_columns_from_input(input.values))

pred = np.multiply(list(estimator.predict(input.values)), SCALE_NUM_TRIPS )
print(pred)


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Using a neural network results in similar performance to the linear model when I ran it -- it might be because there isn't enough data for the NN to do much better. (NN training is a non-convex optimization, and you will get different results each time you run the above code).
Running a trained model
So, we have trained a model, and saved it to a file. Let's use this model to predict taxicab demand given the expected weather for three days.
Here we make a Dataframe out of those inputs, load up the saved model (note that we have to know the model equation -- it's not saved in the model file) and use it to predict the taxicab demand.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast.ipynb".
The first task is:
Variables
Let's pull out the minimum and maximum daily temperature (in Fahrenheit) as well as the amount of rain (in inches) for La Guardia airport.
Can you write Python code for it?

%bq query -n wxquery
SELECT EXTRACT (DAYOFYEAR FROM CAST(CONCAT(@YEAR,'-',mo,'-',da) AS TIMESTAMP)) AS daynumber,
       MIN(EXTRACT (DAYOFWEEK FROM CAST(CONCAT(@YEAR,'-',mo,'-',da) AS TIMESTAMP))) dayofweek,
       MIN(min) mintemp, MAX(max) maxtemp, MAX(IF(prcp=99.99,0,prcp)) rain
FROM `bigquery-public-data.noaa_gsod.gsod*`
WHERE stn='725030' AND _TABLE_SUFFIX = @YEAR
GROUP BY 1 ORDER BY daynumber DESC

query_parameters = [
  {
    'name': 'YEAR',
    'parameterType': {'type': 'STRING'},
    'parameterValue': {'value': 2015}
  }
]
weather = wxquery.execute(query_params=query_parameters).result().to_dataframe()
weather[:5]


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h2> Analyzing taxicab demand </h2>

Let's pull the number of trips for each day in the 2015 dataset using Standard SQL.
Can you write Python code for it?

query_to_dataframe("""
SELECT
  EXTRACT (DAYOFYEAR from pickup_datetime) AS daynumber
FROM `bigquery-public-data.new_york.tlc_yellow_trips_2015`
LIMIT 5
""")


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h2> Linear regression with tf.contrib.learn </h2>

We scale the number of taxicab rides by 400,000 so that the model can keep its predicted values in the [0-1] range. The optimization goes a lot faster when the weights are small numbers.  We save the weights into ./trained_model_linear and display the root mean square error on the test dataset.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h3> Benchmark </h3>

Often, a reasonable estimate of something is its historical average. We can therefore benchmark our machine learning model against the historical average.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h3> Exploratory analysis </h3>

Is there a relationship between maximum temperature and the number of trips?
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h3> Merge datasets </h3>

Let's use Pandas to merge (combine) the taxi cab and weather datasets day-by-day.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:

 Modular queries and Pandas dataframe

Let's use the total number of trips as our proxy for taxicab demand (other reasonable alternatives are total trip_distance or total fare_amount).  It is possible to predict multiple variables using Tensorflow, but for simplicity, we will stick to just predicting the number of trips.
We will give our query a name 'taxiquery' and have it use an input variable '$YEAR'. We can then invoke the 'taxiquery' by giving it a YEAR.  The to_dataframe() converts the BigQuery result into a Pandas dataframe.
Can you write Python code for it?

def taxiquery(year):
  return """
WITH trips AS (
  SELECT EXTRACT (DAYOFYEAR from pickup_datetime) AS daynumber
  FROM `bigquery-public-data.new_york.tlc_yellow_trips_*`
  where _TABLE_SUFFIX = '{}'
)
SELECT daynumber, COUNT(1) AS numtrips FROM trips
GROUP BY daynumber ORDER BY daynumber
""".format(year)

trips = query_to_dataframe(taxiquery(2015))
trips[:5]


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
<h3> Variables </h3>

Let's pull out the minimum and maximum daily temperature (in Fahrenheit) as well as the amount of rain (in inches) for La Guardia airport.
Can you write Python code for it?

def wxquery(year):
  return """
SELECT EXTRACT (DAYOFYEAR FROM CAST(CONCAT('{0}','-',mo,'-',da) AS TIMESTAMP)) AS daynumber,
       MIN(EXTRACT (DAYOFWEEK FROM CAST(CONCAT('{0}','-',mo,'-',da) AS TIMESTAMP))) dayofweek,
       MIN(min) mintemp, MAX(max) maxtemp, MAX(IF(prcp=99.99,0,prcp)) rain
FROM `bigquery-public-data.noaa_gsod.gsod*`
WHERE stn='725030' AND _TABLE_SUFFIX = '{0}'
GROUP BY 1 ORDER BY daynumber DESC
""".format(year)

weather = query_to_dataframe(wxquery(2015))
weather[:5]


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
Explore table
Can you write Python code for it?

import pandas as pd
import numpy as np
import shutil

query_to_dataframe("""
SELECT * FROM `bigquery-public-data.new_york.tlc_yellow_trips_2015` LIMIT 10
""")


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
Hurrah, we seem to have found a predictor. It appears that people use taxis more later in the week. Perhaps New Yorkers make weekly resolutions to walk more and then lose their determination later in the week, or maybe it reflects tourism dynamics in New York City.
Perhaps if we took out the confounding effect of the day of the week, maximum temperature will start to have an effect. Let's see if that's the case:
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
Removing the confounding factor does seem to reflect an underlying trend around temperature. But ... the data are a little sparse, don't you think?  This is something that you have to keep in mind -- the more predictors you start to consider (here we are using two: day of week and maximum temperature), the more rows you will need so as to avoid <em> overfitting </em> the model.
<h3> Adding 2014 and 2016 data </h3>

Let's add in 2014 and 2016 data to the Pandas dataframe.  Note how useful it was for us to modularize our queries around the YEAR.
Can you write Python code for it?

data2 = data # 2015 data
for year in [2014, 2016]:
    weather = query_to_dataframe(wxquery(year))
    trips = query_to_dataframe(taxiquery(year))
    data_for_year = pd.merge(weather, trips, on='daynumber')
    data2 = pd.concat([data2, data_for_year])
data2.describe()

j = data2[data2['dayofweek'] == 7].plot(kind='scatter', x='maxtemp', y='numtrips')


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
The RMSE here (57K) is lower than the benchmark (62K) indicates that we are doing about 10% better with the machine learning model than we would be if we were to just use the historical average (our benchmark).
<h2> Neural network with tf.contrib.learn </h2>

Let's make a more complex model with a few hidden nodes.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
The data do seem a bit more robust.  If we had even more data, it would be better of course. But in this case, we only have 2014-2016 data for taxi trips, so that's what we will go with.
<h2> Machine Learning with Tensorflow </h2>

We'll use 80% of our dataset for training and 20% of the data for testing the model we have trained. Let's shuffle the rows of the Pandas dataframe so that this division is random.  The predictor (or input) columns will be every column in the database other than the number-of-trips (which is our target, or what we want to predict).
The machine learning models that we will use -- linear regression and neural networks -- both require that the input variables are numeric in nature.
The day of the week, however, is a categorical variable (i.e. Tuesday is not really greater than Monday). So, we should create separate columns for whether it is a Monday (with values 0 or 1), Tuesday, etc.
Against that, we do have limited data (remember: the more columns you use as input features, the more rows you need to have in your training dataset), and it appears that there is a clear linear trend by day of the week. So, we will opt for simplicity here and use the data as-is.  Try uncommenting the code that creates separate columns for the days of the week and re-run the notebook if you are curious about the impact of this simplification.
Can you write Python code for it?


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
The mean here is about 400,000 and the root-mean-square-error (RMSE) in this case is about 52,000. In other words, if we were to estimate that there are 400,000 taxi trips on any given day, that estimate is will be off on average by about 52,000 in either direction.
Let's see if we can do better than this -- our goal is to make predictions of taxicab demand whose RMSE is lower than 52,000.
What kinds of things affect people's use of taxicabs?
<h2> Weather data </h2>

We suspect that weather influences how often people use a taxi. Perhaps someone who'd normally walk to work would take a taxi if it is very cold or rainy.
One of the advantages of using a global data warehouse like BigQuery is that you get to mash up unrelated datasets quite easily.
Can you write Python code for it?

query_to_dataframe("""
SELECT * FROM `bigquery-public-data.noaa_gsod.stations`
WHERE state = 'NY' AND wban != '99999' AND name LIKE '%LA GUARDIA%'
""")


I am working on the file "CPB100/lab4a/demandforecast2.ipynb".
The first task is:
The scatterplot above doesn't look very promising. There appears to be a weak downward trend, but it's also quite noisy.
Is there a relationship between the day of the week and the number of trips?
Can you write Python code for it?


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:
 Note: Make sure you generate an API Key and replace the value above. The sample key will not work.
From the same API console, choose "Dashboard" on the left-hand menu and "Enable API".
Enable the following APIs for your project (search for them) if they are not already enabled:

 Google Translate API

 Google Cloud Vision API

 Google Natural Language API

 Google Cloud Speech API

Finally, because we are calling the APIs from Python (clients in many other languages are available), let's install the Python package (it's not installed by default on Datalab)
Can you write Python code for it?

!pip install --upgrade google-api-python-client


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:
<h2> Invoke Translate API </h2>
Can you write Python code for it?

# running Translate API
from googleapiclient.discovery import build
service = build('translate', 'v2', developerKey=APIKEY)

# use the service
inputs = ['is it really this easy?', 'amazing technology', 'wow']
outputs = service.translations().list(source='en', target='fr', q=inputs).execute()
# print outputs
for input, output in zip(inputs, outputs['translations']):
  print("{0} -> {1}".format(input, output['translatedText']))


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:
<h2> Invoke Vision API </h2>

The Vision API can work off an image in Cloud Storage or embedded directly into a POST message. I'll use Cloud Storage and do OCR on this image: <img src="https://storage.googleapis.com/cloud-training-demos/vision/sign2.jpg" width="200" />.  That photograph is from http://www.publicdomainpictures.net/view-image.php?image=15842
Can you write Python code for it?

# Running the Vision API
import base64
IMAGE="gs://cloud-training-demos/vision/sign2.jpg"
vservice = build('vision', 'v1', developerKey=APIKEY)
request = vservice.images().annotate(body={
        'requests': [{
                'image': {
                    'source': {
                        'gcs_image_uri': IMAGE
                    }
                },
                'features': [{
                    'type': 'TEXT_DETECTION',
                    'maxResults': 3,
                }]
            }],
        })
responses = request.execute(num_retries=3)

# Let's output the `responses`
print(responses)

foreigntext = responses['responses'][0]['textAnnotations'][0]['description']
foreignlang = responses['responses'][0]['textAnnotations'][0]['locale']

# Let's output the `foreignlang` and `foreigntext`
print(foreignlang, foreigntext)


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:
<h2> Sentiment analysis with Language API </h2>

Let's evaluate the sentiment of some famous quotes using Google Cloud Natural Language API.
Can you write Python code for it?

# Evaluating the sentiment with Google Cloud Natural Language API
lservice = build('language', 'v1beta1', developerKey=APIKEY)
quotes = [
  'To succeed, you must have tremendous perseverance, tremendous will.',
  'It’s not that I’m so smart, it’s just that I stay with problems longer.',
  'Love is quivering happiness.',
  'Love is of all passions the strongest, for it attacks simultaneously the head, the heart, and the senses.',
  'What difference does it make to the dead, the orphans and the homeless, whether the mad destruction is wrought under the name of totalitarianism or in the holy name of liberty or democracy?',
  'When someone you love dies, and you’re not expecting it, you don’t lose her all at once; you lose her in pieces over a long time — the way the mail stops coming, and her scent fades from the pillows and even from the clothes in her closet and drawers. '
]
for quote in quotes:
# The `documents.analyzeSentiment` method analyzes the sentiment of the provided text.
  response = lservice.documents().analyzeSentiment(
    body={
      'document': {
         'type': 'PLAIN_TEXT',
         'content': quote
      }
    }).execute()
  polarity = response['documentSentiment']['polarity']
  magnitude = response['documentSentiment']['magnitude']

# Lets output the value of each `polarity`, `magnitude` and `quote`
  print('POLARITY=%s MAGNITUDE=%s for %s' % (polarity, magnitude, quote))


lservice = build('language', 'v1beta1', developerKey=APIKEY)
quotes = [
  'To succeed, you must have tremendous perseverance, tremendous will.',
  'It’s not that I’m so smart, it’s just that I stay with problems longer.',
  'Love is quivering happiness.',
  'Love is of all passions the strongest, for it attacks simultaneously the head, the heart, and the senses.',
  'What difference does it make to the dead, the orphans and the homeless, whether the mad destruction is wrought under the name of totalitarianism or in the holy name of liberty or democracy?',
  'When someone you love dies, and you’re not expecting it, you don’t lose her all at once; you lose her in pieces over a long time — the way the mail stops coming, and her scent fades from the pillows and even from the clothes in her closet and drawers. '
]
for quote in quotes:
  response = lservice.documents().analyzeSentiment(
    body={
      'document': {
         'type': 'PLAIN_TEXT',
         'content': quote
      }
    }).execute()
  polarity = response['documentSentiment']['polarity']
  magnitude = response['documentSentiment']['magnitude']
  print('POLARITY=%s MAGNITUDE=%s for %s' % (polarity, magnitude, quote))


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:

 Speech API

The Speech API can work on streaming data, audio content encoded and embedded directly into the POST message, or on a file on Cloud Storage. Here I'll pass in this audio file in Cloud Storage.
Can you write Python code for it?

# Using the Speech API by passing audio file as an argument
sservice = build('speech', 'v1', developerKey=APIKEY)
# The `speech.recognize` method performs synchronous speech recognition.
# It receive results after all audio has been sent and processed.
response = sservice.speech().recognize(
    body={
        'config': {
            'languageCode' : 'en-US',
            'encoding': 'LINEAR16',
            'sampleRateHertz': 16000
        },
        'audio': {
            'uri': 'gs://cloud-training-demos/vision/audio.raw'
            }
        }).execute()

# Let's output the value of `response`
print(response)

print(response['results'][0]['alternatives'][0]['transcript'])

# Let's output the value of `'Confidence`
print('Confidence=%f' % response['results'][0]['alternatives'][0]['confidence'])


I am working on the file "CPB100/lab4c/mlapis.ipynb".
The first task is:
<h2> Translate sign </h2>
Can you write Python code for it?

inputs=[foreigntext]
outputs = service.translations().list(source=foreignlang, target='en', q=inputs).execute()
# print(outputs)
for input, output in zip(inputs, outputs['translations']):
  print("{0} -> {1}".format(input, output['translatedText']))


inputs=[foreigntext]
outputs = service.translations().list(source=foreignlang, target='en', q=inputs).execute()

# Print the outputs
for input, output in zip(inputs, outputs['translations']):
  print("{0} -> {1}".format(input, output['translatedText']))


I am working on the file "CPB102/tensorflow/tfclassic.ipynb".
The first task is:
TensorFlow 付属のモジュールを使って MNIST データセットをダウンロードします。
Can you write Python code for it?

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)


I am working on the file "CPB102/tensorflow/tfclassic.ipynb".
The first task is:
ここから先は tf.layers を使ったときと全く同じです。
Can you write Python code for it?

cross_entropy = -tf.reduce_mean(y_ph * tf.log(y))

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(y_ph, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

train_op = tf.train.GradientDescentOptimizer(1e-1).minimize(cross_entropy)

init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    for i in range(10001):
        x_train, y_train = mnist.train.next_batch(100)
        sess.run(train_op, feed_dict={x_ph: x_train, y_ph: y_train})
        if i % 100 == 0:
            train_loss = sess.run(cross_entropy, feed_dict={x_ph: x_train, y_ph: y_train})
            test_loss = sess.run(cross_entropy, feed_dict={x_ph: mnist.test.images, y_ph: mnist.test.labels})
            tf.logging.info("Iteration: {0} Training Loss: {1} Test Loss: {2}".format(i, train_loss, test_loss))
    test_accuracy = sess.run(accuracy, feed_dict={x_ph: mnist.test.images, y_ph: mnist.test.labels})
    tf.logging.info("Accuracy: {}".format(test_accuracy))


I am working on the file "CPB102/tensorflow/tfclassic.ipynb".
The first task is:
ニューラルネットの weight を tf.Variable として作成します。
行列積の計算なども自分で記述する形になります。
Can you write Python code for it?

weights = tf.Variable(tf.random_normal([784, 20], stddev=0.1))
biases = tf.Variable(tf.zeros([20]))
hidden = tf.matmul(x_ph, weights) + biases

weights = tf.Variable(tf.random_normal([20, 10], stddev=0.1))
biases = tf.Variable(tf.zeros([10]))
logits = tf.matmul(hidden, weights) + biases

y = tf.nn.softmax(logits)


I am working on the file "CPB102/tensorflow/tflayers.ipynb".
The first task is:
tf.Variable を初期化するオペレーションを作成します。
Can you write Python code for it?


I am working on the file "CPB102/tensorflow/tflayers.ipynb".
The first task is:
ニューラルネットの入力となる Tensor を tf.placeholder で用意します。
学習の際にランダムサンプリングしたデータを使って weight の更新を行うので、後から使うデータを変更できるように tf.placeholder を使います。
また、データをいくつずつ渡していくかも後で自由に決められるように、 Tensor の shape を [None, 784] と指定しています。
TensorFlow では tf.placeholder の shape を指定する時に、不明な場合は None とすることが可能です。
ただし、一部のオペレーションは Tensor の shape がきちんと定義されていないと実行できない場合があるので注意してください。
Can you write Python code for it?

x_ph = tf.placeholder(tf.float32, [None, 784])
y_ph = tf.placeholder(tf.float32, [None, 10])


I am working on the file "CPB102/tensorflow/tflayers.ipynb".
The first task is:
損失関数として cross entropy を定義します。
Can you write Python code for it?

cross_entropy = -tf.reduce_mean(y_ph * tf.log(y))


I am working on the file "CPB102/tensorflow/tflayers.ipynb".
The first task is:
最小化してほしい cross_entropy を渡して、勾配法で tf.Variable を更新してくれるオペレーションを作成します。
tf.layers を使う場合は tf.Variable の存在が隠蔽されていますが、裏ではニューラルネットの辺にあたる部分 (weight) を tf.Variable として作成して計算グラフに追加しています。
Can you write Python code for it?


I am working on the file "CPB102/tensorflow/tflayers.ipynb".
The first task is:
計算グラフを構築し終えたら、あとはオペレーション (ノード) を選んで実行するだけです。
ランダムサンプリングしたデータを tf.placeholder に渡しつつ、繰り返し train_op を実行します。
Can you write Python code for it?

with tf.Session() as sess:
    sess.run(init_op)
    for i in range(3001):
        x_train, y_train = mnist.train.next_batch(100)
        sess.run(train_op, feed_dict={x_ph: x_train, y_ph: y_train})
        if i % 100 == 0:
            train_loss = sess.run(cross_entropy, feed_dict={x_ph: x_train, y_ph: y_train})
            test_loss = sess.run(cross_entropy, feed_dict={x_ph: mnist.test.images, y_ph: mnist.test.labels})
            tf.logging.info("Iteration: {0} Training Loss: {1} Test Loss: {2}".format(i, train_loss, test_loss))
    test_accuracy = sess.run(accuracy, feed_dict={x_ph: mnist.test.images, y_ph: mnist.test.labels})
    tf.logging.info("Accuracy: {}".format(test_accuracy))


I am working on the file "CPB102/tensorflow/tfsaver.ipynb".
The first task is:
sample.ckpt というファイルが生成されていれば無事保存されています。
この中に {"v1": 1, "v2": 2} のような key/value の形式 (実際にこのような JSON で保存されているわけではありません) で名前と値が保存されていると思ってください。
Checkpoint を読み込むときは、以下のように同じ名前の tf.Variable を作ってやれば良いです。
Can you write Python code for it?

with tf.Graph().as_default() as g2:
    v1 = tf.Variable(3., name="v1")
    v2 = tf.Variable(4., name="v2")
    init_op = tf.global_variables_initializer()
    saver = tf.train.Saver()

with tf.Session(graph=g2) as sess:
    sess.run(init_op)
    print(sess.run([v1, v2]))
    saver.restore(sess, os.path.join(CHECKPOINT_DIR, "sample.ckpt"))
    print(sess.run([v1, v2]))


I am working on the file "CPB102/tensorflow/tfsaver.ipynb".
The first task is:
tf.Variable にそれぞれ "v1", "v2" という名前を付けて保存します。
Can you write Python code for it?

with tf.Graph().as_default() as g1:
    v1 = tf.Variable(1., name="v1")
    v2 = tf.Variable(2., name="v2")
    init_op = tf.global_variables_initializer()
    saver = tf.train.Saver()

with tf.Session(graph=g1) as sess:
    sess.run(init_op)
    saver.save(sess, os.path.join(CHECKPOINT_DIR, "sample.ckpt"))


I am working on the file "CPB102/tensorflow/tfsaver.ipynb".
The first task is:
まずは checkpoint を保存するディレクトリを作成しましょう。
Can you write Python code for it?

CHECKPOINT_DIR = "saver_sample"

if not os.path.isdir("saver_sample"):
    os.mkdir("saver_sample")


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
Hello TensorFlow
Python を使って足し算をしてみましょう（決して馬鹿にしているわけではなく大真面目です）！
Can you write Python code for it?


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
TensorFlow
Can you write Python code for it?

a = tf.constant([5, 3, 8])
b = tf.constant([3, -1, 2])
c = tf.add(a, b)

print(c)

with tf.Session() as sess:
    result = sess.run(c)

print(result)


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
TensorFlow + placeholder
計算をするたびにデータフローグラフを作っていると、データフローグラフがどんどん肥大化してしまいます。
そのため、 TensorFlow には一部の値を差し替えつつデータフローグラフの共通部分を再利用するための placeholder という仕組みが存在しています。
Can you write Python code for it?

a = tf.placeholder(dtype=tf.int32, shape=(None,))
b = tf.placeholder(dtype=tf.int32, shape=(None,))
c = tf.add(a, b)

with tf.Session() as sess:
    result1 = sess.run(c, feed_dict={a: [3, 4, 5], b: [-1, 2, 3]})
    result2 = sess.run(c, feed_dict={a: [1, 2, 3], b: [3, 2, 1]})

print(result1)
print(result2)


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
tf.Variable 最初に初期化処理を行う必要があるので init_op を実行します。
その後 v の値を表示すると assign_op を実行する前後で値が書き換えられていることが確認できます。
Can you write Python code for it?

with tf.Session() as sess:
    sess.run(init_op)
    print(sess.run(v))
    sess.run(assign_v)
    print(sess.run(v))


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
典型的な使い方として、学習時に使うデータを placeholder で定義しておくという方法があります。
placeholder に対して少しずつ学習用データを流していくというのが TensorFlow で機械学習のアルゴリズムを実装するときの定石です。
tf.Variable
TensorFlow では、計算に使われた値は基本的に捨てられていきます。
tf.add の結果はセッションを介して実行するたびに計算し直すことになりますし、 tf.constant の値はメモリ上に保持されず必要に応じて再生成されます。
後から tf.constant の値を書き換えることもできません。
ただし tf.Variable だけが例外で、値をメモリ上に保持し続けて後から書き換えることも可能になっています。
ニューラルネットやその他多くの機械学習手法で weight として使うことが想定されています。
Can you write Python code for it?

v = tf.Variable([1, 2])
assign_op = tf.assign(v, [2, 3])
init_op = tf.global_variables_initializer()


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
当然ですが 3.0 と答えが表示されます。
今度は TensorFlow で同じような足し算をやってみましょう。
Can you write Python code for it?

a = tf.constant(1.)
b = tf.constant(2.)
c = tf.add(a, b)

print(c)


I am working on the file "CPB102/tensorflow/tfstart.ipynb".
The first task is:
行列演算
NumPy と TensorFlow で行列やベクトルの計算方法を比較してみましょう。
NumPy
Can you write Python code for it?

a = np.array([5, 3, 8])
b = np.array([3, -1, 2])
c = np.add(a, b)

print(c)


I am working on the file "CPMX6/.ipynb_checkpoints/Example_Ipython-checkpoint.ipynb".
The first task is:

Markdown

You can write text in markdown too!

Check out some markdown basics here: http://nbviewer.ipython.org/url/www.astro.washington.edu/users/vanderplas/Astr599/notebooks/Markdown%20Cells.ipynb!

Python code
This code finds the factorial of a series of number in one line:
Can you write Python code for it?

#this is a terrible and awesome way to do this:
factorial = (lambda k: reduce(int.__mul__, range(1,k+1),1))

map(factorial, range(1,9))


I am working on the file "CPMX6/.ipynb_checkpoints/Example_Ipython-checkpoint.ipynb".
The first task is:
Magic functions
This shows the use of builtin timeit magic using the following %-prefix command:
Can you write Python code for it?

import random
%timeit a = [random.random() for x in range(1000)]


I am working on the file "CPMX6/.ipynb_checkpoints/Example_Ipython-checkpoint.ipynb".
The first task is:
Or images stored on the web like this:
Can you write Python code for it?

<img src="http://cdn.alltheragefaces.com/img/faces/large/misc-you-dont-say-l.png" width=300 height=300/>


I am working on the file "CPMX6/.ipynb_checkpoints/Example_Ipython-checkpoint.ipynb".
The first task is:
The builtin magics include:

Functions that work with code: %run, %edit, etc.

Functions to run code in other languages: %%bash %%perl %%ruby %%javascript, etc.

*See some examples with magic functions here: http://nbviewer.ipython.org/github/ipython/ipython/blob/1.x/examples/notebooks/Cell%20Magics.ipynb.
Display images
You can display images stored locally like this:
Can you write Python code for it?

from IPython.display import display, Image
display(Image(filename='misc-you-dont-say-l.png'))