I am working on the file "Kryptos.ipynb".
The first task is:
Again, the key was found successfully. Interestingly, before finding CRYPTANALYSIS, very similar words of the same length were identified as candidate keys:
    - ACROPARALYSIS
    - CHROMATOLYSIS
    - CRYPTANALYSIS
Keyed Caesar Variants
Keyed Caesar Variants
The Caesar Cipher is a substitution cipher, which substitutes letters by letters at the same index in a shifted version of the alphabet, i.e. in case of rot13, a Caesar cipher with key 'N'
    plain  alphabet       ABCDEFGHIJKLMNOPQRSTUVWXYZ
    cipher alphabet       NOPQRSTUVWXYZABCDEFGHIJKLM (and other shifted versions of it, shown alphabet corresponds to key 'N')

A variant of the Caesar cipher is the Keyed Caesar Cipher, which uses an additional alphabet keyword to create a keyed cipher alphabet, i.e. in case the alphabet keyword is KRYPTOS:
    plain  alphabet       ABCDEFGHIJKLMNOPQRSTUVWXYZ
    cipher alphabet       KRYPTOSABCDEFGHIJLMNQUVWXZ (and other shifted versions of it, shown alphabet corresponds to key 'A')
                          RYPTOSABCDEFGHIJLMNQUVWXZK (and other shifted versions of it, shown alphabet corresponds to key 'B')

Finally, we can also use a keyed alphabet for both the plain alphabet and the cipher alphabet to obtain we will call a Caesar Cipher with a keyed alphabet here (note the difference with Keyed Caesar):
    plain  alphabet       KRYPTOSABCDEFGHIJLMNQUVWXZ
    cipher alphabet       KRYPTOSABCDEFGHIJLMNQUVWXZ (and other shifted versions of it, shown alphabet corresponds to key 'K')
                          RYPTOSABCDEFGHIJLMNQUVWXZK (and other shifted versions of it, shown alphabet corresponds to key 'R')

First, let's define a function for computing a keyed alphabet. It takes an alphabet and a keyword consisting of symbols from that alphabet. Then, it removes all but the first occurence of each character in the keyword (i.e. CRYPTANALYSIS becomes CRYPTANLSI), and then appends the remaining characters of the alphabet in order to obtain the keyed alphabet (i.e. CRYPTANLSIBDEFGHJKMOQUVWXZ)
Can you write Python code for it?

import re

def compute_keyed_alphabet(keyword, alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"):
    # remove double keyword letters, keeping first occurence
    keyword_unique = ""
    for c in keyword:
        if c not in keyword_unique:
            keyword_unique += c
    # compute cipher alphabet
    keyed_alphabet = keyword_unique + re.sub("[%s]" % keyword, "", alphabet)
    return keyed_alphabet

def keyed_caesar(s, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key="A", direction=1):
    # compute cipher alphabet
    keyed_alphabet = compute_keyed_alphabet(alpha_key, alphabet)
    # set source and destination alphabet depending on direction
    if direction == 1:
        src_alphabet = alphabet
        dst_alphabet = keyed_alphabet
    else:
        src_alphabet = keyed_alphabet
        dst_alphabet = alphabet
    # encrypt / decrypt
    keyval = alphabet.find(key)
    t = ""
    for sc in s:
        i = src_alphabet.find(sc)
        t += dst_alphabet[(i + keyval * direction) % len(dst_alphabet)] if i > -1 else sc
    return t


I am working on the file "Kryptos.ipynb".
The first task is:
BETWEEN SUBTLE SHADING AND THE ABSENCE OF LIGHT LIES THE NUANCE OF IQLUSION

Kryptos - K2
Let's start with the rest of the lines of the first panel for K2:
Can you write Python code for it?

ctext = "".join(ctext_kryptos[2:14])
ctext


I am working on the file "Kryptos.ipynb".
The first task is:
Based on the IC value, it looks like we are not dealing with plaintext, or a transposition of plaintext.
K2 - Determine Key Length
Can you write Python code for it?

from matplotlib import pyplot as plt
%matplotlib inline

mean_ic_n = []
for n in range(1,32):
    mean_ic = mean_IC(ctext.upper(), n)
    mean_ic_n.append(mean_ic)
    print "%2d %02f" % (n, mean_ic)
plt.rcParams["figure.figsize"] = [12,4]
plt.bar(range(1,len(mean_ic_n)+1), mean_ic_n, align = 'center')
plt.xlabel("key length")
plt.ylabel("average IC")


I am working on the file "Kryptos.ipynb".
The first task is:
Based on the IC, it looks like this is not plaintext, or a transposition of plaintext.
K1 - Determine Key Length
Can you write Python code for it?


I am working on the file "Kryptos.ipynb".
The first task is:
Caesar Cipher Solution #1: Brute-force
However, what if we do not know the key? One option would be to brute-fortry all possible keys, and just read off the solution. There are only 26 possibilities after all.
Can you write Python code for it?

alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
for key in alphabet:
    decrypted_ctext = rot(ctext, key, direction = -1)
    print "%s:\t%s" % (key, decrypted_ctext)


I am working on the file "Kryptos.ipynb".
The first task is:
Caesar Cipher Solution #2: Frequency Analysis
Alternatively, statistics of the text could be used to find the most likely key. Let's look at the frequency of the letters and see what happens when you encrypt / decrypt with Caesar cipher.
Can you write Python code for it?

from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
%matplotlib inline
plt.rcParams["figure.figsize"] = [12,4]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

# determine letter frequency in plaintext, and plot
ptext_freq = [ptext.count(c)/len(ptext) for c in alphabet]
pfig = plt.figure()
plt.bar(range(len(alphabet)), ptext_freq, tick_label = list(alphabet), align = 'center', color = 'b')
plt.title('plaintext letter frequency')

# determine letter frequency in ciphertext, and plot
ctext_freq = [ctext.count(c)/len(ctext) for c in alphabet]
cfig = plt.figure()
plt.bar(range(len(alphabet)), ctext_freq, tick_label = list(alphabet), align = 'center', color = 'r')
plt.title('ciphertext letter frequency')


I am working on the file "Kryptos.ipynb".
The first task is:
Caesar Cipher Solution #4: Maximum likelihood
TL;DR: for a given decoding, estimate how much like English (or another language) it is. Best key is the one that gives the decoding that is most like English (or another language)...
The final way to solve is to, for each possible key, compute the likelihood that the decrypted text is plaintext in a particular language. This of course is unnecessary for something as simple as the Caesar cipher, but now think of a cipher where you would have thousands or millions of possible keys...
For now, let's focus on understanding how it works. Spoiler: in a next section, this approach will be used to automatically determine the key for Kryptos and decrypted the K1 and K2 parts of it.
In the following we will assume this to be English, but it could just as well be a different language with different statistics. This is something that you should decide or guess from context.
Computing likelihood based on observed n-grams
In order to measure how much like English a certain string is, we need to compare its properties with strings from the English language. The criterion that will be used here to measure this will be so-called n-gram statistics, which in short, measuring for each sequence of n symbols, what is the frequency / probability that it appears in an English text.
Then, the likelihood that the observed decrypted ciphertext is an English text, is the joint probability of the observing the ngrams in English. Assuming the observations are independent and identically distributed, the joint probability of the observed ngrams in the decrypted ciphertext as follows:
$$
\mathcal{L}(English \mid decrypt(ctext, key)) = p(decrypt(ctext, key)) \mid English) = \prod_i p(ngram_i \mid English)
$$
where $ngram_i$ is the $i^{th}$ sequence of $n$ letters in the decrypted ciphertext. For example, in case $decrypt(ctext, key) = \texttt{ANEXAMPLETEXT}$ and using 3-grams, the observed 3-grams are
$$
\texttt{ANE, NEX, EXA, XAM, AMP, MPL, PLE, LET, ETE, TEX, EXT},
$$
and therefore the...
Can you write Python code for it?

import re

def find_ngrams(text, n):
    text = re.sub(r'[^A-Z]', r'', text.upper()) #remove everything that is not A-Z
    return zip(*[text[i:] for i in range(n)]) #determine ngrams


I am working on the file "Kryptos.ipynb".
The first task is:
Cracking Caesar Cipher using Maximum Likelihood
Finally, let's crack Caesar using this. We use function ngram_score (source: practicalcryptography.com) for computing the log-likelihood that a text is a particular language, given a file with ngram counts for that language.
Can you write Python code for it?

from ngram_score import ngram_score
fitness = ngram_score('ngrams/en_sherlock_1grams') # load our ngram statistics

alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
results = []
for key in alphabet:
    decrypted_ctext = rot(ctext, key, direction = -1)
    ll_decrypted_ctext = fitness.score(decrypted_ctext)
    results.append((ll_decrypted_ctext, key, decrypted_ctext))
    print "%s:\t%f\t%s" % (key, ll_decrypted_ctext, decrypted_ctext)
print "\nthe most likely key and plaintext:\n %s" % repr(max(results))


I am working on the file "Kryptos.ipynb".
The first task is:
Cryptanalysis - Caesar with Keyed Alphabet
Effect of Caesar with Keyed Alphabet on Frequency Distributions
Can you write Python code for it?

def caesar_keyed_alphabet(s, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key="", direction=1):
    # compute keyed alphabet
    keyed_alphabet = compute_keyed_alphabet(alpha_key, alphabet)
    return rot(s, key, keyed_alphabet, direction)

ptext = "THESCULPTUREHASBEENBOTHAPUZZLEANDAMYSTERYFORTHOSEWHOHOPETOCRACKTHECYPHEREDMESSAGESCONTAINEDWITHINTHESCULPTURESTWOTHOUSANDALPHABETICLETTERSINTHETWENTYYEARSSINCEKRYPTOSWASERECTEDTHREEOFTHEFOURSECTIONSHAVEBEENCONFIRMEDTOHAVEBEENSOLVEDNOONEHASYETBEENABLETOSOLVETHEREMAININGNINETYSEVENCHARACTERMESSAGE"
alpha_key_ = "KRYPTOS"

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,9]
alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
keyed_alphabet = compute_keyed_alphabet(alpha_key_, alphabet)

def rotate(l, n):
    return l[n:] + l[:n]

@interact(key_i = (0,25,1))
def plot_graphs(key_i = 0):
    # encrypt plaintext using this key
    key = keyed_alphabet[key_i]
    ctext = caesar_keyed_alphabet(ptext, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key=alpha_key_, direction=1)

    # determine letter frequency in plaintext, and plot
    ptext_freq = np.array([ptext.count(c)/len(ptext) for c in keyed_alphabet])
    pfig = plt.subplot(3,1,1)
    plt.bar(range(len(keyed_alphabet)), ptext_freq, tick_label = list(keyed_alphabet), align = 'center', color = 'b')
    plt.title('blue = plaintext letter frequency, red = ciphertext letter frequency')

    # determine letter frequency in ciphertext, and plot, in order keyed alphabet
    ctext_freq = np.array([ctext.count(c)/len(ctext) for c in keyed_alphabet])
    cfig = plt.subplot(3,1,2)
    plt.bar(range(len(keyed_alphabet)), ctext_freq, tick_label = rotate(list(keyed_alphabet), key_i), align = 'center', color = 'r')
   ...


I am working on the file "Kryptos.ipynb".
The first task is:
Cryptanalysis - Keyed Caesar
Effect of Keyed Caesar on Frequency Distributions
Can you write Python code for it?


I am working on the file "Kryptos.ipynb".
The first task is:
Cryptanalysis - Keyed Vigenere
First, let's load some Keyed Vigenere ciphertext, with unknown key and unknown alphabet key.
Can you write Python code for it?


I am working on the file "Kryptos.ipynb".
The first task is:
For easy reference, this is the subtitution when keying the alphabet with 'KRYPTOS', and using key 'N':
    plain  alphabet       ABCDEFGHIJKLMNOPQRSTUVWXYZ
    cipher alphabet       KRYPTOSABCDEFGHIJLMNQUVWXZ (and other shifted versions of it, shown alphabet corresponds to key 'A')
                          GHIJLMNQUVWXZKRYPTOSABCDEF (and other shifted versions of it, shown alphabet corresponds to key 'N')

From this plot, it can be seen that when we do not know what was the keyword for creating the keyed alphabet, the cipher cannot be broken by simply aligning the frequency distributions and a more general approach for breaking the substitution is needed.
Keyed Caesar Solution #1: Maximum Likelihood, Combined with Search Procedure
However, since it is a substitution cipher, we could find the substitution and entire cipher alphabet as follows:
- guessing the individual substitutions based on frequency: assume most frequent ciphertext character is 'E' etc
- given this cipher alphabet, gradually try to improve the substitution by swapping characters in the cipher alphabet
- measure the likelihood of plaintext being English as before, and keep track of best keys found so far
More on this approach can be found here.
Can you write Python code for it?

ctext = keyed_caesar(ptext, "N", alpha_key="KRYPTOS", direction=1)
ctext

def substitution(s, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", direction=1):
    if direction == 1:
        src_alphabet = alphabet
        dst_alphabet = key
    else:
        src_alphabet = key
        dst_alphabet = alphabet

    t = ""
    for c in s:
        if c in src_alphabet:
            t += dst_alphabet[src_alphabet.find(c)]
        else:
            t += c
    return t

import random
import re
from ngram_score import ngram_score
fitness = ngram_score('ngrams/en_sherlock_4grams') # load our quadgram statistics

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
maxkey = list(alphabet)
maxscore = -99e9
parentscore,parentkey = maxscore,maxkey[:]

print "Substitution Cipher solver, you may have to wait several iterations for the correct result. Halt execution by halting the kernel, using the stop button in the toolbar."
# keep going until we are killed by the user
i = 0
rec = nbest(1000)

MAX_ITER = 10

while i < MAX_ITER:
    i = i+1
    random.shuffle(parentkey)
    # KEYED CAESAR WITH KEY "A" IS JUST A SUBSTITUTION CIPHER
    deciphered = substitution(ctext, "".join(parentkey), alphabet, direction=-1)
    parentscore = fitness.score(deciphered)
    count = 0
    while count < 1000:
        a = random.randint(0,len(alphabet)-1)
        b = random.randint(0,len(alphabet)-1)
        child = parentkey[:]
        # swap two characters in the child
        child[a],child[b] = child[b],child[a]
        deciphered = substitution(ctext, "".join(child), alphabet, direction=-1)
        score = fitness.score(deciphered)
        # if the child was better, replace the parent with it
        if score > parentscore:
            parentscore = score
            parentkey = child[:]
            count = 0
        count = count+1
        rec.add((score, child, deciphered))
    # keep track of best score printed so far, and print if improved
    if parentscore > maxscore:
        maxscore = rec[0][0]
        maxkey = rec[0][1]
        #deciphered =...


I am working on the file "Kryptos.ipynb".
The first task is:
From the mean IC and peaks at key length 10 and 20, it looks like the key length might be 10.
K1 - Frequency Analysis of Subsequences #1
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

key_length = 10

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)), g_english, tick_label = rotate(list(alphabet), 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(alphabet)), ctext_freq[i], tick_label = rotate(list(alphabet), 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
From this, it can be seen that only the first 2 lines make up K1.
Can you write Python code for it?

ctext_k1 = "".join(ctext_kryptos[:2])

key = "PALIMPSEST"
alphakey = "KRYPTOS"
ptext_k1 = keyed_vigenere(ctext_k1, key, alpha_key = alphakey, direction = -1)

print ctext_k1
print ptext_k1


I am working on the file "Kryptos.ipynb".
The first task is:
Here we can see that using the different criterion and search algorithm the correct key and plaintext have successfully been found.
Vigenere Solution #3: Maximum Likelihood, Combined with Brute Force
Instead of trying to determine the key character by character, we can also bruteforce it and keep track of the best key as before.
The wordlist used here was downloaded from:
https://packetstormsecurity.com/Crackers/wordlists/dictionaries/
Can you write Python code for it?

ctext = "VYCHVUYPESJMZCJZTXNOOEFSXMBQJTTNQAXWKBWTPDDKTUODCOPGJFNTMOPRLACBZGTWEAEEEOKWAKCXCHVOATLGFMVYZRWBNGHPQUCDRKSGXSGWZRZWMURLSTLCHLZWBAECCIMEESTLLPWVNCCMYLELPKAAPTCZKYCTZQOIKGICRMEQTSPWMGHKFTYOHRDCUBAQEQWTVRBPCFKGPWGGFEQTZFSDWDVCCLOYVPBFWGPVFPLYRTMCWVSDCCIHSBLGCLPWTVKPBNVNRLAVWVPQTOEACSYJIUVVPBXSFARC"

from ngram_score import ngram_score

import re
import pprint as pp

qgram = ngram_score('ngrams/en_sherlock_4grams') # load our 4gram statistics
trigram = ngram_score('ngrams/en_sherlock_3grams') # load our 3gram statistics

# keep track of the 100 best keys
N=100
rec = nbest(N)

f = open('wordlists/websters-dictionary','r')

i = 1
for key in f:
    key = re.sub(r'[^A-Z]','',key.upper())
    decrypted_ctext = vigenere(ctext, key, direction = -1)
    score = qgram.score(decrypted_ctext)
    rec.add((score,key, decrypted_ctext))
    i += 1
    if i % 10000 == 0:
        bestscore = rec[0][0]
        bestkey = rec[0][1]
        decrypted_ctext = vigenere(ctext, bestkey, direction = -1)
        print "%20s\t%5.2f,\t%s,\t%s,\t%s" % (key, bestscore, len(bestkey), bestkey, decrypted_ctext)


I am working on the file "Kryptos.ipynb".
The first task is:
If we look at the frequencies in order of the keyed alphabet, then again the effect of Caesar with Keyed Alphabet can be seen as shifting the frequency distributions. However, this is not the case when the looking at the frequencies in order of the alphabet.
Determine Key by Aligning Frequency Distributions?
Can you write Python code for it?

ptext = "THESCULPTUREHASBEENBOTHAPUZZLEANDAMYSTERYFORTHOSEWHOHOPETOCRACKTHECYPHEREDMESSAGESCONTAINEDWITHINTHESCULPTURESTWOTHOUSANDALPHABETICLETTERSINTHETWENTYYEARSSINCEKRYPTOSWASERECTEDTHREEOFTHEFOURSECTIONSHAVEBEENCONFIRMEDTOHAVEBEENSOLVEDNOONEHASYETBEENABLETOSOLVETHEREMAININGNINETYSEVENCHARACTERMESSAGE"
ctext = caesar_keyed_alphabet(ptext, key="N", alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key="KRYPTOS", direction=1)
ctext

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,6]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

def rotate(l, n):
    return l[n:] + l[:n]

@interact(key_i = (0,25,1))
def plot_graphs(key_i = 0):
    # decrypt ciphertext using this key
    key = alphabet[key_i]
    # NOTE THAT WE DO NOT KNOW THE KEY TO THE ALPHABET, SO WE CANNOT USE THAT!!!
    decrypted_ctext = caesar_keyed_alphabet(ctext, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key="A", direction=-1)

    # determine letter frequency in plaintext, and plot
    ptext_freq = np.array([ptext.count(c)/len(ptext) for c in alphabet])
    pfig = plt.subplot(2,1,1)
    plt.bar(range(len(alphabet)), ptext_freq, tick_label = list(alphabet), align = 'center', color = 'b')
    plt.title('blue = plaintext letter frequency, green = decrypted ciphertext letter frequency')

    # determine letter frequency in ciphertext, and plot
    ctext_freq = np.array([decrypted_ctext.count(c)/len(decrypted_ctext) for c in alphabet])
    cfig = plt.subplot(2,1,2)
    plt.bar(range(len(alphabet)), ctext_freq, tick_label = rotate(list(alphabet), key_i), align = 'center', color = 'g')
    plt.xlabel("key = %s" % key)

    plt.show()
    print decrypted_ctext


I am working on the file "Kryptos.ipynb".
The first task is:
It takes a letter as key, finds its position in the alphabet, and shifts / rotates the alphabet by that amount. Symbols that are not in the alphabet are unaffected. Now, define some text and encrypt it using the function:
Can you write Python code for it?

ptext = '"I HEAR AND I FORGET. I SEE AND I REMEMBER. I DO AND I UNDERSTAND." --CONFUSIUS'
ctext = rot(ptext, 'N')
ctext


I am working on the file "Kryptos.ipynb".
The first task is:
K1 - Frequency Analysis of Subsequences #2
Now, let's try that again, but use the cipher alphabet order.
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
keyed_alphabet = compute_keyed_alphabet("KRYPTOS")

key_length = 10

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
g_english_keyed = [g_english[alphabet.find(c)] for c in keyed_alphabet]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)), g_english_keyed, tick_label = rotate(list(keyed_alphabet), 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in keyed_alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(keyed_alphabet)), ctext_freq[i], tick_label = rotate(list(keyed_alphabet), 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
K1 - Vigenere Key is Unknown, Alphabet Key is Known
Can you write Python code for it?

from itertools import permutations
from ngram_score import ngram_score

import re
import pprint as pp

qgram = ngram_score('ngrams/en_sherlock_4grams') # load our 4gram statistics
trigram = ngram_score('ngrams/en_sherlock_3grams') # load our 3gram statistics

# keep track of the 100 best keys
N=100
L=42 # only process L characters for speed
alphakey = "KRYPTOS"

for KLEN in range(3,16):
    print "="*80
    rec = nbest(N)

    # exhaustively test all possible letters for first 3 entries of the key and keep track of the N best ones
    # if KLEN=7, this will test e.g. FOOAAAA and BARAAAA
    for i in permutations('ABCDEFGHIJKLMNOPQRSTUVWXYZ',3):
        i = "".join(i)
        key = ''.join(i) + 'A'*(KLEN-len(i))
        decrypted_ctext = keyed_vigenere(ctext[:L], key, alpha_key = alphakey, direction = -1)
        score = 0
        for j in range(0,len(ctext),KLEN):
            score += trigram.score(decrypted_ctext[j:j+3])
        rec.add((score,''.join(i), decrypted_ctext))
    next_rec = nbest(N)

    # for the remaining KLEN-3 characters of the key,
    for i in range(0,KLEN-3):
        # go over the N best keys found so far...
        for k in xrange(N):
            # ...and determine the best next character of the key, while keeping best N keys so far
            for c in 'ABCDEFGHIJKLMNOPQRSTUVWXYZ':
                key = rec[k][1] + c
                fullkey = key + 'A'*(KLEN-len(key))
                decrypted_ctext = keyed_vigenere(ctext[:L], fullkey, alpha_key = alphakey, direction = -1)
                score = 0
                for j in range(0,len(ctext),KLEN):
                    score += qgram.score(decrypted_ctext[j:j+len(key)])
                next_rec.add((score, key, decrypted_ctext))
        rec = next_rec
        next_rec = nbest(N)

    # show the results
    bestscore = rec[0][0]
    bestkey = rec[0][1]
    #decrypted_ctext = rec[0][2]
    # always show entire decrypted ctext, even if the above analysis is done only on part of the ctext, e.g. ctext[0:100]
   ...


I am working on the file "Kryptos.ipynb".
The first task is:
K2 - Determining Message Boundaries
Can you write Python code for it?

ctext = "".join(ctext_kryptos[2:])
key = "ABSCISSA"
alphakey = "KRYPTOS"
print ctext
print keyed_vigenere(ctext, key, alpha_key = alphakey, direction = -1)


I am working on the file "Kryptos.ipynb".
The first task is:
K2 - Determining Word Boundaries
Can you write Python code for it?

from word_score import word_score
fitness = word_score()
print fitness.score(ptext_k2)


I am working on the file "Kryptos.ipynb".
The first task is:
K2 - Frequency Analysis of Subsequences #2
Now, let's try that again, but use the cipher alphabet order instead.
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
keyed_alphabet = compute_keyed_alphabet("KRYPTOS")

key_length = 8

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
g_english_keyed = [g_english[alphabet.find(c)] for c in keyed_alphabet]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)), g_english_keyed, tick_label = rotate(list(keyed_alphabet), 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in keyed_alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(keyed_alphabet)), ctext_freq[i], tick_label = rotate(list(keyed_alphabet), 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
K2 - Frequency Analysis of Subsequences #3
For K2, it looks like it might be possible to align the frequency plots by hand to break the key. To make this easier, let's create plots with two concatenated alphabets. That way, we can print them and align them (or do this by yourself on graph paper of course).
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [18,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
keyed_alphabet = compute_keyed_alphabet("KRYPTOS")
ncopies = 2

key_length = 8

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
g_english_keyed = [g_english[alphabet.find(c)] for c in keyed_alphabet]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)*ncopies), g_english_keyed + [0]*26, tick_label = rotate(list(keyed_alphabet)*ncopies, 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in keyed_alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(keyed_alphabet)*ncopies), list(ctext_freq[i])*ncopies, tick_label = rotate(list(keyed_alphabet)*ncopies, 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
Keyed Vigenere
Now we will move on to the Keyed Vigenere cipher, which is very similar to the Caesar cipher with a keyed alphabet, except that the consecutive shifts of the plaintext characters are determined by a keyword. If the key has length $N$, then each $N^{th}$ character of the plaintext will be shifted by the same shift, determined by the corresponding character in the key. This relation of Vigenere being N interleaved Caesar ciphers with keyed alphabet is also visible from the code:
Can you write Python code for it?

def keyed_vigenere(s, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", alpha_key="", direction=1):
    # compute keyed alphabet
    keyed_alphabet = compute_keyed_alphabet(alpha_key, alphabet)
    t = []
    key_i = 0
    for i in range(len(s)):
        if s[i] in alphabet:
            t_i = rot(s[i], key[key_i % len(key)],keyed_alphabet, direction)
            t.append(t_i)
            key_i += 1
        else:
            t.append(s[i])
    return "".join(t)


I am working on the file "Kryptos.ipynb".
The first task is:
Kryptos
Can you write Python code for it?

kryptos_L1 = """
EMUFPHZLRFAXYUSDJKZLDKRNSHGNFIVJ
YQTQUXQBQVYUVLLTREVJYQTMKYRDMFD
VFPJUDEEHZWETZYVGWHKKQETGFQJNCE
GGWHKK?DQMCPFQZDQMMIAGPFXHQRLG
TIMVMZJANQLVKQEDAGDVFRPJUNGEUNA
QZGZLECGYUXUEENJTBJLBQCRTBJDFHRR
YIZETKZEMVDUFKSJHKFWHKUWQLSZFTI
HHDDDUVH?DWKBFUFPWNTDFIYCUQZERE
EVLDKFEZMOQQJLTTUGSYQPFEUNLAVIDX
FLGGTEZ?FKZBSFDQVGOGIPUFXHHDRKF
FHQNTGPUAECNUVPDJMQCLQUMUNEDFQ
ELZZVRRGKFFVOEEXBDMVPNFQXEZLGRE
DNQFMPNZGLFLPMRJQYALMGNUVPDXVKP
DQUMEBEDMHDAFMJGZNUPLGEWJLLAETG"""

kryptos_L2 = """
ENDYAHROHNLSRHEOCPTEOIBIDYSHNAIA
CHTNREYULDSLLSLLNOHSNOSMRWXMNE
TPRNGATIHNRARPESLNNELEBLPIIACAE
WMTWNDITEENRAHCTENEUDRETNHAEOE
TFOLSEDTIWENHAEIOYTEYQHEENCTAYCR
EIFTBRSPAMHHEWENATAMATEGYEERLB
TEEFOASFIOTUETUAEOTOARMAEERTNRTI
BSEDDNIAAHTTMSTEWPIEROAGRIEWFEB
AECTDDHILCEIHSITEGOEAOSDDRYDLORIT
RKLMLEHAGTDHARDPNEOHMGFMFEUHE
ECDMRIPFEIMEHNLSSTTRTVDOHW?OBKR
UOXOGHULBSOLIFBBWFLRVQQPRNGKSSO
TWTQSJQSSEKZZWATJKLUDIAWINFBNYP
VTTMZFPKWGDKZXTJCDIGKUHUAUEKCAR"""

ctext_kryptos = (kryptos_L1+kryptos_L2).replace(" ", "").strip().split("\n")


I am working on the file "Kryptos.ipynb".
The first task is:
Kryptos - K1
Can you write Python code for it?

ctext = "".join(ctext_kryptos[:2])
ctext


I am working on the file "Kryptos.ipynb".
The first task is:
Now, like we did before with the normal Caesar cipher and look at the effect of the cipher on the frequency distribution:
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,9]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
keyed_alphabet = compute_keyed_alphabet("KRYPTOS", alphabet)  # UNCOMMENT IF YOU WANT ORDER THE

def rotate(l, n):
    return l[n:] + l[:n]

@interact(key_i = (0,25,1))
def plot_graphs(key_i = 0):
    # encrypt ptext using this key
    key = alphabet[key_i]
    ctext = keyed_caesar(ptext, key, alpha_key="KRYPTOS", direction=1)

    # determine letter frequency in plaintext, and plot
    ptext_freq = np.array([ptext.count(c)/len(ptext) for c in alphabet])
    pfig = plt.subplot(3,1,1)
    plt.bar(range(len(alphabet)), ptext_freq, tick_label = list(alphabet), align = 'center', color = 'b')
    plt.title('blue = plaintext letter frequency, red = ciphertext letter frequency')

    # determine letter frequency in ciphertext, and plot, ordered by keyed alphabet
    ctext_freq = np.array([ctext.count(c)/len(ctext) for c in keyed_alphabet])
    cfig = plt.subplot(3,1,2)
    plt.bar(range(len(keyed_alphabet)), ctext_freq, tick_label = rotate(list(keyed_alphabet), key_i), align = 'center', color = 'r')
    plt.xlabel("key = %s" % key)

    # determine letter frequency in ciphertext, and plot, ordered by alphabet
    ctext_freq = np.array([ctext.count(c)/len(ctext) for c in alphabet])
    cfig = plt.subplot(3,1,3)
    plt.bar(range(len(alphabet)), ctext_freq, tick_label = rotate(list(alphabet), key_i), align = 'center', color = 'r')
    plt.xlabel("key = %s" % key)

    plt.show()
    print ctext


I am working on the file "Kryptos.ipynb".
The first task is:
Perhaps a bit underwhelming to use this for cracking the Caesar cipher. However, now we have a useful tool in hand that we can use for other more complicated ciphers as well, like the Vigenere cipher which we will now look at.

Vigenere
Now we will move on to the Vigenere cipher, which is very similar to the Caesar cipher, except that the consecutive shifts of the plaintext characters are determined by a keyword. If the key has length $N$, then each $N^{th}$ character of the plaintext will be shifted by the same shift, determined by the corresponding character in the key. This relation of Vigenere being N interleaved Caesar ciphers is also visible from the code:
Can you write Python code for it?

def vigenere(s, key, alphabet="ABCDEFGHIJKLMNOPQRSTUVWXYZ", direction=1):
    t = []
    key_i = 0
    for i in range(len(s)):
        if s[i] in alphabet: # only process symbols from the specified alphabet
            t.append(rot(s[i],key[key_i], alphabet, direction))
            key_i = (key_i + 1) % len(key)
        else:
            t.append(s[i])
    return "".join(t)


I am working on the file "Kryptos.ipynb".
The first task is:
Pretty close, but no cigar. Let's try CRYPTANALYSIS.
Can you write Python code for it?

vigenere(ctext, 'CRYPTANALYSIS', direction = -1)


I am working on the file "Kryptos.ipynb".
The first task is:
Since we know the key we can decrypt it:
Can you write Python code for it?

decrypted_ctext = rot(ctext, 'N', direction = -1)
decrypted_ctext


I am working on the file "Kryptos.ipynb".
The first task is:
So K2 indeed seems to be just the remainder of the first panel.
Can you write Python code for it?

ctext_k2 = "".join(ctext_kryptos[2:14])

key = "ABSCISSA"
alphakey = "KRYPTOS"
ptext_k2 = keyed_vigenere(ctext_k2, key, alpha_key = alphakey, direction = -1)

print ctext_k2
print ptext_k2


I am working on the file "Kryptos.ipynb".
The first task is:
The IC tells us that it is likely not a transposition of English plaintext.
Determine Key Length
Can you write Python code for it?

from matplotlib import pyplot as plt
%matplotlib inline

mean_ic_n = []
for n in range(1,32):
    mean_ic = mean_IC(ctext.upper(), n)
    mean_ic_n.append(mean_ic)
    print "%2d %02f" % (n, mean_ic)
plt.rcParams["figure.figsize"] = [12,4]
plt.bar(range(1,len(mean_ic_n)+1), mean_ic_n, align = 'center')
plt.xlabel("key length")
plt.ylabel("average IC")


I am working on the file "Kryptos.ipynb".
The first task is:
The above function takes a text and a number n and returns that text as ngrams:
- text[i:] for i in range(3) would be
    - $\texttt{ANEXAMPLETEXT}$
    - $\texttt{NEXAMPLETEXT}$
    - $\texttt{EXAMPLETEXT}$
- *[...] passes these as individual arguments into the zip() function
- zip() zips the lists together and returns a list of tuples of the 1st entries in the lists, the 2nd entries in the list, etc until one of the lists runs out:
    - $\texttt{(A,N,E)}$
    - $\texttt{(N,E,X)}$
    - ...
    - $\texttt{(E,X,T)}$
Now, using this function, we can easily create a file of ngrams and their counts in some reference text. These statistics are later used to estimate the likelihood that another string is from the same language.
Can you write Python code for it?

def compute_ngram_statistics(infile, outfile, n):

    # read in file, and remove newlines
    s = ''
    print "reading %s..." % infile
    with open(infile) as f:
         s += " ".join(line.strip() for line in f)

    print "finding and counting %d-grams..." % n
    # compute ngrams
    qgs = find_ngrams(s, n)
    # count them
    d = {}
    for qg in qgs:
        if d.has_key("".join(qg)):
            d["".join(qg)]+=1
        else:
            d["".join(qg)]=1

    # convert to sorted list and write to file
    print "writing results to %s..." % outfile
    qg_list = sorted(list(d.items()), key=lambda x: x[1], reverse=True)
    f = open(outfile, 'w')
    for t in qg_list:
        f.write("%s %d\n" % t)

    print "done!"

compute_ngram_statistics('ngrams/gutenberg_sherlock.txt', 'ngrams/en_sherlock_1grams', 1)
compute_ngram_statistics('ngrams/gutenberg_sherlock.txt', 'ngrams/en_sherlock_2grams', 2)
compute_ngram_statistics('ngrams/gutenberg_sherlock.txt', 'ngrams/en_sherlock_3grams', 3)
compute_ngram_statistics('ngrams/gutenberg_sherlock.txt', 'ngrams/en_sherlock_4grams', 4)


I am working on the file "Kryptos.ipynb".
The first task is:
The average IC for different key lengths tells us that it is likely encrypted with a key of length 9. We therefore have to crack 9 Caesar ciphers with a Keyed Alphabet, each of them with their own shift as defined by the Vigenere key.
Frequency analysis of subsequences
Now, let's look at the frequency distributions for the 9 subsequences:
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

key_length = 9

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)), g_english, tick_label = rotate(list(alphabet), 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(alphabet)), ctext_freq[i], tick_label = rotate(list(alphabet), 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
This suggests a key length of 8.
### K2 - Frequency Analysis of Subsequences #1
Can you write Python code for it?

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,16]

def rotate(l, n):
    return l[n:] + l[:n]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

key_length = 8

g_english = [0.0736, 0.0148, 0.0445, 0.0302, 0.102, 0.0227, 0.0122, 0.0277, 0.0855, 0.000557, 0.00237, 0.0342, 0.0206, 0.0717, 0.103, 0.0246, 0.00181, 0.0735, 0.0608, 0.0889, 0.0392, 0.0153, 0.0173, 0.000557, 0.032, 0.000278]
pfig = plt.subplot(key_length+1,1,1)
plt.bar(range(len(alphabet)), g_english, tick_label = rotate(list(alphabet), 0), align = 'center', color = 'b')

ctext_freq = []
for i in range(key_length):
    ctext_i = ctext[i::key_length]
    ctext_freq.append(np.array([ctext_i.count(c)/len(ctext_i) for c in alphabet]))

    cfig = plt.subplot(key_length+1,1,i+2)
    plt.bar(range(len(alphabet)), ctext_freq[i], tick_label = rotate(list(alphabet), 0), align = 'center', color = 'r')


I am working on the file "Kryptos.ipynb".
The first task is:
We can see that due to the keyed alphabet, the effect of Keyed Caesar is not just a shift of the frequency distributions (unless we look at the frequencies in order of the keyed alphabet).
Determine Key by Aligning Frequency Distributions?
Now, we'll start with a ciphertext resulting from Keyed Caesar encryption, and look at decrypting it by aligning frequency distributions. Note that when just given the ciphertext, we have no idea what the key to create the keyed alphabet is.
Can you write Python code for it?

ctext = keyed_caesar(ptext, "N", alpha_key="KRYPTOS", direction=1)
ctext

# https://blog.dominodatalab.com/interactive-dashboards-in-jupyter/
# http://ipywidgets.readthedocs.io/en/latest/examples/Using%20Interact.html
from __future__ import division
from matplotlib import pyplot as plt
import numpy as np
from ipywidgets import interact, interactive, fixed
#import ipywidgets as widgets
plt.rcParams["figure.figsize"] = [12,6]

alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

def rotate(l, n):
    return l[n:] + l[:n]

@interact(key_i = (0,25,1))
def plot_graphs(key_i = 0):
    # decrypt ciphertext using this key
    key = alphabet[key_i]
    # NOTE THAT WE DO NOT KNOW THE KEY TO THE ALPHABET, SO WE CANNOT USE THAT!!!
    decrypted_ctext = keyed_caesar(ctext, key, alpha_key = "A", direction=-1)
    # determine letter frequency in plaintext, and plot # FIXME: base on separate text
    ptext_freq = np.array([ptext.count(c)/len(ptext) for c in alphabet])
    pfig = plt.subplot(2,1,1)
    plt.bar(range(len(alphabet)), ptext_freq, tick_label = list(alphabet), align = 'center', color = 'b')
    plt.title('blue = plaintext letter frequency, green = decrypted ciphertext letter frequency')

    # determine letter frequency in ciphertext, and plot
    ctext_freq = np.array([decrypted_ctext.count(c)/len(decrypted_ctext) for c in alphabet])
    cfig = plt.subplot(2,1,2)
    plt.bar(range(len(alphabet)), ctext_freq, tick_label = rotate(list(alphabet), key_i), align = 'center', color = 'g')
    plt.xlabel("key = %s" % key)

    plt.show()
    print decrypted_ctext


I am working on the file "Kryptos.ipynb".
The first task is:
which can be seen as:
"I HEAR AND I FORGET. I SEE AND I REMEMBER. I DO AND I UNDERSTAND." --CONFUSIUS
 N ONON ONO N ONONON  O NON ONO N ONONONON  O NO NON O NONONONONO     NONONONON
 ------------------------------------------------------------------------------
"V VROE OAR V TBFTSG. W FSR OAR V FRARAOSE. W QC NBQ W HBQSEGGOAR." --PCATHGVIF

Vigenere Solution #0: Manually Cracking a Vigenere Cipher
Since Vigenere is so similar to the Caesar cipher, we can re-use many of the same principles which we already saw for solving the Caesar cipher. However, before we can attack the Vigenere cipher in the same way as the Caesar cipher, we need to guess the length of the key. Again, for this we can use the frequency distributions, however, now we look at the frequency distribution of every $N^{th}$, since each $N^{th}$ character is shifted with the same key. In summary, we are going to try different key lengths and see for which key length, the N different frequency distributions look most like shifted English distributions. The technical way to measure this is through the Index of Coincidence (IC), which characterizes piece of text by the likelihood of drawing a pair of identical letters at random from that text.

CHALLENGE: Manually Cracking a Vigenere Cipher

Before moving on to the IC, an alternative more visual approach for determining key length is used on http://www.simonsingh.net/The_Black_Chamber/vigenere_cracking_tool.html, which finds common letter combinations and how far they are apart in the text. Since these **distances between common letter combinations are likely a multiple of the key length**, investigating the common factors of these distances will give an indication of the likely key length.

Use the vigenere cracking tool at the above site to crack the following vigenere ciphertext:...
Can you write Python code for it?

from __future__ import division
import numpy as np

# computes the IC for a given string
def IC(s, alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
    s_filtered = filter(s, alphabet) # we only care about the letters in our alphabet
    ic = 0
    for c in alphabet:
        c_count = s_filtered.count(c)
        N = len(s_filtered)
        if c_count > 1:
            ic += (c_count * (c_count - 1)) / (N * (N - 1) / len(alphabet))
    return ic

# helper function to filter out non-alphabet characters
def filter(s, alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
    return "".join([c for c in s if c in alphabet])

# computes the avg IC of subsequences of each n'th character
def mean_IC(s, n, alphabet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
    s_filtered = filter(s, alphabet) # we only care about the letters in our alphabet
    s_filtered_subseq = [s_filtered[i::n] for i in range(n)]
    ic = []
    for i in range(n):
        ic.append(IC(s_filtered_subseq[i]))
    return np.mean(ic)

ctext = "VYCHVUYPESJMZCJZTXNOOEFSXMBQJTTNQAXWKBWTPDDKTUODCOPGJFNTMOPRLACBZGTWEAEEEOKWAKCXCHVOATLGFMVYZRWBNGHPQUCDRKSGXSGWZRZWMURLSTLCHLZWBAECCIMEESTLLPWVNCCMYLELPKAAPTCZKYCTZQOIKGICRMEQTSPWMGHKFTYOHRDCUBAQEQWTVRBPCFKGPWGGFEQTZFSDWDVCCLOYVPBFWGPVFPLYRTMCWVSDCCIHSBLGCLPWTVKPBNVNRLAVWVPQTOEACSYJIUVVPBXSFARC"

from matplotlib import pyplot as plt
mean_ic_n = []
for n in range(1,32):
    mean_ic = mean_IC(ctext.upper(), n)
    mean_ic_n.append(mean_ic)
    print "%2d %02f" % (n, mean_ic)
plt.bar(range(1,len(mean_ic_n)+1), mean_ic_n, align = 'center')
plt.xlabel("key length")
plt.ylabel("average IC")


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
WARNING in the following, I use a very manual approach: I copied the time of each '%timeit' example, to compare speed-up ratios.
So when I rerun the cells, the times might vary (a little bit), and I cannot keep an up-to-date versions of the computations of each ratio, so bear with me the (tiny) incoherences.
KL divergences
Bernoulli
Can you write Python code for it?

test_fs([klBern, klBern_numba, klBern_cython, klBern_c], lambda: (r(), r()))

%timeit klBern(r(), r())

%timeit klBern_numba(r(), r())

%timeit klBern_cython(r(), r())

%timeit klBern_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
As much as possible, one should call @jit(nopython=True) to be sure that numba does not fall back silently to naive Python code. With nopython=True, any call to the generated function will fail if the compilation could not succeed.
KL divergences
Bernoulli distributions
Can you write Python code for it?

@jit(nopython=True)
def klBern_numba(x, y):
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return x * np.log(x / y) + (1 - x) * np.log((1 - x) / (1 - y))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Bernoulli
Can you write Python code for it?

def klucbBern(x, d, precision=1e-6):
    """ KL-UCB index computation for Bernoulli distributions, using :func:`klucb`."""
    upperbound = min(1., klucbGauss(x, d, sig2x=0.25))  # variance 1/4 for [0,1] bounded distributions
    # upperbound = min(1., klucbPoisson(x, d))  # also safe, and better ?
    return klucb(x, d, klBern, upperbound, precision)

klucbBern(0.1, 0.2)
klucbBern(0.5, 0.2)
klucbBern(0.9, 0.2)
klucbBern(0.1, 0.4)
klucbBern(0.1, 0.9)
klucbBern(0.5, 0.4)
klucbBern(0.5, 0.9)
klucbBern(0.9, 0.4)
klucbBern(0.9, 0.9)


klucbBern_c = lambda x, y: kullback.klucbBern(x, y, 1e-6)

test_fs([klucbBern, klucbBern_numba, klucbBern_cython, klucbBern_c], lambda: (r(), r()))

%timeit klucbBern(r(), r())

%timeit klucbBern_numba(r(), r())

%timeit klucbBern_cython(r(), r())

%timeit klucbBern_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Bernoulli
Here, the nopython=True fails as numba has a hard time typing linked function calls.
Can you write Python code for it?

@jit
def klucbBern_numba(x, d, precision=1e-6):
    upperbound = min(1., klucbGauss_numba(x, d, sig2x=0.25))  # variance 1/4 for [0,1] bounded distributions
    # upperbound = min(1., klucbPoisson(x, d))  # also safe, and better ?
    return klucb_numba(x, d, klBern_numba, upperbound, precision)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Binomial
Can you write Python code for it?

test_fs([klBin, klBin_numba, klBin_cython, klBin_c], lambda: (r(), r(), rn()))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Binomial distributions
Can you write Python code for it?

%%cython
from libc.math cimport log
eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]

def klBin_cython(float x, float y, int n) -> float:
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return n * (x * log(x / y) + (1 - x) * log((1 - x) / (1 - y)))


@jit(nopython=True)
def klBin_numba(x, y, n):
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return n * (x * np.log(x / y) + (1 - x) * np.log((1 - x) / (1 - y)))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
But if we remove the time taken by the PRNG (which takes the same time for both the naive Python and the Cython function), we get a larger speed-up, closer to reality, about $50$ times and not just $6$ times faster!
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Clean up
Can you write Python code for it?

%%bash
[ -f kullback.py.old ] && mv -vf kullback.py.old kullback.py


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Distribution-specific KL-UCB indexes
Gaussian
Can you write Python code for it?

@jit(nopython=True)
def klucbGauss_numba(x, d, sig2x=0.25, precision=0.):
    return x + np.sqrt(2 * sig2x * d)


def klucbGauss(x, d, sig2x=0.25, precision=0.):
    """ KL-UCB index computation for Gaussian distributions.

    - Note that it does not require any search.

    .. warning:: it works only if the good variance constant is given.
    """
    return x + np.sqrt(2 * sig2x * d)

klucbGauss(0.1, 0.2)
klucbGauss(0.5, 0.2)
klucbGauss(0.9, 0.2)
klucbGauss(0.1, 0.4)
klucbGauss(0.1, 0.9)
klucbGauss(0.5, 0.4)
klucbGauss(0.5, 0.9)
klucbGauss(0.9, 0.4)
klucbGauss(0.9, 0.9)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Exponential
Can you write Python code for it?

def klucbExp(x, d, precision=1e-6):
    """ KL-UCB index computation for exponential distributions, using :func:`klucb`."""
    if d < 0.77:  # XXX where does this value come from?
        upperbound = x / (1 + 2. / 3 * d - np.sqrt(4. / 9 * d * d + 2 * d))
        # safe, klexp(x,y) >= e^2/(2*(1-2e/3)) if x=y(1-e)
    else:
        upperbound = x * np.exp(d + 1)
    if d > 1.61:  # XXX where does this value come from?
        lowerbound = x * np.exp(d)
    else:
        lowerbound = x / (1 + d - np.sqrt(d * d + 2 * d))
    return klucb(x, d, klGamma, upperbound, lowerbound, precision)

klucbExp(0.1, 0.2)
klucbExp(0.5, 0.2)
klucbExp(0.9, 0.2)
klucbExp(0.1, 0.4)
klucbExp(0.1, 0.9)
klucbExp(0.5, 0.4)
klucbExp(0.5, 0.9)
klucbExp(0.9, 0.4)
klucbExp(0.9, 0.9)


test_fs([klExp, klExp_numba, klExp_cython, klExp_c], lambda: (r(), r()))

%timeit klExp(r(), r())

%timeit klExp_numba(r(), r())

%timeit klExp_cython(r(), r())

%timeit klExp_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
For example, for klucbBern, the two steps are to first compute an upperbound (as precise as possible) and the compute the kl-UCB index:
Can you write Python code for it?

x, d = 0.9, 0.2
upperbound = 1
klucb(x, d, klBern, upperbound, lowerbound=0, precision=1e-3, max_iterations=10)
klucb(x, d, klBern, upperbound, lowerbound=0, precision=1e-6, max_iterations=10)
klucb(x, d, klBern, upperbound, lowerbound=0, precision=1e-3, max_iterations=50)
klucb(x, d, klBern, upperbound, lowerbound=0, precision=1e-6, max_iterations=100)


x, d = 0.9, 0.2
upperbound = 1
klucb_numba(x, d, klBern_numba, upperbound, lowerbound=0, precision=1e-3, max_iterations=10)
klucb_numba(x, d, klBern_numba, upperbound, lowerbound=0, precision=1e-6, max_iterations=10)
klucb_numba(x, d, klBern_numba, upperbound, lowerbound=0, precision=1e-3, max_iterations=50)
klucb_numba(x, d, klBern_numba, upperbound, lowerbound=0, precision=1e-6, max_iterations=100)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
For example, for klucbBern_cython, the two steps are to first compute an upperbound (as precise as possible) and the compute the kl-UCB index:
Can you write Python code for it?

x, d = 0.9, 0.2
upperbound = 1
klucb_cython(x, d, klBern_cython, upperbound, lowerbound=0, precision=1e-3, max_iterations=10)
klucb_cython(x, d, klBern_cython, upperbound, lowerbound=0, precision=1e-6, max_iterations=10)
klucb_cython(x, d, klBern_cython, upperbound, lowerbound=0, precision=1e-3, max_iterations=50)
klucb_cython(x, d, klBern_cython, upperbound, lowerbound=0, precision=1e-6, max_iterations=100)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Gamma
Can you write Python code for it?

klGamma_c = lambda x, y: kullback.klGamma(x, y, 1)

test_fs([klGamma, klGamma_numba, klGamma_cython, klGamma_c], lambda: (r(), r()))

%timeit klGamma(r(), r())

%timeit klGamma_numba(r(), r())

%timeit klGamma_cython(r(), r())

%timeit klGamma_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Gamma distributions
Can you write Python code for it?

@jit(nopython=True)
def klGamma_numba(x, y, a=1):
    if x <= 0 or y <= 0:
        return inf
    else:
        x = max(x, eps)
        y = max(y, eps)
        return a * (x / y - 1 - np.log(x / y))


%%cython
from libc.math cimport log
eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]

def klGamma_cython(float x, float y, float a=1) -> float:
    if x <= 0 or y <= 0:
        return float('+inf')
    else:
        x = max(x, eps)
        y = max(y, eps)
        return a * (x / y - 1 - log(x / y))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Gaussian
Can you write Python code for it?

klGauss_c = lambda x, y: kullback.klGauss(x, y, 0.25)

test_fs([klGauss, klGauss_numba, klGauss_cython, klGauss_c], lambda: (r(), r()))

%timeit klGauss(r(), r())

%timeit klGauss_numba(r(), r())

%timeit klGauss_cython(r(), r())

%timeit klGauss_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Gaussian distributions
Can you write Python code for it?

@jit(nopython=True)
def klGauss_numba(x, y, sig2x=0.25, sig2y=0.25):
    if - eps < (sig2y - sig2x) and (sig2y - sig2x) < eps:
        return (x - y) ** 2 / (2. * sig2x)
    else:
        return (x - y) ** 2 / (2. * sig2y) + 0.5 * ((sig2x/sig2y)**2 - 1 - np.log(sig2x/sig2y))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Generic KL-UCB indexes, with a bisection search
Can you write Python code for it?

def klucb(x, d, kl, upperbound, lowerbound=float('-inf'), precision=1e-6, max_iterations=50):
    """ The generic KL-UCB index computation.

    - x: value of the cum reward,
    - d: upper bound on the divergence,
    - kl: the KL divergence to be used (:func:`klBern`, :func:`klGauss`, etc),
    - upperbound, lowerbound=float('-inf'): the known bound of the values x,
    - precision=1e-6: the threshold from where to stop the research,
    - max_iterations: max number of iterations of the loop (safer to bound it to reduce time complexity).

    .. note:: It uses a **bisection search**, and one call to ``kl`` for each step of the bisection search.
    """
    value = max(x, lowerbound)
    u = upperbound
    _count_iteration = 0
    while _count_iteration < max_iterations and u - value > precision:
        _count_iteration += 1
        m = (value + u) / 2.
        if kl(x, m) > d:
            u = m
        else:
            value = m
    return (value + u) / 2.


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Generic KL-UCB indexes, with a bisection search
For these, they need previously defined functions, which have to be rewritten from inside the cython cell to be accessible from Cython.
To minimize repetitions, I use only one cell to define all functions.
Can you write Python code for it?

%%cython
from libc.math cimport sqrt, log, exp
eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]

def klucbGauss_cython(float x, float d, float sig2x=0.25, float precision=0.) -> float:
    return x + sqrt(2 * sig2x * d)

cdef float klucbGauss_cython_x(float x, float d, float sig2x=0.25, float precision=0.):
    return x + sqrt(2 * sig2x * d)

def klucb_cython(float x, float d, kl, float upperbound,
                 float lowerbound=float('-inf'),
                 float precision=1e-6, int max_iterations=50) -> float:
    cdef float value = max(x, lowerbound)
    cdef float u = upperbound
    cdef int _count_iteration = 0
    cdef float m = 0
    while _count_iteration < max_iterations and u - value > precision:
        _count_iteration += 1
        m = (value + u) / 2.
        if kl(x, m) > d:
            u = m
        else:
            value = m
    return (value + u) / 2.

cdef float klBern_cython_x(float x, float y):
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return x * log(x / y) + (1 - x) * log((1 - x) / (1 - y))

def klucbBern_cython(float x, float d, float precision=1e-6) -> float:
    cdef float upperbound = min(1., klucbGauss_cython_x(x, d, sig2x=0.25))  # variance 1/4 for [0,1] bounded distributions
    # upperbound = min(1., klucbPoisson(x, d))  # also safe, and better ?
    return klucb_cython(x, d, klBern_cython_x, upperbound, precision)

cdef float klPoisson_cython_x(float x, float y):
    x = max(x, eps)
    y = max(y, eps)
    return y - x + x * log(x / y)

def klucbPoisson_cython(float x, float d, float precision=1e-6) -> float:
    cdef float upperbound = x + d + sqrt(d * d + 2 * x * d)  # looks safe, to check: left (Gaussian) tail of Poisson dev
    return klucb_cython(x, d, klPoisson_cython_x, upperbound, precision)

cdef float klGamma_cython_x(float x, float y):
    if x <= 0 or y <= 0:
        return float('+inf')
    else:
        x = max(x, eps)
        y = max(y, eps)
        return x / y - 1 - log(x / y)

def...


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Here we observe a large speed-up. But how large? $6$ times or $50$ times?
It's really important to include the time taken by the Pseudo-Random Number Generator:

Wrong computation of the speed-up gives about $6$ times faster:
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
I will include docstrings and examples only for the naive implementation.
KL divergences
Bernoulli distributions
Can you write Python code for it?

def klBern(x, y):
    r""" Kullback-Leibler divergence for Bernoulli distributions. https://en.wikipedia.org/wiki/Bernoulli_distribution#Kullback.E2.80.93Leibler_divergence

    .. math:: \mathrm{KL}(\mathcal{B}(x), \mathcal{B}(y)) = x \log(\frac{x}{y}) + (1-x) \log(\frac{1-x}{1-y})."""
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return x * np.log(x / y) + (1 - x) * np.log((1 - x) / (1 - y))

klBern(0.5, 0.5)
klBern(0.1, 0.9)
klBern(0.9, 0.1)
klBern(0.4, 0.5)
klBern(0.01, 0.99)
klBern(0, 1)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
If you want to reproduce this notebook, download the kullback_py3.c and follow the build instructions.

Tests and benchmarks
For each of the functions defined in three approaches above, I will do some numerical tests to compare their speed − and memory −  efficiency. Simple.
The benchmark will be to test the computation time on random entries.
It includes a constant time: creating random values! So I also compare the time to simply generate the values.
Can you write Python code for it?

r = np.random.random
rn = lambda: np.random.randint(1000)

%timeit (r(), r())
%timeit (r(), r(), rn())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
KL divergences
Bernoulli distributions
Can you write Python code for it?

%%cython
from libc.math cimport log
eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]

def klBern_cython(float x, float y) -> float:
    x = min(max(x, eps), 1 - eps)
    y = min(max(y, eps), 1 - eps)
    return x * log(x / y) + (1 - x) * log((1 - x) / (1 - y))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
KL divergences and KL-UCB indexes, in naive Python
I will copy and paste parts of this file from my SMPyBandits library.
Can you write Python code for it?

eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
KL-UCB indexes
Gaussian
Can you write Python code for it?

klucbGauss_c = lambda x, y: kullback.klucbGauss(x, y, 0.25)

test_fs([klucbGauss, klucbGauss_numba, klucbGauss_cython, klucbGauss_c], lambda: (r(), r()))

%timeit klucbGauss(r(), r())

%timeit klucbGauss_numba(r(), r())

%timeit klucbGauss_cython(r(), r())

%timeit klucbGauss_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Negative binomial distributions
Can you write Python code for it?

@jit(nopython=True)
def klNegBin_numba(x, y, r=1):
    x = max(x, eps)
    y = max(y, eps)
    return r * np.log((r + x) / (r + y)) - x * np.log(y * (r + x) / (x * (r + y)))


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Others
We could do the same for more distributions, but that's enough.

With Numba
It will be exactly the same code as above, except that the numba.jit decorator will be used for each functions, to let numba try to speed up the code!
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Poisson
Can you write Python code for it?

test_fs([klPoisson, klPoisson_numba, klPoisson_cython, klPoisson_c], lambda: (r(), r()))

%timeit klPoisson(r(), r())

%timeit klPoisson_numba(r(), r())

%timeit klPoisson_cython(r(), r())

%timeit klPoisson_c(r(), r())


klucbPoisson_c = lambda x, y: kullback.klucbPoisson(x, y, 1e-6)

test_fs([klucbPoisson, klucbPoisson_numba, klucbPoisson_cython, klucbPoisson_c], lambda: (r(), r()))

%timeit klucbPoisson(r(), r())

%timeit klucbPoisson_numba(r(), r())

%timeit klucbPoisson_cython(r(), r())

%timeit klucbPoisson_c(r(), r())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Poisson distributions
Can you write Python code for it?

@jit(nopython=True)
def klPoisson_numba(x, y):
    x = max(x, eps)
    y = max(y, eps)
    return y - x + x * np.log(x / y)


def klPoisson(x, y):
    r""" Kullback-Leibler divergence for Poison distributions. https://en.wikipedia.org/wiki/Poisson_distribution#Kullback.E2.80.93Leibler_divergence

    .. math:: \mathrm{KL}(\mathrm{Poisson}(x), \mathrm{Poisson}(y)) = y - x + x \times \log(\frac{x}{y}).
    """
    x = max(x, eps)
    y = max(y, eps)
    return y - x + x * np.log(x / y)

klPoisson(3, 3)
klPoisson(2, 1)
klPoisson(1, 2)
klPoisson(3, 6)
klPoisson(6, 8)
klPoisson(1, 0)
klPoisson(0, 0)


%%cython
from libc.math cimport log
eps = 1e-15  #: Threshold value: everything in [0, 1] is truncated to [eps, 1 - eps]

def klPoisson_cython(float x, float y) -> float:
    x = max(x, eps)
    y = max(y, eps)
    return y - x + x * log(x / y)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
The time to generate random numbers like this is small, but not zero!
Generating a uniform integer, in particular, takes some time (more than 1 µs is not something that can be ignored!).

$\implies$ we will remove this $700$ ns or $2.5$ µs overhead when computing speed-up ratio between naive Python and numb or Cython versions.
But we also need to test that the three versions of each function gives the same results (up-to approximation errors less than
$10^{-6}$ (at least)).
Can you write Python code for it?

def test_fs(fs, inputs, tolerance=1e-5, nb_tests=100):
    for _ in range(nb_tests):
        args = inputs()
        ref_f = fs[0]  # Python version
        output = ref_f(*args)
        for other_f in fs[1:]:
            other_output = other_f(*args)
            if abs(output) > 1:
                rel_diff = (output - other_output) / output
            else:
                rel_diff = (output - other_output)
            assert abs(rel_diff) <= tolerance, "Error: function {} gave {} and function {} gave {} on inputs {}, and the two outputs are too different.".format(ref_f, output, other_f, other_output, args)


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $15$ times faster for Cython, and one more failure case for Numba.
The speed-up for the C version is CRAZY here, and it shows that optimizing loops is the most important!
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $17$ times faster for Cython, and one more failure case for Numba.
Strong victory for the C version, a speed-up of $50$ is impressive!
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $3$ times faster for Numba and $6$ times faster for Cython.
Cython starts to win the race!
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $45$ times faster for Cython, but Numba completely here!
Why? No idea!
C is also slower than the Python version here, due to this lambda x,y: ... trick.
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $5$ times faster for Numba and $10$ times faster for Cython.
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
This is a speed-up ratio of about $6$ times faster for Numba, and $6$ or $5$ times faster for Cython and C.
Note that the C version probably looses a little bit due to this lambda , y: ... trick.
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
Too much numerical difference? Let's try again with a larger tolerance:
Can you write Python code for it?

test_fs([klBin, klBin_numba, klBin_cython, klBin_c], lambda: (r(), r(), rn()), tolerance=1e-3)

%timeit klBin(r(), r(), rn())

%timeit klBin_numba(r(), r(), rn())

%timeit klBin_cython(r(), r(), rn())

%timeit klBin_c(r(), r(), rn())


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
With Cython
It will be almost exactly the same code, by using the cython magic to have cells written in Cython.
Can you write Python code for it?


I am working on the file "Kullback-Leibler_divergences_in_native_Python__Cython_and_Numba.ipynb".
The first task is:
With the C API for Python
It is more tedious, and won't be included here, but Python can easily be extended using C.
It is the best way to obtain close-to-optimal performance for some parts of your code, and I will let you read the introduction to the official documentation if you are curious.
For my SMPyBandits, I reused some code from the py/maBandits project, and the authors implemented some of the previously defined KL-divergences and KL-UCB indexes in pure Python as well as in C optimized.
I copied the compiled library in the current directory, and it can be imported:
Can you write Python code for it?

%%bash
ls -larth *kullback*
[ -f kullback.py ] && mv -vf kullback.py kullback.py.old

!ls -larth kullback*.so

import kullback

help(kullback.klBern)

[ s for s in dir(kullback) if not s.startswith('_') ]

klBern_c = kullback.klBern
klBin_c = kullback.klBin
klExp_c = kullback.klExp
klGamma_c = kullback.klGamma
klGauss_c = kullback.klGauss
klPoisson_c = kullback.klPoisson
klucbBern_c = kullback.klucbBern
klucbExp_c = kullback.klucbExp
klucbGamma_c = kullback.klucbGamma
klucbGauss_c = kullback.klucbGauss
klucbPoisson_c = kullback.klucbPoisson


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
2.Schreibe eine Funktion, die alle Elemente einer Liste, addiert. Es ist verboten mit "sum" zu arbeiten.
Can you write Python code for it?

def summe(mylist):
    long_elem = 0
    for elem in mylist:
        long_elem = long_elem + elem
    return long_elem

summe(lst)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
4.Schreibe eine Funktion, die einen String nimmt, und spiegelt. Also "hallo" zu "ollah".
Can you write Python code for it?

elem = input('Bitte geben Sie ein Wort ein ')
wort = list(elem)
print(wort)

def umkehr(wort):
    elem = input('Bitte geben Sie ein Wort ein ')
    wort = list(elem)
    wort.reverse()
    print(wort)

umkehr(wort)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
5.Schreibe eine Funktion, die prüft, ob eine Zahl in einer bestimmten Zahlenfolge zu finden ist.
Can you write Python code for it?

liste = [45, 34, 64,45]

def such(zahlen):
    zahl = int(input('Bitte geben Sie eine Zahl ein:'))
    if zahl in zahlen:
        return 'Treffer'
    else:
        return 'Kein Treffer'

such(liste)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
6.Lösche die mehrfach genannten Elemente aus der folgenden Liste.
Can you write Python code for it?

liste = [5,5,5,5,3,2,11,5]

def lösch(mehrfach):
    new_mehrfach = []
    for elem in mehrfach:
        if elem in new_mehrfach:
            new_mehrfach.append(elem)
        else:
            continue
    print(new_mehrfach)

lösch(liste)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
7.Drucke die geraden Zahlen aus der folgenden Liste aus:
Can you write Python code for it?

lst = [34,23,22,443,45,78,23,89,23]

def gerade(summe):
    for elem in summe:
        if elem % 2 == 0:
            print(elem)
        else:
            continue

gerade(lst)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
8.Prüfe mit einer Funktionen, wieviele Grossbuchstaben in folgendem Satz zu finden sind.
Can you write Python code for it?

satz = "In Oesterreich zeichnet sich ein Rechtsrutsch ab. OeVP und FPOe haben stark zugelegt. Gemaess der neusten Hochrechnung ist die Partei von Sebastian Kurz mit 31,6 Prozent der Stimmen Wahlsiegerin, auf Platz zwei folgt die SPÖ (26,9 Prozent) vor der FPOe (26,0 Prozent)."

def counting_caps(XXXXX):
    XXXXXX = 0
    for XXXXXX in elem:
        if XXXXXX.isupper():
            XXXXXXX += 1
    return  XXXXXXXX

 counting_caps(satz)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Python Functions, 10 Übungen.ipynb".
The first task is:
Multipliziert Integers oder Floats mit 2
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Rückblic_forloop_dominique.ipynb".
The first task is:
2.Drucke alle die Zahlen von 0 bis 4 aus:
Can you write Python code for it?

for x in range(5):
    print(x)

for x in range(3, 6):
    print(x)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Rückblic_forloop_dominique.ipynb".
The first task is:
6.Addiere nur die Zahlen, die gerade sind
Can you write Python code for it?

evennumber = []
for elem in numbers:
    if elem % 2 == 0:
        evennumber.append(elem)
sum(evennumber)


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Rückblic_forloop_dominique.ipynb".
The first task is:
8.Entwickle ein Programm, das alle Nummern zwischen 2000 und 3200 findet, die durch 7, aber nicht durch 5 teilbar sind. Das Ergebnis sollte auf einer Zeile ausgedruckt werden. Tipp: Schaue Dir hier die Vergleichsoperanden von Python an.
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Dominique Strebel/06/Rückblic_forloop_dominique.ipynb".
The first task is:
9.Schreibe einen For Loop, der die Nummern in der folgenden Liste von int in str verwandelt.
Can you write Python code for it?

lst = range(45,99)

newlst = []
for i in lst:
    i = str(i)
    newlst.append(i)
print(newlst)


I am working on the file "Kursteilnehmer/Sven Millischer/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
10.Schreibe nun ein Programm, das alle Ziffern 4 mit dem Buchstaben A ersetzte, alle Ziffern 5 mit dem Buchtaben B.
Can you write Python code for it?

newnewlist = []
for elem in new_list:
    if '4' in elem:
        elem = elem.replace('4', 'A')
    if '5' in elem:
        elem = elem.replace('5', 'B')
    newnewlist.append(elem)

newnewlist


I am working on the file "Kursteilnehmer/Sven Millischer/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
3.Drucke die Zahlen 3,4,5 aus:
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Sven Millischer/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
5.Addiere alle Zahlen in der Liste
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Sven Millischer/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
6.Addiere nur die Zahlen, die gerade sind
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Sven Millischer/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
7.Drucke mit einem For Loop 5 Mal hintereinander Hello World aus
Can you write Python code for it?

for x in range(5):
    print ("Hello World")

#Lösung


I am working on the file "Kursteilnehmer/Sven Millischer/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
1.Schreibe eine Funktion, die aus einer Liste, die grösste Zahl herauszieht. Es ist verboten mit "max" zu arbeiten. :-)
Can you write Python code for it?

lst = [12, 45, 373, 1028]

def highstnbr(mylist):
    mylist.sort()
    return mylist[-1]

highstnbr(lst)


I am working on the file "Kursteilnehmer/Sven Millischer/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
3.Schreibe eine Funktion, die alle Elemente einer Liste multipliziert.
Can you write Python code for it?

lst = [12, 45, 373, 1028]

def multplr(mylist):
    total=1
    for elem in mylist:
        total*=elem
    return total

multplr(lst)


I am working on the file "Kursteilnehmer/Sven Millischer/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
6.Lösche die mehrfach genannten Elemente aus der folgenden Liste.
Can you write Python code for it?

liste = [5,5,5,5,3,2,11,5]

list(set(liste)) evenlst.append(x) evenlst = []


I am working on the file "Kursteilnehmer/Sven Millischer/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
7.Drucke die geraden Zahlen aus der folgenden Liste aus:
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Sven Millischer/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
9.Prüfe mit einer Funktionen, wieviele 'e's in folgendem Satz zu finden sind.
Can you write Python code for it?

satz = "In Oesterreich zeichnet sich ein Rechtsrutsch ab. OeVP und FPOe haben stark zugelegt. Gemaess der neusten Hochrechnung ist die Partei von Sebastian Kurz mit 31,6 Prozent der Stimmen Wahlsiegerin, auf Platz zwei folgt die SPÖ (26,9 Prozent) vor der FPOe (26,0 Prozent)."

def counting_character(phrase):
    character = 0
    for x in phrase:
        if x.count('e'):
            character += 1
    return character

 counting_character(satz)


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
4.Baue einen For-Loop, indem Du alle geraden Zahlen ausdruckst, die tiefer sind als 237.
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
7.Drucke mit einem For Loop 5 Mal hintereinander Hello World aus
Can you write Python code for it?

for i in range(5):
    print("Hello World")

#Lösung


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
8.Entwickle ein Programm, das alle Nummern zwischen 2000 und 3200 findet, die durch 7, aber nicht durch 5 teilbar sind. Das Ergebnis sollte auf einer Zeile ausgedruckt werden. Tipp: Schaue Dir hier die Vergleichsoperanden von Python an.
Can you write Python code for it?


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /01 Rückblick For-Loop-Übungen.ipynb".
The first task is:
9.Schreibe einen For Loop, der die Nummern in der folgenden Liste von int in str verwandelt.
Can you write Python code for it?

lst = range(45,99)

lst = list(lst)
index = 0
print(lst)
for number in lst:
    lst[index] = str(number)
    index += 1


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
1.Schreibe eine Funktion, die aus einer Liste, die grösste Zahl herauszieht. Es ist verboten mit "max" zu arbeiten. :-)
Can you write Python code for it?

def highestNumber(lst):
    highestNumber= 0
    for number in lst:
        if number > highestNumber:
            highestNumber = number
    return highestNumber

highestNumber([0, 5, 10, 200])


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
10.Schreibe eine Funktion, die auf eine andere Funktion nutzt.
Can you write Python code for it?

numbers = [34,23,22,443,45,78,23,11,11,12,89,23]

def addUniqueNumbers(lst):
    new_list = deleteDuplicates(lst)
    print(new_list)
    added = addNumbers(new_list)
    return added

addUniqueNumbers(pipi)


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
2.Schreibe eine Funktion, die alle Elemente einer Liste, addiert. Es ist verboten mit "sum" zu arbeiten.
Can you write Python code for it?

def addNumbers(lst):
    added = 0
    for number in lst:
        added += number
    return added

addNumbers([2, 5, 10])


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
3.Schreibe eine Funktion, die alle Elemente einer Liste multipliziert.
Can you write Python code for it?

def multiplyNumbers(lst):
    multiplied = 1
    for number in lst:
        multiplied *= number
    return multiplied

multiplyNumbers([1, 2, 10])


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
4.Schreibe eine Funktion, die einen String nimmt, und spiegelt. Also "hallo" zu "ollah".
Can you write Python code for it?

def mirrorString(string):
    mirrored = []
    newString = ""
    for char in string:
        mirrored.insert(0, char)

    for char in mirrored:
        newString = newString + char
    return newString

mirrorString("hallo")


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
5.Schreibe eine Funktion, die prüft, ob eine Zahl in einer bestimmten Zahlenfolge zu finden ist.
Can you write Python code for it?

zahl = 56
liste = [45, 34, 64,45]

def isInList(number, lst):
    for elem in lst:
        if(elem == number):
            return True
    return False

isInList(zahl, liste)


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
7.Drucke die geraden Zahlen aus der folgenden Liste aus:
Can you write Python code for it?

lst = [34,23,22,443,45,78,23,89,23]

def printEvenNumbers(lst):
    for number in lst:
        if number%2 == 0:
            print(number)

printEvenNumbers(lst)


I am working on the file "Kursteilnehmer/Thierry Seiler/06 /03 Python Functions, 10 Übungen.ipynb".
The first task is:
Multipliziert Integers oder Floats mit 2
Can you write Python code for it?


I am working on the file "L0 - python3 and jupyter.ipynb".
The first task is:
Exercise 2: Root finding
Use Newton's method to find a root of an equation numerically. Newton's method starts from $x_0$ and iteratively computes $$x_{i+1} = x_i - \frac{f(x_i)}{f'(x_i)}.$$
Let us consider find a $x$ such that $f(x) = x \ln(x) - 16 = 0$. First, we plot the function and it seems $x$ is close to 8.0.
Can you write Python code for it?

import matplotlib
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import math

def f(x):
    return x * math.log(x) - 16.0

xvals = np.arange(0.01, 10, 0.01)
yvals = np.array([f(x) for x in xvals])
plt.plot(xvals, yvals)
plt.plot(xvals, 0*xvals)
plt.show()


I am working on the file "L0 - python3 and jupyter.ipynb".
The first task is:
Exercise 3: Enumerating Combinations
You need to write a function, comb(n, k), that returns a list consisting of all the combinations of $k$ elements out of a set ${1, 2, \ldots, n}$. Your implementation should be using recursions instead of for loop (you may use list comprehension though).
Can you write Python code for it?


I am working on the file "L0 - python3 and jupyter.ipynb".
The first task is:
You can test your implementation using the following code.
Can you write Python code for it?


I am working on the file "L1 - List Intersection.ipynb".
The first task is:
As you can see above, the galloping search algorithm quickly skip over the list to reposition the cursor. In the additional note, it is proved that its complexity is $O(\log x)$, where $x$ is the difference between the correct cursor position and the initial cursor position.
The Naive Intersection Algorithm
Can you write Python code for it?

def intersection_naive(a, b):
    # just in case these lists have been traversed.
    a.reset()
    b.reset()

    ret = []
    while not a.eol() and not b.eol():
        if a.elem() < b.elem():
            p("move a")
            a.next()
        elif a.elem() > b.elem():
            p("move b")
            b.next()
        else:
            p("found one")
            ret.append(a.elem())
            a.next()
            b.next()
    return ret


I am working on the file "L1 - List Intersection.ipynb".
The first task is:
If you have implemented the gallop_to() correctly, you should pass the following test. You may change the call to skip_to() for now to see how it should work.
Can you write Python code for it?

def test_gallop(l, val):
    print("=> gallop_to({})".format(val))
    l.reset()
    l.gallop_to(val)
    l.dump()

test_a = [val - 1 for val in a.get_list()]
test_a.append(a.get_list()[-1] + 100)

DEBUG = False # to show the overview
a.dump()
for t_a in test_a:
    test_gallop(a, t_a)

DEBUG = True # to show the details. Your detailed debugging info may vary.
test_gallop(a, test_a[-2])


I am working on the file "L1 - List Intersection.ipynb".
The first task is:
Intersection Algorithm using gallop_to()
Here, we call gallop_to explicitly to show some detailed info if you have it implemented correctly. You can change the calls to skip_to in a more general setting.
Can you write Python code for it?

def intersection_galloping(a, b):
    # just in case these lists have been traversed.
    a.reset()
    b.reset()

    ret = []
    while not a.eol() and not b.eol():
        if a.elem() == b.elem():
            p("found one")
            ret.append(a.elem())
            a.next()
        else:
            if a.elem() < b.elem():
                a.gallop_to(b.elem())
            else:
                b.gallop_to(a.elem())
    # end_while
    return ret

intersection_galloping(a, b)


I am working on the file "L1 - List Intersection.ipynb".
The first task is:
Test
Can you write Python code for it?


I am working on the file "L1 - List Intersection.ipynb".
The first task is:
The InvertedList class
We implement this class which encapsulates all the important operations one can perform on an inverted list.
Can you write Python code for it?

from pprint import pprint
from bisect import *

## You may find many similarity between this class and the file operation (in C/Java).
class InvertedList:
    def __init__(self, l):
        self.data = l[:] # make a copy
        self.cur = 0     # the cursor

    def get_list(self):
        return self.data

    def eol(self):
        # we use cur == len(list) to indicate EOL
        return False if self.cur < len(self.data) else True

    def next(self, val = 1):
        # does not allow cur to be out-of-range, but use len(list) to indicate EOL
        self.cur = min(self.cur + val, len(self.data))

    def elem(self):
        if self.eol():
            return None
        else:
            return self.data[self.cur]

    def peek(self, pos):
        # look at the element under the current cursor, but does not advance the cursor.
        if pos < len(self.data):
            return self.data[pos]
        else:
            return None

    def reset(self):
        self.cur = 0

    def dump(self):
        print("* {} : {} (cur = {})".format(self.data[:self.cur], self.data[self.cur:], self.cur))

    def skip_to(self, val):
        ##!! This is a naive and inefficient implementation.
        ##!! Once you have implemented gallop_to(), you can just call it here instead.
        # move the cursor to the first element no smaller than `val`

        while not self.eol() and self.elem() < val:
            self.next()

    def gallop_to(self, val):
        ##!! Not implemented yet.
        pass


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
GROUP BY
In pandas, we uses groupby() to split a dataset into groups; we can apply some function (e.g., aggregation) and combine the groups together.
We use size() to return the number of rows of each group (like COUNT in SQL)
Can you write Python code for it?


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
Import data
We can easily create a DataFrame from a CSV or a tab-delimited file. You can creata a dataframe object in other ways, e.g., from multiple Series or from a dictionary.
Then we can
* use shape to see the number of rows and columns
* use head(x) or tail(x) to view the first or last x rows of the dataFrame. The default value of x is 5.
* pandas is also smart in that it will intelligently print part of the dataframe content if it is too large.
Can you write Python code for it?

df = pd.read_csv('./asset/lecture_data.txt', sep='\t')  # to read an excel file, use read_excel()
df.head()

df

df.describe()


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
JOIN
Can you write Python code for it?

# import another table about the population of each city
df2 = pd.read_csv('./asset/population_1.txt', sep='\t')
df2.head()


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
Pivoting and Visualization
Can you write Python code for it?

table = pd.pivot_table(df, index = 'location', columns = 'time', aggfunc=np.sum)
table

pd.pivot_table(df, index = ['location', 'item'], columns = 'time', aggfunc=np.sum, margins=True)

trace1 = go.Bar(
    x=table.index,
    y=table.dollars_sold.Q1,
    name='Q1'
)
trace2 = go.Bar(
    x=table.index,
    y=table.dollars_sold.Q2,
    name='Q2'
)
trace3 = go.Bar(
    x=table.index,
    y=table.dollars_sold.Q3,
    name='Q3'
)
trace4 = go.Bar(
    x=table.index,
    y=table.dollars_sold.Q4,
    name='Q4'
)

data = [trace1, trace2, trace3, trace4]
layout = go.Layout(
    barmode='group'
)

fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='sales-plot')


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
WHERE
Can you write Python code for it?


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
We can also specify the join type
Can you write Python code for it?


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
We use agg() to apply multiple functions at once, and pass a list of columns to groupby() to grouping multiple columns
Can you write Python code for it?


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
We use pd.merge() to join two DataFrames, where you can specify the join key
Can you write Python code for it?


I am working on the file "L1 - Pandas-1.ipynb".
The first task is:
use pd.concat() to union two tables without removing duplicates (i.e., UNION ALL in SQL)
Can you write Python code for it?


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
Advanced Filtering over Pivot Table
We firstly build a pivot table
Can you write Python code for it?

table = pd.pivot_table(df, index=['council'], columns=['type'], values=['sold'], aggfunc=np.sum, margins=True)
table


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
For simplicity, we will stick with the default aggregation function.
We also want to see value, but we need to change it into floats first
Can you write Python code for it?


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
Note that the default aggregation function is np.mean. We can specify the aggregation function in the aggfunc parameter, as shown below.
Can you write Python code for it?

pd.pivot_table(df, index=['type'], aggfunc={'distance_to_CBD':np.mean, 'sold':np.sum})


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
Pivot Table
In order to build up a pivot table, we must specify an index.
Can you write Python code for it?


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
We can just look at data from one city
Can you write Python code for it?


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
We should use avg for value but sum for sold, and we do not want to see distance_to_CBD for now
Can you write Python code for it?

pd.pivot_table(df, index=['council','suburb'],
               columns=['type'],
               values=['sold', 'value'],
               aggfunc={'sold':np.sum, 'value':np.mean},
               margins=True)


I am working on the file "L1 - Pandas-2.ipynb".
The first task is:
columns provide an additional way to segment the data
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
1. ndarray in NumPy
NumPy’s main object is the homogeneous multidimensional array (ndarray). It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers.
A list of elements can be expressed as an ndarray of rank 1, i.e. a 1D array; a matrix can be expressed as an ndarray of rank 2.
Here lists some important attributes of ndarray:

ndarray.ndim: the rank of the ndarray. For instance, a matrix has a rank of 2.
ndarray.shape: the dimensions of the ndarray as a tuple of integers. For an array having 10 elements, the shape is (10,) (note the trailing comma, not the same as (10, 1)); for a matrix having 20 rows and 30 columns, the shape is (20, 30).
ndarray.size: the total number of elements in the ndarray, which is equal to the product of the dimensions.
ndarray.dtype: an object describing the type of the elements in the array.
Can you write Python code for it?

arr = np.array(range(1, 10))
print('arr\t\t', arr)
print('arr.ndim\t', arr.ndim)
print('arr.shape\t', arr.shape)
print('arr.size\t', arr.size)
print('arr.dtype\t', arr.dtype)


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
3.1 How to Get the Shape of an ndarray
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
3.2 How to Reshape the ndarray:
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
4.4 Matrix multiplication
$$
\begin{bmatrix}
  1 & 2\
  3 & 4
\end{bmatrix}
\begin{bmatrix}
  -1 & 1\
  2 & 1
\end{bmatrix}
=
%% do it column wise
\begin{bmatrix}
\left(
    -1 \cdot
    \begin{bmatrix}
      1 \
      3
    \end{bmatrix}
    + 2 \cdot
    \begin{bmatrix}
      2\
      4
    \end{bmatrix}
\right)
&
\left(
    1 \cdot
    \begin{bmatrix}
      1 \
      3
    \end{bmatrix}
    + 1 \cdot
    \begin{bmatrix}
      2\
      4
    \end{bmatrix}
\right)
\end{bmatrix}
=
\begin{bmatrix}
  3 & 3\
  5 & 7
\end{bmatrix}
$$
Since Python 3.5, we can use the infix operator * for element-wise multiplication, and the infix operator @ for matrix multiplication.
Can you write Python code for it?

A = np.array([[1,2],[3,4]])
B = np.array([[-1,1],[2,1]])
print(np.matmul(A, B)) # preferred over np.dot()
print(A @ B) # A and B must be numpy arrays
print(A * B)
print(B @ A)

x = np.array([1, 2]).reshape((2,1))
print(A @ x)


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
Both ways create a new ndarray to hold the results without modifying the original ones.
4.3 Norms
See https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html for more details.
Can you write Python code for it?

a = np.arange(3)-1
a

np.linalg.norm(a)

np.linalg.norm(a, ord=1)

np.linalg.norm(a, ord=np.inf)


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
Due to the finite precision of floating point numbers, however, it's better to use linspace when we are trying to create a sequence of floating point numbers, specifying how many elements we want, instead of the step.
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
Note that 1d arrays and 2d arrays are different.
Can you write Python code for it?

xx = arr.reshape((1,9))
xx

xx.shape

xx[0]

arr

arr.shape


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
Note that there is only one argument. So never do this:
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
The default dtype is numpy.float64 for these functions.
2.3 Using arange, linspace
Akin to range in Python, arange in NumPy returns a sequence of numbers in a ndarray. Use the dtype parameter to change the type, or use astype() function to cast into another type.
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
The reshape function returns a new ndarray with the shape changed without modifying the original one.
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
To calculate the dot product, the sentence in the following cell is intuitive but WRONG:
Can you write Python code for it?

a * b  # calculating elementwise product!


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
To correctly calculate the dot products of two ndarrays, use numpy.dot or the dot function on the ndarray object.
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
To create a matrix, call array on a sequence of sequence.
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
To directly modify the shape of an ndarray:
Can you write Python code for it?


I am working on the file "L1 - numpy-fundamentals.ipynb".
The first task is:
To solve $\mathbf{A z = B}$, we have $\mathbf{z = A^{-1} B}$
Can you write Python code for it?


I am working on the file "L2 - Nets.ipynb".
The first task is:
Видим, что точность сильно растёт при увеличении размера до $\approx 2^7$, однако затем она значительно не меняется, а время обучения растёт.
Зависимость от количества слоёв
Can you write Python code for it?

acc=[]
for i in range(10):
    np.random.seed(1337)  # for reproducibility

    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    for layer in range(i):
        model.add(Dense(512))
        model.add(Activation('relu'))
    model.add(Dense(10))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

    history = model.fit(X_train, Y_train,
                        batch_size=batch_size, nb_epoch=nb_epoch,
                        verbose=0, validation_data=(X_test, Y_test))
    score = model.evaluate(X_test, Y_test, verbose=0)
    acc.append(score[1])

plt.plot(list(range(1,11)),acc)
plt.show()


I am working on the file "L2 - Nets.ipynb".
The first task is:
Воспользовались автоэнкодером иначе: натренировав его на получение похожего на входной результата, уберём декодер и взамен добавим нужный нам классификатор. Точность, правда, всё равно ниже :(
Can you write Python code for it?

import matplotlib.pyplot as plt

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(X_train[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()