I am working on the file "Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Testing
After you're satisified with the training output and accuracy, you can then run the network on the test data set to measure it's performance! Remember, only do this after you've done the training and are satisfied with the results.
A good result will be higher than 95% accuracy. Some simple models have been known to get up to 99.7% accuracy!
Can you write Python code for it?

# Compare the labels that our model predicts with the actual labels

# Find the indices of the most confident prediction for each item. That tells us the predicted digit for that sample.
predictions = np.array(model.predict(testX)).argmax(axis=1)

# Calculate the accuracy, which is the percentage of times the predicated labels matched the actual labels
actual = testY.argmax(axis=1)
test_accuracy = np.mean(predictions == actual, axis=0)

# Print out the result
print("Test accuracy: ", test_accuracy)


I am working on the file "Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively.
Too few epochs don't effectively train your network, and too many take a long time to execute. Choose wisely!
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=100, n_epoch=20)


I am working on the file "Handwritten Digit Recognition with TFLearn.ipynb".
The first task is:
Visualize the training data
Provided below is a function that will help you visualize the MNIST data. By passing in the index of a training example, the function show_digit will display that training image along with it's corresponding label in the title.
Can you write Python code for it?

# Visualizing the data
import matplotlib.pyplot as plt
%matplotlib inline

# Function for displaying a training image by it's index in the MNIST set
def show_digit(index):
    label = trainY[index].argmax(axis=0)
    # Reshape 784 array into 28x28 image
    image = trainX[index].reshape([28,28])
    plt.title('Training data, index: %d,  Label: %d' % (index, label))
    plt.imshow(image, cmap='gray_r')
    plt.show()

# Display the first (index 0) training image
show_digit(0)


I am working on the file "Hansard mining.ipynb".
The first task is:
Analysis
Using the raw data discover the following:

Average number of words spoken by party
word cloud of spoken words by party
plot of total words spoken for each MP highlighted by party

Then apply NLTK methods and look at vocabulary:

Average vocab of each party
replot word clouds used token analysis
plot of total vocab for each MP highlighted by party

Other ideas

NLTK dispersion plot of words like "austerity"
frequency distribution by party top words by token and their context
word redundancy by party - i.e. unique words which parties repeat the same shit over and over (lexical_diversity)
longest word by party
which MP uses the longest words on average - distribution
collocations
http://www.nltk.org/book/ch01.html#counting-vocabulary

Party-wise summaries

Total speech per MP per Party
Can you write Python code for it?

total_speech = []
for speaker, party_text in words_spoken_by_mp.items():
    total_speech.append((speaker, party_text[0], len(party_text[1])))

average_totals = []
for party, mp_number in parties.items():
    average_totals.append((party, len(party_talk[party])/mp_number))
average_totals = sorted(average_totals, key=lambda x: x[1], reverse=True)

for party, average in average_totals:
    print("{0:<40s}: {1:>10,.1f}".format(party, average))

y_pos = np.arange(len(average_totals))
plt.figure(figsize=(10,10))
plot_mp_numbers = [parties[x[1]] for x in average_totals]
plt.bar(y_pos, [x[1] for x in average_totals], align='center', alpha=0.4)
plt.xticks(y_pos, [x[0] for x in average_totals], rotation=90)
plt.ylabel('Average words spoken per MP')
plt.xlabel('Party')
plt.show()


I am working on the file "Hansard mining.ipynb".
The first task is:
Parse the xml of debates
Generate 3 datastructs from xml and mp_data.csv mentioned below:

named_text (a list of tuples of MPs names and their speeches over time i.e. will appear multiple times if they speak several times)
words_spoken_by_mp (a dict keyed with the MPs name and valued by a tuple representing their party and concatenated text)
party_talk (a dict keyed by the party name and valued by concatenated text of all speeches for all MPs belonging to it)

Additionally create the ancillary dict parties which contains the list of "real parties" and their respective number of MPs and the party_dict containing all MPs respective parties
named_text
Can you write Python code for it?

def get_speaker_and_text(xml):
    """
    For a given xml file return a list of tuples
    of speeches and their speakers
    """
    with open(xml, 'rb') as fh:
        parsed_xml = fh.read()

    data = []
    root = lxml.etree.fromstring(parsed_xml)
    speeches = root.xpath('speech')
    for speech in speeches:
        #if speech.get('nospeaker') is not None:
        speech_list = [x.text for x in speech.getchildren() if x.text is not None]

        speech_list = " ".join(speech_list)

        speaker =  speech.get('speakername')
        data.append((speaker, speech_list))

    return data

named_text = []
for xml in glob.glob(os.path.join("data", "*.xml")):
    named_text = named_text + get_speaker_and_text(xml)

named_text[-1]


I am working on the file "Hansard mining.ipynb".
The first task is:
Wordclouds
Can you write Python code for it?

def make_wordcloud(text, image_filename=None):
    """
    Generate a wordcloud
    """
    # remove minor words
    wordcloud.STOPWORDS.add("hon")
    wordcloud.STOPWORDS.add("Government")
    wordcloud.STOPWORDS.add("government")
    wordcloud.STOPWORDS.add("Minister")
    wordcloud.STOPWORDS.add("minister")
    wordcloud.STOPWORDS.add("S")
    wordcloud.STOPWORDS.add("s")
    wordcloud.STOPWORDS.add("Member")
    wordcloud.STOPWORDS.add("Friend")

    if image_filename:
        mask = imread(image_filename)
        wc = wordcloud.WordCloud(background_color="white", max_words=2000, mask=mask,
                   stopwords=wordcloud.STOPWORDS)
    else:
        wc = wordcloud.WordCloud(background_color="white", max_words=2000, stopwords=wordcloud.STOPWORDS)

    wc.generate(text)

    return wc

plt.figure(figsize=(20,10))

plt.subplot(241)
plt.imshow(make_wordcloud(party_talk['Conservative']))
plt.axis("off")
plt.title('Conservative')

plt.subplot(242)
plt.imshow(make_wordcloud(party_talk['Labour']))
plt.axis("off")
plt.title('Labour')

plt.subplot(243)
plt.imshow(make_wordcloud(party_talk['Scottish National Party']))
plt.axis("off")
plt.title('SNP')

plt.subplot(244)
plt.imshow(make_wordcloud(party_talk['Liberal Democrat']))
plt.axis("off")
plt.title('Lib Dem')

plt.subplot(245)
plt.imshow(make_wordcloud(party_talk['DUP']))
plt.axis("off")
plt.title('DUP')

plt.subplot(246)
plt.imshow(make_wordcloud(party_talk['Green']))
plt.axis("off")
plt.title('Green')

plt.subplot(247)
plt.imshow(make_wordcloud(party_talk['UKIP']))
plt.axis("off")
plt.title('UKIP')

plt.subplot(248)
plt.imshow(make_wordcloud(party_talk['Plaid Cymru']))
plt.axis("off")
plt.title('PC')


I am working on the file "Hansard mining.ipynb".
The first task is:
parties and party_dict
Using mp_data.csv manually downloaded from http://www.theyworkforyou.com/mps/?f=csv to acquire party affiliations for each mp and generate a dict for quick lookup of an MP's party using their name
Can you write Python code for it?

#Encoding issue thus the weird kwarg
with open('data/mp_data.csv',  encoding = "ISO-8859-1") as fh:
    mp_data = [x for x in csv.reader(fh)]

party_dict = {}
for mp in mp_data[1:]:
    # forename and surname fields
    name = mp[1] + ' ' + mp[2]
    party = mp[3]
    party_dict.update({name: party})

#Manual clean up of this data
#Add the speakers
party_dict.update({'Mr Speaker': 'Speaker'})
party_dict.update({'Mr Deputy Speaker': 'Speaker'})
party_dict.update({'Mr Speaker-Elect': 'Speaker'})
party_dict.update({'The Chairman': 'Speaker'})
party_dict.update({'The Second Deputy Chairman': 'Speaker'})
party_dict.update({'Madam Deputy Speaker': 'Speaker'})

#add Stephen Philip Rotheram for some reason
party_dict.update({'Stephen Philip Rotheram': 'Labour'})

#add multiple speakers
party_dict.update({'My Lords and Members of the House of Commons':'Multiple'})
party_dict.update({'Members of the House of Commons':'Multiple'})
party_dict.update({'Hon. Members':'Multiple'})
party_dict.update({'Several hon. Members':'Multiple'})

#add none
party_dict.update({None: 'None'})

# parties and MP number
parties = {'UKIP': 1,
           'Sinn Féin': 4,
           'Conservative': 330,
           'Labour': 232,
           'Liberal Democrat': 8,
           'Independent': 1,
           'UUP': 2,
           'DUP': 8,
           'Scottish National Party': 56,
           'Green': 1,
           'Social Democratic and Labour Party': 3,
           'Plaid Cymru': 3}


I am working on the file "Hansard mining.ipynb".
The first task is:
party_talk
Also generate a dict for each party valued by the concatenated string of everything their MPs have said
Can you write Python code for it?

# Initialise party talk dict with each party name
party_talk = {}
for party in parties.keys():
    party_talk.update({party:''})

for speaker, party_and_text in words_spoken_by_mp.items():
    # will throw error if party doesn't exist in parties list
    party, text = party_and_text

    if len(party_talk[party]) == 0:
        party_talk[party] = text
    else:
        party_talk[party] = party_talk[party] + ' ' + text

con_sentences = nltk.tokenize.sent_tokenize(party_talk['Conservative'])
con_sentences = ["{0} {1} {2}".format("START_TOKEN", w, "END_TOKEN") for w in con_sentences]
con_sentences = [nltk.word_tokenize(sentence) for sentence in con_sentences]

word_freqs = nltk.FreqDist(con_sentences)
vocab = word_freqs.most_common(9999)

idx_2_word = [x[0] for x in vocab]
idx_2_word.append("TOO_INFREQUENT_WORD")
word_2_idx = dict([(w,i) for i,w in enumerate(idx_2_word)])

for i, sentence in enumerate(con_sentences):
    con_sentences[i] = [word if word in word_2_idx else "TOO_INFREQUENT_WORD"  for word in sentence]

X_train = np.asarray([[word_2_idx[w] for w in sent[:-1]] for sent in con_sentences])
Y_train = np.asarray([[word_2_idx[w] for w in sent[1:]] for sent in con_sentences])


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
Finding the dynamical matrix with numerical derivatives
We start from a function $V(x)$. If we want to calculate a derivative,
we just use the difference formula but don't take the delta too small.
Using $\delta x = 10^{-6}$ is safe.
$$
F = -\frac{dV(x)}{dx} \approx
\frac{V(x+\Delta x) - V(x-\Delta x)}{2\Delta x}
$$
Note that it is more accurate to do this symmetric difference formula
than it would be to use the usual forward derivative from calculus class.
It's easy to see this formula is just calculating the slope of the function using points near $x$.
Can you write Python code for it?

def V(x):
    return 0.5 * x**2

deltax = 1e-6

def F_approx(x):
    return ( V(x + deltax) - V(x - deltax) ) / (2 * deltax)

[(x, F_approx(x)) for x in np.linspace(-2,2,9)]


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
Next, we can find the second derivative by using the difference formula twice.
We find the nice expression,
$$
\frac{d^2V}{dx^2} \approx \frac{V(x+\Delta x) - 2V(x) + V(x-\Delta x)}{(\Delta x)^2}.
$$
This formula has the nice interpretation of comparing the value of $V(x)$ to
the average of points on either side. If it is equal to the average, the line
is straight and the second derivative is zero.
If average of the outer values is larger than $V(x)$, then the ends curve upward,
and the second derivative is positive.
Likewise, if the average of the outer values is less than $V(x)$, then the ends curve downward,
and the second derivative is negative.
Can you write Python code for it?

def dV2dx2_approx(x):
    return ( V(x + deltax) - 2 * V(x) + V(x - deltax) ) / deltax**2

[(x, dV2dx2_approx(x)) for x in np.linspace(-2,2,9)]


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
Now we can use these derivative formulas to calcuate the dynamical matrix
for the two masses on one spring. Well use $k=1$ and $m=1$ for simplicity.
Can you write Python code for it?

def V2(x1, x2):
    return 0.5 * (x1 - x2)**2

x1, x2 = -1, 1
mat = np.array(
[[(V2(x1+deltax, x2) - 2 * V2(x1,x2) + V2(x1-deltax, x2)) / deltax**2 ,
  (V2(x1+deltax, x2+deltax) - V2(x1-deltax, x2+deltax)
   - V2(x1+deltax, x2-deltax) + V2(x1+deltax, x2+deltax)) / (2*deltax)**2],
 [(V2(x1+deltax, x2+deltax) - V2(x1-deltax, x2+deltax)
   - V2(x1+deltax, x2-deltax) + V2(x1+deltax, x2+deltax)) / (2*deltax)**2,
  (V2(x1, x2+deltax) - 2 * V2(x1,x2) + V2(x1, x2-deltax)) / deltax**2 ]]
)

mat

freq, vectors = np.linalg.eig(mat)
vectors = vectors.transpose()

for f,v in zip(freq, vectors):
    print("freqency", f, ", eigenvector", v)


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
The first mode is a vibrational mode were the masses vibrate against each other (moving in opposite directions). This can be seen from the eigenvector.
Can you write Python code for it?


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
The frequencies of the two modes of vibration are (in multiples of $\sqrt{k/m}$)
Can you write Python code for it?


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
We can interactively illustrate the vibrational mode.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
from IPython.html import widgets

def make_plot(t):
    fig, ax = plt.subplots()
    x,y = np.array([-1,1]), np.array([0,0])
    plt.plot(x, y, 'k.')
    plt.plot(x + 0.3 * vectors[0] * t, y, 'bo')
    plt.xlim(-1.5,1.5)
    plt.ylim(-1.5,1.5)

widgets.interact(make_plot, t=(-1,1,0.1))


I am working on the file "HarmonicOscillator.ipynb".
The first task is:
We want to generalize this result to several massess connected by several springs.
The spring constant as a second derivative of potential
The force related to poential energy by
$$ F = -\frac{d}{dx}V(x).$$
Ths equation comes directly from the definition that work is force times distance.
Integrating this, we find the potential energy of a mass on a spring,
$$ V(x) = \frac{1}{2}kx^2. $$
In fact, the spring contant can be defined to be the second derivative of the potential,
$$ k = \frac{d^2}{dx^2} V(x).$$ We take the value of the second derivative at the minimum
of the potential, which assumes that the oscillations are not very far from equilibrium.
We see that Hooke's law is simply
$$F = -\frac{d^2 V(x)}{dx^2} x, $$
where the second derivative is evaluated at the minimum of the potential.
For a general potential, we can write the equation of motion as
$$ \frac{d^2}{dt^2} x = -\frac{1}{m}\frac{d^2V(x)}{dx^2} x.$$
The expression on the right hand side is known as the dynamical matrix,
though this is a trivial 1x1 matrix.
Two masses connected by a spring
Now the potential depends on two corrdinates,
$$ V(x_1, x_2) = \frac{1}{2} k (x_1 - x_2 - d),$$
where $d$ is the equilibrium separation of the particles.
Now the force on each particle depends on the positions of both of the particles,
$$
\begin{pmatrix}F_1 \ F_2\end{pmatrix}
= -
\begin{pmatrix}
\frac{\partial^2 V}{\partial x_1^2} &
\frac{\partial^2 V}{\partial x_1\partial x_2} \
\frac{\partial^2 V}{\partial x_1\partial x_2} &
\frac{\partial^2 V}{\partial x_2^2} \
\end{pmatrix}
\begin{pmatrix}x_1 \ x_2\end{pmatrix}
$$
For performing the derivatives, we find
$$
\begin{pmatrix}F_1 \ F_2\end{pmatrix}
= -
\begin{pmatrix}
k & -k \
-k & k \
\end{pmatrix}
\begin{pmatrix}x_1 \ x_2\end{pmatrix}
$$
The equations of motion are coupled,
$$
\begin{pmatrix}
\frac{d^2x_1}{dt^2} \
\frac{d^2x_2}{dt^2} \
\end{pmatrix}
= -
\begin{pmatrix}
k/m & -k/m \
-k/m & k/m \
\end{pmatrix}
\begin{pmatrix}x_1 \ x_2\end{pmatrix}
$$
To decouple the equations, we find the eigenvalues and...
Can you write Python code for it?

import numpy as np
a = np.array([[1, -1], [-1, 1]])
freq, vectors = np.linalg.eig(a)
vectors = vectors.transpose()


I am working on the file "Harmonicka_analyza--nizke_a_vysoke_harmonicke_slozky.ipynb".
The first task is:
Vyšší harmonické složky -- rychlé změny
Vyšší harmonické složky představují rychlé změny.
Na předchozích obrázcích je patrné, že čím více vyššíh hramonických složek je zachováno, tím jsou hrany strměší.
Pokud odfiltrujeme nižsí harmonické složky a ponecháme pouze vyšší harmonické složky původní tvar už nebude možné
rozeznat, ale zůstanou nám práve rychlé změny -- v místě původní strmé hrany se oběvý zákmit.
Can you write Python code for it?

Q=copy.deepcopy(U)
Q[:11]=0
soucet(Q,u'Obdélníkové napětí -- Jen vyšší harmonicé složky')

Q=copy.deepcopy(U)
Q[:31]=0
soucet(Q,u'Obdélníkové napětí -- Jen vyšší harmonicé složky')

Q=copy.deepcopy(U)
Q[:51]=0
soucet(Q,u'Obdélníkové napětí -- Jen vyšší harmonicé složky')

Q=copy.deepcopy(U)
Q[:30]=0
Q[51:]=0
soucet(Q,u'Obdélníkové napětí -- Jen vyšší harmonicé složky')


I am working on the file "Harry Tight Binding.ipynb".
The first task is:
Interpolated SiGe band structure
The band structure for Si(1-x)Ge(x) is supposedly well approximated by a linear interpolation of the Si and Ge band structures.
Can you write Python code for it?

def SiGe_band(x=0.2):
    Si_data = TB.bandpts(TB.Si)
    Ge_data = TB.bandpts(TB.Ge)
    data = (1-x)*Si_data + x*Ge_data
    TB.bandplt("SiGe, %%Ge=%.2f" % x,data)
    return

SiGe_band(0)

SiGe_band(0.1)

SiGe_band(0.25)

SiGe_band(0.37)


I am working on the file "Harry Tight Binding.ipynb".
The first task is:
Plotting misc parts of Brillouin zones
Compare Si and Ge CBs.
Can you write Python code for it?

Ge_CB = TB.bandpts(TB.Ge)[:,4]
Si_CB = TB.bandpts(TB.Si)[:,4]

nk = len(Si_CB)
n = (nk-2)//3

plt.plot(Si_CB)
plt.plot(Ge_CB)
TB.band_labels(n)
plt.axis(xmax=3*n+1)

plt.plot(Si_CB,label='Si')
plt.plot(Ge_CB,label='Ge')
plt.plot(0.9*Si_CB + 0.1*Ge_CB,label='Si_0.9 Ge_0.1')
TB.band_labels(n)
plt.axis(xmax=3*n+1)
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

min_Si = min(Si_CB)
min_Ge = min(Ge_CB)
print min_Si, min_Ge
# min_Si - min_Ge = 0.12
Si_CB_shifted = Si_CB - min_Si + min_Ge + 0.12


I am working on the file "Harvesting ArcGIS REST.ipynb".
The first task is:
Harvest
Hammertime? Hammertime.
Can you write Python code for it?

harvest_arcgis_service(services[0],
                       ckan,
                       owner_org_id = ckan.action.organization_show(id="mrwa")["id"],
                       author = "Main Roads Western Australia",
                       author_email = "irissupport@mainroads.wa.gov.au",
                       debug=False)

a = ["SLIP Classic", "Harvested"]
a.append("test")
a


I am working on the file "Hashing.ipynb".
The first task is:
Ordinal Hash
When dealing with strings, we can use the ordinal values of the constituent characters of a given word, to create a hash.
It's important to notice, however, that anagrams can produce hash collisions, as shown below.
Can you write Python code for it?

def ord_hash(string, table_size):
    hash_val = 0
    for position in range(len(string)):
        hash_val = hash_val + ord(string[position])

    return hash_val % table_size

print(ord_hash("cat", 11))
print(ord_hash("tac", 11))


I am working on the file "Hashing.ipynb".
The first task is:
Used a little more interestingly, we can use the weighted ordinal hash function that we've defined above, combined with the lp_hash function that we've just defined, to store string(s) for later lookup.
Can you write Python code for it?

animal_items = ["cat", "dog", "goat",
         "chicken", "pig", "horse",
        "ostrich", "lion", "puma"]

def rehash(oldhash, table_size):
    return (oldhash+1) % table_size

def weighted_ord_hash(string, table_size):
    hash_val = 0
    for position in range(len(string)):
        hash_val = hash_val + (ord(string[position]) * position)
    return hash_val % table_size

def lp_hash(item_list, table_size):

    lp_hash_table = dict([(i,None) for i,x in enumerate(range(table_size))])

    for item in item_list:
        i = weighted_ord_hash(item, table_size)
        print("%s hashed == %s \n" %(item, i))
        if lp_hash_table[i] == None:
            lp_hash_table[i] = item
        elif lp_hash_table[i] != None:
            print("Collision, attempting linear probe \n")
            next_slot = rehash(i, table_size)
            print("Setting next slot to %s \n" % next_slot)
            while lp_hash_table[next_slot] != None:
                next_slot = rehash(next_slot, len(lp_hash_table.keys()))
                print("Next slot was not empty, trying next slot %s \n" % next_slot)
            if lp_hash_table[next_slot] == None:
                lp_hash_table[next_slot] = item
    return lp_hash_table

print(lp_hash(animal_items, 11))


I am working on the file "Hashing.ipynb".
The first task is:
We can see there were multiple collisions within this dataset. Specifically hashes of 1, 9, and 10. And we can see in the resulting table that only the last computed hashes are stored in the respective table slots.
Below we'll implement an lp_hash function that will perform linear probing over the slots available within the table for any collisions that occur.
Can you write Python code for it?

items = [25,54,34,67,75,21,77,31]

def rehash(oldhash, table_size):
    return (oldhash+1) % table_size

def lp_hash(item_list, table_size):

    lp_hash_table = dict([(i,None) for i,x in enumerate(range(table_size))])

    for item in item_list:
        i = item % table_size
        print("%s hashed == %s \n" %(item, i))
        if lp_hash_table[i] == None:
            lp_hash_table[i] = item
        elif lp_hash_table[i] != None:
            print("Collision, attempting linear probe \n")
            next_slot = rehash(i, table_size)
            print("Setting next slot to %s \n" % next_slot)
            while lp_hash_table[next_slot] != None:
                next_slot = rehash(next_slot, len(lp_hash_table.keys()))
                print("Next slot was not empty, trying next slot %s \n" % next_slot)
            if lp_hash_table[next_slot] == None:
                lp_hash_table[next_slot] = item
    return lp_hash_table

print(lp_hash(items, 11))


I am working on the file "Hathi Catalog Builder.ipynb".
The first task is:
Here's where we write it out.
Can you write Python code for it?

if __name__=="__main__":
    all_hathi = All_Hathi("/home/bschmidt/data/hathi_metadata/")
    dump = gzip.open("~/hathi_metadata/jsoncatalog_full.txt.gz","w")
    for i,vol in enumerate(all_hathi):
        if i % 250000 == 0:
            sys.stdout.write("Reading item no. " + str(i) + "\n")
        dump.write(json.dumps(vol) + "\n")


I am working on the file "Hathi Catalog Builder.ipynb".
The first task is:
This class has the base names of the files and my directory structure hard-coded in: run elsewhere, you'll need to change that
Can you write Python code for it?

demo = []
for i,entry in enumerate(all_hathi):
    demo.append(entry)
    if i > 10:
        break
print demo


I am working on the file "Hathi Catalog Builder.ipynb".
The first task is:
This converts a json object into a MARC class so the existing methods will work
Can you write Python code for it?


I am working on the file "Haversine/Haversine.ipynb".
The first task is:
Calculate Excursion Distance
Can you write Python code for it?

start = (43.059535, -71.013171)
end = (43.083620, -70.892085)

central_angle, km = haversine(*(start + end))

print("Central Angle of %g radians" % central_angle)
print("Arc length distance of %g km" % km)


I am working on the file "HealthDemo.ipynb".
The first task is:
It would appear that nosmokers are older, more educated, more 'coupled' and heavier - but of similar height and blood pressure
Can you write Python code for it?

# Lets draw two samples of equal size
n_sample = 200
smoker_sample = smokers.sample(n_sample)
nosmoker_sample = nosmokers.sample(n_sample)

weight = Table([nosmoker_sample['weight'],smoker_sample['weight']],['NoSmoke','Smoke'])
weight.hist(overlay=True,bins=30,normed=True)

bins=np.arange(39,139,5)
weight_dist = weight.bin(bins=bins, normed=True)
weight_dist['diff']=weight_dist['NoSmoke density']-weight_dist['Smoke density']
print('TVD: ',sum(np.abs(weight_dist['diff'])))
weight_dist.show()

weight_dist.select(['bin','diff']).bar('bin')

weight.stats(summary_ops)

np.mean(weight['NoSmoke'])-np.mean(weight['Smoke'])


I am working on the file "HealthSample.ipynb".
The first task is:
Descriptive statistics - smoking
Can you write Python code for it?

def firstQtile(x) : return np.percentile(x,25)
def thirdQtile(x) : return np.percentile(x,25)
summary_ops = (min, firstQtile, np.median, np.mean, thirdQtile, max, sum)

# Let's try what is the effect of smoking
smokers = health.where('smoker',1)
nosmokers = health.where('smoker',0)
print(smokers.num_rows, ' smokers')
print(nosmokers.num_rows, ' non-smokers')

smokers.stats(summary_ops)

nosmokers.stats(summary_ops)

help(smokers.hist)


I am working on the file "HealthSample.ipynb".
The first task is:
Permutation tests
Can you write Python code for it?

# Lets draw two samples of equal size
n_sample = 200
smoker_sample = smokers.sample(n_sample)
nosmoker_sample = nosmokers.sample(n_sample)
weight = Table().with_columns([('NoSmoke', nosmoker_sample['weight']),('Smoke', smoker_sample['weight'])])
weight.hist(overlay=True,bins=30,normed=True)

weight.stats(summary_ops)


I am working on the file "HealthSample.ipynb".
The first task is:
The 4.5 kg difference is certainly not an artifact of the sample we started with.  The smokers definitely weigh less. At the same time, these are not light people in this study.  Better go back and understand what was the purpose of the study that led to the selection of these six thousand individuals.
Other Factors
Can you write Python code for it?

# A sense of the overall population represented - older
health.select(['age','education']).hist(bins=20)

# How does education correlate with age?
health.select(['age','education']).scatter('age', fit_line=True)

health.pivot_hist('race','education',normed=True)

# How are races represented in the dataset and how does hispanic overlay the three?
race = health.select(['race', 'hispanic'])
race['count']=1
by_race = race.group('race',sum)
by_race['race frac'] = by_race['count sum']/np.sum(by_race['count sum'])
by_race['hisp frac'] = by_race['hispanic sum'] / by_race['count sum']
by_race

health.select(['height','weight']).scatter('height','weight',fit_line=True)


I am working on the file "HealthSample.ipynb".
The first task is:
What is the effect of smoking on weight?
Can you write Python code for it?

smokers.hist('weight', bins=20)

nosmokers.hist('weight', bins=20)

np.mean(nosmokers['weight'])-np.mean(smokers['weight'])


I am working on the file "Health_Economics.ipynb".
The first task is:
Import UK Health Data // Source: http://www.phoutcomes.info/public-health-outcomes-framework#page/9/gid/1000049/pat/6/par/E12000004/ati/101/are/E06000015/nn/nn-1-E06000015
Can you write Python code for it?

healthd = pd.read_excel('health.xlsx',sheetname='District & UA')
healthi = pd.read_excel('health.xlsx',sheetname='Indicator Metadata')


I am working on the file "Health_Economics.ipynb".
The first task is:
Merge Data Sets
Can you write Python code for it?

df = gva.merge(datar, left_index=True, right_index=True, how='outer')
df


I am working on the file "Health_Economics.ipynb".
The first task is:
Plot Graph
Can you write Python code for it?

plt.figure()
plt.plot(df['GVA % Change'], '-o', df['Child LIF % Change'], '-o');
plt.gca().set_title('Percentage Change in GVA & Children in Low Income Families (South Oxfordshire)');
plt.gca().set_ylabel('% Change from 2006 Baseline');
plt.gca().legend(['Gross Value Added','Reduction in Children in Low Income Families']);


I am working on the file "Health_Economics.ipynb".
The first task is:
Selected Indicator
Can you write Python code for it?

Id = '1.01ii'

ind = healthi[healthi['Indicator'].str.contains('{} '.format(Id)) == True]
nm = ind.iloc[0]['Indicator']
unit = ind.iloc[0]['Unit']

#Print out information about selected indicator.
ind

# Select indicator from main dataset
data = healthd[healthd['Indicator'].str.contains('{} '.format(Id)) == True]
data = data[[1,4,6]]

# Get list of time periods
xx = data[[0]]
xx = xx.drop_duplicates()
xl = xx['Time Period'].tolist()

# Loop to create column for each year
for x in range(len(xl)):
    inputs = data[data['Time Period'].str.contains(xl[x]) == True]
    inputs = inputs.drop('Time Period', 1)

    # Average male and female values
    inputs = inputs.groupby(inputs['Area Code']).mean()
    inputs.columns = [xl[x]]
    if x == 0:
        datar = inputs
    else:
        datar = datar.merge(inputs, left_index=True, right_index=True, how='left')

# Select data for South Oxfordshire
datar = datar[datar.index.str.contains(ac) == True]
datar = datar.transpose()
datar.columns = ['Children LIF']
bl = datar.iloc[0]['Children LIF']
datar['Child LIF % Change'] = ((datar['Children LIF'] - bl)/bl) * -100
datar = datar.set_index([xi],drop=True)
datar


I am working on the file "Heaps.ipynb".
The first task is:
(2) The left child is largest. In the case if we swap the parent node with the child, the heap properties are restored (i.e. the top node is larger than either of its children).
Can you write Python code for it?

heap1 = [5, 10, 3]
heap2 = [10, 5, 3]
with Canvas(400, 80) as ctx:
    draw_binary_tree(ctx, (100, 20), heap1)
    draw_binary_tree(ctx, (300, 20), heap2)


I am working on the file "Heaps.ipynb".
The first task is:
A quick example of using the code:
Can you write Python code for it?

heap = []
heap_insert(heap, 20)
print("adding 20: ", heap) # [20]
heap_insert(heap, 5)
print("adding 5:  ", heap) # [5, 20]
heap_insert(heap, 1)
print("adding 1:  ", heap) # [1, 20, 5]
heap_insert(heap, 50)
print("adding 50: ", heap) # [1, 20, 5, 50]
heap_insert(heap, 6)
print("adding 6:  ", heap) # [1, 5, 6, 50, 20]

with Canvas(400, 150) as ctx:
    draw_binary_tree(ctx, (200, 50), heap)


I am working on the file "Heaps.ipynb".
The first task is:
Finally, putting everything together, we have heap_pop:
Can you write Python code for it?

def heap_pop(heap):
    # swap root with last value
    heap[0], heap[-1] = heap[-1], heap[0]

    # remove last value
    result = heap.pop()

    # restore heap properties
    for i in range(len(heap)):
        percolate_down(heap, 0, len(heap))

    return result


I am working on the file "Heaps.ipynb".
The first task is:
Operations
Inserting a new item into the heap
To insert a new item into the heap, we start by first adding it to the end. Once added, we will percolate the item up until its parent is larger than the item.
Can you write Python code for it?

def parent(i): return (i-1)/2
def left(i): return 2*i+1
def right(i): return 2*i+2

def percolate_up(heap, startpos, pos):
    ppos = parent(pos)
    while pos > startpos and heap[ppos] < heap[pos]:
        # percolate value up by swapping current position with parent position
        heap[pos], heap[ppos] = heap[ppos], heap[pos]

        # move up one node
        pos = ppos
        ppos = parent(pos)

def heap_insert(heap, value):
    # add value to end
    heap.append(value)

    # move value up heap until the nodes below it are smaller
    percolate_up(heap, 0, len(heap)-1)


I am working on the file "Heaps.ipynb".
The first task is:
To see heap_pop in action:
Can you write Python code for it?

heap = []
heap_insert(heap, 1)
heap_insert(heap, 100)
heap_insert(heap, 20)
heap_insert(heap, 5)
heap_insert(heap, 3)
print(heap)

print(heap_pop(heap))
print(heap_pop(heap))
print(heap_pop(heap))
print(heap_pop(heap))
print(heap_pop(heap))


I am working on the file "Heaps.ipynb".
The first task is:
To see why this works, we can visualize the algorithm. We start with a new value of 100 (highlighted with red). That is inserted into the bottom of the heap. We percoluate 100 up (each swap is highlighted) until it gets placed into the root note. Once finished, the heap's properties are now restored, and every child will have a smaller value than its parent.
To get a good sense of how percolute_up works, try putting different values in for the heap. Note that, it won't work correctly if the initial value isn't a proper heap.
Can you write Python code for it?

heap = [16, 14, 10, 8, 7, 9, 3, 2, 4]
heap.append(100)
insert_item_to_heap_example(heap)


I am working on the file "Heapsort.ipynb".
The first task is:
Heap Sort
Summary
| Performance                 | Complexity       |
|-----------------------------|------------------|
|Worst-case                   | $O(n\log{n})$    |
|Best-case                    | $O(n\log{n})$    |
|Average                      | $O(n\log{n})$    |
|Worst-case space             | $O(1)$           |
Notes
The time complexity for heap sort in all cases make it a good choice when there need to be guarantees for runtime. Additionally, heap sort requires a constant amount of memory, as it can sort in-place.
Algorithm
It is possible implement heap sort by using heap_insert and heap_pop (see notes on heaps). However, both of these operations require the allocation of additional space. Instead, we can perform the same operations in place.
The function build_heap applies the function percoluate_down on each non-leaf node in the tree starting at the bottom.
Can you write Python code for it?

def build_heap(lst):
    # last non-leaf node
    nonleaf_nodes = len(lst)/2

    # start at bottom work up for each node
    for i in range(nonleaf_nodes-1, -1, -1):
        percolate_down(lst, i, len(lst))


I am working on the file "Heapsort.ipynb".
The first task is:
a visualize demonstration of what build_heap is doing:
Can you write Python code for it?


I am working on the file "Heapsort.ipynb".
The first task is:
which allows us to define:
Can you write Python code for it?

def heap_sort(lst):
    build_heap(lst)

    # build up list starting from largset at back to smallest up front
    for i in range(len(lst)-1, 0, -1):
        # move largest value to the back
        lst[0], lst[i] = lst[i], lst[0]

        # re-heapify tree
        percolate_down(lst, 0, i)

    return lst


I am working on the file "HeatMapQuestionaire.ipynb".
The first task is:
Which, I hope you agree, is a very attractive and informative plot.
As an aside, we can also use some additional matplotlib modules to put all fo the heat maps into a single figure, each with their own colour scheme and colour bar:
Can you write Python code for it?

from mpl_toolkits.axes_grid1 import make_axes_locatable
from matplotlib.ticker import MultipleLocator

fig, axs = plt.subplots(3, 1, figsize=(10,15))

#Add nice light grey boundaries and grid:
for ax in axs.reshape(-1):
    ax.tick_params(axis='both', color='#E6E6E6')
    ax.spines['top'].set_color('#E6E6E6')
    ax.spines['bottom'].set_color('#E6E6E6')
    ax.spines['left'].set_color('#E6E6E6')
    ax.spines['right'].set_color('#E6E6E6')
    ax.vlines(np.arange(0.5, 15, 1), 0.5, 5.5, lw=2, colors='#E6E6E6')
    ax.hlines(np.arange(0.5, 6, 1), 0.5, 14.5, lw=2, colors='#E6E6E6')

#Plot the heatmap data
p = axs[0].imshow(before_responses, origin='lower', cmap='Blues',
                  extent=[0.5, 14.5, 0.5, 5.5], interpolation='nearest')
div1 = make_axes_locatable(axs[0])
cbloc = div1.append_axes("right", size="5%", pad=0.05)
cb1 = plt.colorbar(p, cax=cbloc, ticks=MultipleLocator(1.0))
axs[0].set_title('Before')

p = axs[1].imshow(after_responses, origin='lower', cmap='RdPu',
                  extent=[0.5, 14.5, 0.5, 5.5], interpolation='nearest')
div2 = make_axes_locatable(axs[1])
cbloc = div2.append_axes("right", size="5%", pad=0.05)
cb2 = plt.colorbar(p, cax=cbloc, ticks=MultipleLocator(1.0))
axs[1].set_title('After')

p = axs[2].imshow(form_responses, origin='lower', cmap='Greens',
                  extent=[0.5, 14.5, 0.5, 5.5], interpolation='nearest')
div3 = make_axes_locatable(axs[2])
cbloc = div3.append_axes("right", size="5%", pad=0.05)
cb3 = plt.colorbar(p, cax=cbloc, ticks=MultipleLocator(1.0))
axs[2].set_title('Form')

for ax in axs:
    ax.set_yticklabels(['', 'Completely agree', '', 'Neither', '',
                        'Completely Disagree'])
    ax.set_xlabel('Statement')


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
As a quick diagnostic, we start and stop capture frames.
Can you write Python code for it?


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
As we can see, the measured heater values from the housekeeping are roughly the same as the values set values, up to an affine transformation. To see this, it is helpful to compute measured - set to see the errors:
Can you write Python code for it?

err1 = map(lambda y,x: y-x, y1, x1)
err2 = map(lambda y,x: y-x, y2, x2)
err3 = map(lambda y,x: y-x, y3, x3)

f = plt.figure(figsize=(10,10))
ax1 = f.add_subplot(331)
ax1.set_title('Error for Heater 1')
ax1.locator_params(nbins=4)
ax1.set_xlim([-20,160])
ax1.scatter(x1,err1,color='blue')

ax2 = f.add_subplot(332)
ax2.set_title('Error for Heater 2')
ax2.locator_params(nbins=4)
ax2.set_xlim([-20,160])
ax2.scatter(x2,err2,color='red')

ax3 = f.add_subplot(333)
ax3.set_title('Error for Heater 3')
ax3.locator_params(nbins=4)
ax3.set_xlim([-20,160])
ax3.scatter(x3,err3,color='green')
plt.show()


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
From the above we can infer that in order to correct for the error we need to disregard the measured values when the heaters are set to 0 mA, since they are apparently outliars.
Calibration
We next turn to calibrating the heaters.
The above analysis only really looks at 3 heaters; we can see that the error measurements have an outliar at 0 mA, but it is of interest to reproduce this analysis for more than N=3 components.  This should be as simple as rerunning this notebook, however.
For the time being we are proceeding with calibration by performing a simple linear transformation on the set value so that it will properly correspond with the desired observed value.
To calculate this, we first compute via linear regression the constants $m$ and $c$ in the equation below:
$$ \texttt{measured_value} = m \cdot \texttt{set_value} + c $$
We make sure to avoid $\texttt{set_value} = 0$ where outliers have been observed.
Can you write Python code for it?

def get_set_observed(heater_current, measurements):
    return ([i["set"][heater_current] for i in measurements
             if i["set"][heater_current] != 0],
            [j["measured"][heater_current] for j in measurements
             if j["set"][heater_current] != 0])

def calibrate(heater_current, measurements):
    from scipy import stats
    x,y = get_set_observed(heater_current, measurements)
    return stats.linregress(x,y)


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
Make sure to run the following cell so that the heaters can "chill out".
Can you write Python code for it?

fpe1.ops.heater_1_current = fpe1.ops.heater_1_current.low
fpe1.ops.heater_2_current = fpe1.ops.heater_2_current.low
fpe1.ops.heater_3_current = fpe1.ops.heater_3_current.low
fpe1.ops.send()


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
Now we collect the measurements; this will make 10 sets of samples for heater currents that are between 0 and 150 mA, spaced out by 10 mA.
Can you write Python code for it?

from tessfpe.data.operating_parameters import operating_parameters
operating_parameters["heater_1_current"]
high = operating_parameters["heater_1_current"]["high"]

measurements = []
for i in range(5,int(high)*10,10):
    a = i % high
    b = (i + 100) % high
    c = abs((high - i) % high)
    fpe1.ops.heater_1_current = a
    fpe1.ops.heater_2_current = b
    fpe1.ops.heater_3_current = c
    fpe1.ops.send()
    measurements.append({"set": {"heater_1_current": a,
                                 "heater_2_current": b,
                                 "heater_3_current": c},
                         "measured": {"heater_1_current": fpe1.house_keeping["analogue"]["heater_1_current"],
                                      "heater_2_current": fpe1.house_keeping["analogue"]["heater_2_current"],
                                      "heater_3_current": fpe1.house_keeping["analogue"]["heater_3_current"]}})
len(measurements)


I am working on the file "Heater Calibration Tests.ipynb".
The first task is:
Plots
Let's first plot the set value on the $x$ axis versus the measured value on the $y$ axis to test how well calibrated the sensors are:
Can you write Python code for it?

x1 = [i["set"]["heater_1_current"] for i in measurements]
y1 = [j["measured"]["heater_1_current"] for j in measurements]
x2 = [i["set"]["heater_2_current"] for i in measurements]
y2 = [j["measured"]["heater_2_current"] for j in measurements]
x3 = [i["set"]["heater_3_current"] for i in measurements]
y3 = [j["measured"]["heater_3_current"] for j in measurements]

%matplotlib inline
%config InlineBackend.figure_format = 'svg'

import numpy as np
import matplotlib.pyplot as plt

f = plt.figure(figsize=(10,10))
ax1 = f.add_subplot(331, aspect='equal')
ax1.set_title('Heater 1')
ax1.scatter(x1,y1,color='blue')
ax1.locator_params(nbins=4)
ax1.set_xlim([0,160])
ax1.set_ylim([0,160])

ax2 = f.add_subplot(332, aspect='equal')
ax2.set_title('Heater 2')
ax2.scatter(x2,y2,color='red')
ax2.locator_params(nbins=4)
ax2.set_xlim([0,160])
ax2.set_ylim([0,160])

ax3 = f.add_subplot(333, aspect='equal')
ax3.set_title('Heater 3')
ax3.scatter(x3,y3,color='green')
ax3.locator_params(nbins=4)
ax3.set_xlim([0,160])
ax3.set_ylim([0,160])
plt.show()


I am working on the file "Heatmap for Single Gene.ipynb".
The first task is:
In the code above, I created a heatmap using the code from the heatmap above ( http://stackoverflow.com/questions/2369492/generate-a-heatmap-in-matplotlib-using-a-scatter-data-set ) and from the data of some of the expression vectors for genes in cells in the aorta.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

aortaData = []
aa1 = []
aa2 = []
aa3 = []
aa4 = []
aa5 = []
with open ("genomicdata.csv") as csvfile:
    readCSV = csv.reader(csvfile, delimiter= '\t') #gives access to the CSV file
    for col in readCSV:
        if col[7] == '':
            aortaData.append('0')
        else:
            aortaData.append(col[7]) #save the data from the side
    #need smaller arrays
    #Each array needs 5 elements
    aa1 = list(map(float, aortaData[1:6]))
    aa2 = list(map(float, aortaData[6:11]))
    aa3 = list(map(float, aortaData[11:16]))
    aa4 = list(map(float, aortaData[16:21]))
    aa5 = list(map(float, aortaData[21:26]))

x = [1, 2, 3, 4, 5]
y = [0.1, 0.2, 0.3, 0.4, 0.5]

#take the 5 small arrays
intensity = [
    aa1,
    aa2,
    aa3,
    aa4,
    aa5
]

#setup the 2D grid with Numpy
x, y = np.meshgrid(x, y)

#convert intensity (list of lists) to a numpy array for plotting
intensity = np.array(intensity)

plt.pcolormesh(x, y, intensity) #plots the intensity on the grid with colors
plt.colorbar() #colors are set from the colorbar. The colorbar shows the intensity
plt.show()


I am working on the file "Heatmap for Single Gene.ipynb".
The first task is:
The previous bit of code is to make sure that the Heatmap plots in my Python 3 notebook.
Can you write Python code for it?

import numpy as np
import numpy.random
import matplotlib.pyplot as plt
import csv

aortaData = []
aa1 = []
aa2 = []
aa3 = []
aa4 = []
aa5 = []
with open ("genomicdata.csv") as csvfile:
    readCSV = csv.reader(csvfile, delimiter= '\t') #gives access to the CSV file
    for col in readCSV:
        if col[7] == '':
            aortaData.append('0')
        else:
            aortaData.append(col[7]) #save the data from the side
    #need smaller arrays
    #Each array needs 5 elements
    aa1 = list(map(float, aortaData[1:6]))
    aa2 = list(map(float, aortaData[6:11]))
    aa3 = list(map(float, aortaData[11:16]))
    aa4 = list(map(float, aortaData[16:21]))
    aa5 = list(map(float, aortaData[21:26]))

# Generate some test data
x = np.random.randn(8873)
y = np.random.randn(8873)

heatmap, xedges, yedges = np.histogram2d(x, y, bins=5) #if array is 5x5 (has 25 components), need 5 bins
extent = [xedges[0], xedges[-1], yedges[0], yedges[-1]]

plt.clf()
plt.imshow(heatmap.T, extent=extent, origin='lower')
plt.colorbar()
plt.show()


I am working on the file "Heatmap for Single Gene.ipynb".
The first task is:
This is a test to manually see how the heat map should be expressed. I took the array from the previous example, and wrote it out manually to test how the data will be expressed.
http://seaborn.pydata.org/generated/seaborn.heatmap.html
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

aortaData = []
aa1 = []
aa2 = []
aa3 = []
aa4 = []
aa5 = []
aa6 = []
aa7 = []
aa8 = []
aa9 = []
aa10 = []
with open ("genomicdata.csv") as csvfile:
    readCSV = csv.reader(csvfile, delimiter= '\t') #gives access to the CSV file
    for col in readCSV:
        if col[7] == '':
            aortaData.append('0')
        else:
            aortaData.append(col[7]) #save the data from the side
    #need smaller arrays
    #Each array needs 5 elements
    aa1 = list(map(float, aortaData[1:11]))
    aa2 = list(map(float, aortaData[11:21]))
    aa3 = list(map(float, aortaData[21:31]))
    aa4 = list(map(float, aortaData[31:41]))
    aa5 = list(map(float, aortaData[41:51]))
    aa6 = list(map(float, aortaData[51:61]))
    aa7 = list(map(float, aortaData[61:71]))
    aa8 = list(map(float, aortaData[71:81]))
    aa9 = list(map(float, aortaData[81:91]))
    aa10 = list(map(float, aortaData[91:101]))

x = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
y = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]

#take the 10 small arrays
intensity = [
    aa1,
    aa2,
    aa3,
    aa4,
    aa5,
    aa6,
    aa7,
    aa8,
    aa9,
    aa10
]

#setup the 2D grid with Numpy
x, y = np.meshgrid(x, y)

#convert intensity (list of lists) to a numpy array for plotting
intensity = np.array(intensity)

plt.pcolormesh(x, y, intensity) #plots the intensity on the grid with colors
plt.colorbar() #colors are set from the colorbar. The colorbar shows the intensity
plt.show()


I am working on the file "Heatmap for Single Gene.ipynb".
The first task is:
https://stackoverflow.com/questions/33282368/plotting-a-2d-heatmap-with-matplotlib
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

#here's our data to plot, all normal Python lists
x = [1, 2, 3, 4, 5]
y = [0.1, 0.2, 0.3, 0.4, 0.5]

intensity = [
    [3, 0, 20, 2, .6],
    [4, 33, 13, 2, 5],
    [.6, 1, 9, 10, 10],
    [0, 7, 0, 6, 4],
    [18, 15, 94, 0, 0]
]

#setup the 2D grid with Numpy
x, y = np.meshgrid(x, y)

#convert intensity (list of lists) to a numpy array for plotting
intensity = np.array(intensity)

#now just plug the data into pcolormesh, it's that easy!
plt.pcolormesh(x, y, intensity)
plt.colorbar() #need a colorbar to show the intensity scale
plt.show() #boom


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:

Checkup Exercise Set II

Exercise: Implement the following search procedure to find a good model

 You are given a list of possible values of `C` below

 For each C:

 Create a logistic regression model with that value of C

 Find the average score for this model using the `cv_score` function **only on the training set** `(Xlr, ylr)`

 Pick the C with the highest average score

Your goal is to find the best model parameters based *only* on the training set, without showing the model test set at all (which is why the test set is also called a *hold-out* set).

Can you write Python code for it?

#the grid of parameters to search over
Cs = [0.001, 0.1, 1, 10, 100]

# your turn

best_score = 0
best_C = 0

for C in Cs:
    model = LogisticRegression(C=C)
    score = cv_score(model, Xlr, ylr)
    print("C =", C, " Score =", score)

    if score > best_score:
        best_score = score
        best_C = C

print("\nThe best C is", best_C, "with a score of", best_score)


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
A Motivating Example Using sklearn: Heights and Weights
We'll use a dataset of heights and weights of males and females to hone our understanding of classifiers. We load the data into a dataframe and plot it.
Can you write Python code for it?

dflog = pd.read_csv("data/01_heights_weights_genders.csv")
dflog.head()


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Below is an example of using the cv_score function for a basic logistic regression model without regularization.
Can you write Python code for it?

clf = LogisticRegression()
score = cv_score(clf, Xlr, ylr)
print(score)


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Discriminative vs Generative Classifier
Logistic regression is what is known as a discriminative classifier as we learn a soft boundary between/among classes. Another paradigm is the generative classifier where we learn the distribution of each class. For more examples of generative classifiers, look here.
Let us plot the probabilities obtained from predict_proba, overlayed on the samples with their true labels:
Can you write Python code for it?

plt.figure()
ax = plt.gca()
points_plot_prob(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, psize=20, alpha=0.1);


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
In the figure here showing the results of the logistic regression, we plot the actual labels of both the training(circles) and test(squares) samples. The 0's (females) are plotted in red, the 1's (males) in blue. We also show the classification boundary, a line (to the resolution of a grid square). Every sample on the red background side of the line will be classified female, and every sample on the blue side, male. Notice that most of the samples are classified well, but there are misclassified people on both sides, as evidenced by leakage of dots or squares of one color ontothe side of the other color. Both test and traing accuracy are about 92%.
The Probabilistic Interpretaion
Remember we said earlier that if $h > 0.5$ we ought to identify the sample with $y=1$? One way of thinking about this is to identify $h(\v{w}\cdot\v{x})$ with the probability that the sample is a '1' ($y=1$). Then we have the intuitive notion that lets identify a sample as 1 if we find that the probabilty of being a '1' is $\ge 0.5$.
So suppose we say then that the probability of $y=1$ for a given $\v{x}$ is given by $h(\v{w}\cdot\v{x})$?
Then, the conditional probabilities of $y=1$ or $y=0$ given a particular sample's features $\v{x}$ are:
\begin{eqnarray}
P(y=1 | \v{x}) &=& h(\v{w}\cdot\v{x}) \
P(y=0 | \v{x}) &=& 1 - h(\v{w}\cdot\v{x}).
\end{eqnarray}
These two can be written together as
$$P(y|\v{x}, \v{w}) = h(\v{w}\cdot\v{x})^y \left(1 - h(\v{w}\cdot\v{x}) \right)^{(1-y)} $$
Then multiplying over the samples we get the probability of the training $y$ given $\v{w}$ and the $\v{x}$:
$$P(y|\v{x},\v{w}) = P({y_i} | {\v{x}i}, \v{w}) = \prod{y_i \in \cal{D}} P(y_i|\v{x_i}, \v{w}) = \prod_{y_i \in \cal{D}} h(\v{w}\cdot\v{x_i})^{y_i} \left(1 - h(\v{w}\cdot\v{x_i}) \right)^{(1-y_i)}$$
Why use probabilities? Earlier, we talked about how the regression function $f(x)$ never gives us the $y$ exactly, because of noise. This hold for classification too. Even with identical features, a different sample may be classified differently.
We said...
Can you write Python code for it?


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Remember that the form of data we will use always is

with the "response" or "label" $y$ as a plain array of 0s and 1s for binary classification. Sometimes we will also see -1 and +1 instead. There are also multiclass classifiers that can assign an observation to one of $K > 2$ classes and the labe may then be an integer, but we will not be discussing those here.
y = [1,1,0,0,0,1,0,1,0....].

Checkup Exercise Set I

 Exercise: Create a scatter plot of Weight vs. Height

 Exercise: Color the points differently by Gender

Can you write Python code for it?

# your turn

plt.scatter(dflog.Weight[dflog.Gender == "Male"], dflog.Height[dflog.Gender == "Male"], alpha=0.2, c="red")
plt.scatter(dflog.Weight[dflog.Gender == "Female"], dflog.Height[dflog.Gender == "Female"], alpha=0.2, c="blue")
plt.xlabel('Weight')
plt.ylabel('Height')
plt.title('Weight vs Height')
plt.show()


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Similar values were obtained using built-in tools.
A Walkthrough of the Math Behind Logistic Regression
Setting up Some Demo Code
Let's first set some code up for classification that we will need for further discussion on the math. We first set up a function cv_optimize which takes a classifier clf, a grid of hyperparameters (such as a complexity parameter or regularization parameter) implemented as a dictionary parameters, a training set (as a samples x features array) Xtrain, and a set of labels ytrain. The code takes the traning set, splits it into n_folds parts, sets up n_folds folds, and carries out a cross-validation by splitting the training set into a training and validation section for each foldfor us. It prints the best value of the parameters, and retuens the best classifier to us.
Can you write Python code for it?

def cv_optimize(clf, parameters, Xtrain, ytrain, n_folds=5):
    gs = sklearn.model_selection.GridSearchCV(clf, param_grid=parameters, cv=n_folds)
    gs.fit(Xtrain, ytrain)
    print("BEST PARAMS", gs.best_params_)
    best = gs.best_estimator_
    return best


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
So we then come up with our rule by identifying:
$$z = \v{w}\cdot\v{x}.$$
Then $h(\v{w}\cdot\v{x}) \ge 0.5$ if $\v{w}\cdot\v{x} \ge 0$ and $h(\v{w}\cdot\v{x}) \lt 0.5$ if $\v{w}\cdot\v{x} \lt 0$, and:
\begin{eqnarray}
y = 1 &if& h(\v{w}\cdot\v{x}) \ge 0.5\
y = 0 &if& h(\v{w}\cdot\v{x}) \lt 0.5.
\end{eqnarray}
We will show soon that this identification can be achieved by minimizing a loss in the ERM framework called the log loss :
$$ R_{\cal{D}}(\v{w}) = - \sum_{y_i \in \cal{D}} \left ( y_i \log(h(\v{w}\cdot\v{x})) + ( 1 - y_i) \log(1 - h(\v{w}\cdot\v{x})) \right )$$
We will also add a regularization term:
$$ R_{\cal{D}}(\v{w}) = - \sum_{y_i \in \cal{D}} \left ( y_i \log(h(\v{w}\cdot\v{x})) + ( 1 - y_i) \log(1 - h(\v{w}\cdot\v{x})) \right ) + \frac{1}{C} \v{w}\cdot\v{w},$$
where $C$ is the regularization strength (equivalent to $1/\alpha$ from the Ridge case), and smaller values of $C$ mean stronger regularization. As before, the regularization tries to prevent features from having terribly high weights, thus implementing a form of feature selection.
How did we come up with this loss? We'll come back to that, but let us see how logistic regression works out.
Can you write Python code for it?

dflog.head()

clf_l, Xtrain_l, ytrain_l, Xtest_l, ytest_l  = do_classify(LogisticRegression(),
                                                           {"C": [0.01, 0.1, 1, 10, 100]},
                                                           dflog, ['Weight', 'Height'], 'Gender','Male')

plt.figure()
ax=plt.gca()
points_plot(ax, Xtrain_l, Xtest_l, ytrain_l, ytest_l, clf_l, alpha=0.2);


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
This value of C scored high and performed better than the default model. Cross-validation and grid search is necessary in order to optimize your model. You improve your prediction results and also increase the probability that your model will perform as well with new, unseen data. You lower the probability that your model only did well by chance on a particular set of data.
Black Box Grid Search in sklearn
Scikit-learn, as with many other Python packages, provides utilities to perform common operations so you do not have to do it manually. It is important to understand the mechanics of each operation, but at a certain point, you will want to use the utility instead to save time...

Checkup Exercise Set IV

Exercise: Use scikit-learn's [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html) tool to perform cross validation and grid search.

* Instead of writing your own loops above to iterate over the model parameters, can you use GridSearchCV to find the best model over the training set?
* Does it give you the same best value of `C`?
* How does this model you've obtained perform on the test set?
Can you write Python code for it?

# your turn

from sklearn.model_selection import GridSearchCV

model2 = LogisticRegression()
params = {'C': Cs}
model_cv = GridSearchCV(model2, param_grid=params, cv=5, scoring="accuracy")
model_cv.fit(Xlr, ylr)
print("Best params =", model_cv.best_params_)
print("Best score =", model_cv.best_score_)

model2 = model_cv.best_estimator_
model2.fit(Xlr, ylr)
print("The score is", accuracy_score(model2.predict(Xtestlr), ytestlr))


I am working on the file "Heights and Weights Using Logistic Regression/Mini_Project_Logistic_Regression.ipynb".
The first task is:
Training and Test Datasets
When fitting models, we would like to ensure two things:

We have found the best model (in terms of model parameters).
The model is highly likely to generalize i.e. perform well on unseen data.

Purpose of splitting data into Training/testing sets

 We built our model with the requirement that the model fit the data well.

 As a side-effect, the model will fit THIS dataset well. What about new data?

 We wanted the model for predictions, right?

 One simple solution, leave out some data (for testing) and train the model on the rest

 This also leads directly to the idea of cross-validation, next section.

First, we try a basic Logistic Regression:

Split the data into a training and test (hold-out) set
Train on the training set, and test for accuracy on the testing set
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Split the data into a training and test set.
Xlr, Xtestlr, ylr, ytestlr = train_test_split(dflog[['Height','Weight']].values,
                                              (dflog.Gender == "Male").values,random_state=5)

clf = LogisticRegression()
# Fit the model on the trainng data.
clf.fit(Xlr, ylr)
# Print the accuracy from the testing data.
print(accuracy_score(clf.predict(Xtestlr), ytestlr))


I am working on the file "Hello,_Colaboratory.ipynb".
The first task is:
Forms
Forms can be used to parameterize code. See the forms example notebook for more details.
Can you write Python code for it?

#@title Examples

text = 'value' #@param
date_input = '2018-03-22' #@param {type:"date"}
number_slider = 0 #@param {type:"slider", min:-1, max:1, step:0.1}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]


I am working on the file "Hello,_Colaboratory.ipynb".
The first task is:
TensorFlow execution
Colaboratory allows you to execute TensorFlow code in your browser with a single click. The example below adds two matrices.
$\begin{bmatrix}
  1. & 1. & 1. \
  1. & 1. & 1. \
\end{bmatrix} +
\begin{bmatrix}
  1. & 2. & 3. \
  4. & 5. & 6. \
\end{bmatrix} =
\begin{bmatrix}
  2. & 3. & 4. \
  5. & 6. & 7. \
\end{bmatrix}$
Can you write Python code for it?

import tensorflow as tf

input1 = tf.ones((2, 3))
input2 = tf.reshape(tf.range(1, 7, dtype=tf.float32), (2, 3))
output = input1 + input2

with tf.Session():
  result = output.eval()
result


I am working on the file "Hello,_Colaboratory.ipynb".
The first task is:
Want to use a new library?  pip install it at the top of the notebook. Then that library can be used anywhere else in the notebook. For recipes to import commonly used libraries, refer to the importing libraries example notebook.
Can you write Python code for it?

!pip install -q matplotlib-venn

from matplotlib_venn import venn2
_ = venn2(subsets = (3, 2, 1))


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
1 * 0.8 의 출력 값은 예상대로 0.8을 리턴합니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
SummaryWriter를 사용하여 log_simple_graph 디렉토리에 sess에서 만들어진 세션 그래프 정보를 기록합니다.
Can you write Python code for it?

summary_writer = tf.train.SummaryWriter('log_simple_graph', sess.graph)


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
get_operations 로 받은 리스트에는 하나의 엘리먼트가 들어 있고 엘리먼트는 Operation 클래스의 인스턴스입니다.
Can you write Python code for it?

ops = graph.get_operations()
len(ops), ops[0].__class__


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
input_value는 상수 텐서를 위한 일종의 연산 노드이며 값이 들어 있지 않습니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
ops 의 첫번째 노드(여기서는 상수노드)의 정의를 조회하면 프로토콜 버퍼 형식으로 연산 노드를 표현하고 있음을 알 수 있습니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
계산된 그래디언트를 가중치에 반영하면 학습속도가 0.025 이므로 0.025 * 1.6 = 0.04 가 되어 w 가 0.04 만큼 감소됩니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
그래프의 노드를 다시 조회하면 mul 노드가 추가된 것을 확인할 수 있습니다.
Can you write Python code for it?

ops = graph.get_operations()
for op in ops:
    print(op.name)


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
두 수를 곱하는 연산을 만들어 보기 위해 weight 변수를 만듭니다. 상수 노드는 tensorflow.python.framework.ops.Tensor 의 객체인데 변수 노드는 tensorflow.python.ops.variables.Variable 의 객체입니다.
Can you write Python code for it?

weight = tf.Variable(0.8, name='weight')
weight


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
쉘에서 tensorboard --logdir=log_simple_graph 명령으로 텐서보드를 실행하고 브라우저로 http://localhost:6006 으로 접속한 후 그래프 탭을 클릭하면 아래와 같은 그림이 보입니다. 이 그래프는 init 연산에서 weight 변수를 초기화하고 mul 연산에서 input_value 와 wegiht 를 사용하고 있다는 것을 잘 보여 주고 있습니다.

같은 값을 가지는 상수와 변수를 x, w 로 다시 만듭니다. 그리고 곱셉 노드를 파이썬의 곱셉 연산자를 사용하지 않고 텐서플로우에서 제공하는 수학 함수인 mul을 사용하여 표현할 수 있습니다.
Can you write Python code for it?

x = tf.constant(1.0, name='x')
w = tf.Variable(0.8, name='w')
y = tf.mul(w, x, name='y')

init = tf.initialize_all_variables()
sess.run(init)


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
실수 1.0 값을 가지는 상수 input_value 를 만듭니다. name 옵션을 사용하여 이름을 지정하면 텐서보드의 그래프에서 구분하기 편리합니다.
Can you write Python code for it?

input_value = tf.constant(1.0, name='input_value')


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
실제 참 값(y_)을 0이라고 정의하고 예측값과의 차이의 제곱을 에러 함수(loss function or error function)로 정의합니다.
Can you write Python code for it?

y_ = tf.constant(0.0)
loss = (y - y_)**2


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
이 과정을 반복하도록 루프를 작성하고 summary_writer를 이용하여 결과 y 값을 기록하면 텐서보드에서 그래프로 값의 변화를 확인할 수 있습니다.
Can you write Python code for it?

train_step = tf.train.GradientDescentOptimizer(0.025).minimize(loss)
summary_y = tf.scalar_summary('output', y)
for i in range(100):
    summary_str = sess.run(summary_y)
    summary_writer.add_summary(summary_str, i)
    sess.run(train_step)


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
이제 연산 노드는 다섯 개로 늘어납니다. 변수 텐서를 생성하면 그래프에 초기값과 할당, 조회에 관련된 노드가 추가로 생성되기 때문입니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
텐서플로우의 디폴트 그래프는 직접 접근을 할 수 없고 get_default_graph 메소드를 이용합니다.
Can you write Python code for it?


I am working on the file "Hello-TensorFlow/Hello, TensorFlow!.ipynb".
The first task is:
학습속도를 0.025로 하고 그래디언트 디센트 최적화 방식을 선택합니다.
Can you write Python code for it?

optim = tf.train.GradientDescentOptimizer(learning_rate=0.025)
grads_and_vars = optim.compute_gradients(loss)

grads_and_vars


I am working on the file "HelloTF-Android/Hello World TensorFlow-Android.ipynb".
The first task is:
Cross check whether input and output nodes are present in graph def
Can you write Python code for it?

output_graph_def

with tf.gfile.GFile(output_optimized_graph_name, "wb") as f:
            f.write(output_graph_def.SerializeToString())


I am working on the file "HelloTF-Android/Hello World TensorFlow-Android.ipynb".
The first task is:
Step 3 : Make the model ready for inference
Freezing the model
Can you write Python code for it?

# Freeze the graph
MODEL_NAME = 'iris'
input_graph_path = 'model_files/savegraph.pbtxt'
checkpoint_path = 'model_files/model.ckpt'
input_saver_def_path = ""
input_binary = False
output_node_names = "output"
restore_op_name = "save/restore_all"
filename_tensor_name = "save/Const:0"
output_frozen_graph_name = 'model_files/frozen_model_'+MODEL_NAME+'.pb'
output_optimized_graph_name = 'model_files/optimized_inference_model_'+MODEL_NAME+'.pb'
clear_devices = True

freeze_graph.freeze_graph(input_graph_path, input_saver_def_path,
                          input_binary, checkpoint_path, output_node_names,
                          restore_op_name, filename_tensor_name,
                          output_frozen_graph_name, clear_devices, "")

output_graph_def = optimize_for_inference_lib.optimize_for_inference(
        sess.graph_def,
        ["Input"], # an array of the input node(s)
        ["output"], # an array of output nodes
        tf.float32.as_datatype_enum)


I am working on the file "HelloWorld.ipynb".
The first task is:
Now we submit the topo object that represents the application's topology to the ANALYTICS_SERVICE context, passing in the VCAP services information in the configuration.
Can you write Python code for it?

cfg = {}
cfg[ConfigParams.VCAP_SERVICES] = vs
cfg[ConfigParams.SERVICE_NAME] = service_name
submit("ANALYTICS_SERVICE", topo, cfg)
print("Done - Submitted job!")


I am working on the file "Hello_Colaboratory.ipynb".
The first task is:
Forms
Forms can be used to parameterize code. See the forms example notebook for more details.
Can you write Python code for it?

#@title Examples

text = 'value' #@param
date_input = '2018-03-22' #@param {type:"date"}
number_slider = 0.5 #@param {type:"slider", min:-1, max:1, step:0.1}
dropdown = '1st option' #@param ["1st option", "2nd option", "3rd option"]


I am working on the file "Hello_Colaboratory.ipynb".
The first task is:
TensorFlow execution
Colaboratory allows you to execute TensorFlow code in your browser with a single click. The example below adds two matrices.
$\begin{bmatrix}
  1. & 1. & 1. \
  1. & 1. & 1. \
\end{bmatrix} +
\begin{bmatrix}
  1. & 2. & 3. \
  4. & 5. & 6. \
\end{bmatrix} =
\begin{bmatrix}
  2. & 3. & 4. \
  5. & 6. & 7. \
\end{bmatrix}$
Can you write Python code for it?

import tensorflow as tf
import numpy as np

with tf.Session():
  input1 = tf.constant(1.0, shape=[2, 3])
  input2 = tf.constant(np.reshape(np.arange(1.0, 7.0, dtype=np.float32), (2, 3)))
  output = tf.add(input1, input2)
  result = output.eval()

result


I am working on the file "Hello_Colaboratory.ipynb".
The first task is:
Visualization
Colaboratory includes widely used libraries like matplotlib, simplifying visualization.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

x = np.arange(20)
y = [x_i + np.random.randn(1) for x_i in x]
a, b = np.polyfit(x, y, 1)
_ = plt.plot(x, y, 'o', np.arange(20), a*np.arange(20)+b, '-')


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
X = [Titan, Naga, Djinn, Mage, Golem, Gargoyle, Gremlin, Sold gems]
Can you write Python code for it?

loss_function = -np.hstack([units.health, [0]])
A = [(-units.health * units.distant).tolist() + [0],
     (-units.health * units.air).tolist() + [0],
     units.mercury.tolist() + [0],
     units.gems.tolist() + [1],
     units.gold.tolist() + [-gem_price]]
b = [-distant_min_health, -air_min_health, mercury, gems, gold]
bounds = [(0, available) for available in units.available] + [(0, gems)]

%%time
result = scipy.optimize.linprog(loss_function, A, b, bounds=bounds)


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Дальнобойные, воздушные
Can you write Python code for it?

np.dot(optimal_army, units.health * units.distant), np.dot(optimal_army, units.health * units.air)


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Для общего случая ILP дальше не всегда можно просто докрутить до целых чисел. А это прекрасный повод забыть про родной L-BFGS и окунуться в мир целочисленных линейных задач.
Integer Linear Problem
Вообще, известно как с помощью дополняющих условий (Cutting-plane method) сводить задачу к целочисленной, но делать это на scipy не стоит.
Я начал копать с этой страницы: http://prod.sandia.gov/techlib/access-control.cgi/2013/138847.pdf
И остановился сразу как только нашел слово GNU. Интересно, что основатель проекта из МАИ. Список python биндингов: https://en.wikibooks.org/wiki/GLPK/Python
Can you write Python code for it?

import pulp

problem = pulp.LpProblem("Heroes III", pulp.LpMaximize)

titan = pulp.LpVariable('titan', lowBound=0, upBound=units.loc['titan'].available, cat='Integer')
naga = pulp.LpVariable('naga', lowBound=0, upBound=units.loc['naga'].available, cat='Integer')
djinn = pulp.LpVariable('djinn', lowBound=0, upBound=units.loc['djinn'].available, cat='Integer')
mage = pulp.LpVariable('mage', lowBound=0, upBound=units.loc['mage'].available, cat='Integer')
golem = pulp.LpVariable('golem', lowBound=0, upBound=units.loc['golem'].available, cat='Integer')
gargoyle = pulp.LpVariable('gargoyle', lowBound=0, upBound=units.loc['gargoyle'].available, cat='Integer')
gremlin = pulp.LpVariable('gremlin', lowBound=0, upBound=units.loc['gremlin'].available, cat='Integer')
sold_gems = pulp.LpVariable('sold gems', lowBound=0, upBound=gems, cat='Integer')
army = [titan, naga, djinn, mage, golem, gargoyle, gremlin]

# gain function
problem += np.dot(army, units.health.values)

# restrictions
problem += np.dot(army, units.health * units.distant) >= distant_min_health
problem += np.dot(army, units.health * units.air) >= air_min_health
problem += np.dot(army, units.mercury) <= mercury
problem += np.dot(army + [sold_gems], units.gems.tolist() + [1]) <= gems
problem += np.dot(army + [sold_gems], units.gold.tolist() + [-gem_price]) <= gold

%%time
pulp.LpStatus[problem.solve()]

solution = pd.DataFrame([{'value': parameter.value()} for parameter in problem.variables()],
                        index=[parameter.name
                               for parameter in problem.variables()])
solution.loc[['sold_gems'] + units.name.tolist()]


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Затраченные ресурсы
Can you write Python code for it?

np.dot(optimal_army, units.mercury), \
    np.dot(optimal_army + [sold_gems.value()], units.gems.tolist() + [1]), \
    np.dot(optimal_army + [sold_gems.value()], units.gold.tolist() + [-gem_price]), \


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Последний элемент ответа 0 - камни продавать не нужно. Сила набранной армии: 12875.
Сила существ дальнего боя / воздушной армии:
Can you write Python code for it?

-np.dot(result.x, np.array(A[0])), -np.dot(result.x, A[1])


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Решение найдено почти мгновенно.
Can you write Python code for it?


I am working on the file "Heroes of Might and Magic III.ipynb".
The first task is:
Теперь пришлось продавать камни.
Can you write Python code for it?

optimal_army = [unit.value() for unit in army]


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
Breush-Pagan Test
BP統計量を計算するにあたって、statsmodels.stats.diagnostic.het_breushpagan()を使う。
```

statsmodels.stats.diagnostic.het_breushpagan(resid, exog_het)
Parameters:
    resid : arraylike, (nobs,)
        For the Breush-Pagan test, this should be the residual of a regression.
    exog_het : array_like, (nobs, nvars)
        This contains variables that might create data dependent heteroscedasticity.
Returns:
    lm : float
        lagrange multiplier statistic
    lm_pvalue :float :
        p-value of lagrange multiplier test
    fvalue : float
        f-statistic of the hypothesis that the error variance does not depend on x
    f_pvalue : float
        p-value for the f-statistic

```
Can you write Python code for it?


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
P値は0.003より有意水準1%でも帰無仮説BP=0を棄却することができ、不均一分散であると結論することができます。
例題6-3
「k0602.csv」を用いた不均一分散のデータである場合の回帰分析。
不均一分散が存在する場合、変数を対数化することによって不均一分散の状態の解消を行います。
先ほど読み込んだ「k0602.csv」のデータから変数の対数化を行います。
dataの中に新しいコラム「lnX」「lnY」を追加します。
Can you write Python code for it?

data["lnX"] = np.log(data["X"])
data["lnY"] = np.log(data["Y"])
data

# 説明変数設定
X = data['lnX']
X = sm.add_constant(X)
X

# 被説明変数設定
Y = data['lnY']
Y

# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())
# グラフ生成
plt.plot(data["lnX"], data["lnY"], 'o', label="data")
plt.plot(data["lnX"], results.fittedvalues, label="OLS")
plt.xlim(min(data["lnX"])-0.5, max(data["lnX"])+0.5)
plt.ylim(min(data["lnY"])-0.5, max(data["lnY"])+0.5)
plt.title('Example6-3')
plt.legend(loc=2)
plt.show()

# BPテスト
BP = smsdia.het_breushpagan(results.resid, results.model.exog)
print('Breusch-Pagan test : ')
print('BP = ', BP[0])
print('p-value = ', BP[1])


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
これの1つ目がBP統計量で、2つ目がP値。
見やすいように整えておく。
Can you write Python code for it?

BP = smsdia.het_breushpagan(results.resid, results.model.exog)
print('Breusch-Pagan test : ')
print('BP = ', BP[0])
print('p-value = ', BP[1])


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
やっぱりX, Yの値が変わってる笑
とりあえず「k0603.csv」の場合の回帰も行います。
Can you write Python code for it?

X = data6_3['lnX']
X = sm.add_constant(X)
Y = data6_3['lnY']
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())

plt.plot(data6_3["lnX"], data6_3["lnY"], 'o', label="data")
plt.plot(data6_3["lnX"], results.fittedvalues, label="OLS")
plt.xlim(min(data6_3["lnX"])-0.5, max(data6_3["lnX"])+0.5)
plt.ylim(min(data6_3["lnY"])-0.5, max(data6_3["lnY"])+0.5)
plt.title('Example6-3')
plt.legend(loc=2)
plt.show()

BP = smsdia.het_breushpagan(results.resid, results.model.exog)
print('Breusch-Pagan test : ')
print('BP = ', BP[0])
print('p-value = ', BP[1])


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
よって、 BP=0.14630.1463となり0に近い値であることがわかる。
P値は0.702となり有意水準10%でも帰無仮説BP=0を棄却することはできず、均一分散であると結論することができます。
例題6-2
「k0602.csv」を用いた不均一分散のデータである場合の回帰分析。
BP統計量による不均一分散の有無の仮説検定を行います。
データを読み込みます。
Can you write Python code for it?

data = pd.read_csv('example/k0602.csv')

# 説明変数設定
X = data['X']
X = sm.add_constant(X)
X

# 被説明変数設定
Y = data['Y']
Y

# OLSの実行(Ordinary Least Squares: 最小二乗法)
model = sm.OLS(Y,X)
results = model.fit()
print(results.summary())

# グラフ生成
plt.plot(data["X"], data["Y"], 'o', label="data")
plt.plot(data["X"], results.fittedvalues, label="OLS")
plt.xlim(min(data["X"])-1, max(data["X"])+1)
plt.ylim(min(data["Y"])-1, max(data["Y"])+1)
plt.title('Example6-2')
plt.legend(loc=2)
plt.show()

BP = smsdia.het_breushpagan(results.resid, results.model.exog)
print('Breusch-Pagan test : ')
print('BP = ', BP[0])
print('p-value = ', BP[1])


I am working on the file "Heteroscedasticity.ipynb".
The first task is:
例題6-4
「k0602.csv」を用いた不均一分散のデータである場合の回帰分析。
不均一分散が存在する場合、第3の変数を基準とした比率に変換することによって不均一分散の状態の解消を行います。
例題6-4で使う「k0604.csv」も「k0602.csv」とデータが異なるので、「k0602.csv」を基にしたデータの加工は省き、直接行います。
Can you write Python code for it?


I am working on the file "HeuristicLVSegmentation.ipynb".
The first task is:
Given the significant overlap of the two pixel populations, I would speculate that a simple thresholding scheme would not work. To further demonstrate that, we can plot the ROC curve of the simple thresholding scheme. Note that I treat blood pool as positiv and heart muscle as negative.
Can you write Python code for it?

TPR = [1.0]
FPR = [1.0]
blood_pixels.sort()
muscle_pixels.sort()
cond_pos = len(blood_pixels)
cond_neg = len(muscle_pixels)
false_pos = cond_neg
false_neg = 0
for v in muscle_pixels:
    false_pos -= 1
    while false_neg < cond_pos and blood_pixels[false_neg] <= v:
        false_neg += 1
    TPR.append(1.0 - false_neg/cond_pos)
    FPR.append(false_pos/cond_neg)
fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 9))
ax1.set_xlabel('False Positive Rate')
ax1.set_ylabel('True Positive Rate')
ax1.plot(FPR, TPR)
ax2.set_xlabel('False Positive Rate')
ax2.set_ylabel('TPR - FPR')
ax2.plot(FPR, [t-f for f, t in zip(FPR, TPR)])


I am working on the file "HeuristicLVSegmentation.ipynb".
The first task is:
Simple Thresholding
To be able to segment a region reliably using simple thresholding, the pixel values of the foreground and background should have a significant statistical difference. To check that, I plot the histogram of pixel values within the blood pool and within the muscle.
Can you write Python code for it?

# Walk through all filtered records and collect pixel values of both regions
blood_pixels = []
muscle_pixels = []
for r in records:
    img, ic, oc = r.data.dicom, r.data.ic_mask, r.data.oc_mask
    blood_mask = ic
    # muscle is within oc but not in ic
    muscle_mask = np.logical_and(oc, np.logical_not(ic))
    blood_pixels.extend(list(img[blood_mask]))
    muscle_pixels.extend(list(img[muscle_mask]))
fig, ax = plt.subplots(1, 1, figsize=(12, 9))
ax.hist(np.array(blood_pixels), bins=256, alpha=0.5, fc='g')
ax.hist(np.array(muscle_pixels), bins=256, alpha=0.5, fc='r')
ax.set_xlabel('Pixel Value')
ax.set_ylabel('Counts')


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
From the above expressions we can infer the topological phase diagram of the system. While keeping a finite value of $\Delta$ tuning $\mu^2+\Delta^2-B^2$ in the interval $[0,4\gamma^2]$ the system has a Winding number of one and hence is topological, otherwhise it is trivial.
Let us examine the winding and the spectrum as we tune the parameters of the system:
Can you write Python code for it?

figsize(12,6)
@interact(mu=(-3,3,0.1),B=(0,2,0.1),gamma=fixed(1),Delta=(0,2,0.1))
def spectr_wire(mu=0,B=0,gamma=1,Delta=0):
    # this part produces the spectrum
    subplot(121)
    k=linspace(0,2*pi,100)
    I=1j
    # evaluating the BdG spectra
    plot(k,real(eval(str(bdgspectr[0]))),
         'o',lw=3,label='BdG',mec='green',mfc='green',alpha=0.5,mew=0)
    for i in [1,2,3]:
        plot(k,real(eval(str(bdgspectr[i]))),
             'o',lw=3,mec='green',mfc='green',alpha=0.5,mew=0)
    # evaluating the particle and the hole spectra without superconductivity
    plot(k,eval(str(wspect[0])),'r-',lw=3,label='particle')
    plot(k,eval(str(wspect[1])),'r-',lw=3)
    plot(k,-eval(str(wspect[0])),'b--',lw=3,label='hole')
    plot(k,-eval(str(wspect[1])),'b--',lw=3)

    plot(k,0*k,'k-',lw=4)
    grid()
    xlim(0,2*pi)
    ylim(-3,3)
    xlabel(r'$k$',fontsize=20)
    ylabel(r'$E$',fontsize=20)
    legend(fontsize=20)
    # this part produces the winding plot
    subplot(122)
    plot(0,0,'ko',ms=8)
    plot(eval(str(sympy.re(detblock))),
         eval(str(sympy.im(detblock))),lw=3)
    xlabel(r'$\mathrm{Re}(\mathrm{det}(h))$',fontsize=20)
    ylabel(r'$\mathrm{Im}(\mathrm{det}(h))$',fontsize=20)

    grid()
    xlim(-5,5)
    ylim(-5,5)


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
Looking at the imaginary and real part of this quantity we recognize that it describes an ellipse:
Can you write Python code for it?


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
The Winding number of the determinant of the nonzero subblock will serve as a topological invariant for this model:
Can you write Python code for it?

detblock=sympy.simplify((
    ((( P*( Hk  )*P.inv() )[:2,2:]).det())
).rewrite(sin))


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
The eigenvectors of this matrix give the unitary transformation necessary to block off-diagonalize the $\mathcal{H}$ matrix
Can you write Python code for it?


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
This is the Chiral symmetry operator
Can you write Python code for it?


I am working on the file "HgTe_edge_proximity.ipynb".
The first task is:
We can diagonalize this yealding the eigenvalues of the system:
Can you write Python code for it?

# this is where we will keep the eigenvalues of the BdG matrix
bdgspectr=list(Hk.eigenvals())
# in theis list we keep the eigenvalues of the particle block
wspect=list(
    ((sympy.exp(sympy.I*k).rewrite(sin)*T+
      sympy.exp(-sympy.I*k).rewrite(sin)*T.transpose()+
      U)[:2,:2]).eigenvals()
)
wspect


I am working on the file "Hidden Markov Models.ipynb".
The first task is:
A tutorial on hidden Markov models and selected applications in speech recognition.

Rabiner, Lawrence R. "A tutorial on hidden Markov models and selected applications in speech recognition." Proceedings of the IEEE 77.2 (1989): 257-286.

I. INTRODUCTION
markov, hidden markovについて
流行の原因は？

数学的に柔軟なモデルであり，様々な応用分野で理論的な基礎が形成されている
実際にとても良く機能する

1960sからHMMとHMM音声認識は存在していたが，広く理解されるようになったのはここ最近(pub: 1989)のこと．

HMMの基礎理論はエンジニアにあまり読まれていなかった
初期に出た論文はチュートリアルがなかった

II. DISCRETE MARKOV MODEL
どの時点でも確率変数が状態集合のどこかにいるモデル．
$ N $ を状態の数， $ t $ を時刻とする．
そして $ q_t $ を時刻 $ t $ のときの状態とする．
\begin{align}
& S &=& { S_1, S_2, \dots, S_N } \
& q_t \in S
\end{align}
すると，本来であれば時刻 $ t $ のときに状態 $ q_t $ である確率は，それまでの時刻の情報を用いて
\begin{align}
P(q_t = S_j| q_{t-1} = S_i, q_{t-2} = S_k, \dots)
\end{align}
と表すことができる．
（単純？）マルコフ連鎖とは， $ q_t $ の状態は $ q_{t-1} $ ，すなわち1時刻前の状態にのみ依存して決定されるという仮定をおいたモデルであり，先ほどの確率を以下のように近似する．
\begin{align}
P(q_t = S_j| q_{t-1} = S_i, q_{t-2} = S_k, \dots) \approx P(q_t = S_j| q_{t-1} = S_i)
\end{align}
確率 $ P(q_t = S_j| q_{t-1} = S_i) $ を状態遷移確率と呼ぶ．
これは状態数 $ n $ を次元数とした正方行列を使って表すことができる．
状態遷移確率を表す行列を $ A $ として，以下のように定義する（パラメータは論文に載ってる通り）．
Can you write Python code for it?


I am working on the file "Hidden Markov Models.ipynb".
The first task is:
また，無限級数の公式を使って厳密な値を求めると以下．
\begin{align}
\sum_{d=1}^{\infty} d a_{ii}^{d-1} (1-a_{ii}) =　\frac{1}{1-a_{ii}}
\end{align}
結果が一致した．
Can you write Python code for it?

for i in range(len(A)):
    d_bar = 1 / (1 - A[i, i])
    print('{weather}: {d_bar}'.format(weather=S_label[i], d_bar=d_bar))


I am working on the file "Hidden Markov Models.ipynb".
The first task is:
同じ天気が続くような系列を考える．
ある天気について，それが連続するときの連続日数に関する期待値は，
\begin{align}
\bar{d} &=& \sum_{d=1}^{\infty} d p_i(d) \
          &=& \sum_{d=1}^{\infty} d a_{ii}^{d-1} (1-a_{ii})
\end{align}
で与えることが出来て，近似的に計算すると結果は以下の通り．
Can you write Python code for it?

for i in range(len(A)):
    d_bar = sum(d * pow(A[i, i], d-1) * (1 - A[i, i])  for d in range(1, 100000))
    print('{weather}: {d_bar}'.format(weather=S_label[i], d_bar=d_bar))


I am working on the file "Hidden Markov Models.ipynb".
The first task is:
確率は以下のようになる．
reduceは総積を計算している．
Can you write Python code for it?

probability = pi[O[0]] * reduce(lambda x, y: x * y, [A[O[i], O[i+1]] for i in range(len(O)-1)])
print(probability)


I am working on the file "HiddenMarkovModel_PoSTaggingFromScratch.ipynb".
The first task is:
Build the vanilla Viterbi based POS tagger
Can you write Python code for it?

#First We need to build the Transition Probability (Tag to Tag) and Emission Probability (Tag to Vocab Word) Matrix


I am working on the file "HiddenMarkovModel_PoSTaggingFromScratch.ipynb".
The first task is:
Creating the HMM Model with Viterbi Heuristics
Can you write Python code for it?

#Tagging Using Hiden Markov Model and Viterbi Heuristic
def HMMTaggerWithViterbi(wordList, train_bag = WordTagPairList):
    PredictedTagList = []

    TagsList = list(set([pair[1] for pair in train_bag]))

    for key, word in enumerate(wordList):
        #initialise list of probability column for a given observation
        CandidateProbabilityList = []
        for tag in TagsList:
            if key == 0:
                #Probability of Tag Just after Start State
                TransitionProbability = TransitionProbabilityMatrixDF.loc['.', tag]
            else:
                #Probability of Tag after Previous Tag
                TransitionProbability = TransitionProbabilityMatrixDF.loc[PredictedTagList[-1], tag]

            #Emission Probability
            EmissionCount, TotalCount = getEmissionProbability(wordList[key], tag)
            EmissionProbability = EmissionCount/TotalCount

            #State Probability
            StateProbability = EmissionProbability * TransitionProbability
            CandidateProbabilityList.append(StateProbability)

        maximumStateProbability = max(CandidateProbabilityList)
        # getting state for which probability is maximum
        PredictedTag = TagsList[CandidateProbabilityList.index(maximumStateProbability)]
        PredictedTagList.append(PredictedTag)
    return list(zip(wordList, PredictedTagList))


I am working on the file "HiddenMarkovModel_PoSTaggingFromScratch.ipynb".
The first task is:
Exploring and Processing Data
Can you write Python code for it?

#check some details
print(nltk_data[:5])

# we will need to split here as train and test needs to happen on entire sentences
# Splitting into train and test
random.seed(100)
train_set, validation_set = train_test_split(nltk_data,test_size=0.05)

print("Total Training Sentences  : " + str(len(train_set)))
print("Total Validations Sentences : " + str(len(validation_set)))
print(train_set[:5])

# Create List of Word and Tokens
WordTagPairList = [pair for sent in train_set for pair in sent]
print(WordTagPairList[:5])

# Finding Total Data, Unique words and Tags
print("Total Word Tag Pairs : " + str(len(WordTagPairList)))

wordTokens = [pair[0] for pair in WordTagPairList]
tagTokens = [pair[1] for pair in WordTagPairList]

#Unique word tokens is the vocabulary
Vocab = set(wordTokens)
#Unique Tags are the tag set
TagSet = set(tagTokens)

print("Vocab List : " +str(Vocab))
print("Unique Vocab Size : " + str(len(Vocab)))
print ("Tags List : " + str(TagSet))
print("Unique Tags Size : " + str(len(TagSet)))


I am working on the file "HiddenMarkovModel_PoSTaggingFromScratch.ipynb".
The first task is:
Solving "The learning problem" Stage
Can you write Python code for it?

# compute probability next tag based on previous tag

def getTransitionProbability(nextTag, baseTag, train_bag = WordTagPairList):
    tags = [pair[1] for pair in train_bag]
    TotalCountOfBaseTag = len([t for t in tags if t==baseTag])
    TotalCountNextTagFollowingBaseTag = 0
    for index in range(len(tags)-1):
        if tags[index]==baseTag and tags[index+1] == nextTag:
            TotalCountNextTagFollowingBaseTag += 1
    return (TotalCountNextTagFollowingBaseTag, TotalCountOfBaseTag)

def getProbability(Numerator, Denominator):
    return Numerator/Denominator

#Test

print(getTransitionProbability(nextTag='VERB', baseTag='NOUN'))
count1 , count2 = getTransitionProbability(nextTag='VERB', baseTag='NOUN')
print(getProbability(count1, count2))
print(getTransitionProbability(nextTag='VERB', baseTag='ADV'))
print(getTransitionProbability(nextTag='VERB', baseTag='.'))
print(getTransitionProbability(nextTag='DET', baseTag='.'))

#create the Transition Probability Matrix
TransitionProbabilityMatrix = np.zeros((len(TagSet), len(TagSet)), dtype='float32')
for i, baseTag in enumerate(list(TagSet)):
    for j, nextTag in enumerate(list(TagSet)):
        nextTagOnBaseTagCount, baseTagCount = getTransitionProbability(nextTag, baseTag)
        TransitionProbabilityMatrix[i, j] = nextTagOnBaseTagCount/baseTagCount

# checking values of Transition Probabilities
print(TransitionProbabilityMatrix)
print("Dimensions of Transition Matrix are : " +str(len(TransitionProbabilityMatrix)) + "," + str(len(TransitionProbabilityMatrix[0])) )

TransitionProbabilityMatrixDF = pd.DataFrame(TransitionProbabilityMatrix, columns = list(TagSet), index=list(TagSet))
TransitionProbabilityMatrixDF.head(5)

#checking the heatmap for how good the dependencies are.
plt.figure(figsize=(12, 12))
sns.heatmap(TransitionProbabilityMatrixDF)
plt.show()

#Creating Vocab Emission Probability based on Tag State (TagSize x Vocab)matrix
tagCount = len(TagSet)
vocabCount = len(Vocab)
EmissionProbabilityMatrix = np.zeros((tagCount,...


I am working on the file "HiddenMarkovModel_PoSTaggingFromScratch.ipynb".
The first task is:
Validation Set Evaluation
Can you write Python code for it?

random.seed(100)

# choose random 5 sents
rndom = [random.randint(1,len(validation_set)) for x in range(5)]

# list of validation sentences
validationSentences = [validation_set[i] for i in rndom]

#list of validation Tuples created from Validation Sentences
validationTuples = [tup for sent in validationSentences for tup in sent]
print(validationTuples)

#list of only Validation words, stripped off the tags. These will be predicted by HMMTagger and later compared for accuracy
ValidationWords = [tup[0] for sent in validationSentences for tup in sent]
print(ValidationWords)

def getHMMAccuracy(testSetList, OriginalWordTagList):
    start = time.time()
    TaggedWordSequence = HMMTaggerWithViterbi(testSetList)
    end = time.time()
    difference = end-start
    print("Time taken in seconds: ", difference)
    print(TaggedWordSequence)
    print(OriginalWordTagList)

    correctlyTaggedWordCount = [i for i, j in zip(TaggedWordSequence, OriginalWordTagList) if i == j]
    accuracy = len(correctlyTaggedWordCount)/len(TaggedWordSequence)
    print (accuracy)

getHMMAccuracy(ValidationWords, validationTuples )


I am working on the file "Hierarchical+Clustering (1).ipynb".
The first task is:
Samples clustering using scipy
First, we'll implement the clustering using scipy modules
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import pdist, squareform
from scipy.cluster.hierarchy import linkage, dendrogram

data_dist = pdist(data_array[1:10]) # computing the distance
data_link = linkage(data_dist) # computing the linkage

dendrogram(data_link,labels=data.dtype.names)
plt.xlabel('Samples')
plt.ylabel('Distance')
plt.suptitle('Samples clustering', fontweight='bold', fontsize=14);

plt.show()

# Compute and plot first dendrogram.
fig = plt.figure(figsize=(8,8))
# x ywidth height
ax1 = fig.add_axes([0.05,0.1,0.2,0.6])
Y = linkage(data_dist, method='single')
Z1 = dendrogram(Y, orientation='right',labels=data.dtype.names) # adding/removing the axes
ax1.set_xticks([])

# Compute and plot second dendrogram.
ax2 = fig.add_axes([0.3,0.71,0.6,0.2])
Z2 = dendrogram(Y)
ax2.set_xticks([])
ax2.set_yticks([])

#Compute and plot the heatmap
axmatrix = fig.add_axes([0.3,0.1,0.6,0.6])
idx1 = Z1['leaves']
idx2 = Z2['leaves']
D = squareform(data_dist)
D = D[idx1,:]
D = D[:,idx2]
im = axmatrix.matshow(D, aspect='auto', origin='lower', cmap=plt.cm.YlGnBu)
axmatrix.set_xticks([])
axmatrix.set_yticks([])

# Plot colorbar.
axcolor = fig.add_axes([0.91,0.1,0.02,0.6])
plt.colorbar(im, cax=axcolor)

plt.show()


I am working on the file "Hierarchical+Clustering (1).ipynb".
The first task is:
note numpy also has recfromcsv() and pandas can read_csv, with pandas DF.values giving a numpy array
Can you write Python code for it?

len(data)

len(data.dtype.names)

data.dtype.names

type(data)

data

data_array = data.view((np.float, len(data.dtype.names)))

data_array

data_array = data_array.transpose()

print(data_array)

data_array[1:10]


I am working on the file "Hierarchical+Clustering (1).ipynb".
The first task is:
the fastcluster module http://math.stanford.edu/~muellner/fastcluster.html?section=0
Can you write Python code for it?

! pip install fastcluster

from fastcluster import *
%timeit data_link = linkage(data_array[1:10], method='single', metric='euclidean', preserve_input=True)
dendrogram(data_link,labels=data.dtype.names)
plt.xlabel('Samples')
plt.ylabel('Distance')
plt.suptitle('Samples clustering', fontweight='bold', fontsize=14);
plt.show()


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Analysis

Period I: Before the announcement on 2nd July, there were some rumors about the discovery of a Higgs-like boson at Tevatron;

Period II: On 2nd July at 1 PM GMT, scientists from CDF and D0 experiments, based at Tevatron, presented results indicating that the Higgs particle should have a mass between 115 and 135 GeV/c2 (corresponding to about 123-144 times the mass of the proton) [7];

Period III: After 2nd July and before 4th of July there were many rumors about the Higgs boson dis- covery at LHC [8];

Period IV: The main event was the announce- ment on 4th July at 8 AM GMT by the scientists from the ATLAS and CMS experiments, based at CERN, presenting results indicating the existence of a new particle, compatible with the Higgs bo- son, with mass around 125 GeV/c2 [9, 10]. After 4th July, popular media covered the event.

Extract causal multilayer graph activated components
The graph is already given by the activity. Thus we don't use our framework is this special case.
Can you write Python code for it?

ACTIVITY = parse_activity(os.path.join(DATA_DIR, 'HiggsDiscovery_multiplex_time.txt'), reverse=True)
# COMPS = create_activated_components(ACTIVITY, 10, 'min', 'RT', SOCIAL_RE)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Export component from plotting using Gephi
Can you write Python code for it?

path = os.path.join(DATA_DIR, g.name)
# nx.write_edgelist(g, path + '.csv', data=False)
nx.write_graphml(g, path + '.graphml')


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Extract the biggest component subgraph for further analysis
Can you write Python code for it?

g = sptgraph.component_to_networkx(big_comp, H, layer_to_ts)
print nx.info(g)

g_un = g.to_undirected()
partition = community.best_partition(g_un)

print 'Number of communities:', len(set(partition.values()))

print 'Modularity:', community.modularity(partition, g_un)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Filter components and extract global statistics
Can you write Python code for it?

FILT_COMPS = COMPS[COMPS['node_count'] > 20]  # filter too small components
C = FILT_COMPS.set_index('start', drop=False).sort_index()
# Resample do get statistics (averages the values)
C.resample('12H')

fig = plot_spatial_spread(FILT_COMPS.iloc[0]['component'])
show(fig)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
First scale 10 min RT
Using only tgt_id as activation
Can you write Python code for it?

signal_10min_rt_tgt = create_signal(ACTIVITY, 10, 'min', 'RT', subset='tgt_id')
gl_comps_10min_rt_tgt, H, max_id = extract_components_gl(signal_10min_rt_tgt, SOCIAL_GL_RE)
layer_to_ts = dict(signal_10min_rt_tgt[['layer', 'timestamp']].values)

gl_comps_10min_rt_tgt[['component_id', 'node_count', 'width', 'height', 'start', 'end']]


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Helper functions
Can you write Python code for it?

def round_df(df, freq, unit):
    mul = 1
    if unit == 'min':
        mul = 60
        unit = 'Min'

    elif unit == 'hour':
        mul = 60 * 60
        unit = 'H'

    elif unit == 'day':
        mul = 60 * 60 * 24
        unit = 'D'
    else:
        mul = 1
        unit = 'S'

    ns_min = freq * mul * 1000000000
    idx = pd.DatetimeIndex(((df.index.astype(np.int64) // ns_min) * ns_min))
    idx.name = df.index.name
    df.index = idx
    return df, unit

def create_layers(df, freq, unit):
    # Trick: use pandas resample to generate continuous indexes
    dfR = df.resample(str(freq) + unit)
    layer_map = dict(itertools.izip(dfR.index.astype(np.int64), itertools.count()))
    df['layer'] = np.vectorize(lambda x: layer_map[x])(df.index.astype(np.int64))
    # delta between layers (timedelta)
    delta = dfR.index[1] - dfR.index[0]
    return df, delta

def create_layered_graph(input_df, freq=1, unit='min', action='RT', social_graph=None):

    h = nx.DiGraph()
    h.name = 'Higgs layered ' + action + ' - ' + str(freq) + ' ' + unit

    df = input_df.copy()
    if action != '' and action is not None:
        df = df[df['action'] == action]

    df, unit = round_df(df, freq, unit)
    df, delta_ts = create_layers(df, freq, unit)
    # Maximum id in whole dataframe, min should not be 0
    max_id = np.max(np.max(df[['src_id', 'tgt_id']]).values)

    for idx, row in df.iterrows():
        base_src = row['src_id']
        base_tgt = row['tgt_id']
        layer = row['layer']
        act = row['action']

        src = base_src + (layer * max_id)
        tgt = base_tgt + ((layer + 1) * max_id)

        # Add nodes
        h.add_node(src, {'base_id': base_src, 'layer': layer, 'timestamp': idx})
        h.add_node(tgt, {'base_id': base_tgt, 'layer': layer + 1, 'timestamp': idx + delta_ts})

        # Add edge
        e_d = {'action': act, 'timestamp': idx}
        if social_graph is not None:
            if social_graph.has_edge(base_src,...


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Overlap between retweet and social network
Can you write Python code for it?

def overlap_graph(g1, g2):
    common_edges = 0
    for u, v in g1.edges_iter():
        if g2.has_edge(u, v):
            common_edges += 1

    res = common_edges * 100 / float(nx.number_of_edges(g1))
    print 'Percentage of overlap (', g1.name, ',', g2.name, '):', res
    print 'Number of common edges:', common_edges
    return res, common_edges

# overlap_graph(REPLY, SOCIAL)
overlap_graph(RETWEET, SOCIAL)
# overlap_graph(MENTION, SOCIAL)


I am working on the file "Higgs twitter analysis.ipynb".
The first task is:
Parse data
Parse retweet graph
Can you write Python code for it?

start = time.time()
# path = os.path.join(DATA_DIR, 'HiggsDiscovery_RT.edges.gz')
# RETWEET = nx.read_edgelist(path, create_using=nx.DiGraph(),
#                            nodetype=int, data=(('weight', int),))
# RETWEET.name = 'Higgs RT'
# nx.write_gpickle(RETWEET, os.path.join(DATA_DIR, 'retweet.gpickle'))
RETWEET = nx.read_gpickle(os.path.join(DATA_DIR, 'retweet.gpickle'))
print 'Retweet graph loaded in:', time.time() - start
print nx.info(RETWEET)