I am working on the file "2016-2017.Meetings/fall/05.DIY_naive_bayes/naive_bayes_classifier_solution/naive_bayes_tweet_classifier.ipynb".
The first task is:
Let's take a moment to verify that tokenize does what we want it to.
Can you write Python code for it?

sample_tweet = "I am growing the Republican Party tremendously - just look at the numbers, way up! Democrats numbers are significantly down from years past."
sample_tokens = tokenize(sample_tweet)
sample_tokens

# open the file with train data and refer to it as "data", and
# open the file with train labels and refer to it as "labels"
with open(TRAIN_DATA_PATH, 'r') as data, open(TRAIN_LABELS_PATH, 'r') as labels:
    # zip data and labels together to create a list of tuples of (tweet,label),
    # which we then iterate over. the tweet variable refers to an individual tweet
    # and the label variable refers to an individual label
    for tweet, label in zip(data, labels):
        # if it is a Hillary tweet, increment the "hillary" element of `total_tweet_count`
        # if it is a Trump tweet, increment the "trump" element of `total_tweet_count`.
        total_tweet_count[LABEL_MAP[label]] += 1.0

        tokens = tokenize(tweet)

        # iterate over the words we found in the tweet
        for word in tokens:
            # if we haven't seen a word yet, let's add it to our dictionary with a
            # count of 2 * how much we smoothing up by (`SMOOTHING`),
            # (since we presume that both hillary and trump have seen it)

            # we essentially pretend that there is a tweet in the hillary
            # and trump train sets that contain ALL of the words in the vocabulary
            # and a tweet that contains NONE of the words in the vocabulary.
            if word not in vocabulary:
                vocabulary[word] = 2 * SMOOTHING
                # similarily, since we pretend that we saw an extra tweet
                # with this unknown word and one without, we add SMOOTHING
                # to hillary and trump.
                word_counts_per_candidate["trump"][word] = SMOOTHING
                word_counts_per_candidate["hillary"][word] = SMOOTHING
            # now add one to the global dictionary
            # and the associated candidate's
        ...


I am working on the file "2016-2017.Meetings/fall/05.DIY_naive_bayes/naive_bayes_classifier_solution/naive_bayes_tweet_classifier.ipynb".
The first task is:
Now let's take a look at some of our calculated word probabilities for each candidate.
Can you write Python code for it?

first_10_word_probabilities_per_candidate_H = {k: word_probabilities_per_candidate["hillary"][k] for k in word_probabilities_per_candidate["hillary"].keys()[:10]}
first_10_word_probabilities_per_candidate_H

first_10_word_probabilities_per_candidate_T = {k: word_probabilities_per_candidate["trump"][k] for k in word_probabilities_per_candidate["trump"].keys()[:10]}
first_10_word_probabilities_per_candidate_T


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/02.index_processed_data.ipynb".
The first task is:
If you compare the output of the first indexed example with the first raw example, you will see that each word has been assigned a unique index and words that are the same across sentences have the same index.
Now, we'll repackage the lists into a slightly more digestible format for the model. We will have one list of lists (note that each "question" now is a list of integers) for all of the question_1's, and one list of lists for all of the question_2's. Then, we'll have a list of labels.
These lists should correspond index-wise, so that label[i] should correspond to the correct label of the data point with indexed_question_1s[i] and indexed_question_2s[i].
Can you write Python code for it?

indexed_question_1s = []
indexed_question_2s = []
labels = []

for indexed_train_line in tqdm(indexed_train_lines):
    # Unpack the tuple into 3 variables
    indexed_question_1, indexed_question_2, label = indexed_train_line

    # Now add each of the individual elements of one train instance to their
    # separate lists.
    indexed_question_1s.append(indexed_question_1)
    indexed_question_2s.append(indexed_question_2)
    labels.append(label)

# Print the first element from each of the lists, it should be the same as the
# first element of the combined dataset above.
print("First indexed_question_1s: {}".format(indexed_question_1s[0]))
print("First indexed_question_2s: {}".format(indexed_question_2s[0]))
print("First label: {}".format(labels[0]))


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/02.index_processed_data.ipynb".
The first task is:
Right now, each data point consists of two strings and an integer label. Computers don't like dealing with strings directly very much, so we need to convert these strings to lists of integers.
The way we do this is: we will assign each string a unique integer ID, and then replace all occurences of the string with that integer ID. In this way, we can encode to the model what the various input strings are. This is called "indexing" the data.
Can you write Python code for it?

padding_token = "@@PADDING@@"
oov_token = "@@UNKOWN@@"
word_indices = {padding_token: 0, oov_token: 1}

for train_instance in tqdm(raw_train_lines):
    # unpack the tuple into 3 variables
    question_1, question_2, label = train_instance

    # iterate over the tokens in each question, and add them to the word
    # indices if they aren't in there already
    for word in question_1:
        if word not in word_indices:
            # by taking the current length of the dictionary
            # to be the index, we can guarantee that each unique word
            # will get a unique index.
            index = len(word_indices)
            word_indices[word] = index

    for word in question_2:
        if word not in word_indices:
            # by taking the current length of the dictionary
            # to be the index, we can guarantee that each unique word
            # will get a unique index.
            index = len(word_indices)
            word_indices[word] = index

# The number of unique tokens in our corpus
len(word_indices)


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Compiling the model
Now, we compile our model into a Tensorflow/Theano graph. Keras handles this for us, but we need to specify an optimization algorithm to use, as well as a loss function. adam is generally a good choice of optimizer, and binary_crossentropy is appropriate for a binary classification task like the one we have.
We can also specify a list of metrics to be printed during training and testing, so we'll print the accuracy.
Can you write Python code for it?

duplicate_questions_model.compile('adam', 'binary_crossentropy', metrics=['accuracy'])

# Print a summary of the layers of our model and their inputs and outputs
duplicate_questions_model.summary()


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Encoding the words
Now, our data consists of matrices of shape (batch_size, maxlen, embedding_dimension). It might be hard to intuitively think about what this means, but you've intuitively replaced each "int" index in the sentence with a vector of size embedding dimension (so from (batch_size, maxlen) to (batch_size, maxlen, embedding_dimension)).
Now that we have embedded our questions, it's time to encode them. A popular choice in modern NLP is to use a recurrent neural networks, especially the Bidirectional LSTM (biLSTM). An LSTM essentially takes a single question as input (something of shape (maxlen, embedding_dimension) in this case), and squeezes it into a fixed-length vector of size (LSTM_output_units). In this manner, you can think of the LSTM as "encoding" the question. into a single vector.
The "Bidirectional" part comes from the idea that you should run the question (a sequence of vectors) through the LSTM, and then reverse the question and run it through another LSTM. Then, you take the vector that was outputted from both and concatenate it. This intuitively lets the LSTM "read from both directions".
Can you write Python code for it?

# Now we take the embedded questions, and we encode them with a bidirectional LSTM.
# Think of a LSTM as converting/encoding a sequence of vectors into a fixed length vector.
# In this case, it takes in a single question of size (100, 128) and returns something of
# size (2*LSTM_output_units). Since it is batched, we go from (32, 100, 128) to (32, 2*LSTM_output_units)

# Bidirectional LSTM encoder for question_1_embedded
question_1_encoder = Bidirectional(LSTM(units=64))

# Bidirectional LSTM encoder for question_2_embedded
question_2_encoder = Bidirectional(LSTM(units=64))

# Now, we apply the Bidirectional LSTM encoders to our embedded questions.
# shape: (batch_size, 2*LSTM_output_units), or (32, 128) by default
question_1_encoded = question_1_encoder(question_1_embedded)
print("question_1_encoded: {}".format(question_1_encoded))

# shape: (batch_size, 2*LSTM_output_units), or (32, 128) by default
question_2_encoded = question_2_encoder(question_2_embedded)
print("question_2_encoded: {}".format(question_2_encoded))


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Great, each of our questions is now of length maxlen.
At this point, we need to get the "vocabulary" of the training data. This is the number of unique indices in the data, so in this case it's easy to calculate by taking the length of the word_indices dictionary.
Can you write Python code for it?

vocabulary_size = len(word_indices)
print("Vocabulary size: {}".format(vocabulary_size))


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
In the above printed representation of our tensor, you'll notice that the shape is a weird (?, maxlen). In this case, the ? refers to a dimension that can be of any size. Since that is our batch_size, we can vary the batch size to be whatever we want and the model will still work.
The Embedding Layer
Now that our questions are in the graph, we want to use an embedding layer to project each int index (which actually represents one word) into a higher-dimensional space. The way we do this is by using an Embedding layer. This layer replaces each index with a vector, and the vector should ideally represent the semantic meaning of the index. In this way, the model can get some notion of "meaning" between the indices.
In this model, the Embedding layer is randomly initialized --- every index is assigned a random vector at first. As the model trains, it will tweak the vector assigned to each word in order to minimize the loss. However, this naturally leads to a lot more parameters to tune, which makes the model harder to learn.
It's thus common practice in the field to use pre-trained embeddings. Pre-trained embeddings are what they sound like, embeddings for a word that already have gotten to a pretty good representation. By using these pretrained embeddings and not updating them (so keeping them fixed and not letting the model change them), you drastically lower the amount of parameters the model has to fiddle with and also prevent the model from overfitting (as it can make the embeddings overly domain-specific, while the pretrained embeddings are quite general).
Can you write Python code for it?

# Embedding layer for question 1. For each word in the question, it'll
# transform it into a fixed-length vector of size 128.
embedding_layer_1 = Embedding(input_dim=vocabulary_size, output_dim=128,
                              mask_zero=True, input_length=maxlen)

# Embedding layer for question 2. For each word in the question, it'll
# transform it into a fixed-length vector of size 128.
embedding_layer_2 = Embedding(vocabulary_size, 128,
                              mask_zero=True, input_length=maxlen)

# Now, we apply the embedding layers that we constructed to the input
# shape: (batch_size, question_1_length, embedding_output_dim) or (32, 100, 128) by default
question_1_embedded = embedding_layer_1(question_1_input)
print("question_1_embedded {}".format(question_1_embedded))

# shape: (batch_size, question_2_length, embedding_output_dim) or (32, 100, 128) by default
question_2_embedded = embedding_layer_2(question_2_input)
print("question_2_embedded {}".format(question_2_embedded))


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Padding
We're almost ready to train our model. There's just one hitch though: neural networks take as input fixed-length vectors. What are we to do, since our questions are sequences of ints with variable length?
The answer is to pad the shorter instances to the length of the longest instance, thus making them all the same length! We'll pad with the 0 character -- this is why we set the padding character to have a 0 index in the word to index dictionary. Keras will automatically figure out that these 0's are padding, and not take them into account when doing model computations (this is called masking).
It's common to also truncate sequences. For example, say that the average length of our questions is 10 words, but there's one outlier with 900 words. Padding all of the other questions to 900 words would be a huge waste of space, when we could simply truncate that one outlier with 900 words to 10 words. Thus, we'll set a max length of 100 words; if a question is less than 100 words, it'll be padded up, and if it's longer it'll be truncated.
Note that since the two questions are actually separate inputs to the model, as you'll see later, their max length could be set to different values if you wanted. This is useful if you're comparing, say, a question and a document -- you'd expect the question to be much shorter than the document, and adjust your lengths accordingly.
Can you write Python code for it?

maxlen = 100
max_training_instances=10000

# It takes a long time to train on all 400,000 samples on CPU (5 hours/epoch) --- let's cut it down to
# max_training_instances size. The dataset itself is a bit unbalanced, around 67% non-duplicate
# / 33% duplicate. We can use this opportunity to make it more balanced as well.
indices_with_0 = [index for index,value in enumerate(labels_list) if value==0]
indices_with_1 = [index for index,value in enumerate(labels_list) if value==1]

reduced_indexed_question_1s = []
reduced_indexed_question_2s = []
reduced_labels_list = []

for i in range(max_training_instances):
    # if i is even (~50%), pull something from indices_with_0 and add it to
    # the truncated dataset. Else, pull something from indices_with_1 and
    # add it to the truncatd dataset. If any of the list of indices are empty,
    # use the other one.
    # TODO: I'm pretty sure this if can be refactored, but it's late and I can't think
    # right now.
    if i % 2 == 0:
        if indices_with_0:
            index = indices_with_0.pop()
        else:
            index = indices_with_1.pop()
    else:
        if indices_with_1:
            index = indices_with_1.pop()
        else:
            index = indices_with_0.pop()
    reduced_indexed_question_1s.append(indexed_question_1s[index])
    reduced_indexed_question_2s.append(indexed_question_2s[index])
    reduced_labels_list.append(labels_list[index])

print(len(reduced_indexed_question_1s))
print(len(reduced_indexed_question_2s))
print(len(reduced_labels_list))

# Now we want to pad / truncate our instances to a max length.
# Keras has a handy function to do this, but it isn't hard to implement yourself as well.
padded_question_1s = sequence.pad_sequences(reduced_indexed_question_1s, maxlen=maxlen)
padded_question_2s = sequence.pad_sequences(reduced_indexed_question_2s, maxlen=maxlen)

padded_question_1s_shape = padded_question_1s.shape
padded_question_2s_shape = padded_question_2s.shape

# We also want to convert our list of labels to a numpy...


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Training our model
Now, we can finally pass in our input arrays and output labels and watch the model train!
Can you write Python code for it?

# Now, we can finally fit our model on training data!
# Note that the order of the input x matters.
duplicate_questions_model.fit(x=[padded_question_1s, padded_question_2s], y=labels,
                              batch_size=batch_size, epochs=4, validation_split=0.1)


I am working on the file "2016-2017.Meetings/spring/01.keras_tutorial_duplicate_questions/03.train_model.ipynb".
The first task is:
Wrapping up the model
Now that we've successfully strung together a bunch of our layers and inputs to get a final probability, we can create a keras Model to seamlessly take the input numpy arrays, run them through the computation graph we built in the way we specified, to get an output probability that it will automatically compare to the label in order to adjust the loss.
To do all of this, we just need to create an instance of the Model class and specify which Input layers are our inputs, and what value from the graph is our final output. Note that since we have multiple inputs, we need to pass a list of Input tensors.
Can you write Python code for it?

# These duplicate probabilties are what we want to output from our model, so we'll create
# the model now.
duplicate_questions_model = Model(inputs=[question_1_input, question_2_input], outputs=duplicate_probabilities)


I am working on the file "2016/2016-12-07-jupyter/python4astronomers_jupyter.ipynb".
The first task is:
Did it work?
Can you write Python code for it?


I am working on the file "2016/2016-12-07-jupyter/python4astronomers_jupyter.ipynb".
The first task is:
Let's see what files we just extracted:
Can you write Python code for it?


I am working on the file "2016/2016-12-07-jupyter/python4astronomers_jupyter.ipynb".
The first task is:
Now prep the notebook for plotting:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "2016/BBGÀgora 20161201.ipynb".
The first task is:
Manage big tsv files
Can you write Python code for it?

import bgdata, csv, gzip, os, pandas
from pprint import pprint

domains = os.path.expanduser('~/tmp/domains.tsv.gz')
# If you want to test use this:
# domains = os.path.join(bgdata.get_path('tcgi', 'oncodrivemut', '1.1'), 'ensembl75_pfam_domain_coordinates.tsv.gz')

%%time
df = pandas.read_csv(domains, sep='\t')
result = df[df['Ensembl Gene ID'] == 'ENSG00000261258'].head(1).to_dict(orient='records')
pprint(result)
print('\n')

%%time
with gzip.open(domains, 'rt') as fd:
    for r in csv.DictReader(fd, delimiter='\t'):
        if r['Ensembl Gene ID'] == 'ENSG00000261258':
            pprint(r)
            print('\n')
            break

%%time
with gzip.open(domains, 'rt') as fd:
    reader = csv.reader(fd, delimiter='\t')
    header = next(reader)
    for r in reader:
        if r[0] == 'ENSG00000261258':
            pprint({h: v for h,v in zip(header, r)})
            print('\n')
            break

%%time
with gzip.open(domains, 'rt') as fd:
    header = next(fd).split('\t')

    reader = csv.reader((l for l in fd if l.startswith('ENSG00000261258')), delimiter='\t')
    for r in reader:
        if r[0] == 'ENSG00000261258':
            pprint({h: v for h,v in zip(header, r)})
            print('\n')
            break


I am working on the file "2016/BBGÀgora 20161201.ipynb".
The first task is:
Python iterables and iterators
Can you write Python code for it?

# ITERABLE: Anything that you can use in a for is an iterable
for a in [1,2,3]:
    print(a)

# ITERATOR: A iteration of an iterable
list_iterator = iter([1,2,3])
print(next(list_iterator))
print(next(list_iterator))
print(next(list_iterator))

list_iterator = iter([1,2,3])
print(next(list_iterator))
print(next(list_iterator))
print(next(list_iterator))
print(next(list_iterator))


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
A slice of an array is a view into the same data, so modifying it will modify the original array.
Can you write Python code for it?

print a[0, 1]
b[0, 0] = 77    # b[0, 0] is the same piece of data as a[0, 1]
print a[0, 1]


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Array indexing
Numpy offers several ways to index into arrays.
Slicing: Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array:
Can you write Python code for it?

import numpy as np

# Create the following rank 2 array with shape (3, 4)
# [[ 1  2  3  4]
#  [ 5  6  7  8]
#  [ 9 10 11 12]]
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])

# Use slicing to pull out the subarray consisting of the first 2 rows
# and columns 1 and 2; b is the following array of shape (2, 2):
# [[2 3]
#  [6 7]]
b = a[:2, 1:3]
print b


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Boolean array indexing: Boolean array indexing lets you pick out arbitrary elements of an array. Frequently this type of indexing is used to select the elements of an array that satisfy some condition. Here is an example:
Can you write Python code for it?

import numpy as np

a = np.array([[1,2], [3, 4], [5, 6]])

bool_idx = (a > 2)  # Find the elements of a that are bigger than 2;
                    # this returns a numpy array of Booleans of the same
                    # shape as a, where each slot of bool_idx tells
                    # whether that element of a is > 2.

print bool_idx

# We use boolean array indexing to construct a rank 1 array
# consisting of the elements of a corresponding to the True values
# of bool_idx
print a[bool_idx]

# We can do all of the above in a single concise statement:
print a[a > 2]


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Broadcasting typically makes your code more concise and faster, so you should strive to use it where possible.
This brief overview has touched on many of the important things that you need to know about numpy, but is far from complete. Check out the numpy reference to find out much more about numpy.
Matplotlib
Matplotlib is a plotting library. In this section give a brief introduction to the matplotlib.pyplot module, which provides a plotting system similar to that of MATLAB.
Can you write Python code for it?


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
By running this special iPython command, we will be displaying plots inline:
Can you write Python code for it?


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Classes
The syntax for defining classes in Python is straightforward:
Can you write Python code for it?

class Greeter:

    # Constructor
    def __init__(self, name):
        self.name = name  # Create an instance variable

    # Instance method
    def greet(self, loud=False):
        if loud:
            print 'HELLO, %s!' % self.name.upper()
        else:
            print 'Hello, %s' % self.name

g = Greeter('Fred')  # Construct an instance of the Greeter class
g.greet()            # Call an instance method; prints "Hello, Fred"
g.greet(loud=True)   # Call an instance method; prints "HELLO, FRED!"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Dictionary comprehensions: These are similar to list comprehensions, but allow you to easily construct dictionaries. For example:
Can you write Python code for it?

nums = [0, 1, 2, 3, 4]
even_num_to_square = {x: x ** 2 for x in nums if x % 2 == 0}
print even_num_to_square


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
For brevity we have left out a lot of details about numpy array indexing; if you want to know more you should read the documentation.
Datatypes
Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Here is an example:
Can you write Python code for it?

x = np.array([1, 2])  # Let numpy choose the datatype
y = np.array([1.0, 2.0])  # Let numpy choose the datatype
z = np.array([1, 2], dtype=np.int64)  # Force a particular datatype

print x.dtype, y.dtype, z.dtype


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
If you want access to keys and their corresponding values, use the iteritems method:
Can you write Python code for it?

d = {'person': 2, 'cat': 4, 'spider': 8}
for animal, legs in d.iteritems():
    print 'A %s has %d legs' % (animal, legs)


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
If you want access to the index of each element within the body of a loop, use the built-in enumerate function:
Can you write Python code for it?

animals = ['cat', 'dog', 'monkey']
for idx, animal in enumerate(animals):
    print '#%d: %s' % (idx + 1, animal)


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
List comprehensions can also contain conditions:
Can you write Python code for it?

nums = [0, 1, 2, 3, 4]
even_squares = [x ** 2 for x in nums if x % 2 == 0]
print even_squares


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
List comprehensions:
When programming, frequently we want to transform one type of data into another. As a simple example, consider the following code that computes square numbers:
Can you write Python code for it?

nums = [0, 1, 2, 3, 4]
squares = []
for x in nums:
    squares.append(x ** 2)
print squares


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Loops
You can loop over the elements of a list like this:
Can you write Python code for it?

animals = ['cat', 'dog', 'monkey']
for animal in animals:
    print animal


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Loops: Iterating over a set has the same syntax as iterating over a list; however since sets are unordered, you cannot make assumptions about the order in which you visit the elements of the set:
Can you write Python code for it?

animals = {'cat', 'dog', 'fish'}
for idx, animal in enumerate(animals):
    print '#%d: %s' % (idx + 1, animal)
# Prints "#1: fish", "#2: dog", "#3: cat"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Note that unlike many languages, Python does not have unary increment (x++) or decrement (x--) operators.
Python also has built-in types for long integers and complex numbers; you can find all of the details in the documentation.
Booleans
Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols (&&, ||, etc.):
Can you write Python code for it?

t, f = True, False
print type(t) # Prints "<type 'bool'>"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Numpy
Numpy is the core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays. If you are already familiar with MATLAB, you might find this tutorial useful to get started with Numpy.
To use Numpy, we first need to import the numpy package:
Can you write Python code for it?


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Numpy also provides many functions to create arrays:
Can you write Python code for it?

a = np.zeros((2,2))  # Create an array of all zeros
print a

b = np.ones((1,2))   # Create an array of all ones
print b

c = np.full((2,2), 7) # Create a constant array
print c

d = np.eye(2)        # Create a 2x2 identity matrix
print d

e = np.random.random((2,2)) # Create an array filled with random values
print e


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Numpy broadcasting allows us to perform this computation without actually creating multiple copies of v. Consider this version, using broadcasting:
Can you write Python code for it?

import numpy as np

# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
y = x + v  # Add v to each row of x using broadcasting
print y


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Numpy provides many useful functions for performing computations on arrays; one of the most useful is sum:
Can you write Python code for it?

x = np.array([[1,2],[3,4]])

print np.sum(x)  # Compute sum of all elements; prints "10"
print np.sum(x, axis=0)  # Compute sum of each column; prints "[4 6]"
print np.sum(x, axis=1)  # Compute sum of each row; prints "[3 7]"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
One useful trick with integer array indexing is selecting or mutating one element from each row of a matrix:
Can you write Python code for it?

# Create a new array from which we will select elements
a = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
print a

# Create an array of indices
b = np.array([0, 2, 0, 1])

# Select one element from each row of a using the indices in b
print a[np.arange(4), b]  # Prints "[ 1  6  7 11]"

# Mutate one element from each row of a using the indices in b
a[np.arange(4), b] += 10
print a


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Python versions
There are currently two different supported versions of Python, 2.7 and 3.4. Somewhat confusingly, Python 3.0 introduced many backwards-incompatible changes to the language, so code written for 2.7 may not work under 3.4 and vice versa. For this class all code will use Python 2.7.
You can check your Python version at the command line by running python --version.
Basic data types
Numbers
Integers and floats work as you would expect from other languages:
Can you write Python code for it?

x = 3
print x, type(x)

print x + 1   # Addition;
print x - 1   # Subtraction;
print x * 2   # Multiplication;
print x ** 2  # Exponentiation;

x += 1
print x  # Prints "4"
x *= 2
print x  # Prints "8"

y = 2.5
print type(y) # Prints "<type 'float'>"
print y, y + 1, y * 2, y ** 2 # Prints "2.5 3.5 5.0 6.25"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Set comprehensions: Like lists and dictionaries, we can easily construct sets using set comprehensions:
Can you write Python code for it?

from math import sqrt
print {int(sqrt(x)) for x in range(30)}


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Sets
A set is an unordered collection of distinct elements. As a simple example, consider the following:
Can you write Python code for it?

animals = {'cat', 'dog'}
print 'cat' in animals   # Check if an element is in a set; prints "True"
print 'fish' in animals  # prints "False"

animals.add('fish')      # Add an element to a set
print 'fish' in animals
print len(animals)       # Number of elements in a set;

animals.add('cat')       # Adding an element that is already in the set does nothing
print len(animals)
animals.remove('cat')    # Remove an element from a set
print len(animals)


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
String objects have a bunch of useful methods; for example:
Can you write Python code for it?

s = "hello"
print s.capitalize()  # Capitalize a string; prints "Hello"
print s.upper()       # Convert a string to uppercase; prints "HELLO"
print s.rjust(7)      # Right-justify a string, padding with spaces; prints "  hello"
print s.center(7)     # Center a string, padding with spaces; prints " hello "
print s.replace('l', '(ell)')  # Replace all instances of one substring with another;
                               # prints "he(ell)(ell)o"
print '  world '.strip()  # Strip leading and trailing whitespace; prints "world"


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Subplots
You can plot different things in the same figure using the subplot function. Here is an example:
Can you write Python code for it?

# Compute the x and y coordinates for points on sine and cosine curves
x = np.arange(0, 3 * np.pi, 0.1)
y_sin = np.sin(x)
y_cos = np.cos(x)

# Set up a subplot grid that has height 2 and width 1,
# and set the first such subplot as active.
plt.subplot(2, 1, 1)

# Make the first plot
plt.plot(x, y_sin)
plt.title('Sine')

# Set the second subplot as active, and make the second plot.
plt.subplot(2, 1, 2)
plt.plot(x, y_cos)
plt.title('Cosine')

# Show the figure.
plt.show()


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
The line y = x + v works even though x has shape (4, 3) and v has shape (3,) due to broadcasting; this line works as if v actually had shape (4, 3), where each row was a copy of v, and the sum was performed elementwise.
Broadcasting two arrays together follows these rules:

If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.
The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.
The arrays can be broadcast together if they are compatible in all dimensions.
After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.
In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension

If this explanation does not make sense, try reading the explanation from the documentation or this explanation.
Functions that support broadcasting are known as universal functions. You can find the list of all universal functions in the documentation.
Here are some applications of broadcasting:
Can you write Python code for it?

# Compute outer product of vectors
v = np.array([1,2,3])  # v has shape (3,)
w = np.array([4,5])    # w has shape (2,)
# To compute an outer product, we first reshape v to be a column
# vector of shape (3, 1); we can then broadcast it against w to yield
# an output of shape (3, 2), which is the outer product of v and w:

print np.reshape(v, (3, 1)) * w

# Add a vector to each row of a matrix
x = np.array([[1,2,3], [4,5,6]])
# x has shape (2, 3) and v has shape (3,) so they broadcast to (2, 3),
# giving the following matrix:

print x + v

# Add a vector to each column of a matrix
# x has shape (2, 3) and w has shape (2,).
# If we transpose x then it has shape (3, 2) and can be broadcast
# against w to yield a result of shape (3, 2); transposing this result
# yields the final result of shape (2, 3) which is the matrix x with
# the vector w added to each column. Gives the following matrix:

print (x.T + w).T

# Another solution is to reshape w to be a row vector of shape (2, 1);
# we can then broadcast it directly against x to produce the same
# output.
print x + np.reshape(w, (2, 1))

# Multiply a matrix by a constant:
# x has shape (2, 3). Numpy treats scalars as arrays of shape ();
# these can be broadcast together to shape (2, 3), producing the
# following array:
print x * 2


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
This works; however when the matrix x is very large, computing an explicit loop in Python could be slow. Note that adding the vector v to each row of the matrix x is equivalent to forming a matrix vv by stacking multiple copies of v vertically, then performing elementwise summation of x and vv. We could implement this approach like this:
Can you write Python code for it?

vv = np.tile(v, (4, 1))  # Stack 4 copies of v on top of each other
print vv                 # Prints "[[1 0 1]
                         #          [1 0 1]
                         #          [1 0 1]
                         #          [1 0 1]]"

y = x + vv  # Add x and vv elementwise
print y


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Tuples
A tuple is an (immutable) ordered list of values. A tuple is in many ways similar to a list; one of the most important differences is that tuples can be used as keys in dictionaries and as elements of sets, while lists cannot. Here is a trivial example:
Can you write Python code for it?

d = {(x, x + 1): x for x in range(10)}  # Create a dictionary with tuple keys
t = (5, 6)       # Create a tuple
print type(t)
print d[t]
print d[(1, 2)]

t[0] = 1


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
Two ways of accessing the data in the middle row of the array.
Mixing integer indexing with slices yields an array of lower rank,
while using only slices yields an array of the same rank as the
original array:
Can you write Python code for it?

row_r1 = a[1, :]    # Rank 1 view of the second row of a
row_r2 = a[1:2, :]  # Rank 2 view of the second row of a
row_r3 = a[[1], :]  # Rank 2 view of the second row of a
print row_r1, row_r1.shape
print row_r2, row_r2.shape
print row_r3, row_r3.shape

# We can make the same distinction when accessing columns of an array:
col_r1 = a[:, 1]
col_r2 = a[:, 1:2]
print col_r1, col_r1.shape
print
print col_r2, col_r2.shape


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
We will often define functions to take optional keyword arguments, like this:
Can you write Python code for it?

def hello(name, loud=False):
    if loud:
        print 'HELLO, %s' % name.upper()
    else:
        print 'Hello, %s!' % name

hello('Bob')
hello('Fred', loud=True)


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
With just a little bit of extra work we can easily plot multiple lines at once, and add a title, legend, and axis labels:
Can you write Python code for it?

y_sin = np.sin(x)
y_cos = np.cos(x)

# Plot the points using matplotlib
plt.plot(x, y_sin)
plt.plot(x, y_cos)
plt.xlabel('x axis label')
plt.ylabel('y axis label')
plt.title('Sine and Cosine')
plt.legend(['Sine', 'Cosine'])


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
You can also mix integer indexing with slice indexing. However, doing so will yield an array of lower rank than the original array. Note that this is quite different from the way that MATLAB handles array slicing:
Can you write Python code for it?

# Create the following rank 2 array with shape (3, 4)
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print a


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
You can find a list of all string methods in the documentation.
Containers
Python includes several built-in container types: lists, dictionaries, sets, and tuples.
Lists
A list is the Python equivalent of an array, but is resizeable and can contain elements of different types:
Can you write Python code for it?

xs = [3, 1, 2]   # Create a list
print xs, xs[2]
print xs[-1]     # Negative indices count from the end of the list; prints "2"

xs[2] = 'foo'    # Lists can contain elements of different types
print xs

xs.append('bar') # Add a new element to the end of the list
print xs

x = xs.pop()     # Remove and return the last element of the list
print x, xs


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
You can find all you need to know about dictionaries in the documentation.
It is easy to iterate over the keys in a dictionary:
Can you write Python code for it?

d = {'person': 2, 'cat': 4, 'spider': 8}
for animal in d:
    legs = d[animal]
    print 'A %s has %d legs' % (animal, legs)


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
You can make this code simpler using a list comprehension:
Can you write Python code for it?

nums = [0, 1, 2, 3, 4]
squares = [x ** 2 for x in nums]
print squares


I am working on the file "2016/Python 101 Workshop- NMIMS/cs228-python-tutorial.ipynb".
The first task is:
You can read all about numpy datatypes in the documentation.
Array math
Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module:
Can you write Python code for it?

x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)

# Elementwise sum; both produce the array
print x + y
print np.add(x, y)

# Elementwise difference; both produce the array
print x - y
print np.subtract(x, y)

# Elementwise product; both produce the array
print x * y
print np.multiply(x, y)

# Elementwise division; both produce the array
# [[ 0.2         0.33333333]
#  [ 0.42857143  0.5       ]]
print x / y
print np.divide(x, y)

# Elementwise square root; produces the array
# [[ 1.          1.41421356]
#  [ 1.73205081  2.        ]]
print np.sqrt(x)


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
Bernoulli Distribution
The Bernoulli distribution, named after Jacob Bernoulli, is the probability distribution of a random variable which takes the value 1 (success) with probability $p$ and the value 0 (failure) with probability $q=1-p$.
The Bernoulli distribution with $p=0.5$ (implying $q=0.5$) describes a 'fair' coin toss where 1 and 0  represent "heads" and "tails", respectively. If the coin is unfair, then we would have that $p\neq 0.5$.
Can you write Python code for it?

n = 1000;
coin_flips = bernoulli.rvs(p=0.5, size=n)
print(coin_flips)


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
Hypothesis testing
So what is the likelihood of flipping a coin 1000 times and seeing less than 545 heads?
The CLT tells us that this is approximately
$$
\int_{-\infty}^{545} p(x) dx = F(545).
$$
This is something that we can easily evaluate using the cumulative distribution function (CDF).
Can you write Python code for it?

n = 1000
p = 0.5
mu = n*p
sigma = sc.sqrt(n*p*(1-p))
print(norm.cdf(545, loc=mu, scale=sigma))

# a plot illustrating the integral
x = sc.arange(mu-4*sigma,mu+4*sigma,0.001);
plt.plot(x, norm.pdf(x, loc=mu, scale=sigma), linewidth=2, color='k')
x2 = sc.arange(mu-4*sigma,545,0.001)
plt.fill_between(x2, y1= norm.pdf(x2,loc=mu, scale=sigma), facecolor='red', alpha=0.5)
plt.xlim([mu-4*sigma,mu+4*sigma])
plt.show()


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
So $99.8\%$ of the time, we would see fewer than 545 heads.  So seeing 545 heads is very unlikely! It happens only $0.2\%$ of the time. This is so unlikely that we might declare that the coin is not fair!
This is precisely what hypothesis testing is.
In hypothesis testing, we make a null hypothesis, denoted $H_0$. In this case, the null hypothesis is
$$
H_0: \text{the coin is fair, i.e., $p=0.5$}.
$$
The alternative hypothesis, $H_a$, is typically the hypothesis that the researcher wants to validate. In this case, that the coin is unfair, i.e., $p\neq 0.5$.
We also choose a significance level for the test, $\alpha$, traditionally $1\%$ or $5\%$.
In this case, let's choose a significance level of $\alpha = 1\%$. We then perform an experiment. In this case, we flip the coin 1000 times and count the number of heads (in this case 545).
Finally, assuming the null hypothesis is true, we compute how how likely it is to see a number that is at least as far from the expected value as the number obtained. To do this, we compute the integral
$$
\int_{455}^{545} p(x) dx = F(545) - F(455)
$$
Question: why this lower bound?
Can you write Python code for it?

val_integral = norm.cdf(545, loc=mu, scale=sigma) - norm.cdf(455, loc=mu, scale=sigma)
print(val_integral)
print(1-val_integral)


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
Some facts about the binomial distribution:
* The mean is $np$
* The variance is $np(1-p)$
Mathematical aside: Binomial (and Bernoulli) random variables are examples of discrete random variables since they can take only discrete values. A Bernoulli random variable can take values $0$ or $1$. A binomial random variable  can only take values
$$
0,1,\ldots, n.
$$
One can compute the probability that the variable takes each value. This is called the probability mass function.
For a Bernoulli random variable, the probability mass function is given by
$$
f(k) = \begin{cases} p & k=1 \ 1-p & k = 0 \end{cases}
$$
For a binomial random variable, the probability mass function is given by
$$
f(k) = \binom{n}{k} p^k (1-p)^{n-k}.
$$
Here, $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ is the number of ways to arrange the
$k$ heads among the $n$ flips. For a fair coin, we have $p=0.5$ and $f(k) = \binom{n}{k} \frac{1}{2^n}$. This is the number of ways to arrange $k$ heads among $n$ outcomes divided by the total number of outcomes.
The probability mass function can be plotted using the scipy library as follows.
Can you write Python code for it?

f = lambda k: binom.pmf(k, n=n,p=p)

x = sc.arange(n+1);
plt.plot(x, f(x),'*-')
plt.title("The probability mass function for a Binomial random variable")
plt.xlim([0,n])
plt.show()


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
This integral can be computed using the cumulative distribution function (CDF)
$$
F(x) = \int_{-\infty}^x f(x) dx.
$$
We have that
$$
\int_a^b f(x) dx = F(b) - F(a)
$$
Can you write Python code for it?

norm.cdf(mu+sigma, loc=mu, scale=sigma) - norm.cdf(mu-sigma, loc=mu, scale=sigma)


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
This means that 68% of the time, this normal random variable will have values between $\mu-\sigma$ and $\mu+\sigma$.
You used to have to look these values up in a table!
Let's see what it looks like if we sample 1,000,000 normal random variables and then plot a histogram.
Can you write Python code for it?

norm_vars = norm.rvs(loc=mu,scale=sigma,size=1000000)
print(norm_vars[:100])

plt.hist(norm_vars, bins=100,normed=True)
plt.plot(x, pdf, linewidth=2, color='k')
plt.title("A histogram of normal random variables")
plt.show()


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
Thus, $99.6\%$ of the time we see a value less extreme than 545. In other words, we would see either more than 545 heads or less than 455 heads only 0.4% of the time. The is called the P-value. Since the P-value is smaller than the chosen significance level, we reject the null hypothesis and declare the coin to be unfair.
Some comments about the p-value:
1. A p-value is a probability calculated assuming that $H_0$ is true.
+ The smaller the p-value, the stronger the evidence against $H_0$.
+ A p-value is not the probability that the null hypothesis is true of false. It is the probability that an erroneous conclusion is reached.  (More on this next lecture)
Example: "Freshman 15", Fact or Fiction
This example was taken from Devore, pp.314-315.
"A common belief among the lay public is that body weight increases after entry into college, and the phrase 'freshman 15' has been coined to describe the 15 puunds that students presumably gain over their freshman year."
Let $\mu$ denote the true average weight gain in the first year of college. We take the null hypothesis to be
$$
H_0: \mu = 15
$$
We suppose a random sample of $n$ students is selected, their weights (before and after the first year of college) are measured, and the sample mean $\bar{x}$ and sample standard deviation $s$ are computed. An article in the journal Obesity (2006) cites that for a sample of $n=137$ students, the sample mean weight gain was $\bar{x}=2.42$ lb and with a sample standard deviation of $s=5.72$ lb. Assuming $H_0$ to be true, how unlikely is it that we would observe such a small value?
We take a normal distribution with mean given by the null value ($\mu = 15$) and variance given by $s^2/n = (5.72)^2/137=0.2388$.
Can you write Python code for it?

mu = 15
sigma = sc.sqrt(5.72**2/137)
print(2*norm.cdf(2.42, loc=mu, scale=sigma))


I am working on the file "2016/assets/slides/03-hypothesis-testing-1.ipynb".
The first task is:
What if we flip the coin more times?
Can you write Python code for it?

n = 1000000
coin_flips = bernoulli.rvs(p=0.5, size=n)
print(sum(coin_flips)/n)


I am working on the file "2016/assignment 3.ipynb".
The first task is:
How many nodes and edges are there in the graph?
Can you write Python code for it?


I am working on the file "2016/assignment 3.ipynb".
The first task is:
Most lines in the file contain a string representation of a python dictionary that represents a tweet. A tweet is a message that has been generated on twitter. Below you see the example of one tweet (one line in the file).
{u'contributors': None, u'coordinates': None, u'created_at': u'Fri Sep 26 08:01:38 +0000 2014', u'entities': {u'hashtags': [], u'symbols': [], u'trends': [], u'urls': [{u'display_url': u'bbc.in/1qBcZ4G', u'expanded_url': u'http://bbc.in/1qBcZ4G', u'indices': [126, 140], u'url': u'http://t.co/8kPs90syqo'}], u'user_mentions': [{u'id': 2190056023, u'id_str': u'2190056023', u'indices': [3, 9], u'name': u'BBC Outside Source', u'screen_name': u'BBCOS'}]}, u'favorite_count': 0, u'favorited': False, u'filter_level': u'medium', u'geo': None, u'id': 515410856616935425, u'id_str': u'515410856616935425', u'in_reply_to_screen_name': None, u'in_reply_to_status_id': None, u'in_reply_to_status_id_str': None, u'in_reply_to_user_id': None, u'in_reply_to_user_id_str': None, u'lang': u'en', u'place': None, u'possibly_sensitive': False, u'retweet_count': 0, u'retweeted': False, u'retweeted_status': {u'contributors': None, u'coordinates': None, u'created_at': u'Fri Sep 26 07:44:43 +0000 2014', u'entities': {u'hashtags': [], u'symbols': [], u'trends': [], u'urls': [{u'display_url': u'bbc.in/1qBcZ4G', u'expanded_url': u'http://bbc.in/1qBcZ4G', u'indices': [115, 137], u'url': u'http://t.co/8kPs90syqo'}], u'user_mentions': []}, u'favorite_count': 0, u'favorited': False, u'filter_level': u'low', u'geo': None, u'id': 515406597829693440, u'id_str': u'515406597829693440', u'in_reply_to_screen_name': None, u'in_reply_to_status_id': None, u'in_reply_to_status_id_str': None, u'in_reply_to_user_id': None, u'in_reply_to_user_id_str': None, u'lang': u'en', u'place': None, u'possibly_sensitive': False, u'retweet_count': 3, u'retweeted': False, u'source': u'SocialFlow', u'text': u"EU's anti-terrorism chief tells BBC the number of Europeans joining Islamist...
Can you write Python code for it?


I am working on the file "2016/assignment 3.ipynb".
The first task is:
Tweets
Run the following cell to assign the data to an RDD.
Can you write Python code for it?


I am working on the file "2016/assignment 3.ipynb".
The first task is:
Use this cell to explain / expand your answer.
Can you write Python code for it?


I am working on the file "2016/assignment 3.ipynb".
The first task is:
Use this cell to explain / expand your answer.
Problem 5: Pagerank (30 points)
Some tweets are replies to tweets written by other Twitter users. You can tell which tweets are replies by checking the in_reply_to_screen_name value of a tweet: if it is None, then it is not a reply - otherwise, it is a reply to the user with the screen_name mentioned therein.
For example, in_reply_to_screen_name: None signifies that the tweet is not a reply to another tweet, but in_reply_to_screen_name: northern_bytes signifies that the tweet is a reply to another tweet generated by user northern_bytes.
We are going to construct a graph in the following steps:
1. We place one node in the graph for each user who has produced more than $20$ english tweets (with distinct ids) in the data;
2. We place one directed edge from node $u$ to node $v$ iff the total number of english replies from user $u$ to user $v$ in the data is more than $10$. In that case, we will say that '$u$ is connected to $v$'.
1. Graph construction
Construct a pair RDD named linksRDD to store the adjacency list of each node.
Specifically, for each node (user) $u$ in the graph, you should have one element of the following form
(screen_name, [screen_name_1, screen_name_2, ..., screen_name_v, ...])
where screen_name corresponds to user $u$ and screen_name_v corresponds to a user $v$ that $u$ is connected to.
Can you write Python code for it?


I am working on the file "2016/day05-python.ipynb".
The first task is:
Part 1: Join the 5th character of each of the first 8 interesting hashes
Can you write Python code for it?

def password1(prefix):
    return "".join(h[5] for h in itertools.islice(interestingHashes(prefix), 8))

password1(doorId)


I am working on the file "2016/day05-python.ipynb".
The first task is:
Part 2: More complex password algorithm

If the 5th character of an interesting hash is a digit between 0 and 7, it tells us a position in the password.
This position is assigned to the 6th character of the interesting hash.
An interesting hash is ignored if the position given by its 5th character has been seen already.
Can you write Python code for it?

def password2(prefix):
    result = [None] * 8
    for h in interestingHashes(prefix):
        if h[5] in "01234567":
            pos = int(h[5])
            if result[pos] is None:
                result[pos] = h[6]
                if all(c is not None for c in result):
                    return "".join(result)

password2(doorId)


I am working on the file "2016/day11-python.ipynb".
The first task is:
A state is a tuple the contains the elevator position and the compressed representation of the item positions
Can you write Python code for it?

initialState = (1, compressedItems(initialItems))
print(initialState)


I am working on the file "2016/day11-python.ipynb".
The first task is:
Check if a state is valid
A state is valid unless there is a floor
* which has at least one generator, and
* which has at least one microchip which is not accompanied by the matching generator.
Can you write Python code for it?

def isValidState(state):
    currentFloor, items = state
    floorsWithGenerators = set(generatorFloor for generatorFloor, microchipFloor in items)
    floorsWithVulnerableMicrochips = set(microchipFloor
                                         for generatorFloor, microchipFloor in items
                                         if generatorFloor != microchipFloor)
    return len(floorsWithGenerators & floorsWithVulnerableMicrochips) == 0

assert     isValidState((1, ((2, 2), (2, 3), (4, 3), (4, 4))))
assert not isValidState((1, ((2, 2), (2, 3), (4, 2), (4, 4))))


I am working on the file "2016/day11-python.ipynb".
The first task is:
Compact representation of the item positions
Our current representation of the positions of the microchips and generators is inefficient. Assuming that there is exactly one microchip and one generator per element, we can make the following simplifications:
* For each element, it is sufficient to store the positions of the generator and the microchip in a tuple with two elements.
* For the solution of the problem, the element names are irrelevant. Therefore, it is sufficient to store only the tuples with the positions of the generator and the microchip for each element, and ignore the element name.
* In order to reduce the problem space, the list of tuples can be sorted: for the number of moves that are needed to solve the puzzle, it does not matter if the positions for two elements are ((2, 3), (1, 1)) or ((1, 1), (2, 3)).
Helper function that generates a position tuple for a single element: tupleForElement
Can you write Python code for it?

# Takes an iterable that yields two (element, type, floor) tuples, where
# * the element should be the same for both tuples,
# * the first item should be a generator (type 'G'),
# * the second item should be a microchip (type 'M').
# Returns a tuple that contains only the floors where the generator and the microchip are.
def tupleForElement(items):
    result = tuple(floor for element, itemType, floor in items)
    assert len(result) == 2
    return result

assert tupleForElement((("iron", "G", 3), ("iron", "M", 1))) == (3, 1)


I am working on the file "2016/day11-python.ipynb".
The first task is:
Functions for parsing the initial state
Map the floors to integers
Can you write Python code for it?

floors = {
    "first"  : 1,
    "second" : 2,
    "third"  : 3,
    "fourth" : 4,
}


I am working on the file "2016/day11-python.ipynb".
The first task is:
Functions for working with states
Can you write Python code for it?

def isFinalState(state, targetFloor=4):
    currentFloor, items = state
    return currentFloor == targetFloor and all(item == (targetFloor, targetFloor) for item in items)


I am working on the file "2016/day11-python.ipynb".
The first task is:
Parse all items on a floor
Can you write Python code for it?

def parseFloor(line):
    m = re.fullmatch("The ([a-z]+) floor contains (.*).", line)
    floor, itemsStr = m.groups()
    return tuple(sorted(parseItem(item[2:]) + (floors[floor],)
                        for item in re.split("(?:,\ )?(?:\ ?and\ )?", itemsStr)
                        if item.startswith("a ")))
assert (parseFloor("The first floor contains a hydrogen-compatible microchip and a lithium generator.") ==
        (("hydrogen", "M", 1),
         ("lithium", "G", 1)))
assert (parseFloor("The second floor contains a hydrogen generator, and a lithium-compatible microchip.") ==
        (("hydrogen", "G", 2),
         ("lithium", "M", 2)))
assert (parseFloor("The second floor contains a hydrogen generator.") ==
        (("hydrogen", "G", 2),))
assert (parseFloor("The third floor contains a lithium-compatible microchip.") ==
        (("lithium", "M", 3),))
assert (parseFloor("The fourth floor contains nothing relevant.") ==
        ())


I am working on the file "2016/day11-python.ipynb".
The first task is:
Parse an item (microchip or generator)
Can you write Python code for it?

def parseItem(item):
    microchipMatch = re.fullmatch("([a-z]+)-compatible microchip", item)
    if microchipMatch is not None:
        return microchipMatch.group(1), "M"
    generatorMatch = re.fullmatch("([a-z]+) generator", item)
    assert generatorMatch is not None
    return generatorMatch.group(1), "G"

assert parseItem("hydrogen-compatible microchip") == ("hydrogen", "M")
assert parseItem("lithium generator") == ("lithium", "G")


I am working on the file "2016/day11-python.ipynb".
The first task is:
Use these functions for parsing the initial items on all floors
Can you write Python code for it?

initialItems = tuple(sorted(itertools.chain.from_iterable(parseFloor(line) for line in inputLines)))
print(initialItems)


I am working on the file "2016/decision-tree/DecisionTree.ipynb".
The first task is:
Agora ok!
Vamos ao que interessa...
Can you write Python code for it?

dTree = DecisionTreeClassifier(criterion="entropy")
dTree.fit(trainData.ix[:,0:6], trainData.index)

dTree.predict(testData.ix[:, 0:6])
dTree.score(testData.ix[:, 0:6], testData.index)


I am working on the file "2016/decision-tree/DecisionTree.ipynb".
The first task is:
Faremos a separação dos dados em conjunto de treino e teste
Can you write Python code for it?

size = len(data)
trainSize = int(math.floor(size * 0.7))
trainData = data[:trainSize]
testData = data[trainSize:]


I am working on the file "2016/decision-tree/DecisionTree.ipynb".
The first task is:
No código acima, faremos a leitura do arquivo, informando que não há cabeçário (obrigatório) e a nossa coluna 6 (0-6) representa a label
Can you write Python code for it?


I am working on the file "2016/decision-tree/DecisionTree.ipynb".
The first task is:
O problema é que nossos dados categóricos são strings. Então precisamos converter em representantes numéricos para aplicarmos no algoritmo
Can you write Python code for it?

for h in headers:
    data[h] = data[h].astype('category')
    data[h] = data[h].cat.codes

data.set_index("class", inplace=True)
data.head()


I am working on the file "2016/ferran/day10.ipynb".
The first task is:
Since we had +1 the Bot labels for convenience, the sought answer is 73.
Challenge 10.2
Can you write Python code for it?

handle_dict = parse_bot_commands(myinput)
community = BotSwarm(210)
community.initialize(handle_dict)
while (community.output[0] is None) or (community.output[1] is None) or (community.output[2] is None):
    community.update(handle_dict)
print(community.output[:3])
community.output[0]*community.output[1]*community.output[2]


I am working on the file "2016/jordi/Day 1 - No Time for a Taxicab.ipynb".
The first task is:
Process input
Can you write Python code for it?

with open("inputs/day1a.txt") as fd:
    print("Easter Bunny HQ is {} blocks away".format(blocksaway(fd.read())))


I am working on the file "2016/jordi/Day 1 - No Time for a Taxicab.ipynb".
The first task is:
Test examples
Can you write Python code for it?

blocksaway("R2, L3")

blocksaway("R2, R2, R2")

blocksaway("R5, L5, R5, R3")


I am working on the file "2016/knn/knn.ipynb".
The first task is:
Agora vamos fazer o mesmo para um knn que utiliza pesos relativos a distâncias:
Can you write Python code for it?

wknn3 = KNeighborsClassifier(n_neighbors=3,weights='distance')
wknn3.fit(trfeatures, trlabels)
wknn3.score(ttfeatures,ttlabels)


I am working on the file "2016/knn/knn.ipynb".
The first task is:
Agora vamos separar os conjuntos de treino e teste
Can you write Python code for it?

ndata = np.random.permutation(data)

size = len(ndata)
nt = int(math.floor(size*0.7))
trfeatures = ndata[0:nt,0:3]
ttfeatures = ndata[nt:size,0:3]
trlabels = ndata[0:nt,3]
ttlabels = ndata[nt:size,3]


I am working on the file "2016/knn/knn.ipynb".
The first task is:
Com os nossos dados separados podemos criar um classificador e treiná-lo
Can you write Python code for it?

knn3 = KNeighborsClassifier(n_neighbors=3)
knn3.fit(trfeatures, trlabels)


I am working on the file "2016/knn/knn.ipynb".
The first task is:
E avaliá-lo usando score:
Can you write Python code for it?


I am working on the file "2016/knn/knn.ipynb".
The first task is:
Prática
Verifique no intervalo de k = 1 a 10, qual o melhor valor de k e o melhor tipo de pesso a ser utilizado
Can you write Python code for it?

print("K \t Uniform \t Distance")
for k in range(1,11):
    UniformKnnClassifier = KNeighborsClassifier(n_neighbors=k,weights='uniform')
    UniformKnnClassifier.fit(trfeatures, trlabels)
    uScore = UniformKnnClassifier.score(ttfeatures, ttlabels)

    DistanceKnnClassifier = KNeighborsClassifier(n_neighbors=k,weights='distance')
    DistanceKnnClassifier.fit(trfeatures, trlabels)
    dScore = DistanceKnnClassifier.score(ttfeatures, ttlabels)

    print k,"\t{:f} \t{:f}".format(uScore,dScore)


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
(Pseudo) set operations
Can you write Python code for it?

oneRDD = sc.parallelize([1, 1, 1, 2, 3, 3, 4, 4])
oneRDD.persist()
otherRDD = sc.parallelize([1, 4, 4, 7])
otherRDD.persist()

oneRDD.union(otherRDD).collect()

oneRDD.subtract(otherRDD).collect()

oneRDD.distinct().collect()

oneRDD.intersection(otherRDD).collect() # removes duplicates

oneRDD.cartesian(otherRDD).collect()[:5]


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
(inner) join
Can you write Python code for it?

course_a = sc.parallelize([ ("Antti", 8), ("Tuukka", 10), ("Leena", 9)])
course_b = sc.parallelize([ ("Leena", 10), ("Tuukka", 10)])

result = course_a.join(course_b)
result.collect()


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
Be careful, though: if the function that you pass as argument to an RDD operation
* is an object method, or
* references an object field,
then Spark will ship the entire object to the cluster nodes along with the function.
This is demonstrated in the piece of code below.
Can you write Python code for it?

class SearchFunctions(object):
    def __init__(self, query):
        self.query

    def is_match(self, s):
        return self.query in s

    def get_matches_in_rdd_v1(self, rdd):
        return rdd.filter(self.is_match) # the function is an object method

    def get_matches_in_rdd_v2(self, rdd):
        return rdd.filter(lambda x: self.query in x) # the function references an object field


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
Creating RDDS
We saw that we can create RDDs by loading files from disk. We can also create RDDs from Python collections or transforming other RDDs.
Can you write Python code for it?

data = sc.parallelize([0,1,2,3,4,5,6,7,8,9]) # create RDD from Python collection

data_squared = data.map(lambda num: num ** 2) # transformation


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
First examples
Can you write Python code for it?

text = sc.textFile("myfile.txt") # load data
text.count() # count lines

text = sc.textFile("myfile.txt") # load data

# count only lines that mention "Spark"
spark_lines = text.filter(lambda line: 'Spark' in line)
spark_lines.count() # count lines


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
How long does it take to collect primesRDD? Let's time the operation.
Can you write Python code for it?


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
If we do not need primesRDD in memory anymore, we can tell Spark to discard it.
Can you write Python code for it?


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
Lazy evaluation
RDDs are evaluated lazily. This means that Spark will not materialize an RDD until it has to perform an action. In the example below, primesRDD is not evaluated until action collect() is performed on it.
Can you write Python code for it?

def is_prime(num):
    """ return True if num is prime, False otherwise"""
    if num < 1 or num % 1 != 0:
        raise Exception("invalid argument")
    for d in range(2, int(np.sqrt(num) + 1)):
        if num % d == 0:
            return False
    return True

numbersRDD = sc.parallelize(range(1, 1000000)) # creation of RDD
primesRDD = numbersRDD.filter(is_prime) # transformation

# primesRDD has not been materialized until this point

primes = primesRDD.collect() # action
print(primes[:15]) # this code does not involve Spark computation


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
The following is a better way to implement the two methods above (get_matches_in_rdd_v1 and get_matches_in_rdd_v2), if we want to avoid sending a SearchFunctions object for computation to the cluster.
Can you write Python code for it?

class SearchFunctions(object):
    def __init__(self, query):
        self.query

    def is_match(self, s):
        return self.query in s

    def get_matches_in_rdd(self, rdd):
        query = self.query
        return rdd.filter(lambda x: query in x)


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
Warning
In the example above, we update the value of an accumulator within a transformation (map). This is not recommended, unless for debugging purposes! The reason is that, if there are failures during the materialization of llengthRDD, some of its partitions will be re-computed, possibly causing the accumulator to double-count some the the long lines.
It is advisable to use accumulators within actions - and particularly with the foreach action, as demonstrated below.
Can you write Python code for it?

text = sc.textFile("myfile.txt")
long_lines = sc.accumulator(0)

def line_len(line):
    global long_lines
    length = len(line)
    if length > 30:
        long_lines += 1

text.foreach(line_len)
long_lines.value


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
When I ran the above on my laptop, it took about more than 10s. That's because Spark had to evaluate primesRDD before performing collect on it.
How long would it take if primesRDD was already in memory?
Can you write Python code for it?

primesRDD.persist()

%%timeit
primes = primesRDD.collect()


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
aggregate
Can you write Python code for it?

data = sc.parallelize([1,43,62,23,52])
aggr = data.aggregate(zeroValue = (0,0),
                      seqOp = (lambda x, y: (x[0] + y, x[1] + 1)),
                      combOp = (lambda x, y: (x[0] + y[0], x[1] + y[1])))
aggr[0] / aggr[1] # average value of RDD elements


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
combineByKey
Can you write Python code for it?

pairRDD = sc.parallelize([ ('$APPL', 100.64), ('$GOOG', 706.2), ('$AMZN', 552.32), ('$APPL', 100.52), ('$AMZN', 552.32) ])

aggr = pairRDD.combineByKey(createCombiner = lambda x: (x, 1),
                           mergeValue = lambda x, y: (x[0] + y, x[1] + 1),
                           mergeCombiners = lambda x, y: (x[0] + y[0], x[1] + y[1]))

aggr.map(lambda x: (x[0], x[1][0]/x[1][1])).collect() # average value per key


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
map and flatmap
Can you write Python code for it?

phrases = sc.parallelize(["hello world", "how are you", "how do you do"])

words = phrases.flatMap(lambda phrase: phrase.split(" "))

words.collect()

phrases = sc.parallelize(["hello world", "how are you", "how do you do"])

words = phrases.map(lambda phrase: phrase.split(" "))

words.collect()


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
reduce
Can you write Python code for it?

data = sc.parallelize([1,43,62,23,52])
data.reduce(lambda x, y: x + y)

data.reduce(lambda x, y: x * y)

data.reduce(lambda x, y: x**2 + y**2) # this does NOT compute the sum of squares of RDD elements

data.reduce(lambda x, y: np.sqrt(x**2 + y**2)) ** 2


I am working on the file "2016/lecture_7_spark.ipynb".
The first task is:
reduceByKey
Can you write Python code for it?

pairRDD = sc.parallelize([ ('$APPL', 100.64), ('$GOOG', 706.2), ('$AMZN', 552.32), ('$APPL', 100.52), ('$AMZN', 552.32) ])

pairRDD.reduceByKey(lambda x, y: "x + y").collect() # sum of values per key


I am working on the file "2016/logistic-regression/plot_iris_logistic.ipynb".
The first task is:
Logistic Regression 3-class Classifier
Show below is a logistic-regression classifiers decision boundaries on the
iris <https://en.wikipedia.org/wiki/Iris_flower_data_set>_ dataset. The
datapoints are colored according to their labels.
Can you write Python code for it?

print(__doc__)

# Code source: Gaël Varoquaux
# Modified for documentation by Jaques Grobler
# License: BSD 3 clause

import numpy as np
import matplotlib.pyplot as plt
from sklearn import linear_model, datasets

# import some data to play with
iris = datasets.load_iris()
X = iris.data[:, :2]  # we only take the first two features.
Y = iris.target

h = .02  # step size in the mesh

logreg = linear_model.LogisticRegression()

# we create an instance of Neighbours Classifier and fit the data.
logreg.fit(X, Y)

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, x_max]x[y_min, y_max].
x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
Z = logreg.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1, figsize=(4, 3))
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)

# Plot also the training points
plt.scatter(X[:, 0], X[:, 1], c=Y, edgecolors='k', cmap=plt.cm.Paired)
plt.xlabel('Sepal length')
plt.ylabel('Sepal width')

plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
plt.xticks(())
plt.yticks(())

plt.show()


I am working on the file "2016/logistic-regression/plot_iris_logistic.ipynb".
The first task is:
Utilize o classificador logistico na dataset haberman.txt (o mesmo utilizado na aula do knn) usando todos os atributos. Não esqueça de separar os dados em treino e teste

Implemente as funções precision e recall para avaliar os resultados
Can you write Python code for it?

import numpy as np
from sklearn import linear_model

data = np.loadtxt('haberman.data',delimiter=',')
#vamos embaralhar os dados antes de dividir treino e teste
rdata = np.random.permutation(data)
l,c = rdata.shape
X = rdata[:,:c-1]
y = rdata[:,c-1]

nt = int(l * 0.8)
X_train = X[:nt,:]
X_test = X[nt:,:]
y_train = y[:nt]
y_test = y[nt:]

logreg = linear_model.LogisticRegression(C=1e5)
logreg.fit(X_train,y_train)
logreg.score(X_test,y_test)

#Na aula do dia 05-10-16 implementamos uma função, vamos alterá-la para
#podermos calcular os valores por classe
def confusionMatrix(predicted, actual, class_value):
    if len(predicted) != len(actual): return -1
    tp = 0.0
    fp = 0.0
    tn = 0.0
    fn = 0.0
    for i in range(len(actual)):
        if actual[i] == class_value: #labels that are equal to the actual class  (positive examples)
            if predicted[i] == class_value:
                tp += 1.0 #correctly predicted positive
            else:
                fn += 1.0 #incorrectly predicted negative
        else:              #labels that are different than the actual class (negative examples)
            if predicted[i] != class_value:
                tn += 1.0 #correctly predicted negative
            else:
                fp += 1.0 #incorrectly predicted positive
    rtn = [tp, fn, fp, tn]
    return rtn

def precision(tp,fp):
    return tp/(tp+fp)

def recall (tp,fn):
    return tp/(tp+fn)

yb = logreg.predict(X_test)
#vamos verificar os valores para as 2 classes
tp, fn, fp, tn = confusionMatrix(yb,y_test,1.0) #Classe 1
print tp, fn, fp, tn
print "Precision: " + str(precision(tp,fp))
print "Recall: " + str(recall(tp,fn))

tp, fn, fp, tn = confusionMatrix(yb,y_test,2.0) #Classe 2
print tp, fn, fp, tn
print "Precision: " + str(precision(tp,fp))
print "Recall: " + str(recall(tp,fn))


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Alternatively, you can import the dataset by running the same command on a terminal.

mongoimport --db moderndb --collection restaurants --drop --file dataset.json

The dataset contains documents that look like the one below.
 Restaurant Example
{
  "address": {
     "building": "1007",
     "coord": [ -73.856077, 40.848447 ],
     "street": "Morris Park Ave",
     "zipcode": "10462"
  },
  "borough": "Bronx",
  "cuisine": "Bakery",
  "grades": [
     { "date": { "$date": 1393804800000 }, "grade": "A", "score": 2 },
     { "date": { "$date": 1378857600000 }, "grade": "A", "score": 6 },
     { "date": { "$date": 1358985600000 }, "grade": "A", "score": 10 },
     { "date": { "$date": 1322006400000 }, "grade": "A", "score": 9 },
     { "date": { "$date": 1299715200000 }, "grade": "B", "score": 14 }
  ],
  "name": "Morris Park Bake Shop",
  "restaurant_id": "30075445"
}
Can you write Python code for it?

restaurants = db.restaurants # our new collection

# how many restaurants?
restaurants.count()


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
As we see in this example, MongoDB makes use of an index (as indicated by keyword "IXSCAN") -- and particularly the index ('borough_1') we constructed to execute the query.
What if we had not built this index?
Can you write Python code for it?

restaurants.drop_indexes() # we drop all indexes first -- use this with care!
my_cursor = restaurants.find({"borough": "brooklyn"}) # submit query to find restaurants from specific borough
my_cursor.explain()["queryPlanner"]["winningPlan"] # ask mongodb to explain execution plan


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
If we call the collection's function find(), we get back a cursor.
Can you write Python code for it?


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
In that case, MongoDB simply performs a scan over the collection (as indicated by keyword "COLLSCAN").
Joins
Until very recently, MongoDB did not support joins.
It was up to the user to implement a join if needed.
For example, the following double for-loop emulated "page-oriented nested-loops join".
Can you write Python code for it?

for a in restaurants.find({"borough": "Manhattan"}).limit(7):
    for b in restaurants.find({"borough": "Bronx"}).limit(5):
        if a["cuisine"] == b["cuisine"]:
            print(a["cuisine"], a["address"]["zipcode"], b["address"]["zipcode"])


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Indexing
MongoDb automatically creates an index on the _id field upon creating a collection.
We can use create_index() to create index on one or more fields of a collection.
Single-field index
Can you write Python code for it?

# note that the argument is a list of tuples
# [(<field>: <type>), ...]
# here, we specify only one such tuple for one field
restaurants.create_index([("borough", pm.ASCENDING)])


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Inserting and finding documents
Our collection coll is currently empty. Let's add one document to it.
Can you write Python code for it?


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Joins with \$lookup
This is a new aggregation stage that implements left outer equi-joins.
"A left outer equi-join produces a result set that contains data for all documents from the left table (collection) together with data from the right table (collection) for documents where there is a match with documents from the left table (collection)."
Can you write Python code for it?

# create first collection
orders_docs = [{ "_id" : 1, "item" : "abc", "price" : 12, "quantity" : 2 },
{ "_id" : 2, "item" : "jkl", "price" : 20, "quantity" : 1 },
{ "_id" : 3  }]
orders = db.orders
orders.drop()
orders.insert_many(orders_docs)

# create second collection
inventory_docs = [
    { "_id" : 1, "item" : "abc", "description": "product 1", "instock" : 120 },
{ "_id" : 2, "item" : "def", "description": "product 2", "instock" : 80 },
{ "_id" : 3, "item" : "ijk", "description": "product 3", "instock" : 60 },
{ "_id" : 4, "item" : "jkl", "description": "product 4", "instock" : 70 },
{ "_id" : 5, "item": None, "description": "Incomplete" },
{ "_id" : 6 }
]
inventory = db.inventory
inventory.drop()
inventory.insert_many(inventory_docs)

result = orders.aggregate([ # "orders" is the outer collection
    {
      "$lookup":
        {
          "from": "inventory", # the inner collection
          "localField": "item", # the join field of the outer collection
          "foreignField": "item", # the join field of the outer collection
          "as": "inventory_docs" # name of field with array of joined inner docs
        }
   }
])

print_my_docs(result, 10)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Loading a larger dataset
Download file primer-dataset.json and load it into mongodb by running the command below.
Can you write Python code for it?

%%bash
mongoimport --db moderndb --collection restaurants --drop --file primer-dataset.json


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Multi-key index
An index for a fields with array value.
Can you write Python code for it?

restaurants.find_one()

restaurants.create_index([("address.coord", 1)])

restaurants.create_index([("grades.score", 1)])

restaurants.create_index([("grades.grade", 1), ("grades.score", 1)])


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Multiple Conditions
Can you write Python code for it?

# logical AND
cursor = restaurants.find({"cuisine": "Italian", "address.zipcode": "10075"})

next(cursor)

# logical OR
cursor = restaurants.find({"$or": [{"cuisine": "Italian"},
                                          {"address.zipcode": "10075"}]})

print_my_docs(cursor, 3)

# logical AND, differently
cursor = restaurants.find({"$and": [{"cuisine": "Italian"},
                                          {"address.zipcode": "10075"}]})

next(cursor)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Note on the semantics of the \$not operator
The operator selects documents that do not match the specified condition on the specified field. These documents include ones that do not contain the field.
To demonstrate this, consider the following simple example of a collection.
Can you write Python code for it?

# simple example of a collection
mycoll = db.mycoll
mycoll.drop()

# insert three documents
mycoll.insert_one({"grades": [7, 7]})
mycoll.insert_one({"grades": [7, 3]})
mycoll.insert_one({"grades": [3, 3]})
mycoll.insert_one({"grades": []})
mycoll.insert_one({})


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Projecting fields
We can use find() not only to retrieve documents that match a condition, but also to project only those fields that we are interested in.
For example, to suppress the "_id" field from appearing in the results, we can provide a second argument to find(), as follows.
Can you write Python code for it?

query_result = coll.find({"name": "Michael"}, {"_id": 0})
for doc in query_result:
    print(doc)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Querying the Dataset
Can you write Python code for it?

# retrieve a cursor over all documents in the collection
cursor = restaurants.find()

# define printing function
def print_my_docs(cursor, num):
    for i in range(num): # print only up to num next documents from cursor
        try:
            print(next(cursor))
            print()
        except:
            break

# let's print a few documents
print_my_docs(cursor, 3)

next(cursor) # get one more document


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Question: Does MongoDB optimize the stages of an aggregation pipeline?
The question was asked in relation to the "limit" query we saw above ("Limiting the number of results").
Indeed, MongoDB does optimize the execution of the aggregation pipeline, as explained here. In relation to the aforementioned query, see, in particular, the part on sort+limit coalescence.

Credits and references
We used and consulted material from:
* the offficial PyMongo tutorial as well as this shorter one,
* the JSON and BSON documentation, as well as SON,
* these posts on the MongoDB blog about the new (v.3.2) left outer equi-join functionality,
* this StackOverflow thread.

Updates to the notebook

Compared to the in-class tutorial, we've added answers to two in-class questions.
[Feb 26] Added sub-sections projecting fields, storing the result as a collection, and  retrieving the execution plan.

Setup
Run the following commands before the rest of the notebook.
Can you write Python code for it?

import pymongo as pm
client = pm.MongoClient()
client.drop_database("moderndb")

import bson.son as son


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Retrieving the execution plan
We can retrieve the execution plan for a find() query by calling the explain() function on the result cursor. We demonstrate this in the following example.
Can you write Python code for it?

restaurants.drop_indexes() # we drop all indexes first -- use this with care!
restaurants.create_index([("borough", pm.ASCENDING)]) # build an index on field "borough", in ascending order
my_cursor = restaurants.find({"borough": "brooklyn"}) # submit query to find restaurants from specific borough
my_cursor.explain()["queryPlanner"]["winningPlan"] # ask mongodb to explain execution plan


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
SQL Query
SELECT zipcode, cuisine, COUNT(*) as count
FROM restaurants
WHERE borough = "Manhattan"
GROUP BY zipcode, cuisine
HAVING COUNT(*) > 3
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan"}},
        {"$group": {"_id": {"zipcode": "$address.zipcode", "cuisine": "$cuisine"}, "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 3}}}
    ]
)

print_my_docs(cursor, 5)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
SQL Query
SELECT zipcode, cuisine, COUNT(*) as count
FROM restaurants
WHERE borough = "Manhattan"
GROUP BY zipcode, cuisine
HAVING COUNT(*) > 3
ORDER BY count
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$match": {"borough": "Manhattan"}},
        {"$group": {"_id": {"zipcode": "$address.zipcode", "cuisine": "$cuisine"}, "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 3}}},
        {"$sort": {"count": 1}}
    ]
)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
SQL query
SELECT borough, cuisine, COUNT(*) as count
FROM restaurants
GROUP BY borough, cuisine
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": {"borough": "$borough", "cuisine": "$cuisine"}, "count": {"$sum": 1}}}
    ]
)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
SQL query
SELECT borough, cuisine, COUNT(*) as count
FROM restaurants
GROUP BY borough, cuisine
HAVING COUNT(*) > 3
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": {"borough": "$borough", "cuisine": "$cuisine"}, "count": {"$sum": 1}}},
        {"$match": {"count": {"$gt": 3}}}
    ]
)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
SQL to Aggregation
Here we explore the correspondence between SQL queries and the aggregation framework.
 SQL query
SELECT COUNT(*) AS count
FROM restaurants
Can you write Python code for it?

cursor = restaurants.aggregate(
    [
        {"$group": {"_id": None, "count": {"$sum": 1}} }
    ]
)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Sorting
Can you write Python code for it?

cursor = restaurants.find()

# to sort, specify list of sorting criteria,
# each criterion given as a tuple
# (field_name, sort_order)
# here we have only one
sorted_cursor = cursor.sort([("borough", pm.ASCENDING)])

print_my_docs(cursor, 2)

another_sorted_cursor = restaurants.find().sort([("borough", pm.ASCENDING),
                                     ("address.zipcode", pm.DESCENDING)])

print_my_docs(another_sorted_cursor, 3)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Specify Range Conditions
Can you write Python code for it?

cursor = restaurants.find({"grades.score": {"$gt": 30}})

cursor = restaurants.find({"grades.score": {"$lt": 10}})

next(cursor)["grades"]


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
Specify equality conditions
Can you write Python code for it?

# top-level field
cursor = restaurants.find({"borough": "Manhattan"})

print_my_docs(cursor, 2)

# nested field (in embedded document)
cursor = restaurants.find({"address.zipcode": "10075"})

print_my_docs(cursor, 2)

# query by field in array
cursor = restaurants.find({"grades.grade": "B"})

# print one document from the query result
next(cursor)['grades']

# exact array match
cursor = restaurants.find({"address.coord": [-73.98513559999999, 40.7676919]})

print_my_docs(cursor, 10)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
The following will not work!
We cannot currently have compound multi-key indexes.
Can you write Python code for it?

restaurants.create_index([("address.coord", 1), ("grades.score", 1)]) # NOPE!


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
We can remove such documents from the result as a post-processing step. (Exercise: how?)
Second approach: aggregation pipeline
Can you write Python code for it?

# using aggregation
mycursor = restaurants.aggregate(
    [
        # unwind the grades array
        {"$unwind": "$grades"}, #now each document contains one "grades" value

        # group by document "_id" and count:
        # (i) the total number of documents in each group as `count`
        # -- this is the same as the number of elements in the original array
        # (ii) the number of documents that satisfy the condition (grade = "A") as `num_satisfied`
        {"$group": {"_id": "$_id", "count": {"$sum": 1}, "num_satisfied": {"$sum": {"$cond": [{"$eq": ["$grades.grade", "A"]}, 1, 0]}}}},

        # create a field (named `same`) that is 1 if (count = num_satisfied) and 0 otherwise
        {"$project": {"_id": 1, "same_count": {"$cond": [{"$eq": ["$count", "$num_satisfied"]} , 1, 0]}}},

        # keep only the document ids for which (same = 1)
        {"$match": {"same_count": 1}}
    ]
)

print_my_docs(mycursor, 5)


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
We can use the cursor to iterate over all documents in the collection.
Can you write Python code for it?


I am working on the file "2016/mongodb.tutorial.ipynb".
The first task is:
What if we're interested in keeping only some of the rest of the fields -- let's say, only "grades"?
Can you write Python code for it?

query_result = coll.find({"name": "Michael"}, {"_id": 0, "grades": 1})
for doc in query_result:
    print(doc)


I am working on the file "2016/naive-bayes/NaiveBayes.ipynb".
The first task is:
Modelo
Como o modelo é representado basicamente pela frequência das palavras, precisamos categorizar os possíveis valores das features. Após esse processo, fazemos a contagem.

countFrequencies: faz a contagem que cada valor de feature e label aparecem em todo o dataset de treino, independentemente.
countCondFrequencies: faz a contagem que cada valor de feature aparece para cada possível label.
Can you write Python code for it?

def countFrequencies(self)

def countCondFrequencies(self)


I am working on the file "2016/naive-bayes/NaiveBayes.ipynb".
The first task is:
Treino
Vamos treinar o nosso modelos. O que deve ser composto na função de treino?
Can you write Python code for it?


I am working on the file "2016/network-analysis/(Response) Centrality.ipynb".
The first task is:
Agora você deve escrever a sua função que computa a betweenness centrality para um grafo nx:

Dica: recomendo fortemente o uso da all_shortest_paths da Networkx
Requisito: Não vale usar a função de betweenness centrality do networkx
Can you write Python code for it?

# ALGORITHM: Betweenness Centrality
# INPUT: G(V, E)
# OUTPUT: Dictionary(V, Weights)
#
# BEGIN
#   result <- init_empty_betweenness(V)
#
#   FOR EACH vi, vj in V THEN
#
#     IF vi < vj THEN
#
#       shortest_paths <- get_shortest_paths(G, vi, vj)
#
#       contribuition <- 1 / SIZE_OF(shortest_paths)
#
#       FOR EACH path IN shortest_paths THEN
#
#         FOR EACH pi IN path IF pi != vi AND pi != pj
#
#           result[pi] += contribuition
#
# RETURN result
# END

def betweenness_centrality(nx_graph):
    """ Return the weight dictionary {node: betweenness centrality} """

    nodes = nx_graph.nodes()
    edges = nx_graph.edges()
    btwns_dict = { node: 0.0 for node in nodes }

    for node in nodes:
        for other in nodes:
            if (node < other): # don't double count
                paths = [p for p in nx.all_shortest_paths(nx_graph, node, other)]
                contrib = 1.0 / len(paths)
                for path in paths:
                    for n in path:
                        if n not in [node, other]:
                            btwns_dict[n] += contrib

    return btwns_dict


I am working on the file "2016/network-analysis/Centrality.ipynb".
The first task is:
Agora que temos nossa função de centralidade pronta, podemos obter a centralidades para os dois gráficos:
Can you write Python code for it?

facebook_graph = nau.facebook_nx_graph()
random_graph = nau.random_nx_grap()

facebook_btwns = betweenness_centrality(facebook_graph)
random_btwns = betweenness_centrality(random_graph)


I am working on the file "2016/network-analysis/Centrality.ipynb".
The first task is:
Agora você deve escrever a sua função que computa a betweenness centrality para um grafo nx:

Dica: recomendo fortemente o uso da all_shortest_paths da Networkx
Requisito: Não é permitido o uso da função de betweenness centrality da Networkx
Can you write Python code for it?

# ALGORITHM: Betweenness Centrality
# INPUT: G(V, E)
# OUTPUT: Dictionary(V, Weights)
#
# BEGIN
#   result <- init_empty_betweenness(V)
#
#   FOR EACH vi, vj in V THEN
#
#     IF vi < vj THEN
#
#       shortest_paths <- get_shortest_paths(G, vi, vj)
#
#       contribuition <- 1 / SIZE_OF(shortest_paths)
#
#       FOR EACH path IN shortest_paths THEN
#
#         FOR EACH pi IN path IF pi != vi AND pi != pj
#
#           result[pi] += contribuition
#
# RETURN result
# END

def betweenness_centrality(nx_graph):
    """ Return the weight dictionary {node: betweenness centrality} """
    return {}


I am working on the file "2016/network-analysis/Centrality.ipynb".
The first task is:
E agora geraremos os gráficos dos grafos para análise visual:
Can you write Python code for it?

nau.write_btwns_graph(facebook_graph, facebook_btwns, 'facebook_btwns_graph.png')
nau.write_btwns_graph(random_graph, random_btwns, 'random_btwns_graph.png')


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
A VectorPoint is a point with a direction
I use them to plot the point on the graph, and also to represent the
direction of movement.
Can you write Python code for it?

VectorPoint = namedtuple('VectorPoint', ('x', 'y'))


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
Paths will save all visited paths, but we do not need the constraint that they must be unique, since we are checking for intersecting points.
Can you write Python code for it?


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
The final distance from origin is the distance of x and y axis.
Can you write Python code for it?

distance = abs(point.x) + abs(point.y)
# print(distance)


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
The point to be tracked, this is the position on graph.
Can you write Python code for it?


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
Tokenize the token into turn and number of blocks
The first letter represents the direction to turn to, and will only be 'L' for left, and 'R' for right.
The numbers after that are the number of blocks to walk.

Get mask based on direction to turn
The direction is the swapped direction, with the mask applied. So the previous (x,y) becomes (y,x) * mask(x,y).
The point is now the number of blocks added with the direction applied.
Can you write Python code for it?

for token in data:
    turn, blocks = token[:1], int(token[1:])

    if turn == turns.left:
        mask = mask_left
    if turn == turns.right:
        mask = mask_right
    #
    direction = VectorPoint(direction.y * mask.x, direction.x * mask.y)
    point = VectorPoint(
        point.x + blocks * direction.x,
        point.y + blocks * direction.y)