I am working on the file "11_realtime/evaluation.ipynb".
The first task is:
Categorical Features
Can you write Python code for it?

%%bigquery df2
SELECT
  ROUND(CAST(dep_delay AS FLOAT64), 0) AS dep_delay,
  AVG(IF(origin='JFK', predicted_ontime.scores[OFFSET(0)], NULL)) AS JFK,
  AVG(IF(origin='SEA', predicted_ontime.scores[OFFSET(0)], NULL)) AS SEA,
FROM dsongcp.ch10_automl_evaluated
GROUP BY dep_delay
ORDER BY dep_delay ASC

ax = df2.plot(x='dep_delay', y='JFK', xlim=[-10,100], style='r--');
df2.plot(x='dep_delay', y='SEA', xlim=[-10,100], ax=ax);

%%bigquery df2
SELECT
  carrier,
  ROUND(CAST(dep_delay AS FLOAT64), 0) AS dep_delay,
  AVG(predicted_ontime.scores[OFFSET(0)]) AS prob_ontime
FROM dsongcp.ch10_automl_evaluated
GROUP BY carrier, dep_delay
ORDER BY carrier ASC, dep_delay ASC

df = df2.copy()
df.head()

df = df2.set_index('dep_delay')
df.head()

dfg = df2[df2['dep_delay'] == 20.0].sort_values(by='prob_ontime').reset_index(drop=True)
dfg

print(dfg.loc[0]['carrier'])
print(dfg.loc[0]['prob_ontime'])

fig, ax = plt.subplots(figsize=(15,15))
df.groupby('carrier')['prob_ontime'].plot(xlim=[-10,60], legend=True, ax=ax, lw=4);
ax.annotate(dfg.loc[0]['carrier'],
            xy=(20.0, dfg.loc[0]['prob_ontime']),
            xycoords='data', xytext=(-50, -30),
            textcoords='offset points', arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=-0.2'));
ax.annotate(dfg.loc[4]['carrier'],
            xy=(20.0, dfg.loc[4]['prob_ontime']),
            xycoords='data', xytext=(-80, -60),
            textcoords='offset points', arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=-0.2'));
ax.annotate(dfg.loc[11]['carrier'],
            xy=(20.0, dfg.loc[11]['prob_ontime']),
            xycoords='data', xytext=(50, 60),
            textcoords='offset points', arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=-0.2'));

ax.annotate(dfg.loc[16]['carrier'],
            xy=(20.0, dfg.loc[16]['prob_ontime']),
            xycoords='data', xytext=(50, 30),
            textcoords='offset points', arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=-0.2'));


I am working on the file "11_realtime/evaluation.ipynb".
The first task is:
Confusion matrix
Let's find the fraction of true on-time flights at some threshold:
Can you write Python code for it?

%%bigquery
SELECT
  *,
  num_pred_ontime / num_ontime AS frac_1_as_1,
  num_pred_late   / num_ontime AS frac_1_as_0,
  num_pred_ontime / num_late AS frac_0_as_1,
  num_pred_late   / num_late AS frac_0_as_0
FROM (
    SELECT
      0.7 AS thresh,
      SUM(IF(CAST(ontime AS FLOAT64) > 0.5, 1, 0)) AS num_ontime,
      SUM(IF(CAST(ontime AS FLOAT64) <= 0.5, 1, 0)) AS num_late,
      SUM(IF(predicted_ontime.scores[OFFSET(0)] > 0.7, 1, 0)) AS num_pred_ontime,
      SUM(IF(predicted_ontime.scores[OFFSET(0)] <= 0.7, 1, 0)) AS num_pred_late,
    FROM dsongcp.ch10_automl_evaluated
)

%%bigquery
WITH counts AS (
    SELECT
      thresh,
      COUNTIF(CAST(ontime AS FLOAT64) > 0.5 AND predicted_ontime.scores[OFFSET(0)] > thresh) AS num_1_as_1,
      COUNTIF(CAST(ontime AS FLOAT64) > 0.5 AND predicted_ontime.scores[OFFSET(0)] <= thresh) AS num_1_as_0,
      COUNTIF(CAST(ontime AS FLOAT64) <= 0.5 AND predicted_ontime.scores[OFFSET(0)] > thresh) AS num_0_as_1,
      COUNTIF(CAST(ontime AS FLOAT64) <= 0.5 AND predicted_ontime.scores[OFFSET(0)] <= thresh) AS num_0_as_0
    FROM UNNEST([0.5, 0.7, 0.8]) AS thresh, dsongcp.ch10_automl_evaluated
    GROUP BY thresh
)

SELECT
  *,
  ROUND(num_1_as_1 / (num_1_as_1 + num_1_as_0), 2) AS frac_1_as_1,
  ROUND(num_1_as_0 / (num_1_as_1 + num_1_as_0), 2) AS frac_1_as_0,
  ROUND(num_0_as_1 / (num_0_as_1 + num_0_as_0), 2) AS frac_0_as_1,
  ROUND(num_0_as_0 / (num_0_as_1 + num_0_as_0), 2) AS frac_0_as_0
FROM counts
ORDER BY thresh ASC

%%bigquery df
WITH counts AS (
    SELECT
      thresh,
      COUNTIF(CAST(ontime AS FLOAT64) > 0.5 AND predicted_ontime.scores[OFFSET(0)] > thresh) AS num_1_as_1,
      COUNTIF(CAST(ontime AS FLOAT64) > 0.5 AND predicted_ontime.scores[OFFSET(0)] <= thresh) AS num_1_as_0,
      COUNTIF(CAST(ontime AS FLOAT64) <= 0.5 AND predicted_ontime.scores[OFFSET(0)] > thresh) AS num_0_as_1,
      COUNTIF(CAST(ontime AS FLOAT64) <= 0.5 AND predicted_ontime.scores[OFFSET(0)] <= thresh) AS num_0_as_0
    FROM UNNEST(GENERATE_ARRAY(0.0, 1.0, 0.01)) AS thresh,...


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Build the graph
Here, we'll build the graph. First up, defining the hyperparameters.

lstm_size: Number of units in the hidden layers in the LSTM cells. Usually larger is better performance wise. Common values are 128, 256, 512, etc.
lstm_layers: Number of LSTM layers in the network. I'd start with 1, then add more if I'm underfitting.
batch_size: The number of reviews to feed the network in one training pass. Typically this should be set as high as you can go without running out of memory.
learning_rate: Learning rate
Can you write Python code for it?

lstm_size = 256
lstm_layers = 1
batch_size = 500
learning_rate = 0.001


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Create an array features that contains the data we'll pass to the network. The data should come from review_ints, since we want to feed integers to the network. Each row should be 200 elements long.
For reviews shorter than 200 words, left pad with 0s. That is, if the review is ['best', 'movie', 'ever'], [117, 18, 128] as integers, the row will look like [0, 0, 0, ..., 0, 117, 18, 128].
For reviews longer than 200, use on the first 200 words as the feature vector.
Can you write Python code for it?

seq_len = 200
features = np.zeros([len(reviews_ints), seq_len], dtype=int)
for i, review in enumerate(reviews_ints):
    if len(review) < seq_len:
        features[i, (seq_len-len(review)):] = review
    else:
        features[i, :] = review[:seq_len]


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Data preprocessing
The first step when building a neural network model is getting your data into the proper form to feed into the network. Since we're using embedding layers, we'll need to encode each word with an integer. We'll also want to clean it up a bit.
You can see an example of the reviews data above. We'll want to get rid of those periods. Also, you might notice that the reviews are delimited with newlines \n. To deal with those, I'm going to split the text into each review using \n as the delimiter. Then I can combined all the reviews back together into one big string.
First, let's remove all punctuation. Then get all the text without the newlines and split it into individual words.
Can you write Python code for it?

from string import punctuation
all_text = ''.join([c for c in reviews if c not in punctuation])
reviews = all_text.split('\n')

all_text = ' '.join(reviews)
words = all_text.split()

all_text[:2006]

len(words)


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Embedding
Now we'll add an embedding layer. We need to do this because there are 74000 words in our vocabulary. It is massively inefficient to one-hot encode our classes here. You should remember dealing with this problem from the word2vec lesson. Instead of one-hot encoding, we can have an embedding layer and use that layer as a lookup table. You could train an embedding layer using word2vec, then load it here. But, it's fine to just make a new layer and let the network learn the weights.
tf.nn.embedding_lookup function takes the embedding matrix and an input tensor, such as the review vectors. Then, it'll return another tensor with the embedded vectors. So, if the embedding layer has 200 units, the function will return a tensor with size [batch_size, 200].
Can you write Python code for it?

num_words = len(vocab_to_int)
num_words

# Size of the embedding vectors (number of units in the embedding layer)
embed_size = 300

with graph.as_default():
    # embedding weight matrix
    embedding = tf.Variable(tf.random_uniform([num_words, embed_size], -1, 1))
    # the lookup (instead of matmul)
    embed = tf.nn.embedding_lookup(embedding, inputs_)


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
For the network itself, we'll be passing in our 200 element long review vectors. Each batch will be batch_size vectors. We'll also be using dropout on the LSTM layer, so we'll make a placeholder for the keep probability.
Can you write Python code for it?

n_words = len(vocab_to_int) + 1 # Adding 1 because we use 0's for padding, dictionary started at 1

# Create the graph object
graph = tf.Graph()
# Add nodes to the graph
with graph.as_default():
    # first dimaeniton is batch size, and second is sequence length
    inputs_ = tf.placeholder(tf.int32, [None, None], name='inputs_')
    # `labels_` needs to be two-dimensional to work with some functions later.
    labels_ = tf.placeholder(tf.int32, [None, None], name='lables_')
    # `keep_prob` is a scalar (a 0-dimensional tensor), so no need for size argument
    keep_prob = tf.placeholder(tf.float32, name='keep_prob')


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
If you built labels correctly, you should see the next output.
Can you write Python code for it?

from collections import Counter
review_lens = Counter([len(x) for x in reviews_ints])
print("Zero-length reviews: {}".format(review_lens[0]))
print("Maximum review length: {}".format(max(review_lens)))


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Testing
Can you write Python code for it?

test_acc = []
with tf.Session(graph=graph) as sess:
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
    test_state = sess.run(cell.zero_state(batch_size, tf.float32))
    for ii, (x, y) in enumerate(get_batches(test_x, test_y, batch_size), 1):
        feed = {inputs_: x,
                labels_: y[:, None],
                keep_prob: 1,
                initial_state: test_state}
        batch_acc, test_state = sess.run([accuracy, final_state], feed_dict=feed)
        test_acc.append(batch_acc)
    print("Test accuracy: {:.3f}".format(np.mean(test_acc)))


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Training
Below is the typical training code. If you want to do this yourself, feel free to delete all this code and implement it yourself. Before you run this, make sure the checkpoints directory exists.
Can you write Python code for it?

epochs = 10

with graph.as_default():
    saver = tf.train.Saver()

with tf.Session(graph=graph) as sess:
    sess.run(tf.global_variables_initializer())
    iteration = 1
    for e in range(epochs):
        state = sess.run(initial_state)

        for ii, (x, y) in enumerate(get_batches(train_x, train_y, batch_size), 1):
            feed = {inputs_: x,
                    labels_: y[:, None],
                    keep_prob: 0.5,
                    initial_state: state}
            loss, state, _ = sess.run([cost, final_state, optimizer], feed_dict=feed)

            if iteration%5==0:
                print("Epoch: {}/{}".format(e, epochs),
                      "Iteration: {}".format(iteration),
                      "Train loss: {:.3f}".format(loss))

            if iteration%25==0:
                val_acc = []
                val_state = sess.run(cell.zero_state(batch_size, tf.float32))
                for x, y in get_batches(val_x, val_y, batch_size):
                    feed = {inputs_: x,
                            labels_: y[:, None],
                            keep_prob: 1,
                            initial_state: val_state}
                    batch_acc, val_state = sess.run([accuracy, final_state], feed_dict=feed)
                    val_acc.append(batch_acc)
                print("Val acc: {:.3f}".format(np.mean(val_acc)))
            iteration +=1
    saver.save(sess, "checkpoints/sentiment.ckpt")


I am working on the file "11_sentiment-rnn/Sentiment_RNN.ipynb".
The first task is:
Training, Validation, Test
Can you write Python code for it?

split_frac = 0.8
split = int(split_frac * len(features))

train_x, val_x = features[:split, :], features[split:, :]
train_y, val_y = labels[:split], labels[split:]

split_frac = 0.5
split = int(split_frac * len(val_x))

val_x, test_x = val_x[:split, :], val_x[split:, :]
val_y, test_y = val_y[:split], val_y[split:]

print("\t\t\tFeature Shapes:\t Label shapes:")
print("Train set: \t\t{}\t{}".format(train_x.shape, train_y.shape),
      "\nValidation set: \t{}\t{}".format(val_x.shape, val_y.shape),
      "\nTest set: \t\t{}\t{}".format(test_x.shape, test_y.shape))


I am working on the file "11a--nyc-whiners-followup.ipynb".
The first task is:
Solution 1
This solution first queries the database for the total number of complaints by type. It then uses these data to normalize the counts by city.
The main idea is to use queries to get data frames storing the values you need, and then combining them within Python.
Can you write Python code for it?

t1a = time.time ()

# Determine the number of complaints by type
query = '''
  SELECT ComplaintType, COUNT(*) AS NumComplaints
    FROM data
    GROUP BY ComplaintType COLLATE NOCASE
    ORDER BY -NumComplaints
'''
df = pd.read_sql_query (query, disk_engine)

t1a = time.time () - t1a
print "[+%gs] Part A" % t1a
print df.head ()

t1b = time.time ()
# Convert this data into a lookup table (dictionary)
total_complaints_by_type = \
  dict (zip ([x.capitalize () for x in df.ComplaintType],
             df.NumComplaints))
t1b = time.time () - t1b
print "[+%gs] Part B" % t1b

# Print a few entries just as a sanity check
print list (itertools.islice (total_complaints_by_type.items (), 5))

t1c = time.time ()

def capitalize (string_list):
    """
    Given a list of strings, returns a new list with standardized
    capitalization.
    """
    return [s.capitalize () for s in string_list]

def gather (key_list, dictionary):
    """
    Given a list of keys, returns a list of corresponding values from a
    dictionary.
    """
    return [dictionary[key] for key in key_list]

traces1 = []
for city in TOP_CITIES: # Determines the complaint counts by city
    print ("[+%gs] Processing %s ..." % (time.time () - t1c, city)) ; sys.stdout.flush ()

    query = '''
      SELECT ComplaintType, COUNT(*) as NumComplaints
        FROM data
        WHERE City = "{}" COLLATE NOCASE
        GROUP BY ComplaintType COLLATE NOCASE
        ORDER BY -NumComplaints
    '''.format (city)
    df = pd.read_sql_query (query, disk_engine)

    # Normalize complaint counts
    complaint_types = capitalize (df.ComplaintType)
    totals = gather (complaint_types, total_complaints_by_type)
    percent_complaints = 100.0 * df.NumComplaints / totals

    # Add this city as a new trace
    traces1.append (Bar (x=complaint_types,
                         y=percent_complaints,
                         name=city.capitalize ()))

t1c = time.time () - t1c
print "[+%gs] Part C" % t1c

# Check it!
print "==> Total time for Solution 1: %gs" %...


I am working on the file "11a--nyc-whiners-followup.ipynb".
The first task is:
Solution 3
This third solution introduces a new concept, namely, the idea of a subquery.
The basic idea is that, within a SELECT statement, you can reference a table generated "on-the-fly" from another SELECT statement. Notice how this solution basically merges the two queries used in the previous solutions into just a single query.
Can you write Python code for it?

t3 = time.time ()

traces3 = []
for city in TOP_CITIES: # Determines the complaint counts by city
    print ("[+%gs] Processing %s ..." % (time.time () - t3, city)) ; sys.stdout.flush ()

    query = '''
      SELECT D.ComplaintType,
             (100.0 * COUNT(*) / T.NumComplaints) AS PercentComplaints
        FROM data AS D,
             (SELECT ComplaintType, COUNT(*) AS NumComplaints
                FROM data
                GROUP BY ComplaintType COLLATE NOCASE) AS T
        WHERE (City = "{}" COLLATE NOCASE)
              AND (D.ComplaintType = T.ComplaintType COLLATE NOCASE)
        GROUP BY D.ComplaintType COLLATE NOCASE
        ORDER BY -T.NumComplaints
    '''.format (city)
    df = pd.read_sql_query (query, disk_engine)
    traces3.append (Bar (x=capitalize (df.ComplaintType),
                         y=df.PercentComplaints,
                         name=city.capitalize ()))

t3 = time.time () - t3
print "[+%gs] Total" % t3

print "==> Total time for Solution 3: %gs" % t3
iplot_percent_complaints_by_type_and_city (traces3)


I am working on the file "11a--nyc-whiners-followup.ipynb".
The first task is:
Solution 4 (variation of 2)
This next solution is a variation on Solution 2, except instead of creating a view, we create an actual table with the totals.
By storing the table, we can speed up Solution 2 a lot. The downside is that we now have to be careful to maintain this totals table, in the event there are updates to the underlying dataset from which it derives.
Can you write Python code for it?

t4a = time.time ()

query = '''
  DROP TABLE IF EXISTS TotalComplaints
'''
c = disk_engine.cursor ()
c.execute (query)

query = '''
  CREATE TABLE TotalComplaints AS
    SELECT ComplaintType, COUNT(*) AS NumComplaints
      FROM data
      GROUP BY ComplaintType COLLATE NOCASE
      ORDER BY -NumComplaints
'''
c.execute (query)

t4a = time.time () - t4a
print "[+%gs] Part A" % t4a

t4b = time.time ()

traces4 = []
for city in TOP_CITIES: # Determines the complaint counts by city
    print ("[+%gs] Processing %s ..." % (time.time () - t4b, city)) ; sys.stdout.flush ()

    query = '''
      SELECT D.ComplaintType,
             (100.0 * COUNT(*) / T.NumComplaints) AS PercentComplaints
        FROM data AS D, TotalComplaints AS T
        WHERE (City = "{}" COLLATE NOCASE)
              AND (D.ComplaintType = T.ComplaintType COLLATE NOCASE)
        GROUP BY D.ComplaintType COLLATE NOCASE
        ORDER BY -T.NumComplaints
    '''.format (city)
    df = pd.read_sql_query (query, disk_engine)
    traces4.append (Bar (x=capitalize (df.ComplaintType),
                         y=df.PercentComplaints,
                         name=city.capitalize ()))

t4b = time.time () - t4b
print "[+%gs] Part B" % t4b

print "==> Total time for Solution 4: %gs" % (t4a + t4b)
iplot_percent_complaints_by_type_and_city (traces4)


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
Look at this list of tags (https://pythonprogramming.net/natural-language-toolkit-nltk-part-speech-tagging)
So lets try to use this on out Daphne data
Can you write Python code for it?

df = pd.read_csv('allPostText_test.csv')

df.info()

def vec(name):
    tokens = nltk.word_tokenize(name)
    tagged = nltk.pos_tag(tokens)
    return tagged

df['Text'].apply(vec).head(10)

df['Tags'] = df['Text'].apply(vec)


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
NTLK Documentation
Understanding how NTLK works
Can you write Python code for it?

sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""

tokens = nltk.word_tokenize(sentence)

tokens

tagged = nltk.pos_tag(tokens)

tagged


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
Not great, so lets look around for a better solution.
Can you write Python code for it?


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
Only consider the ones, with NNP as second item
Can you write Python code for it?

def token(tags):
    mini_list = []
    for elem in tags:
        if elem[1] == 'NNP':
            mini_list.append(elem[0])
    return mini_list

df['People list'] = df['Tags'].apply(token)

df['People list'].head(10)


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
Only getting the peolple
Can you write Python code for it?

lst = list(df['people'])

lst = [x for x in lst if x !=[]]

flat_list = [item for sublist in lst for item in sublist]

name_list = []
for name in flat_list:
    if name[0] == 'PERSON':
        name_list.append(name[1])

pd.DataFrame(name_list)[0].value_counts()


I am working on the file "12 beatifulsoup, regex, nltk stemming/03 NLTK.ipynb".
The first task is:
sudo python -m nltk.downloader all if you have problems
Code stolen from Stackoverflow
Can you write Python code for it?

for sent in nltk.sent_tokenize(df['Text'][1]):
    for chunk in nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(sent))):
        if hasattr(chunk, 'label'):
            print(chunk.label()+',', ' '.join(c[0] for c in chunk))

def peopled(elem):
    mini_list = []
    for sent in nltk.sent_tokenize(elem):
        for chunk in nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(sent))):
            if hasattr(chunk, 'label'):
                p = chunk.label(), ' '.join(c[0] for c in chunk)
                mini_list.append(p)
    return mini_list

df['people'] = df['Text'].apply(peopled)


I am working on the file "12--pagerank.ipynb".
The first task is:
Exercise. Compute a view called Outdegrees, which contains the following columns:

Id: vertex ID
Degree: the out-degree of this vertex.

To help you test your view, the following snippet includes a second query that selects from your view but adds a Url field and orders the results in descending order of degree. It also prints first few and last few rows of this query, so you can inspect the URLs as a sanity check. (Perhaps it also provides a small bit of entertainment!)
Can you write Python code for it?

# Complete this query:
query = '''
  CREATE VIEW IF NOT EXISTS Outdegrees AS
    ...
'''
c = conn.cursor ()
c.execute (query)

from IPython.display import display

query = '''
  SELECT Outdegrees.Id, Degree, Url
    FROM Outdegrees, Vertices
    WHERE Outdegrees.Id = Vertices.Id
    ORDER BY -Degree
'''
df_outdegrees = pd.read_sql_query (query, conn)
print "==> A few entries with large out-degrees:"
display (df_outdegrees.head ())
print "\n==> A few entries with small out-degrees:"
display (df_outdegrees.tail ())


I am working on the file "12--pagerank.ipynb".
The first task is:
Exercise. Implement a function to multiply a sparse matrix by a dense vector, assuming a dense vector defined as follows.
Can you write Python code for it?

def dense_vector (n, init_val=0.0):
    """
    Returns a dense vector of length `n`, with all entries set to
    `init_val`.
    """
    return [init_val] * n

# Implement this routine:
def spmv (n, A, x):
    """Returns a dense vector y of length n, where y = A*x."""
    pass


I am working on the file "12--pagerank.ipynb".
The first task is:
Exercise. Query the database to extract a report of which URLs point to which URLs. Also include the source vertex out-degree and order the rows in descending order by it.
Part 2: Implement PageRank
The following exercises will walk you through a possible implementation of PageRank for this dataset.
Exercise. Build a sparse matrix, A_1, that stores $G^TD^{-1}$, where $G^T$ is the transpose of the connectivity matrix $G$, and $D^{-1}$ is the diagonal matrix of inverse out-degrees.
Can you write Python code for it?

from cse6040utils import sparse_matrix

A_1 = sparse_matrix () # Initially all zeros, with no rows or columns

# Insert your code here


I am working on the file "12--pagerank.ipynb".
The first task is:
Exercise. Write a snippet of code to verify that the vertex IDs are dense in some interval $[1, n]$. That is, there is a minimum value of $1$, some maximum value $n$, and no missing values between $1$ and $n$.
Can you write Python code for it?


I am working on the file "12-Generators.ipynb".
The first task is:
A generator expression, on the other hand, is used-up after one iteration:
Can you write Python code for it?

G = (n ** 2 for n in range(12))
list(G)

list(G)


I am working on the file "12-Generators.ipynb".
The first task is:
Notice that printing the generator expression does not print the contents; one way to print the contents of a generator expression is to pass it to the list constructor:
Can you write Python code for it?


I am working on the file "12-Generators.ipynb".
The first task is:
Similarly, here we have two ways of constructing equivalent generators:
Can you write Python code for it?

G1 = (n ** 2 for n in range(12))

def gen():
    for n in range(12):
        yield n ** 2

G2 = gen()
print(*G1)
print(*G2)


I am working on the file "12-Generators.ipynb".
The first task is:
The difference is that a generator expression does not actually compute the values until they are needed.
This not only leads to memory efficiency, but to computational efficiency as well!
This also means that while the size of a list is limited by available memory, the size of a generator expression is unlimited!
An example of an infinite generator expression can be created using the count iterator defined in itertools:
Can you write Python code for it?

from itertools import count
count()

for i in count():
    print(i, end=' ')
    if i >= 10: break


I am working on the file "12-Generators.ipynb".
The first task is:
This can be very useful because it means iteration can be stopped and started:
Can you write Python code for it?

G = (n**2 for n in range(12))
for n in G:
    print(n, end=' ')
    if n > 30: break

print("\ndoing something in between")

for n in G:
    print(n, end=' ')


I am working on the file "12-Generators.ipynb".
The first task is:
You might see what we're getting at here: if we were to expand the list of factors appropriately, what we would have the beginnings of is a prime number generator, using the Sieve of Eratosthenes algorithm. We'll explore this more momentarily.
A list can be iterated multiple times; a generator expression is single-use
This is one of those potential gotchas of generator expressions.
With a list, we can straightforwardly do this:
Can you write Python code for it?

L = [n ** 2 for n in range(12)]
for val in L:
    print(val, end=' ')
print()

for val in L:
    print(val, end=' ')


I am working on the file "12-JupyterLab.ipynb".
The first task is:
Output
Can you write Python code for it?


I am working on the file "12-JupyterLab.ipynb".
The first task is:
Terminal
Can you write Python code for it?


I am working on the file "12-JupyterLab.ipynb".
The first task is:
Text editor (a place to type code)
Can you write Python code for it?


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
(b) Der ConsecutiveNPChunker, dessen Code Sie in der nächsten Zelle sehen, basiert auf einem Klassifikator.
Can you write Python code for it?

class ConsecutiveNPChunkTagger(nltk.TaggerI):

    def __init__(self, train_sents, npchunk_features):
        self.extract_features = npchunk_features
        train_set = []
        for tagged_sent in train_sents:
            untagged_sent = nltk.tag.untag(tagged_sent)
            history = []
            for i, (word, tag) in enumerate(tagged_sent):
                featureset = npchunk_features(untagged_sent, i, history)
                train_set.append( (featureset, tag) )
                history.append(tag)
        self.classifier = nltk.NaiveBayesClassifier.train(train_set)

    def tag(self, sentence):
        history = []
        for i, word in enumerate(sentence):
            featureset = self.extract_features(sentence, i, history)
            tag = self.classifier.classify(featureset)
            history.append(tag)
        return zip(sentence, history)

class ConsecutiveNPChunker(nltk.ChunkParserI):
    def __init__(self, train_sents, npchunk_features):
        tagged_sents = [[((w,t),c) for (w,t,c) in
                         nltk.chunk.tree2conlltags(sent)]
                        for sent in train_sents]
        self.tagger = ConsecutiveNPChunkTagger(tagged_sents, npchunk_features)

    def parse(self, sentence):
        tagged_sents = self.tagger.tag(sentence)
        conlltags = [(w,t,c) for ((w,t),c) in tagged_sents]
        return nltk.chunk.conlltags2tree(conlltags)


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
(c) Fügen Sie weitere Features für
- das aktuelle Wort,
- das vorhergehende POS-Tag und
- das vorhergehende Chunk-Tag
zur Extraktion hinzu und beobachten Sie jeweils die Auswirkungen auf die Performanz in der Evaluation.
Can you write Python code for it?

def word_feature(sentence, i, history):
    word, pos = sentence[i]
    return {"pos": pos, "word": word}

def previous_pos(sentence, i, history):
    word, pos = sentence[i]
    if i == 0:
        prevword, prevpos = "<START>", "<START>"
    else:
        prevword, prevpos = sentence[i-1]

    return {"pos": pos, "word": word, "prevpos": prevpos}

def previous_chunk(sentence, i, history):
    word, pos = sentence[i]
    if i == 0:
        prevword, prevpos = "<START>", "<START>"
        prevchunk = "<START>"
    else:
        prevword, prevpos = sentence[i-1]
        prevchunk = history[-1]

    return {
        "pos": pos, "word": word,
        "prevpos": prevpos, "prevchunk": prevchunk
    }

chunker = ConsecutiveNPChunker(train_sents, word_feature)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, previous_pos)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, previous_chunk)
print(chunker.evaluate(test_sents))


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Ablation Study
Can you write Python code for it?

def ablate(feat_extr, feat_name):
    def ablated_feat_extr(sentence, i, history):
        feat_dict = feat_extr(sentence, i, history)
        feat_dict.pop(feat_name, None)
        return feat_dict
    return ablated_feat_extr

# in diese Liste sind die Namen der Features einzutragen, die oben jeweils vergeben wurden
for feat_name in []:
    print("Ablated Feature:", feat_name)
    feature_extractor = ablate(tags_since_dt, feat_name)
    chunker = ConsecutiveNPChunker(train_sents, feature_extractor)
    print(chunker.evaluate(test_sents))


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Aufgabe 2     Herunterladen von Ressourcen
Das CoNLL 2000 Korpus ist ein POS- und Chunk-getaggtes Korpus (IOB- Format), das
in ein Test- und ein Trainingskorpus aufgeteilt ist. Wir werden es zum Training und zur
Evaluation von Chunk-Parsern verwenden. Laden Sie es sich dafür zunächst über die
Ressource corpora/conll2000 herunter.
Can you write Python code for it?


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Aufgabe 4     Datenbasiertes Chunking
(a) Betrachten Sie den folgenden Code für einen Chunker, der für jedes POS-Tag das
wahrscheinlichste Chunk-Tag berechnet (Training) und dieses dann zur Testzeit
ausgibt.
Can you write Python code for it?

class UnigramChunker(nltk.ChunkParserI):
    def __init__(self, train_sents):
        train_data = [
            [
                (t,c)
                for w,t,c in nltk.chunk.tree2conlltags(sent)
            ]
            for sent in train_sents
        ]
        self.tagger = nltk.UnigramTagger(train_data)

    def parse(self, sentence):
        pos_tags = [pos for (word,pos) in sentence]
        tagged_pos_tags = self.tagger.tag(pos_tags)
        chunktags = [
            chunktag for (pos, chunktag) in tagged_pos_tags
        ]
        conlltags = [
            (word, pos, chunktag)
            for ((word, pos), chunktag)
            in zip(sentence, chunktags)
        ]
        return nltk.chunk.conlltags2tree(conlltags)


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Betrachten Sie folgende Daten. Es handelt sich um ein vereinfachtes Tagging-Schema fürs Chunking, bei dem nur zwischen „Teil einer NP“ (1) und „nicht Teil einer NP“ (0) unterschieden wird.
Can you write Python code for it?


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Das chunk_types-Argument dient der Auwahl von Chunk-Typen (in diesem Beispiel
Nominalphrasen).
Aufgabe 3     Chunking mit regulären Ausdrücken
Erstellen Sie einen einfachen RegexpParser, der für Nominalphrasen charakteristische
Tags zu NPs zusammenfasst. Solche charakteristischen Tags sind z.B. Kardinalzahlen (CD),
Artikel (DT), Adjektive (JJ, JJR, JJS) und natürlich Substantive (NN, NNS, NNP, NNPS).
Weitere interessante Tags wären PDT (z.B. both, a lot of), POS ('s), PRP (Personalpronomen), PRP$ (Possessivpronomen).
Can you write Python code for it?

regex = r"NP: {<PDT|POS|PRP|PRP\$|CD|DT|JJ[RS]?|NNP?S?>+}"


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Dies erlaubt uns, die Features, die für die Klassifikation extrahiert werden, genauer zu bestimmen.
Ein Feature-Extraktor lässt sich als Funktion z.B. so definieren:
Can you write Python code for it?

def pos_feature(sentence , i, history):
    word, pos = sentence[i]
    return {"pos": pos}


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Dieser Feature-Extraktor extrahiert genau ein Feature, nämlich das POS-Tag, das
auch der UnigramChunker verwendet hat.
Evaluieren Sie den ConsecutiveNPChunker mit diesem Feature-Extraktor und vergleichen Sie seine Performanz mit der des UnigramChunkers.
Can you write Python code for it?

chunker = ConsecutiveNPChunker(train_sents, pos_feature)
print(chunker.evaluate(test_sents))


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Features schrittweise hinzufügen
Can you write Python code for it?

chunker = ConsecutiveNPChunker(train_sents, next_pos)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, prevcur_pos)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, curnext_pos)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, next_word)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, tags_since_dt)
print(chunker.evaluate(test_sents))


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Hausaufgaben
Aufgabe 5     Feature Engineering
Anschließend an Aufgabe 4 sollen Sie hier weitere Features implementieren und jeweils deren Auswirkungen auf die Performanz beobachten.
Fügen Sie von oben nach unten immer jeweils ein neues Feature nach dem anderen hinzu
und behalten Sie alle bisherigen Features bei. Dann notieren Sie jeweils den Performanz-
gewinn (oder -verlust) im Vergleich zur vorherigen Zeile in untenstehender Tabelle.
Fügen Sie außerdem noch zwei neue Features an das Ende der Tabelle, die Sie sich selbst
ausdenken!
Zum Schluss können Sie auch eine sogenannte ablation study durchführen. Dabei wird
immer genau eines der Features entfernt, um anhand des Performanzverlustes dessen
Beitrag zum Ergebnis zu bemessen.
| Neues Feature                                                   | Performanzgewinn (F1) | Ablation Study |
|-----------------------------------------------------------------|-----------------------|----------------|
| Lookahead: nächstes POS-Tag                                     |                       |                |
| Paar-Feature: vorhergehendes und aktuelles POS-Tag konkateniert |                       |                |
| Paar-Feature: aktuelles und nächstes POS-Tag konkateniert       |                       |                |
| Lookahead: nächstes Wort                                        |                       |                |
| (Sortierte) Menge aller POS-Tags seit dem Artikel (DT)        |                       |                |
|                                                                 |                       |                |
|                                                                 |                       |                |
Can you write Python code for it?

def next_pos(sentence, i, history):
    word, pos = sentence[i]
    return {
        "pos": pos
    }

def prevcur_pos(sentence, i, history):
    word, pos = sentence[i]
    return {
        "pos": pos
    }

def curnext_pos(sentence, i, history):
    word, pos = sentence[i]
    return {
        "pos": pos
    }

def next_word(sentence, i, history):
    word, pos = sentence[i]
    return {
        "pos": pos
    }

def tags_since_dt(sentence, i, history):
    def tags_since_dt_helper(sentence, i):
        tags = set()
        return '+'.join(sorted(tags))

    word, pos = sentence[i]
    return {
        "pos": pos,
        "tags-since-dt": tags_since_dt_helper(sentence, i)
    }


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Trainieren und evaluieren Sie den UnigramChunker auf dem CoNLL 2000 Korpus.
Can you write Python code for it?

train_sents = conll2000.chunked_sents('train.txt', chunk_types=['NP'])
uc = UnigramChunker(train_sents)
print(uc.evaluate(test_sents))


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Um uns einen Überblick darüber zu verschaffen, was der Chunker gelernt hat, können wir ihn für jedes mögliche POS-Tag eine Vorhersage treffen lassen:
Can you write Python code for it?

postags = sorted(set(pos for sent in train_sents for (word,pos) in sent.leaves()))
uc.tagger.tag(postags)


I am working on the file "12-notebook-after-class.ipynb".
The first task is:
Wenn Sie es erfolgreich heruntergeladen haben, können Sie folgendermaßen darauf zugrei-
fen:
Can you write Python code for it?

from nltk.corpus import conll2000
conll2000.chunked_sents('train.txt', chunk_types=['NP'])[99]


I am working on the file "12-notebook.ipynb".
The first task is:
(b) Der ConsecutiveNPChunker, dessen Code Sie in der nächsten Zelle sehen, basiert auf einem Klassifikator.
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
(c) Fügen Sie weitere Features für
- das aktuelle Wort,
- das vorhergehende POS-Tag und
- das vorhergehende Chunk-Tag
zur Extraktion hinzu und beobachten Sie jeweils die Auswirkungen auf die Performanz in der Evaluation.
Can you write Python code for it?

def word_feature(sentence, i, history):
    word, pos = sentence[i]
    return {"pos": pos}

def previous_pos(sentence, i, history):
    word, pos = sentence[i]
    return {"pos": pos}

def previous_chunk(sentence, i, history):
    word, pos = sentence[i]
    return {"pos": pos}

chunker = ConsecutiveNPChunker(train_sents, word_feature)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, previous_pos)
print(chunker.evaluate(test_sents))

chunker = ConsecutiveNPChunker(train_sents, previous_chunk)
print(chunker.evaluate(test_sents))


I am working on the file "12-notebook.ipynb".
The first task is:
Ablation Study
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
Aufgabe 4     Datenbasiertes Chunking
(a) Betrachten Sie den folgenden Code für einen Chunker, der für jedes POS-Tag das
wahrscheinlichste Chunk-Tag berechnet (Training) und dieses dann zur Testzeit
ausgibt.
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
Betrachten Sie folgende Daten. Es handelt sich um ein vereinfachtes Tagging-Schema fürs Chunking, bei dem nur zwischen „Teil einer NP“ (1) und „nicht Teil einer NP“ (0) unterschieden wird.
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
Das chunk_types-Argument dient der Auwahl von Chunk-Typen (in diesem Beispiel
Nominalphrasen).
Aufgabe 3     Chunking mit regulären Ausdrücken
Erstellen Sie einen einfachen RegexpParser, der für Nominalphrasen charakteristische
Tags zu NPs zusammenfasst. Solche charakteristischen Tags sind z.B. Kardinalzahlen (CD),
Artikel (DT), Adjektive (JJ, JJR, JJS) und natürlich Substantive (NN, NNS, NNP, NNPS).
Weitere interessante Tags wären PDT (z.B. both, a lot of), POS ('s), PRP (Personalpronomen), PRP$ (Possessivpronomen).
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
Evaluieren Sie Ihren Parser anschließend auf dem CoNLL 2000 Korpus:
Can you write Python code for it?

test_sents = conll2000.chunked_sents('test.txt', chunk_types=['NP'])
cp = nltk.RegexpParser(regex)
print(cp.evaluate(test_sents))


I am working on the file "12-notebook.ipynb".
The first task is:
Features schrittweise hinzufügen
Can you write Python code for it?


I am working on the file "12-notebook.ipynb".
The first task is:
Trainieren und evaluieren Sie den UnigramChunker auf dem CoNLL 2000 Korpus.
Can you write Python code for it?

train_sents = conll2000.chunked_sents('train.txt', chunk_types=['NP'])
# uc = ...


I am working on the file "12-notebook.ipynb".
The first task is:
Um uns einen Überblick darüber zu verschaffen, was der Chunker gelernt hat, können wir ihn für jedes mögliche POS-Tag eine Vorhersage treffen lassen:
Can you write Python code for it?

postags = sorted(set(pos for sent in train_sents for (word,pos) in sent.leaves()))
# uc.tagger.tag(postags)


I am working on the file "12-notebook.ipynb".
The first task is:
Wenn Sie es erfolgreich heruntergeladen haben, können Sie folgendermaßen darauf zugrei-
fen:
Can you write Python code for it?


I am working on the file "12. MLP/MLP-Readable.ipynb".
The first task is:
Now lets plot our XOR in anger function
Can you write Python code for it?


I am working on the file "12. MLP/MLP-Readable.ipynb".
The first task is:
Now lets plot our average error energy against the epoch
Can you write Python code for it?

fig = plt.figure(figsize=(7,7))
plt.plot(avg_square_error);
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.xlabel(r'Epoch',fontsize=20)
plt.ylabel(r'Average squared error energy',fontsize=20)


I am working on the file "12. MLP/MLP-Readable.ipynb".
The first task is:
Now the trivial part is to plot the results
Can you write Python code for it?

fig = plt.figure(figsize=(7,7))
levels = np.arange(-0.010, 1.010, 0.05)
plt.contour(y231, levels, origin='lower',linewidths=0.85, cmap=plt.get_cmap('seismic'), extent=(lower_left_corner,upper_right_corner,lower_left_corner,upper_right_corner))
CS = plt.contour(y231, [0], linewidths=1.25, cmap=plt.get_cmap('seismic'), extent=(lower_left_corner,upper_right_corner,lower_left_corner,upper_right_corner))

plt.plot(training_set[trueNegIndices,0],training_set[trueNegIndices,1],'b',marker='o',markersize=10,linestyle='None');
plt.plot(training_set[truePosIndices,0],training_set[truePosIndices,1],'r',marker='^',markersize=12,linestyle='None');
plt.plot(training_set[falseNegIndices,0],training_set[falseNegIndices,1],'b',marker='^',markersize=12,linestyle='None');
plt.plot(training_set[falsePosIndices,0],training_set[falsePosIndices,1],'r',marker='o',markersize=10,linestyle='None');

plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.xlabel(r'$x_1$',fontsize=25)
plt.ylabel(r'$x_2$',fontsize=25)


I am working on the file "12. 추정 및 검정/02. 검정과 유의 확률.ipynb".
The first task is:
이 값은 5% 보다는 작고 1% 보다는 크기 때문에 유의 수준이 5% 라면 기각할 수 있으며(즉 공정한 동전이 아니라고 말할 수 있다.) 유의 수준이 1% 라면 기각할 수 없다.(즉, 공정한 동전이 아니라고 말할 수 없다.)

문제2

어떤 트레이더의 일주일 수익률은 다음과 같다.:

-2.5%, -5%, 4.3%, -3.7% -5.6%

이 트레이더는 돈을 벌어다 줄 사람인가, 아니면 돈을 잃을 사람인가?

수익률이 정규 분포를 따른 다고 가정하면 이 트레이더의 검정통계량은 다음과 같이 계산된다.
$$ t = \dfrac{m}{\frac{s}{\sqrt{N}}} = -1.4025 $$
이 검정 통계량에 대한 유의 확률은 11.67%이다.
$$ F(t=-1.4025;4) = 0.1167 $$
Can you write Python code for it?


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
5 개의 데이터만으로도  두 평균이 다르다는 것을 유의 확률(p-value) 0.2%의 정확도로 알아내었음을 확인할 수 있다.
카이 제곱 분산 검정 (Chi-Square Test for the Variance)
지금까지는 정규 분포의 기댓값을 비교하는 검정을 살펴보았다. 이제는 정규 분포의 분산에 대해 살펴보자.
카이 제곱 분산 검정(Chi-Square Test for the Variance)은 정규 분포의 샘플 분산 값은 정규화 하면 카이 제곱 분포를 따른다는 점을 이용하는 검정 방법이다.
그러나 SciPy는 카이 제곱 분산 검정에 대한 명령이 없으므로 chi2 클래스를 사용하여 직접 구현해야 한다.
Can you write Python code for it?

def chi2var_test(x, sigma2=1):
    v = x.var(ddof=1)
    t = (len(x) - 1)*v/sigma2
    return t, sp.stats.chi2(df=len(x)-1).sf(np.abs(t))

N = 10
mu_0 = 0
sigma_0 = 1.1
np.random.seed(0)
x = sp.stats.norm(mu_0, sigma_0).rvs(N)
sns.distplot(x, kde=False, fit=sp.stats.norm)
plt.show()
x.std()

chi2var_test(x)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
데이터의 갯수를 50개와 100개로 증가시킨 경우에 유의 확률은 0.8%로 감소하였다. 따라서 두 확률 분포의 기댓값이 일치한다는 귀무 가설은 기각할 수 있다.
대응 표본 t-검정 (Paired-two-sample t-test)
대응 표본 t-검정은 독립 표본 t-검정을 두 집단의 샘플이 1대1 대응하는 경우에 대해 수정한 것이다. 즉, 독립 표본 t-검정과 마찬가지로 두 정규 분포의 기댓값이 같은지 확인하기 위한 검정이다.
예를 들어 어떤 반의 학생들이 특강을 수강하기 전과 수강한 이후에 각각 시험을 본 시험 점수의 경우에는 같은 학생의 두 점수는 대응할 수 있다. 이 대응 정보를 알고 있다면 보통의 독립 표본 t-검정에서 발생할 수 있는 샘플간의 차이의 영향을 없앨 수 있기 때문에 특강 수강의 영향을 보다 정확하게 추정할 수 있다.

scipy.stats.ttest_rel
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_rel.html

$\mu_1 = 0$, $\mu_2 = 0.5$로 평균이 달라진 경우에 대해 대응 표본 t-검정을 실시해 보자. 데이터 갯수 $N$은 5 이다.
Can you write Python code for it?

N = 5
mu_1 = 0
mu_2 = 0.5
np.random.seed(1)
x1 = sp.stats.norm(mu_1).rvs(N)
x2 = x1 + sp.stats.norm(mu_2, 0.1).rvs(N)
sns.distplot(x1, kde=False, fit=sp.stats.norm)
sns.distplot(x2, kde=False, fit=sp.stats.norm)
plt.show()

sp.stats.ttest_rel(x1, x2)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
유의 확률(p-value)이 1.96%이므로 만약 유의 수준이 5% 이상 이라면 귀무 가설을 기각할 수 있다. 따라서 $\mu \neq 0$이다. 이 경우는 검정 결과가 오류인 예라고 볼 수 있다. 검정 결과가 오류로 나온 이유는 데이터 수가 10개로 부족하기 때문이다.
오류의 유형 중에서 이러한 오류는 귀무 가설이 진실임에도 불구하고 거짓으로 나온 경우로 유형 1 오류(Type 1 Error)라고 한다.
데이터 갯수 $N=100$, 실제 모수 $\mu_0=0$인 경우 대해 단일 표본 z-검정 명령을 실시해 보자.
Can you write Python code for it?

N = 100
mu_0 = 0
np.random.seed(0)
x = sp.stats.norm(mu_0).rvs(N)

ztest_1samp(x)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
유의 확률(p-value)이 34%로 높으므로 귀무 가설을 기각할 수 없다. 따라서 $\theta=0.5$이다.
데이터 갯수 $N=100$, 실제 모수 $\theta_0=0.5$인 경우 대해 이항 검정 명령을 실시해 보자.
Can you write Python code for it?

N = 100
theta_0 = 0.5
np.random.seed(0)
x = sp.stats.bernoulli(theta_0).rvs(N)
n = np.count_nonzero(x)
n

sp.stats.binom_test(n, N)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
유의 확률(p-value)이 68.4%이므로 귀무 가설을 기각할 수 없다. 따라서 $\mu_1 \neq \mu_2$이다. 이 경우는 검정 결과가 오류인 예라고 볼 수 있다.
오류의 유형 중에서 이러한 오류는 귀무 가설이 거짓임에도 불구하고 진실로 나온 경우로 유형 2 오류(Type 2 Error)라고 한다.
데이터 수가 증가하면 이러한 오류가 발생할 가능성이 줄어든다.
Can you write Python code for it?

N_1 = 50; mu_1 = 0; sigma_1 = 1
N_2 = 100; mu_2 = 0.5; sigma_2 = 1
np.random.seed(0)
x1 = sp.stats.norm(mu_1, sigma_1).rvs(N_1)
x2 = sp.stats.norm(mu_2, sigma_2).rvs(N_2)
sns.distplot(x1, kde=False, fit=sp.stats.norm)
sns.distplot(x2, kde=False, fit=sp.stats.norm)
plt.show()

sp.stats.ttest_ind(x1, x2, equal_var=True)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
유의 확률(p-value)이 92%로 높으므로 귀무 가설을 기각할 수 없다. 따라서 $\theta=0.5$이다.
데이터 갯수 $N=100$, 실제 모수 $\theta_0=0.35$인 경우 대해 이항 검정 명령을 실시해 보자.
Can you write Python code for it?

N = 100
theta_0 = 0.35
np.random.seed(0)
x = sp.stats.bernoulli(theta_0).rvs(N)
n = np.count_nonzero(x)
n

sp.stats.binom_test(n, N)


I am working on the file "12. 추정 및 검정/03. SciPy를 사용한 기초적인 검정.ipynb".
The first task is:
정규성 검정
회귀 분석 등에서는 확률 분포가 가우시안 정규 분포를 따르는지 아닌지를 확인하는 것이 중요하다. 이러한 검정을 정규성 검정(normality test)이라고 한다. 정규성 분포 그 중요도 만큼 여러가지 검정 방법들이 개발되어 있으며 Scipy 패키지 이외에도 statsmodels 패키지에도 다양한 정규성 검정 명령어를 제공한다.
statsmodels에서 제공하는 정규성 검정 명령어

Omnibus Normality test

statsmodels.stats.stattools.omni_normtest

http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.stattools.omni_normtest.html

Jarque–Bera test

statsmodels.stats.stattools.jarque_bera

http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.stattools.jarque_bera.html

Kolmogorov-Smirnov test

statsmodels.stats.diagnostic.kstest_normal

http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.diagnostic.kstest_normal.html

Lilliefors test

statsmodels.stats.diagnostic.lillifors
http://statsmodels.sourceforge.net/devel/generated/statsmodels.stats.diagnostic.lillifors.html

SciPy 에서 제공하는 정규성 검정 명령어

Kolmogorov-Smirnov test
scipy.stats.ks_2samp

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ks_2samp.html

Shapiro–Wilk test

scipy.stats.shapiro

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.shapiro.html

Anderson–Darling test

scipy.stats.anderson

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.anderson.html

D'Agostino's K-squared test

scipy.stats.mstats.normaltest
http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mstats.normaltest.html

이 중에서 Kolmogorov-Smirnov 검정은 사실 정규 분포에 국한되지 않고 두 샘플이 같은 분포를 따르는지 확인할 수 있는 방법이다.
Can you write Python code for it?

np.random.seed(0)
N1 = 50
N2 = 100
x1 = sp.stats.norm(0, 1).rvs(N1)
x2 = sp.stats.norm(0.5, 1.5).rvs(N2)
sns.distplot(x1)
sns.distplot(x2)
plt.show()

sp.stats.ks_2samp(x1, x2)


I am working on the file "12. 추정 및 검정/06. MLE 모수 추정.ipynb".
The first task is:
다변수 정규 분포의 모수 추정
MLE for Multivariate Gaussian Normal Distribution

각각의 시도 $x_i$에 대한 확률은 다변수 정규 분포

$$ p(x ; \theta ) = N(x ; \mu, \Sigma) = \dfrac{1}{(2\pi)^{D/2} |\Sigma|^{1/2}} \exp \left( -\dfrac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right) $$

$N$개의 독립 샘플 $x_{1:N}$ 이 있는 경우,
$$ L(\theta;x_{1:N}) = p(x_{1:N};\theta) = \prod_{i=1}^N  \dfrac{1}{(2\pi)^{D/2} |\Sigma|^{1/2}} \exp \left( -\dfrac{1}{2} (x_i-\mu)^T \Sigma^{-1} (x_i-\mu) \right)$$

Log-Likelihood
$$
\begin{eqnarray}
\log L
&=& \log P(x_{1:N};\theta)  \
&=& \sum_{i=1}^N \left{ -\log((2\pi)^{D/2} |\Sigma|^{1/2}) - \dfrac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu) \right} \
&=& C -\dfrac{N}{2} \log|\Sigma| - \dfrac{1}{2} \sum (x-\mu)^T \Sigma^{-1} (x-\mu)
\end{eqnarray}
$$

precision matrix $\Lambda = \Sigma^{-1}$

$$
\begin{eqnarray}
\log L
&=& C + \dfrac{N}{2} \log|\Lambda| - \dfrac{1}{2} \sum(x-\mu)^T \Lambda (x-\mu)
\end{eqnarray}
$$
$$ \dfrac{\partial L}{\partial \mu} = -  \dfrac{\partial}{\partial \mu}  \sum_{i=1}^N (x_i-\mu)^T \Lambda (x_i-\mu) =  \sum_{i=1}^N 2\Lambda (x_i - \mu) = 0 $$
$$ \mu = \dfrac{1}{N}\sum_{i=1}^N x_i $$
$$ \dfrac{\partial L}{\partial \Lambda} = \dfrac{\partial}{\partial \Lambda} \dfrac{N}{2} \log|\Lambda| - \dfrac{\partial}{\partial \Lambda}  \dfrac{1}{2} \sum_{i=1}^N \text{tr}( (x_i-\mu)(x_i-\mu)^T\Lambda) =0  $$
$$ \dfrac{N}{2} \Lambda^{-T} = \dfrac{1}{2}\sum_{i=1}^N (x_i-\mu)(x_i-\mu)^T $$
$$ \Sigma = \dfrac{1}{N}\sum_{i=1}^N (x_i-\mu)(x_i-\mu)^T $$
Can you write Python code for it?

np.random.seed(0)
mu0 = np.array([0, 1])
sigma0 = np.array([[1, 0.2], [0.2, 4]])
x = sp.stats.multivariate_normal(mu0, sigma0).rvs(1000)
xbar = x.mean(axis=0)
S2 = np.cov(x, rowvar=0)
print(xbar)
print(S2)


I am working on the file "12. 추정 및 검정/06. MLE 모수 추정.ipynb".
The first task is:
카테고리 분포의 모수 추정

각각의 시도 $x_i$에 대한 확률은 카테고리 분포

$$ P(x ; \theta ) = \text{Cat}(x ; \theta) = \prod_{k=1}^K \theta_k^{x_k} $$
$$ \sum_{k=1}^K \theta_k = 1 $$

$N$개의 독립 샘플 $x_{1:N}$ 이 있는 경우,
$$ L(\theta ; x_{1:N}) = P(x_{1:N};\theta) = \prod_{i=1}^N \prod_{k=1}^K \theta_k^{x_{i,k}} $$

Log-Likelihood
$$
\begin{eqnarray}
\log L
&=& \log P(x_{1:N};\theta)  \
&=& \sum_{i=1}^N \sum_{k=1}^K  {x_{i,k}} \log\theta_k \
&=& \sum_{k=1}^K \log\theta_k \sum_{i=1}^N {x_{i,k}}
\end{eqnarray}
$$

$x_k$가 나온 횟수 $N_k = \sum_{i=1}^N {x_{i,k}}$이라고 표시

따라서  Log-Likelihood는
$$
\begin{eqnarray}
\log L
&=& \sum_{k=1}^K \log\theta_k  N_k
\end{eqnarray}
$$

추가 조건
$$ \sum_{k=1}^K \theta_k = 1 $$

Log-Likelihood Derivative with Lagrange multiplier

$$
\begin{eqnarray}
\dfrac{\partial \log L}{\partial \theta_k}
&=& \dfrac{\partial}{\partial \theta_k} \left{ \sum_{k=1}^K \log\theta_k N_k  + \lambda \left(1- \sum_{k=1}^K \theta_k\right)  \right} = 0 \
\dfrac{\partial \log L}{\partial \lambda}
&=& \dfrac{\partial}{\partial \lambda} \left{ \sum_{k=1}^K \log\theta_k N_k  + \lambda \left(1- \sum_{k=1}^K \theta_k \right)  \right} = 0\
\end{eqnarray}
$$
$$
\dfrac{N_1}{\theta_1}  = \dfrac{N_2}{\theta_2} = \cdots = \dfrac{N_K}{\theta_K} = \lambda
$$
$$
\sum_{k=1}^K N_k = N
$$
$$
\lambda \sum_{k=1}^K \theta_k  = \lambda = N
$$
$$
\theta_k = \dfrac{N_k}{N}
$$
Can you write Python code for it?

np.random.seed(0)
theta0 = np.array([0.1, 0.3, 0.6])
x = np.random.choice(np.arange(3), 1000, p=theta0)
N0, N1, N2 = np.bincount(x, minlength=3)
N = N0 + N1 + N2
theta = np.array([N0, N1, N2]) / N
theta


I am working on the file "12. 추정 및 검정/07. 베이지안 모수 추정.ipynb".
The first task is:
정규 분포의 기댓값 모수 추정
이번에는 정규 분포의 기댓값 모수를 베이지안 방법으로 추정한다. 분산 모수 $\sigma^2$은 알고 있다고 가정한다.
기댓값은 $-\infty$부터 $\infty$까지의 모든 수가 가능하기 때문에 모수의 사전 분포로는 정규 분포를 사용한다.
$$ P(\mu) = N(\mu_0, \sigma^2_0) = \dfrac{1}{\sqrt{2\pi\sigma_0^2}} \exp \left(-\dfrac{(\mu-\mu_0)^2}{2\sigma_0^2}\right)$$
데이터는 모두 독립적인 정규 분포의 곱이므로 우도는 다음과 같이 된다.
$$ P(x_{1},\ldots,x_{N} \mid \mu) = \prod_{i=1}^N N(x_i \mid \mu )  = \prod_{i=1}^N  \dfrac{1}{\sqrt{2\pi\sigma^2}} \exp \left(-\dfrac{(x_i-\mu)^2}{2\sigma^2}\right)  $$
$$
\begin{eqnarray}
P(\theta \mid x_{1},\ldots,x_{N})
&\propto & P(x_{1},\ldots,x_{N} \mid \theta) P(\theta) \
&\propto & \exp \left(-\dfrac{(\mu-\mu'_0)^2}{2\sigma_0^{'2}}\right) \
\end{eqnarray}
$$
베이지안 규칙을 사용하여 사후 분포를 구하면 다음과 같이 갱신된 하이퍼 모수  를 가지는 정규 분포가 된다.
$$
\begin{eqnarray}
\mu'_0 &=& \dfrac{\sigma^2}{N\sigma_0^2 + \sigma^2}\mu_0 + \dfrac{N\sigma_0^2}{N\sigma_0^2 + \sigma^2} \dfrac{\sum x_i}{N} \
\dfrac{1}{\sigma_0^{'2}} &=& \dfrac{1}{\sigma_0^{2}} + \dfrac{N}{\sigma^{'2}}
\end{eqnarray}
$$
Can you write Python code for it?

mu, sigma2 = 2, 4

mu0, sigma20 = 0, 1
xx = np.linspace(1, 3, 1000)

np.random.seed(0)

N = 10
x = sp.stats.norm(mu).rvs(N)
mu0 = sigma2/(N*sigma20 + sigma2) * mu0 + (N*sigma20)/(N*sigma20 + sigma2)*x.mean()
sigma20 = 1/(1/sigma20 + N/sigma2)
plt.plot(xx, sp.stats.norm(mu0, sigma20).pdf(xx), label="1st");
print(mu0)

N = 20
x = sp.stats.norm(mu).rvs(N)
mu0 = sigma2/(N*sigma20 + sigma2) * mu0 + (N*sigma20)/(N*sigma20 + sigma2)*x.mean()
sigma20 = 1/(1/sigma20 + N/sigma2)
plt.plot(xx, sp.stats.norm(mu0, sigma20).pdf(xx), label="2nd");
print(mu0)

N = 50
x = sp.stats.norm(mu).rvs(N)
mu0 = sigma2/(N*sigma20 + sigma2) * mu0 + (N*sigma20)/(N*sigma20 + sigma2)*x.mean()
sigma20 = 1/(1/sigma20 + N/sigma2)
plt.plot(xx, sp.stats.norm(mu0, sigma20).pdf(xx), label="3rd");
print(mu0)

N = 100
x = sp.stats.norm(mu).rvs(N)
mu0 = sigma2/(N*sigma20 + sigma2) * mu0 + (N*sigma20)/(N*sigma20 + sigma2)*x.mean()
sigma20 = 1/(1/sigma20 + N/sigma2)
plt.plot(xx, sp.stats.norm(mu0, sigma20).pdf(xx), label="4th");
print(mu0)

plt.axis([1, 3, 0, 20])
plt.legend()
plt.show()


I am working on the file "12. 추정 및 검정/07. 베이지안 모수 추정.ipynb".
The first task is:
카테고리 분포의 모수 추정
다음으로 클래스 갯수가 $K$인 카테고리 분포의 모수 $\theta$ 벡터를 베이지안 추정법으로 추정해 본다.
카테고리 분포의 모수의 각 원소는 모두 0부터 1사이의 값을 가지므로 사전 분포는 하이퍼 모수 $\alpha_k=\dfrac{1}{K}$인 디리클리 분포로 한다.
$$ P(\theta) \propto \prod_{k=1}^K \theta_k^{\alpha_k - 1} \;\;\; (\alpha_k = 1/K , \; \text{ for all } k) $$
데이터는 모두 독립적인 카테고리 분포의 곱이므로 우도는 다음과 같이 다항 분포가 된다.
$$ P(x_{1},\ldots,x_{N} \mid \theta) = \prod_{i=1}^N  \prod_{k=1}^K \theta_k^{x_{i,k}} $$
베이지안 규칙을 사용하여 사후 분포를 구하면 다음과 같이 갱신된 하이퍼 모수 $\alpha'_i$를 가지는 디리클리 분포가 된다.
$$
\begin{eqnarray}
P(\theta \mid x_{1},\ldots,x_{N})
&\propto & P(x_{1},\ldots,x_{N} \mid \theta)  P(\theta) \
&=& \prod_{i=1}^N  \prod_{k=1}^K \theta_k^{x_{i,k}} \cdot \prod_{k=1}^K \theta_k^{\alpha_k - 1}  \
&=& \prod_{k=1}^K  \theta^{\sum_{i=1}^N x_{i,k} + \alpha_k − 1}   \
&=& \prod_{k=1}^K  \theta^{N_k + \alpha_k −1}   \
&=& \prod_{k=1}^K  \theta^{\alpha'_k −1}   \
\end{eqnarray}
$$
이 경우에도 conjugate prior 임을 알 수 있다.
갱신된 하이퍼 모수의 값은 다음과 같다.
$$ \alpha'_k = N_k + \alpha_k $$
Can you write Python code for it?

def plot_dirichlet(alpha):

    def project(x):
        n1 = np.array([1, 0, 0])
        n2 = np.array([0, 1, 0])
        n3 = np.array([0, 0, 1])
        n12 = (n1 + n2)/2
        m1 = np.array([1, -1, 0])
        m2 = n3 - n12
        m1 = m1/np.linalg.norm(m1)
        m2 = m2/np.linalg.norm(m2)
        return np.dstack([(x-n12).dot(m1), (x-n12).dot(m2)])[0]

    def project_reverse(x):
        n1 = np.array([1, 0, 0])
        n2 = np.array([0, 1, 0])
        n3 = np.array([0, 0, 1])
        n12 = (n1 + n2)/2
        m1 = np.array([1, -1, 0])
        m2 = n3 - n12
        m1 = m1/np.linalg.norm(m1)
        m2 = m2/np.linalg.norm(m2)
        return x[:,0][:, np.newaxis] * m1 + x[:,1][:, np.newaxis] * m2 + n12

    eps = np.finfo(float).eps * 10
    X = project([[1-eps,0,0], [0,1-eps,0], [0,0,1-eps]])

    import matplotlib.tri as mtri

    triang = mtri.Triangulation(X[:,0], X[:,1], [[0, 1, 2]])
    refiner = mtri.UniformTriRefiner(triang)
    triang2 = refiner.refine_triangulation(subdiv=6)
    XYZ = project_reverse(np.dstack([triang2.x, triang2.y, 1-triang2.x-triang2.y])[0])

    pdf = sp.stats.dirichlet(alpha).pdf(XYZ.T)
    plt.tricontourf(triang2, pdf)
    plt.axis("equal")
    plt.show()

theta0 = np.array([0.2, 0.6, 0.2])

np.random.seed(0)
x1 = np.random.choice(3, 20, p=theta0)
N1 = np.bincount(x1, minlength=3)
x2 = np.random.choice(3, 100, p=theta0)
N2 = np.bincount(x2, minlength=3)
x3 = np.random.choice(3, 1000, p=theta0)
N3 = np.bincount(x3, minlength=3)

a0 = np.ones(3) / 3
plot_dirichlet(a0)

a1 = a0 + N1
plot_dirichlet(a1)
print((a1 - 1)/(a1.sum() - 3))

a2 = a1 + N2
plot_dirichlet(a2)
print((a2 - 1)/(a2.sum() - 3))

a3 = a2 + N3
plot_dirichlet(a3)
print((a3 - 1)/(a3.sum() - 3))


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
And here's a nice discussion on the topic from our friends at StackOverflow.
Can you write Python code for it?

# Pandas is very smart, and aligns the series for mathematical operations
series_6 + series_7

# Renaming an Index
series_7.index.name = "Names"
series_7

# Naming a Series
series_7.name = "SV"
series_7


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Apply
We can use the apply function to perform an operation over an axis in a dataframe.
Can you write Python code for it?

import pandas as pd
import numpy as np

df = pd.read_csv("data/cafe_sales2015.csv")
df.head()

df["Date"] = pd.to_datetime(df["Date"])
df.set_index(["Date"], inplace = True)
df.interpolate(method = "linear", inplace = True)

df.head()

#print(df.apply(np.cumsum))
df.apply(np.average)

df.apply(lambda x: x.max() - x.min())

# What columns have missing values?
df.apply(lambda x: sum(x.isnull()),axis=0)

# Using Apply to find missing values
# Obviously don't do this for datasets with thousands or millions of rows!
empty = df.apply(lambda col: pd.isnull(col))
empty


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
ApplyMap
Can you write Python code for it?

import pandas as pd
import numpy as np

df = pd.read_csv("data/cafe_sales2015.csv")
df.head()

def to_int(x):
    if type(x) is float:
        x = int(x)
        return x
    else:
        return x

df.interpolate(method = "linear", inplace = True)
df.applymap(to_int).head()


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Bada Bing!
Can you write Python code for it?

import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt

df = pd.read_excel("Data/Sopranos/sopranos-killings.xlsx")
df.head()

pd.pivot_table(df,index=["Cause of Death"],values = ["Season"], aggfunc="first")

pd.pivot_table(df,index=["Cause of Death"],values = ["Season"], aggfunc="count", margins=True)

whacked = pd.pivot_table(df,index=["Cause of Death"],values = ["Season"], aggfunc="count")
whacked

plt.style.available

plt.rcParams.update(plt.rcParamsDefault)
%matplotlib inline
plt.style.use('seaborn-deep')
import matplotlib.pyplot as plt

plt.rcParams["figure.figsize"] = (15,7)
whacked.plot(kind = "bar", legend=None)
plt.title('How People Died on The Sopranos')

with plt.style.context('ggplot', after_reset=True):
    %matplotlib inline
    import matplotlib.pyplot as plt
    plt.rcParams["figure.figsize"] = (15,7)
    whacked.plot(kind = "bar", legend=None)
    plt.title('How People Died on The Sopranos')

killer = pd.pivot_table(df,index=["Killer"],values = ["Season"], aggfunc="count")

killer = killer.sort_values(by=["Season"], ascending = False)
killer

plt.rcParams.update(plt.rcParamsDefault)
plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = (15,7)

killer[:10].plot(kind = "bar", legend=None)
plt.title('Top 10 Killers')


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Conditional Selection
While exploring data sets, one often has to use conditional selection. Or this could be true for creating subsets to work.
Can you write Python code for it?


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Data Structures
There are three fundamental data structures supported by Pandas:

* Series: a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). For those coming from an R background, Series is much like a Vector.
* DataFrame: a 2-dimensional labeled data structure with columns of potentially different types.
* Panel: also called longitudinal data or cross-sectional time series data, is data where multiple cases (people, firms, countries etc) were observed at two or more time periods. This is rarely used though, and I personally haven't come across this except for some Econometrics courses I had taken in my undergraduate years.
Series
The basic format to creat a series is:

series_a = pd.Series(data, index = index_name)
The default value for the index is 1,2,3,4....and so on, and doesn't not need to be specified, except in the case of scalars.
Can you write Python code for it?

import pandas as pd

# From Scalar Values
series_1 = pd.Series([1,2,3,4,5])
series_1


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Dataframe from a Dictionry
Let's create a new dataframe from a dictionary, and then apply some of the selection techniques we just learnt.
Can you write Python code for it?

dict1 = {'first_name': ['Erlich', 'Richard', "Dinesh", 'Gilfoyle', 'Nelson'],
         'second_name': ['Bachman', 'Hendricks', np.nan, np.nan, 'Bighetti'],
        'occupation': ['Investor', 'Entrepreneur', 'Coder', 'Coder', 'Bench Warmer'],
        'age': [40, 30, 28, 29, 28]}
df = pd.DataFrame(dict1, columns = ['first_name', 'second_name','occupation', 'age'])
df

# Who is under 30 years of age?
df[df["age"]<30]

# Who are the coders?
df[df["occupation"] == "Coder"]

# Multiple Conditions : Coders, below 30
# Not that conditions are Booleans, as shown below
coders = df["occupation"] == "Coder"
und_30 = df["age"]<30
df[coders & und_30]

df[df["second_name"].notnull()]


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
First thing we can do is drop rows with missing values with the dropna() function. By default, rows are dropped, but you can change this to columns as well.
Can you write Python code for it?

df.dropna()

df.dropna(axis = 0)

df.dropna(axis = 1)


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
First, we had to mention the axis. 0 is for rows, 1 is for columns.
Can you write Python code for it?


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Further Reading<br>
Wes McKinney's amazing book covers this issue. Refer to Page 132.
Pivot Tables
Pivot tables are summarisation tables that help the user sort, count, total or average the data available in a dataset. If you have used Excel, you will be very familiar with them. If not, let's look at it from a fresh Pandas perspective.
Typically, there are four parameters, but you don't always have to specify every one of them, as we will see in the examples below.

index: An array of the dataset that will used as indices to our new reshaped and aggregated DataFrame
columns: An array of the dataset that will provide columns to the new DataFrame
values: These are the values we wish to aggregate in each cell.
aggfunc: The function we will use to perform the aggregation

Sales Reports
Can you write Python code for it?

import pandas as pd
import numpy as np

# The 'xlrd' module gets imported automatically, if not, install it with 'pip install xlrd'
df = pd.read_excel("Data/bev-sales.xlsx")
df.head()

df.tail()

df.describe()

help(pd.pivot_table)

df.head()

pd.pivot_table(df,index=["Sales Exec"],values=["Revenue"],aggfunc="sum")

%matplotlib inline
import matplotlib.pyplot as plt

pd.pivot_table(df, index=["Sales Exec"],values=["Revenue"],aggfunc="sum").plot(kind="bar")

pd.pivot_table(df,index=["Sales Exec"],values=["Revenue"],aggfunc="mean")

pd.pivot_table(df, index=["Sales Exec", "Item"], values=["Revenue"], aggfunc="sum")

pd.pivot_table(df,index=["Sales Exec"],values=["Revenue"],aggfunc=[np.sum])

pd.pivot_table(df,index=["Sales Exec"],values=["Units sold", "Revenue"],aggfunc=[np.sum])

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn')
plt.rcParams["figure.figsize"] = (15,7)

pd.pivot_table(df,index=["Sales Exec", "Item"],values=["Revenue"],aggfunc=[np.sum]).plot(kind="bar")
plt.title('January Sales Report')

pd.pivot_table(df,index=["Sales Exec", "Item"],values=["Units sold", "Revenue"],
               columns=["Price per Unit"], aggfunc="sum", margins = True)


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Handling Missing Data
Pay special attention to this section. If needed, spend some extra time to cover all the relevant techniques.

Never in my experience have I come across a 100% clean data set "in the wild". What that means is that of course you will find that most data sets that you train with to be complete, but real world data is messy and incomplete.
Even when working with high quality, financial data from exchanges, they might often have missing data points. The less said about unstructured data like text, the better.
TL/DR: If you're going to fight Mike Tyson, don't train to fight Mr Bean.

What is Missing Data?
Data can be missing because:
* It was never captured
* The data does not exist
* It was captured but got corrupted
In Pandas, missing data will be represented as None or NaN.
Can you write Python code for it?


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Imputation
Using the fillna function, we can replace missing values.
Can you write Python code for it?

df = pd.DataFrame({'NYC':[3,np.nan,7,9,6],
                  'SF':[4,3,8,7,15],
                  'CHI':[4,np.nan,np.nan,14,6],
                  'MIA':[3, 9,12,8,9]}, index = ['Mon','Tue','Wed','Thu','Fri'])
df

df.mean()

df.fillna(value = df.mean(), inplace = True)
df

df = pd.DataFrame({'NYC':[3,np.nan,7,9,6],
                  'SF':[4,3,8,7,15],
                  'CHI':[4,np.nan,np.nan,14,6],
                  'MIA':[3, 9,12,8,9]}, index = ['Mon','Tue','Wed','Thu','Fri'])
df

df3 = df.copy()
df3

median = df3['SF'].median()
median

df3.fillna(value = median, inplace = True)
df3

df3.mode()


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
It's interesting to note that the offical Pandas documentation refers to DataFrames as:

Can be thought of as a dict-like container for Series objects.

You can access it as a Series as below:
Can you write Python code for it?

df['Jan']

print('Series:', type(df['Jan']))
print('DataFrame:',type(df[['Jan']]))


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Join
Can you write Python code for it?

customers = {
        'customer_id': ['101', '102', '103', '104', '105'],
        'first_name': ['Tony', 'Silvio', 'Paulie', 'Corrado', 'Christopher'],
        'last_name': ['Soprano', 'Dante', 'Gualtieri', 'Soprano', 'Moltisanti']}
customers

orders = {
        'customer_id': ['101', '104', '105', '108', '111'],
        'order_date': ['2015-01-01', '2015-01-08', '2015-01-19', '2015-02-10', '2015-02-11'],
        'order_value': ['10000', '25000', '1100', '5000', '4400']}
orders

df1_new = pd.DataFrame.from_dict(customers, orient='columns', dtype=None)

df1_new

df1_new = df1_new.set_index('customer_id')
df1_new

df2_new = pd.DataFrame.from_dict(orders, orient='columns', dtype=None)
df2_new

df2_new = df2_new.set_index('customer_id')
df2_new

df1_new.join(df2_new,how = "inner")

df1_new.join(df2_new,how = "outer")

df1_new.join(df2_new,how = "left")

df1_new.join(df2_new,how = "right")

# Alternate Way : I don't recommend this
df_1.join(df_2, on = "customer_id", lsuffix='_l', rsuffix='_r')


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Map
The map function iterates over each element of a series.
Can you write Python code for it?

import pandas as pd
import numpy as np

df = pd.read_csv("data/cafe_sales2015.csv")
df.head()

df["Latte"] = df["Latte"].map(lambda x: x+2)

df.head()

df.interpolate(method = "linear", inplace = True)
df["Water"] = df["Water"].map(lambda x: x-1 if (x>0) else 0)

df.head()


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
One final resource on why you would want to perform these operations in Pandas - and evidence on how fast it really is! http://wesmckinney.com/blog/high-performance-database-joins-with-pandas-dataframe-more-benchmarks/
Grouping, a.k.a. split-apply-combine
While analysing data, a Data Scientist has to very often perform aggregations, perform transformation ops like standardising data, and filter through the dataset to look at only relevant samples.
This is what the groupby function is primarily used for.
Read more here.
Can you write Python code for it?

import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams["figure.figsize"] = (15,7)

paintball = {'Team': ['Super Ducks','Super Ducks', 'Super Ducks', 'Super Ducks', 'Super Ducks', 'Bobcats', 'Bobcats', 'Bobcats', 'Bobcats', 'Tigers', 'Tigers', 'Tigers', 'Tigers','Tigers','Tigers'],
        'Name': ['Tony', 'Antonio', 'Felipe', 'Ryan', 'Mario', 'Sergio', 'Tanaka', 'Anderson', 'Joe', 'Floyd', 'Manny', 'Chris', 'Junior', 'George','Brock'],
        'Kills': ['1', '1', '1', '4', '3', '2', '2', '2','5', '1', '1', '7', '4','8','5'],
        'Shots Fired Before': [17, 19, 22, 8, 13, 85, 64, 49, 74, 14, 20, 24,13,31,37],
        'Shots Fired After': [41, 73, 57, 30, 74, 37, 28, 40, 43, 18, 19, 21,13,32,39]}
df = pd.DataFrame(paintball, columns = ['Team', 'Name', 'Shots Fired Before', 'Shots Fired After','Kills'])
df

df.groupby('Team').mean()

byteam = df.groupby('Team')
byteam.count()

byteam.describe()

byteam.describe().transpose()['Bobcats']

Team_Before = df[['Shots Fired Before']].groupby(df['Team']).mean()
Team_After = df[['Shots Fired After']].groupby(df['Team']).mean()

Team_Before

Team_After

Team_Before.join(Team_After)

plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = (15,7)
Team_Before.join(Team_After).plot(kind="Bar")


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Personally I am quite partial to ggplot and seaborn, but not so much to fivethirtyeight. Let's try these.
Can you write Python code for it?

plt.style.use('ggplot')
plt.rcParams["figure.figsize"] = (15,7)
Team_Before.join(Team_After).plot(kind="Bar")


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Read about using the Vincent package here.
The latest update to Matplotlib, V 2.0.0 has really improved the quality of the graphics, but it's still not quite production ready, while on the positive side, it is stable and has a large community of people who use it. Niche packages like Vincent can produce some amazing graphics right out of the box with minimal tweaking, but they may not be very mature. Nevertheless, as Data Scientists, it's good to learn about new packages, especially those that help you communicate your results to a non-technical audience. If people don't understand what you do, they won't think what you do is important!
Merge, Join, Concatenate
<img src="images/sql-joins.png">
Image Source: http://www.datapine.com/blog/sql-joins-and-data-analysis-using-sql/
Merge
Can you write Python code for it?

customers = {
        'customer_id': ['101', '102', '103', '104', '105'],
        'first_name': ['Tony', 'Silvio', 'Paulie', 'Corrado', 'Christopher'],
        'last_name': ['Soprano', 'Dante', 'Gualtieri', 'Soprano', 'Moltisanti']}
df_1 = pd.DataFrame(customers, columns = ['customer_id', 'first_name', 'last_name'])
df_1

orders = {
        'customer_id': ['101', '104', '105', '108', '111'],
        'order_date': ['2015-01-01', '2015-01-08', '2015-01-19', '2015-02-10', '2015-02-11'],
        'order_value': ['10000', '25000', '1100', '5000', '4400']}
df_2 = pd.DataFrame(orders, columns = ['customer_id', 'order_date', 'order_value'])
df_2

pd.merge(df_1, df_2, how = 'inner', on = 'customer_id')

pd.merge(df_1, df_2, how = 'left', on = 'customer_id')

pd.merge(df_1, df_2, how = 'right', on = 'customer_id')

pd.merge(df_1, df_2, how = 'outer', on = 'customer_id')


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Read more about these here.
DataFrames
DataFrames is in many respects, the real Pandas. Usually, if you're using Pandas, it will be to use DataFrames.<br>
We will begin with creating DataFrames, and the usual indexing and selection mechanisms. In reality, you will probably never have to 'create' a DataFrame, but practice these skills here to get comfortable with heirarchies, indices and selections. Then we will move on to reading data from multiple formats, including spreadsheets, JSON files and API endpoints.
By the way, during these examples, we will always set seed first when generating random numbers. If you're coming from R, this is the same as set.seed(). In Python, we use the random.seed statement from numpy, which you can read about here. You can set it to any number you like, and I usually set it to 42 just out of habit, but there's not to say you can't set it to an arbitrary number like 27 or 2012. Use the same numbers as this notebook though to replicate the results. Also note that we need to mention it in every cell that we want the results replicated.
You will see later about how this is good practice especially when sharing your work with other members of the team - they will be able to reproduce your work on their machines due to the pseudo-random number that is generated algorithmically.
Can you write Python code for it?

import pandas as pd
import numpy as np

# Let's start with a standard array
arr1 = np.array([[40,40,75,95],[80,85,120,130],
                 [155,160,165,170],[200,245,250,260]])
print(arr1.shape)
print(arr1.size)
print(arr1)

# It is quite common to assign a dataframe the name 'df', although you can
# use a relevant name, such baseball_stats or book_sales
# It's always good to use context driven names - you should code expecting
# someone else to read it a few months down the line

df = pd.DataFrame(arr1, index = "Peter,Clarke,Bruce,Tony".split(","),
                  columns = "Jan,Feb,Mar,Apr".split(","))
df


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Removing Columns
While fundamentally adding and removing columns ought to be similar operations, there are a few differences. Let's see if you can figure it out.
Can you write Python code for it?


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Tips
Can you write Python code for it?

df = pd.read_csv("Data/tips.csv")
df.head()

df["tip_pc"] = df["tip"] / df["total_bill"]

df.head()

pd.pivot_table(df,index=["sex"], values = ["tip_pc"], aggfunc="mean")

pd.pivot_table(df, index = ["smoker", "sex"], values = ["tip_pc"], aggfunc = "mean")

pd.pivot_table(df,index=["sex"], values = ["total_bill","tip"], aggfunc="sum")


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
We can subset and get values from the series.
Can you write Python code for it?

series_4['a'] == series_4[0]

series_4[series_4>3]

series_4[series_4%2==0]

series_5 = pd.Series([1,2,3,4,5], index = ['HP', 'GS', 'IBM', 'AA', 'FB'])
series_5

series_5['IBM']

tech_pf1 = series_5[['HP', 'IBM', 'FB']]
tech_pf1

# From a Dictionary
dict_01 = {'Gavin' : 50, 'Russ' : 100, 'Erlich' : 150}
series_6 = pd.Series(dict_01)
series_6

# Reordering the previous series
index = ['Gavin', 'Russ', 'Erlich', 'Peter']
series_7 = pd.Series(dict_01, index=index)
series_7


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
What about fivethirtyeight?
Can you write Python code for it?

plt.style.use('fivethirtyeight')
plt.rcParams["figure.figsize"] = (15,7)
Team_Before.join(Team_After).plot(kind="Bar")


I am working on the file "12.Introduction_to_Pandas.ipynb".
The first task is:
Why is 'Dec' still there? Here lies the difference - while removing columns, we have to specify that the operation should be inplace. Read about it in the official documentation.
Can you write Python code for it?

df.drop('Dec', axis = 1, inplace = True)
df


I am working on the file "12/311 time series homework.ipynb".
The first task is:
According to your selection of data, how many cases were filed in March? How about May?
Can you write Python code for it?

#confirm this is a date type
# complaint_df['Created Date']

complaint_df.index = complaint_df['Created Date']
# complaint_df.head()

#complaint_df = complaint_df.set_index(['Created Date'])
#complaint_df['Created Date'] = pd.to_datetime(complaint_df['Created Date'])
ax = complaint_df['2015-03'].resample('D').count().plot(y='Unique Key')
# complaint_df['2015-04']['Complaint Type'].count()
ax.set_title('NYC Complaints in March 2015')
ax.set_xlabel('Day of the Month')
ax.set_ylabel('Number of Complaints')
plt.savefig('MarchfreqNYcomplaints.png')

ax = complaint_df['2015-05'].resample('D').count().plot(y='Unique Key')
ax.set_title('NYC Complaints in May 2015')
ax.set_xlabel('Day of the Month')
ax.set_ylabel('Number of Complaints')
plt.savefig('MayfreqNYcomplaints.png')

complaint_df.head()


I am working on the file "12/311 time series homework.ipynb".
The first task is:
First, I made a mistake naming the data set! It's 2015 data, not 2014 data. But yes, still use 311-2014.csv. You can rename it.

Importing and preparing your data
Import your data, but only the first 200,000 rows. You'll also want to change the index to be a datetime based on the Created Date column - you'll want to check if it's already a datetime, and parse it if not.
Can you write Python code for it?

df = pd.read_csv('311-2015.csv', dtype = str)

df.head()

import datetime

def created_date_to_datetime(date_str):
    return datetime.datetime.strptime(date_str, '%m/%d/%Y %I:%M:%S %p')

df['created_datetime'] = df['Created Date'].apply(created_date_to_datetime)

df = df.set_index('created_datetime')


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Graph those same agencies on an annual basis - make it weekly. When do people like to complain? When does the NYPD have an odd number of complaints?
Can you write Python code for it?

def agency_weekly_complaints(agency_name_str):
    agency_complaints = df[df['Agency'] == agency_name_str]
    return agency_complaints['Unique Key'].groupby(agency_complaints.index.week).count()

ax = agency_weekly_complaints('NYPD').plot(label = 'NYPD', legend = True)
for x in ['DOT', 'HPD', 'DPR', 'DSNY']:
    agency_weekly_complaints(x).plot(ax = ax, label = x, legend = True)

NYPD_complaints = df[df['Agency'] == 'NYPD']
NYPD_weekly_complaints = NYPD_complaints['Unique Key'].groupby(NYPD_complaints.index.week).count()
NYPD_weekly_complaints[NYPD_weekly_complaints > 13000]

# # a way to use the function agency_weekly_complaints that's actually longer than not using it.
# week_number = 0
# for week in agency_weekly_complaints('NYPD'):
#     week_number += 1
#     if week > 1500:
#         print('In week', week_number)
#         print('there were', week, 'complaints.')
#         print('')


I am working on the file "12/311 time series homework.ipynb".
The first task is:
I'd like to see all of the 311 complaints called in on April 1st.

Surprise! We couldn't do this in class, but it was just a limitation of our data set
Can you write Python code for it?

print("There were", complaint_df['2015-04-01']['Unique Key'].count(), "complaints on April 1st.")


april_1_complaints = df[(df.index.month == 4) & (df.index.day == 1)][['Unique Key', 'Created Date', 'Complaint Type', 'Descriptor']]
april_1_complaints


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Interesting—it looks cyclical. Let's see what day of the week is most popular:
Can you write Python code for it?

df.index[1]

def get_day_of_wk(timestamp):
    return datetime.datetime.strftime(timestamp, '%a')

df['datetime'] = df.index

df['day_of_wk'] = df['datetime'].apply(get_day_of_wk)

complaining_day_of_wk = df[['Unique Key', 'day_of_wk']].groupby('day_of_wk').count()

complaining_day_of_wk['number_of_day'] = [6, 2, 7, 1, 5, 3, 4]

complaining_day_of_wk_sorted = complaining_day_of_wk.sort_values('number_of_day')

complaining_day_of_wk_sorted

ax = complaining_day_of_wk_sorted.plot(y = 'Unique Key', legend = False)


I am working on the file "12/311 time series homework.ipynb".
The first task is:
It looks like complaints are least popular in around Christmas and New Year's.
Maybe the NYPD deals with different issues at different times? Check the most popular complaints in July and August vs the month of May. Also check the most common complaints for the Housing Preservation Bureau (HPD) in winter vs. summer.
Most popular NYPD complaints in July and August:
Can you write Python code for it?

NYPD_complaints = df[df['Agency'] == 'NYPD']
NYPD_jul_aug_complaints = NYPD_complaints[(NYPD_complaints.index.month == 7) | (NYPD_complaints.index.month == 8)][['Unique Key', 'Complaint Type']]
NYPD_jul_aug_complaints.groupby('Complaint Type').count().sort_values('Unique Key', ascending = False).head()


I am working on the file "12/311 time series homework.ipynb".
The first task is:
It looks like complaints are most popular in May, June, September—generally in the summer.
Can you write Python code for it?


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Most popular HPD complaints in June, July, and August:
Can you write Python code for it?

HPD_complaints = df[df['Agency'] == 'HPD']
HPD_jun_jul_aug_complaints = HPD_complaints[(HPD_complaints.index.month == 6) |
                                           (HPD_complaints.index.month == 7) |
                                           (HPD_complaints.index.month == 8)][['Unique Key', 'Complaint Type']]
HPD_jun_jul_aug_complaints.groupby('Complaint Type').count().sort_values('Unique Key', ascending = False).head()


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Most popular NYPD complaints in May:
Can you write Python code for it?

NYPD_complaints = df[df['Agency'] == 'NYPD']
NYPD_may_complaints = NYPD_complaints[(NYPD_complaints.index.month == 5)][['Unique Key', 'Complaint Type']]
NYPD_may_complaints.groupby('Complaint Type').count().sort_values('Unique Key', ascending = False).head()


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Noise complaints are a big deal. Use .str.contains to select noise complaints, and make an chart of when they show up annually. Then make a chart about when they show up every day (cyclic).
Can you write Python code for it?

ax = complaint_df['Complaint Type'].str.contains('Noise').resample('W').count().plot(y='Unique Key')
ax.set_title('NYC Complaints in 2015')
ax.set_xlabel('Month')
ax.set_ylabel('Number of Complaints')
plt.savefig('AnnualfreqNYcomplaints.png')

ax = complaint_df['Complaint Type'].str.contains('Noise').groupby(by=complaint_df.index.hour).mean().plot(y='Unique Key')

ax.set_title('Daily NYC SoundComplaints in 2015')
ax.set_xlabel('Number of Complaints')
plt.savefig('DailyfreqNYSoundcomplaints.png')

#ax.set_xticks([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])
#ax.set_xticklabels([12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])


I am working on the file "12/311 time series homework.ipynb".
The first task is:
So odd. What's the per-minute breakdown of complaints between 12am and 1am? You don't need to include 1am.
Can you write Python code for it?

midnight_complaints = df[df.index.hour == 0][['Unique Key', 'Complaint Type']]

for minute in range(0, 60):
    top_complaint = midnight_complaints[midnight_complaints.index.minute == minute].groupby('Complaint Type').count().sort_values('Unique Key', ascending = False).head(1)
    if minute < 10:
        minute = '0' + str(minute)
    else:
        minute = str(minute)
    print('12:' + minute + '\'s top complaint was:', top_complaint)
    print('')
# hourly_complaints = df['Unique Key'].groupby(df.index.hour).count()


I am working on the file "12/311 time series homework.ipynb".
The first task is:
What hour of the day are the most complaints? Graph a day of complaints.
Can you write Python code for it?

hourly_complaints = df['Unique Key'].groupby(df.index.hour).count()

ax = hourly_complaints.plot()
ax.set_xticks(range(0,24))
ax.set_xticklabels(['Midnight', '', '', '', '', '',
                    '6 am', '', '', '', '', '',
                    'Noon', '', '', '', '', '',
                    '6 pm', '', '', '', '', ''])
ax.set_ylabel('Number of Complaints')
ax.set_xlabel('Time of Day')
ax.set_title('311 Complaints by Time of Day in 2015')


I am working on the file "12/311 time series homework.ipynb".
The first task is:
What month has the most reports filed? How many? Graph it.
Can you write Python code for it?

complaint_df.resample('M')['Complaint Type'].count().sort_values(ascending=False).head(1)

ax = complaint_df['2015-05'].resample('D').count().plot(y='Unique Key')
ax.set_title('NYC Complaints in May 2015')
ax.set_xlabel('Day of the Month')
ax.set_ylabel('Number of Complaints')
plt.savefig('MayfreqNYcomplaints.png')


I am working on the file "12/311 time series homework.ipynb".
The first task is:
What was the most popular type of complaint on April 1st?
What were the most popular three types of complaint on April 1st
Can you write Python code for it?

complaint_df['2015-04-01'].groupby('Complaint Type')['Complaint Type'].count().sort_values(ascending=False).head(1)

complaint_df['2015-04-01'].groupby('Complaint Type')['Complaint Type'].count().sort_values(ascending=False).head(3)


I am working on the file "12/311 time series homework.ipynb".
The first task is:
What was the most popular type of complaint, and how many times was it filed?
Can you write Python code for it?

freq_complaints = df[['Unique Key', 'Complaint Type']].groupby('Complaint Type').count().sort_values('Unique Key', ascending=False).head()

freq_complaints


I am working on the file "12/311 time series homework.ipynb".
The first task is:
What week of the year has the most reports filed? How many? Graph the weekly complaints.
Can you write Python code for it?

complaint_df.resample('W')['Complaint Type'].count().sort_values(ascending=False).head(3)

complaint_df['2015-05-10'].resample('W')['Complaint Type'].count().plot(y='Unique Key')


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Which borough has the most complaints per capita? Since it's only 5 boroughs, you can do the math manually.
Can you write Python code for it?

df[['Unique Key', 'Borough']].groupby('Borough').count().sort_values('Unique Key', ascending = False)


I am working on the file "12/311 time series homework.ipynb".
The first task is:
Which were the top five days of the year for filing complaints? How many on each of those days? Graph it.
Can you write Python code for it?

top_complaining_days = df['Unique Key'].resample('D').count().sort_values(ascending = False).head()
top_complaining_days

ax = top_complaining_days.plot(kind = 'barh')
ax.set_ylabel('Date')
ax.set_xlabel('Number of Complaints')
ax.set_title('The Top 5 Days for 311 Complaints in 2015')

complaining_days = df['Unique Key'].resample('D').count()
ax = complaining_days.plot()
ax.set_ylabel('Number of Complaints')
ax.set_xlabel('Day of Year')
ax.set_title('Noise Complaints by Day in 2015')


complaint_df.resample('W')['Complaint Type'].count().sort_values(ascending=False).head(5)

#may_17 = complaint_df['2015-05-17']
#ax = may_17.groupby(by=may_17.index.hour).count().plot(y='Unique Key')

may_17 = complaint_df['2015-05-17']
ax = may_17.groupby(by=may_17.index.hour).count().plot(y='Unique Key', label="May 17th")
may_24 = complaint_df['2015-05-24']
may_24.groupby(by=may_24.index.hour).count().plot(y='Unique Key', ax=ax, label = "May 24th")
may_03 = complaint_df['2015-05-03']
may_03.groupby(by=may_03.index.hour).count().plot(y='Unique Key', ax=ax, label = "May 3rd")
may_10 = complaint_df['2015-05-10']
may_10.groupby(by=may_10.index.hour).count().plot(y='Unique Key', ax=ax, label = "May 10th")
may_31 = complaint_df['2015-05-31']
may_31.groupby(by=may_31.index.hour).count().plot(y='Unique Key', ax=ax, label = "May 31st")
ax.set_xticks([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23])
ax.set_xticklabels([12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])

ax

# ax = complaint_df['2015-05-10'].resample('H').count().plot(y='Unique Key')
#ax = complaint_df['2015-05-17'].groupby(by=complaint_df.index.hour).count().plot(y='Unique Key', label = "May 17th")
# complaint_df['2015-05-24'].resample('H')['Complaint Type'].count().plot(y='Unique Key', ax=ax, label = "May 24th")
# complaint_df['2015-05-03'].resample('H')['Complaint Type'].count().plot(y='Unique Key', ax=ax, label = "May 3rd")
# complaint_df['2015-05-31'].resample('H')['Complaint Type'].count().plot(y='Unique Key', ax=ax, label = "May 31st")
# # the least sold in a month (e.g., the worst January ever , the worst July ever, etc)
# complaint_df['2015-05-17'].resample('H')['Complaint Type'].count().plot(y='Unique Key', ax=ax, label = 'May 17th')

# the most sold in a month (e.g., the best January ever , the best July ever, etc)
#df.groupby(by=df.index.month).max().plot(y='val', ax=ax, label = 'Most Sold')


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
According to your selection of data, how many cases were filed in March? How about May?
Can you write Python code for it?

March = df['2015-03']['Unique Key'].count()
May = df['2015-05']['Unique Key'].count()
print("March:", March)
print("May:", May)


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
First, I made a mistake naming the data set! It's 2015 data, not 2014 data. But yes, still use 311-2014.csv. You can rename it.

Importing and preparing your data
Import your data, but only the first 200,000 rows. You'll also want to change the index to be a datetime based on the Created Date column - you'll want to check if it's already a datetime, and parse it if not.
Can you write Python code for it?

df = pd.read_csv("311-2015.csv", nrows=200000)

#total number of rows
print("Total rows: {0}".format(len(df)))

#Alternative way to do it
#df2 = df.ix[:200000]

print("Total rows: {0}".format(len(df))) #20000 data points+1 for columns name

#you can see that df2 dataframe displays 20000 rows of data.
df.tail()

#time variables like "Create Date" and "Closed Date" are not considered datatime
df.info()
df.columns

def parse_date(str_date):
    return dateutil.parser.parse(str_date)

parse_date("07/06/2015 10:58:27 AM")

df['Created Date'].head(2)

df['Date'] = df['Created Date'].apply(parse_date)
df.head(2)

#print(df['Date'])
df.info()

df.index = df['Date']

del df['Date']

df.head(10)


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
I'd like to see all of the 311 complaints called in on April 1st.

Surprise! We couldn't do this in class, but it was just a limitation of our data set
Can you write Python code for it?


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
Looks like midnight is a little bit of an outlier. Why might that be? Take the 5 most common agencies and graph the times they file reports at (all day, not just midnight).
Graph those same agencies on an annual basis - make it weekly. When do people like to complain? When does the NYPD have an odd number of complaints?
Maybe the NYPD deals with different issues at different times? Check the most popular complaints in July and August vs the month of May. Also check the most common complaints for the Housing Preservation Bureau (HPD) in winter vs. summer.
Can you write Python code for it?


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
Make a horizontal bar graph of the top 5 most frequent complaint types.
Can you write Python code for it?

complains = df.groupby('Complaint Type').count()

complains.sort('Unique Key', ascending=False).head(5)

COM = complains.sort('Unique Key', ascending=False)

COM['Unique Key'].head(5)

COM['Unique Key'].head(5).sort_values().plot.barh()

import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
COMG.plot.hist()


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
Noise complaints are a big deal. Use .str.contains to select noise complaints, and make an chart of when they show up annually. Then make a chart about when they show up every day (cyclic).
Can you write Python code for it?

noise = df['Complaint Type'].str.contains("Noise")
noise_df = df[noise]
noise_df.head(2)

noise_df.resample('W')['Unique Key'].count().plot()

noise_df.resample('H')['Unique Key'].count().plot() #resample emember what day, month, hour and year. Waht we want is
#by day only.

noise_df.groupby(by=noise_df.index.hour)['Unique Key'].count().plot()


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
One of the hours has an odd number of complaints. What are the most common complaints at that hour, and what are the most common complaints the hour before and after?
Can you write Python code for it?

midnigth_df = df[df.index.hour == 0]

midnigth_df.head(2)


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
So odd. What's the per-minute breakdown of complaints between 12am and 1am? You don't need to include 1am.
Can you write Python code for it?


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
What hour of the day are the most complaints? Graph a day of complaints.
Can you write Python code for it?


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
What month has the most reports filed? How many? Graph it.
Can you write Python code for it?

df.index.month[:15]

df.groupby(by=df.index.month)['Unique Key'].count()

df.groupby(by=df.index.month)['Unique Key'].count().plot() #all months no matter what year

df.resample('M')['Unique Key'].count() #resample takes care of the month of this year.

df.resample('M')['Unique Key'].count().plot()


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
What was the most popular type of complaint, and how many times was it filed?
Can you write Python code for it?

df['Complaint Type'].describe()
#top= mode. Frequency: 21779 cases

#using mode function in complain type
df['Complaint Type'].mode()


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
What week of the year has the most reports filed? How many? Graph the weekly complaints.
Can you write Python code for it?

df.index.week

df.groupby(by=df.index.week)['Unique Key'].count().plot()

df.resample('W')['Unique Key'].count().plot()


I am working on the file "12/Homework_12_Paolo_Rivas.ipynb".
The first task is:
Which borough has the most complaints per capita? Since it's only 5 boroughs, you can do the math manually.
Can you write Python code for it?

#Now NaNs are 0
df["Incident Zip"].fillna(0, inplace=True)

complains = df['Borough'].value_counts()

#brooklyn 2592000
#staten island 472621
# 1419000
#Man 1626000
#queens 2296000

bcount = pd.DataFrame(df['Borough'].value_counts())
bcount['name'] = bcount.index
bcount

#Merged in pandas need to be learn to solve this!


I am working on the file "12/homework 12 - 311 time series homework.ipynb".
The first task is:
According to your selection of data, how many cases were filed in March? How about May?
Can you write Python code for it?

df['2015-03']['Created Date'].count()

df['2015-05']['Created Date'].count()


I am working on the file "12/homework 12 - 311 time series homework.ipynb".
The first task is:
Graph those same agencies on an annual basis - make it weekly. When do people like to complain? When does the NYPD have an odd number of complaints?
Can you write Python code for it?

ax=df[df['Agency']=='NYPD'].groupby(df[df['Agency']=='NYPD'].index.week)['Unique Key'].count().plot(legend=True, label='NYPD')
df[df['Agency']=='HPD'].groupby(df[df['Agency']=='HPD'].index.week)['Unique Key'].count().plot(ax=ax, legend=True, label='HPD')
df[df['Agency']=='DOT'].groupby(df[df['Agency']=='DOT'].index.week)['Unique Key'].count().plot(ax=ax, legend=True, label='DOT')
df[df['Agency']=='DPR'].groupby(df[df['Agency']=='DPR'].index.week)['Unique Key'].count().plot(ax=ax, legend=True, label='DPR')
df[df['Agency']=='DOHMH'].groupby(df[df['Agency']=='DOHMH'].index.week)['Unique Key'].count().plot(ax=ax, legend=True, label='DOHMH')


I am working on the file "12/homework 12 - 311 time series homework.ipynb".
The first task is:
I'd like to see all of the 311 complaints called in on April 1st.

Surprise! We couldn't do this in class, but it was just a limitation of our data set
Can you write Python code for it?


I am working on the file "12/homework 12 - 311 time series homework.ipynb".
The first task is:
Looks like midnight is a little bit of an outlier. Why might that be? Take the 5 most common agencies and graph the times they file reports at (all day, not just midnight).
Can you write Python code for it?

df.groupby('Agency')['Unique Key'].count().sort_values(ascending=False).head(5)

ax=df[df['Agency']=='NYPD'].groupby(df[df['Agency']=='NYPD'].index.hour)['Unique Key'].count().plot(legend=True, label='NYPD')
df[df['Agency']=='HPD'].groupby(df[df['Agency']=='HPD'].index.hour)['Unique Key'].count().plot(ax=ax, legend=True, label='HPD')
df[df['Agency']=='DOT'].groupby(df[df['Agency']=='DOT'].index.hour)['Unique Key'].count().plot(ax=ax, legend=True, label='DOT')
df[df['Agency']=='DPR'].groupby(df[df['Agency']=='DPR'].index.hour)['Unique Key'].count().plot(ax=ax, legend=True, label='DPR')
df[df['Agency']=='DOHMH'].groupby(df[df['Agency']=='DOHMH'].index.hour)['Unique Key'].count().plot(ax=ax, legend=True, label='DOHMH')