I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
Thinking about your results
Answer these questions about your results. How well does the model predict the data? Where does it fail? Why does it fail where it does?

Note: You can edit the text in this cell by double clicking on it. When you want to render the text, press control + enter

Your answer below
I am not very good at English, so I will use translation software. I hope the translation software can help me express what I mean. I learned the BP algorithm in this lesson. I understand how to update weights. The review helped me understand how to set up hidden nodes. In this project. I think holidays(I means Christmas Day not weekends) should also be a factor. Obviously see the Christmas is not very accurate fitting. The training data does not contain Christmas data. Neural networks are not a good predictor of Christmas. In China may not be a good predictor of Chinese New Year. This special holiday is not well represented in the training data. So it can not be a good fit this special holiday.I tried double the epochs and I found that training and verifying the loss was better than before. However, it is seen that the training and verification losses are getting slower and slower.
Unit tests
Run these unit tests to check the correctness of your network implementation. These tests must all be successful to pass the project.
Can you write Python code for it?

import unittest

inputs = [0.5, -0.2, 0.1]
targets = [0.4]
test_w_i_h = np.array([[0.1, 0.4, -0.3],
                       [-0.2, 0.5, 0.2]])
test_w_h_o = np.array([[0.3, -0.1]])

class TestMethods(unittest.TestCase):

    ##########
    # Unit tests for data loading
    ##########

    def test_data_path(self):
        # Test that file path to dataset has been unaltered
        self.assertTrue(data_path.lower() == 'bike-sharing-dataset/hour.csv')

    def test_data_loaded(self):
        # Test that data frame loaded
        self.assertTrue(isinstance(rides, pd.DataFrame))

    ##########
    # Unit tests for network functionality
    ##########

    def test_activation(self):
        network = NeuralNetwork(3, 2, 1, 0.5)
        # Test that the activation function is a sigmoid
        self.assertTrue(np.all(network.activation_function(0.5) == 1/(1+np.exp(-0.5))))

    def test_train(self):
        # Test that weights are updated correctly on training
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        network.train(inputs, targets)
        self.assertTrue(np.allclose(network.weights_hidden_to_output,
                                    np.array([[ 0.37275328, -0.03172939]])))
        self.assertTrue(np.allclose(network.weights_input_to_hidden,
                                    np.array([[ 0.10562014,  0.39775194, -0.29887597],
                                              [-0.20185996,  0.50074398,  0.19962801]])))

    def test_run(self):
        # Test correctness of run method
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        self.assertTrue(np.allclose(network.run(inputs), 0.09998924))

suite = unittest.TestLoader().loadTestsFromModule(TestMethods())
unittest.TextTestRunner().run(suite)


I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
Time to build the network
Below you'll build your network. We've built out the structure and the backwards pass. You'll implement the forward pass through the network. You'll also set the hyperparameters: the learning rate, the number of hidden units, and the number of training passes.
The network has two layers, a hidden layer and an output layer. The hidden layer will use the sigmoid function for activations. The output layer has only one node and is used for the regression, the output of the node is the same as the input of the node. That is, the activation function is $f(x)=x$. A function that takes the input signal and generates an output signal, but takes into account the threshold, is called an activation function. We work through each layer of our network calculating the outputs for each neuron. All of the outputs from one layer become inputs to the neurons on the next layer. This process is called forward propagation.
We use the weights to propagate signals forward from the input to the output layers in a neural network. We use the weights to also propagate error backwards from the output back into the network to update our weights. This is called backpropagation.

Hint: You'll need the derivative of the output activation function ($f(x) = x$) for the backpropagation implementation. If you aren't familiar with calculus, this function is equivalent to the equation $y = x$. What is the slope of that equation? That is the derivative of $f(x)$.

Below, you have these tasks:
1. Implement the sigmoid function to use as the activation function. Set self.activation_function in __init__ to your sigmoid function.
2. Implement the forward pass in the train method.
3. Implement the backpropagation algorithm in the train method, including calculating the output error.
4. Implement the forward pass in the run method.
Can you write Python code for it?

class NeuralNetwork(object):
    def __init__(self, input_nodes, hidden_nodes, output_nodes, learning_rate):
        # Set number of nodes in input, hidden and output layers.
        self.input_nodes = input_nodes
        self.hidden_nodes = hidden_nodes
        self.output_nodes = output_nodes

        # Initialize weights
        self.weights_input_to_hidden = np.random.normal(0.0, self.hidden_nodes ** -0.5,
                                                        (self.hidden_nodes, self.input_nodes))

        self.weights_hidden_to_output = np.random.normal(0.0, self.output_nodes ** -0.5,
                                                         (self.output_nodes, self.hidden_nodes))
        self.lr = learning_rate

        #### Set this to your implemented sigmoid function ####
        # Activation function is the sigmoid function
        self.activation_function = lambda x : 1/(1+np.exp(-x))
        self.activation_function_derivative= lambda x : x * (1 - x)

    def train(self, inputs_list, targets_list):
        # Convert inputs list to 2d array
        inputs = np.array(inputs_list, ndmin=2).T
        targets = np.array(targets_list, ndmin=2).T

        #### Implement the forward pass here ####
        ### Forward pass ###
        # TODO: Hidden layer
        hidden_inputs = np.dot(self.weights_input_to_hidden, inputs)
        hidden_outputs = self.activation_function(hidden_inputs)

        # TODO: Output layer
        final_inputs = np.dot(self.weights_hidden_to_output, hidden_outputs)
        final_outputs = final_inputs

        #### Implement the backward pass here ####
        ### Backward pass ###

        # TODO: Output error
        output_errors = targets - final_outputs  # Output layer error is the difference between desired target and actual output.
        output_grad = 1  # hidden layer gradients

        # TODO: Backpropagated error
        hidden_errors = np.dot(self.weights_hidden_to_output.T, output_errors)  # errors propagated to the hidden layer
        hidden_grad =...


I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
Training the network
Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.
You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.
Choose the number of epochs
This is the number of times the dataset will pass through the network, each time updating the weights. As the number of epochs increases, the network becomes better and better at predicting the targets in the training set. You'll need to choose enough epochs to train the network well but not too many or you'll be overfitting.
Choose the learning rate
This scales the size of weight updates. If this is too big, the weights tend to explode and the network fails to fit the data. A good choice to start at is 0.1. If the network has problems fitting the data, try reducing the learning rate. Note that the lower the learning rate, the smaller the steps are in the weight updates and the longer it takes for the neural network to converge.
Choose the number of hidden nodes
The more hidden nodes you have, the more accurate predictions the model will make. Try a few different numbers and see how it affects the performance. You can look at the losses dictionary for a metric of the network performance. If the number of hidden units is too low, then the model won't have enough space to learn and if it is too high there are too many options for the...
Can you write Python code for it?

import sys

### Set the hyperparameters here ###
epochs = 2000
learning_rate = 0.01
hidden_nodes = 23
output_nodes = 1

N_i = train_features.shape[1]
network = NeuralNetwork(N_i, hidden_nodes, output_nodes, learning_rate)

losses = {'train':[], 'validation':[]}
for e in range(epochs):
    # Go through a random batch of 128 records from the training data set
    batch = np.random.choice(train_features.index, size=128)
    for record, target in zip(train_features.ix[batch].values,
                              train_targets.ix[batch]['cnt']):
        network.train(record, target)

    # Printing out the training progress
    train_loss = MSE(network.run(train_features), train_targets['cnt'].values)
    val_loss = MSE(network.run(val_features), val_targets['cnt'].values)
    sys.stdout.write("\rProgress: " + str(100 * e/float(epochs))[:4] \
                     + "% ... Training loss: " + str(train_loss)[:5] \
                     + " ... Validation loss: " + str(val_loss)[:5])

    losses['train'].append(train_loss)
    losses['validation'].append(val_loss)

plt.plot(losses['train'], label='Training loss')
plt.plot(losses['validation'], label='Validation loss')
plt.legend()
plt.ylim(ymax=0.5)


I am working on the file "1/dlnd-your-first-neural-network.ipynb".
The first task is:
We'll split the data into two sets, one for training and one for validating as the network is being trained. Since this is time series data, we'll train on historical data, then try to predict on future data (the validation set).
Can you write Python code for it?

# Hold out the last 60 days of the remaining data as a validation set
train_features, train_targets = features[:-60*24], targets[:-60*24]
val_features, val_targets = features[-60*24:], targets[-60*24:]


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:

We can specify row ranges:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Add, subtract, multiply, divide:
Operations are element-wise:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
CSV Files
Can you write Python code for it?

df1.to_csv('datafiles/pandas_df1.csv')
!ls datafiles

df2 = pd.read_csv('datafiles/pandas_df1.csv', index_col=0)
print(df2)


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Concatenating and Merging Data
Bringing together DataFrames or Series objects:
Concatenate
Can you write Python code for it?

df1 = pd.DataFrame(np.zeros([3,3], dtype=np.int))
df1

df2 = pd.concat([df1, df1], axis=0)
df2 = df2.reset_index(drop=True) # Renumber indexing
df2


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Data Input Output
Can you write Python code for it?

df1 = pd.DataFrame(np.random.randn(5,4), columns = ['A','B','C','D'], index=['zero','one','two','three','four'])
print(df1)


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Describe
Excluding NaN values, print some descriptive statistics about the collection of values.
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Fast single element label indexing (.iat) - fast .iloc
Intended for fast, single indexes.
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Grouping Operations
Often, there is a need to summarize the data or change the output of the data to make it easier to work with, especially for categorical data types.
Can you write Python code for it?

dfg = pd.DataFrame({'A': ['clogs','sandals','jellies']*2,
                    'B': ['socks','footies']*3,
                    'C': [1,1,1,3,2,2],
                    'D': np.random.rand(6)})
dfg


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Head and Tail
Print the beginning and ending entries of a pandas data structure
Can you write Python code for it?

df1.head(3) # Show the first n rows, default is 5

df1.tail() # Show the last n rows


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Histogram
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Indexing Data
Pandas provides the means to index data via named columns, or as numpy like indices. Indexing is [row, column], just as it was in numpy.
Data is visible via column:
Can you write Python code for it?

df1['A'].head() # df1.A.head() is equivalent


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Indices can be named:
Can you write Python code for it?

# Rename the columns
df1.columns = ['A','B','C','D']
df1.index = ['zero','one']
df1

# Create the dataframe with the columns
df1 = pd.DataFrame([[1,2,3,4],[10,20,30,40]], columns=['A','B','C',"D"], index=['zero','one'])
df1


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Integer based indexing (.iloc)
Slice based on the index number.
Can you write Python code for it?

df1.iloc[:3,0] # Note that this does not include the upper index like numpy


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Lambda functions may also be used
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Logical indexing
A condition is used to select the values within a slice or the entire Pandas object. Using a conditional statement, a true/false DataFrame is produced:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Logical indexing via isin
It's also possible to filter via the index value:
Can you write Python code for it?

df_types

bool_series = df_types['cow color'].isin(['blue'])
print(bool_series) # Show the logical indexing

df_types[bool_series] # Index where the values are true


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Note that tab completion is enabled for column names:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Operations on Pandas Data Objects
Wether it's the entire data frame or a series within a single dataframe, there are a variety of methods that can be applied. Here's a list of a few helpful ones:
Simple statistics (mean, stdev, etc).
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Pivot Table
Without changing the data in any way, summarize the output in a different format. Specify the indicies, columns, and values:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Plotting
Pandas exposes the matplotlib library for use.
Can you write Python code for it?

n = 100
X = np.linspace(0, 5, n)
Y1,Y2 = np.log((X)**2+2), np.sin(X)+2

dfp = pd.DataFrame({'X' : X, 'Y1': Y1, 'Y2': Y2})
dfp.head()

dfp.plot(x = 'X')
plt.show()


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Print a logical DataFrame where NaN is located:
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Rotation
Note that the values rotated out leave NaN behind:
Can you write Python code for it?

dfg['D']

dfg_Ds = dfg['D'].shift(2)
dfg_Ds


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
SQL-like merging
Pandas can do structured query language (SQL) like merges of data:
Can you write Python code for it?

left = pd.DataFrame({'numbers': ['K0', 'K1', 'K2', 'K3'],
    'English': ['one', 'two', 'three', 'four'],
    'Spanish': ['uno', 'dos', 'tres', 'quatro'],
    'German': ['erste', 'zweite','dritte','vierte']})
left

right = pd.DataFrame({'numbers': ['K0', 'K1', 'K2', 'K3'],
    'French': ['un', 'deux', 'trois', 'quatre'],
    'Afrikaans': ['een', 'twee', 'drie', 'vier']})
right

result = pd.merge(left, right, on='numbers')
result


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Sorting by column
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
Stacking
Column labels can be brought into the rows.
Can you write Python code for it?


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
String functions:
Pandas has access to string methods:
Can you write Python code for it?

dfg['A'].str.title() # Make the first letter uppercase


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
That matrix can then be used to index the DataFrame:
Can you write Python code for it?

df1[df1>0.5].head() # Note that the values that were 'False' are 'NaN'


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
We can also set the 'cow color' column to a category:
Can you write Python code for it?

df_types['cow color'] = df_types['cow color'].astype("category")
df_types.dtypes


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
We can create dataframes of multiple datatypes:
Can you write Python code for it?

col1 = range(6)
col2 = np.random.rand(6)
col3 = ['zero','one','two','three','four','five']
col4 = ['blue', 'cow','blue', 'cow','blue', 'cow']

df_types = pd.DataFrame( {'integers': col1, 'floats': col2, 'words': col3, 'cow color': col4} )
print(df_types)

df_types.dtypes


I am working on the file "10 - Pandas Crash Course.ipynb".
The first task is:
hdf5 files
Can you write Python code for it?

df1.to_hdf('datafiles/pandas_df1.h5', 'df')
!ls datafiles

df2 = pd.read_hdf('datafiles/pandas_df1.h5', 'df')
print(df2)


I am working on the file "10 - Working With Text Data.ipynb".
The first task is:
Working with Text Data
<img src="figures/bag_of_words.svg" width=100%>
Can you write Python code for it?

import pandas as pd
import os

data = pd.read_csv(os.path.join("data", "train.csv"))

len(data)

data

y_train = np.array(data.Insult)

y_train

text_train = data.Comment.tolist()

text_train[6]

data_test = pd.read_csv(os.path.join("data", "test_with_solutions.csv"))

text_test, y_test = data_test.Comment.tolist(), np.array(data_test.Insult)

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer()
cv.fit(text_train)

len(cv.vocabulary_)

print(cv.get_feature_names()[:50])
print(cv.get_feature_names()[-50:])

X_train = cv.transform(text_train)

X_train

text_train[6]

X_train[6, :].nonzero()[1]

X_test = cv.transform(text_test)

from sklearn.svm import LinearSVC
svm = LinearSVC()

svm.fit(X_train, y_train)

svm.score(X_train, y_train)

svm.score(X_test, y_test)

def visualize_coefficients(classifier, feature_names, n_top_features=25):
    # get coefficients with large absolute values
    coef = classifier.coef_.ravel()
    positive_coefficients = np.argsort(coef)[-n_top_features:]
    negative_coefficients = np.argsort(coef)[:n_top_features]
    interesting_coefficients = np.hstack([negative_coefficients, positive_coefficients])
    # plot them
    plt.figure(figsize=(15, 5))
    colors = ["red" if c < 0 else "blue" for c in coef[interesting_coefficients]]
    plt.bar(np.arange(50), coef[interesting_coefficients], color=colors)
    feature_names = np.array(feature_names)
    plt.xticks(np.arange(1, 51), feature_names[interesting_coefficients], rotation=60, ha="right");

visualize_coefficients(svm, cv.get_feature_names())


I am working on the file "10 External files.ipynb".
The first task is:
After reading in the current line readline() then moves on to the next line. So calling readline() again uses the next line in the file. One other thing to bear in mind is that readline() leaves whitespace and in particular the \n character at the end of the line. You can see that above (there's a blank line between the printed lines) in the following example.
Can you write Python code for it?

line = f_hand.readline() # note next line has been read
line # compare to print above


I am working on the file "10 External files.ipynb".
The first task is:
Ex. no 2
Print all the lines in the file where the Age value is in the range [70, 80)
Can you write Python code for it?

f_hand = open(file_loc)
for i, line in enumerate(f_hand):
    if i == 0:
        continue
    line = line.strip()
    _, age, *_ = line.split('\t')
    if 70 <= int(age) < 80:
        print(line)


I am working on the file "10 External files.ipynb".
The first task is:
Ex. no 3
Print the two lines in the files for each gender corresponding to the two entries with the (relative) maximum value of body weight (body wt) plus height (ht).
Sol #1:  Using a Dictionary
Can you write Python code for it?

info = {}  # Dictonary holding per-sex lines info
f_hand = open(file_loc)
lines = f_hand.read().splitlines()
for l in lines[1:]:
    l = l.strip()
    key = l[0]
    info.setdefault(key, [])
    info[key].append(tuple(l.split('\t')))

from pprint import pprint  # pprint is for **pretty printing** structures
pprint(info)

max_male = max(info['M'], key=lambda e: float(e[2]) + float(e[3]))
print(max_male)


I am working on the file "10 External files.ipynb".
The first task is:
Exercises
Show the content of the file using a Shell command
Tip 1: The shell command to be used could be cat
Tip 2: Remember the ! (esclamation mark)
Can you write Python code for it?


I am working on the file "10 External files.ipynb".
The first task is:
If you run the above code a new file should appear in your data directory (notice we opened the writeable file in the /data directory) called test.txt. That file should have 10 lines in it with the word 'Line' and a number from 0-9.
In the above code we first opened (created) the file test.txt and then ran through a range of numbers (from 0 to 9) using a for loop. At each iteration of the loop we concatenated (joined) the word 'Line' to the string representation of the number (note the use of str) and a newline character. Finally we wrote each of the resulting strings to our new file. In the last line we closed the file.
Putting it together!
Write a script that uses the csv module to open a file after getting a filepath from the user. Use the script to open the elderlyHeightWeight.csv file. Write out a new file containing only male data. Remember to close all the files once your done. In addition include a try\except clause to handle the situation where the requested file doesn't exist.
Hint: csv.reader objects are lists. Recall how you .join() lists elements into a string.
Adding data to an existing file
As noted above if you open an existing file and write data to it all that pre-existing data gets over written. That's not usually what you want to do. In fact in general you probably never want to write to any file that has raw data you are going to analyse in it - because you might lose or screw-up your original data. Sometimes however you might want to add new measurements (perhaps taken over time) to an existing file. For these cases there's the 'a' argument to the open() function. The a stands for append. Let's take the file containing only the male data we wrote in the last exercise, open it in append mode and write the female data to that file.
Can you write Python code for it?

import csv

# assumes your file was called male_data.tsv
try:
    with open('data/male_data.tsv', 'a') as new_file, open('data/elderlyHeightWeight.csv', 'r') as f_hand:
        reader = csv.reader(f_hand, delimiter='\t') # define the field delimiter
        male_data = [line for line in reader if line[0] == 'M']
        for line in male_data:
            new_file.write('\t'.join(line)+'\n')
except FileNotFoundError:
    print('The file does not exist.')


I am working on the file "10 External files.ipynb".
The first task is:
In the above example we first create the file handle and then read the entire contents of the file into one string. We check the length of that string (286 characters including whitespace characters like \n) and we print the first 10 characters (refer back to the material on slicing if you're unsure how the [:10] slice works). The print statement interprets the tab stop properly but if we just ask for the first 22 characters to be returned (i.e. we do not use print) we can see the tab stop and the \n. Compare this to the illustration of a .tsv file shown above.
Using the print statement and subsetting is fine but not convenient. You might want to print the whole of the first line. The readline() method will read one line at a time and you can use this to e.g. just display the header line (if you know or suspect that your file has a header line).
Can you write Python code for it?

f_hand = open(file_loc, 'r')
line = f_hand.readline() # reads first line
print(line)
# next line
line=f_hand.readline()
print(line)


I am working on the file "10 External files.ipynb".
The first task is:
In the example above we have opened a small file containing height and weight information on a group of elderly men and women from a body composition study.
In the first line we assign the file path to the variable file_loc. We then use that variable to open a file handle (f_hand) using the open() function. The file location is the first argument to the open() function and the second argument 'r' indicates that we want to read from the file i.e. we want access to the data in the file but we don't want to change the file itself. Finally we use a print statement to print the file handle.
The results of the print statement might surprise you. Rather than printing the contents of the file what we get is a representation of the location in the computer memory of where the file is i.e. at memory location 0x7... etc in the example above.
Whilst we can also open Excel or Word files in python this requires the use of special software libraries. We'll see some of those later in the course. Mostly when we are analysing data in files we open simple text files. Both Excel and Word can save files out as simple text files.
The first few lines of the file we will work with are shown below.
Can you write Python code for it?


I am working on the file "10 External files.ipynb".
The first task is:
It's that simple! Now python makes available to us all the useful code in the csv library. The csv library, unsurprisingly, contains python functions and methods to make dealing with csv (and other) text files easier. Let's first see how to open a text file using the csv library and printing out the first few lines.
To read data from a csv file, we use the reader() function. The reader() function takes each line of the file and makes a reader object containing lists made up of each row in the input data. Objects in programming are containers for both data and methods that act on that data (a bit esoteric so don't worry if you don't quite get that). One method the reader object supports is the .next() method. We can use this to access each row at a time. Notably once we have processed the line it's gone from the reader object.
Note:
From here on, we are going to keep using the with/as statement to handle I/O operations, namely Context Manager objects.
For more information, see this notebook.
Can you write Python code for it?

# import csv - already done
with open('data/elderlyHeightWeight.csv', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter='\t') # define the field delimiter
    header = next(reader)
    print (header)
    print () # blank line

    for i in range(4):
        print (next(reader)) # print the first 4 lines after the header


I am working on the file "10 External files.ipynb".
The first task is:
Just like readline() the readlines() method leaves the trailing \n at the end of the line but you can use strip() to remove it if you have to as we did above. We had to use the strip() method on the individual line rather than on the list of lines as lists strip() does not operate on lists. Try moving the strip() to the end of lines = f_hand.readlines() and see what kind if error you get.
Finally (and perhaps most usefully) there is the splitlines() method that does the same as readlines() but drops the trailing \n automatically.
Can you write Python code for it?

f_hand = open(file_loc, 'r')
lines = f_hand.read().splitlines() # read file, then split lines to lists, drops trailing \n
for i in range(4):
    print (lines[i])


I am working on the file "10 External files.ipynb".
The first task is:
Remember that the zip() function will create an iterator (i.e. zip object) made up of tuples. In the example above the use of zip() creates a sequence the first element of which is all the first elements of our data lists, the second list element is all the second elements etc. It's easier to see this than explain it.
Can you write Python code for it?

zip_sequence = zip(gender, age, weight, height, height_m)
print(type(zip_sequence))

print (gender[:4])
print (age[:4])
print (weight[:4])
print (height[:4])
print (height_m[:4])
print # just a blank line
print (list(zip_sequence)[:4])


I am working on the file "10 External files.ipynb".
The first task is:
Sol. #2: Using a list comprehension
Can you write Python code for it?

## Creating Partial Lists using **List Comprehension**
males = [l.strip().split('\t') for l in lines[1:]
             if l.startswith('M')]
females = [l.strip().split('\t') for l in lines[1:]
               if l.startswith('F')]

males

max_male = max(males, key=lambda e: float(e[2]) + float(e[3]))
print(max_male)


I am working on the file "10 External files.ipynb".
The first task is:
Use of strip() has removed the \n from our selected line. There are also lstrip() and rstrip() methods that strip whitespace from only the left or right sides of a string respectively.
Just to confuse you further there's also a readlines() method that reads all the lines in the file into a list. Again the lines are separated on the invisible \n character. This can be handy because you can assign the list to a variable and then loop through the list to print file lines or simply extract the lines you want using slice notation.
Can you write Python code for it?

f_hand = open(file_loc, 'r')
lines = f_hand.readlines()
print(lines[0:2]) # check we have a list
print(len(lines))

for i in range(4):
    print (lines[i].strip())


I am working on the file "10 External files.ipynb".
The first task is:
Using lambda expressions
Can you write Python code for it?

f_hand = open(file_loc)
male_gender = filter(lambda l: l.startswith('M'), f_hand)
for ml in male_gender:
    print(ml.strip())
f_hand.close()


I am working on the file "10 External files.ipynb".
The first task is:
We can see that the reader() function has processed each line into a single list element based on the field delimiter we supplied. Importantly also note that all the values are now of type str in each list (everything is in quotes). This is important if you want to do calculations on these values.
Using the csv module makes it easy to select whole columns by selecting the data we want from the reader.  We'll use the .next() method to find the column order and then iterate over the rows with a for loop to pull out height and weight.
Can you write Python code for it?

with open('data/elderlyHeightWeight.csv', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter='\t') # define the field delimiter

    # use next() method on reader object to id the headers
    headers = next(reader)
    print(headers)

    # we now know weight index is 2, height index is 3

    weight  = ['Weight'] # list to hold data, put in header
    height = ['Height']

    for row in reader:
        weight.append(row[2])
        height.append(row[3])

    print (weight)
    print (height)


I am working on the file "10 External files.ipynb".
The first task is:
While the file handle does not contain the data from the file (it only points at it) it is easy to construct a for loop to cycle through the lines of the file and carry out some computation. For example we can easily count the number of lines in the file.
Can you write Python code for it?

count = 0 # initialise

for line in f_hand: # iterate
    count = count + 1

print('There are %d lines in the file.' % count    )


I am working on the file "10 External files.ipynb".
The first task is:
Why doesn't open() open the file directly?
It might seem stupid that after using open() a file handle is created but the file contents aren't directly available. The reason the open() function does not read the whole file immediately but just points to it is to do with file size. If you do not know in advance the size of the files you are dealing with (often the case - how often do you check the size of files you open on your computer?) automatically opening very large files could:

Take a long time
Crash the whole computer system - essentially you'd run out of memory

For some biological applications the data files (which may well be text files e.g. in RNA Seq data - see comment here) can be very large. So it's safer to point to the file rather than automatically open it. This is also true of other data or informatics applications.
In the for loop above python splits the file into lines based on the newline character (\n - the split is implicit), increments the count variable by 1 for every line and then discards that line. So there is only ever one line from the file in the computer memory at any given time.
If you know that your file is likely to be small you can read the whole file into memory with the read() method (remember the dot notation!).
This reads the entire contents of the file, newlines and all, into one large string.
Can you write Python code for it?

file_loc = 'data/elderlyHeightWeight.csv'
f_hand = open(file_loc, 'r')
f_data = f_hand.read()
print(f_data)

f_hand.close()
f_hand = open(file_loc)
print(f_hand.readline(), end = '')
print(f_hand.readline(), end = '')
f_hand.close()
f_hand = open(file_loc)
lines = f_hand.readlines()
print(type(lines))

file_loc = 'data/elderlyHeightWeight.csv'
f_hand = open(file_loc, 'r')
f_data = f_hand.read()
f_data[:22]
print(len(f_data))
print( f_data[:10])
r'{}'.format(f_data[:22]) # note the tab stop in the output


I am working on the file "10 External files.ipynb".
The first task is:
filter alternative
Can you write Python code for it?

def filter_function(line):
    return line.startswith('M')

f_hand = open(file_loc)
lines = f_hand.readlines()
male_gender = filter(filter_function, lines)
print(lines[0].strip())
for ml in male_gender:
    print(ml.strip())
f_hand.close()


I am working on the file "10 External files.ipynb".
The first task is:
no. 2
Can you write Python code for it?

for i, line in enumerate(lines):
    if i == 4:
        break
    print(line.strip())


I am working on the file "10 Minutes to pandas.ipynb".
The first task is:
Object creation

Intro to Data Structures — pandas 0.20.1 documentation
Series と DataFrame を作る練習
Can you write Python code for it?

# Creating a Series by passing a list of values, letting pandas create a default integer index:
# リストを指定してシリーズを作成すると、Pandasはデフォルトで数値のインデックスを生成する
s = pd.Series([1,3,5,np.nan,6,8])
s

# まずは日付のインデックスを作成
dates = pd.date_range('20130101', periods=6)
dates

# Creating a DataFrame by passing a numpy array, with a datetime index and labeled columns:
# numpy arrayを渡してDataFrameを作成する時に、インデックスと列名を指定
df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))
df

# Creating a DataFrame by passing a dict of objects that can be converted to series-like.
# DataFrame作成時に辞書を渡すと、それぞれの辞書の値を Series のように扱う

df2 = pd.DataFrame({ 'A' : 1.,
                     'B' : pd.Timestamp('20130102'),
                     'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
                     'D' : np.array([3] * 4,dtype='int32'),
                     'E' : pd.Categorical(["test","train","test","train"]),
                     'F' : 'foo' })
df2

# Having specific dtypes
# それぞれ特定のデータ型(dtypes)を持つ

df2.dtypes

# If you’re using IPython, tab completion for column names (as well as public attributes) is automatically enabled.
# Here’s a subset of the attributes that will be completed:
# IPython使っていたら db2. で TAB を入力すると列名が自動的に補完対象になる
# Jupyter notebok だと TAB を入力すると補完するための一覧がリストボックスで表示される


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Download a sample dataset. Next, grab a copy of today's dataset, which is a small (~ 20%) subset of the full dataset:

[SQLite DB, ~ 257 MiB] http://cse6040.gatech.edu/fa15/NYC-311-2M.db

Connect to this database as you did in the last lab.
Can you write Python code for it?

# SQLite database filename
DB_FILENAME = 'NYC-311-2M.db'

# Connect
import sqlite3 as db
disk_engine = db.connect (DB_FILENAME)


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Exercise. Determine the Top 10 whiniest cities. (That is, the 10 cities with the largest numbers of complaints.)
Can you write Python code for it?

# Insert your answer here
query = '''
'''
df = pd.read_sql_query (query, disk_engine)
df.head (10)


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Finding unique values: DISTINCT qualifier. Yet another common idiom is to ask for the unique values of some attribute, for which you can use the DISTINCT qualifier.
Can you write Python code for it?

query = 'SELECT DISTINCT City FROM data'
df = pd.read_sql_query(query, disk_engine)

print ("Found %d unique cities. The first few are:" % len (df))
df.head()


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
From the above data, what would you conclude about the various areas of NY city?
Part 2: SQLite time series with Pandas
Filter SQLite rows with timestamp strings: YYYY-MM-DD hh:mm:ss
Can you write Python code for it?

query = '''
  SELECT ComplaintType, CreatedDate, City
    FROM data
    WHERE CreatedDate < "2015-09-15 23:59:59"
          AND CreatedDate > "2015-09-15 00:00:00"
'''
df = pd.read_sql_query (query, disk_engine)
df


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Learn more

Find more open data sets on Data.gov and NYC Open Data
Learn how to setup MySql with Pandas and Plotly
Add interactive widgets to IPython notebooks for customized data exploration
Big data workflows with HDF5 and Pandas
Interactive graphing with Plotly
Can you write Python code for it?

#from IPython.core.display import HTML
#import urllib2
#HTML(urllib2.urlopen('https://raw.githubusercontent.com/plotly/python-user-guide/css-updates/custom.css').read())


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
More SQL stuff
Partial queries: LIMIT clause. The preceding command was overkill for what we wanted, which was just to preview the table. Instead, we could have used the LIMIT option to ask for just a few results.
Can you write Python code for it?

query = '''
  SELECT *
    FROM data
    LIMIT 5
'''
start_time = time.time ()
df = pd.read_sql_query(query, disk_engine)
elapsed_time = time.time () - start_time
print ("==> LIMIT version took %g seconds." % elapsed_time)

df


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Multiple series in plotly
Here is another example of how to use a query to extract data and then recombine the results into a plot.
Suppose we want to look at the number of complaints by type and by city.
Furthermore, suppose we want to render these results as a bar chart with "complaints" along the x-axis and cumulative counts, as stacked bars, along the y-axis, where different bars correspond to different cities. The plotly package requires that we create a list of traces, where each trace is a series to plot.
Here's how we might construct such a list of traces.
Can you write Python code for it?

traces = []

for city in TOP_CITIES:
    query = '''
      SELECT ComplaintType, COUNT(*) as NumComplaints
        FROM data
        WHERE City = "{}" COLLATE NOCASE
        GROUP BY ComplaintType
        ORDER BY -NumComplaints
    '''.format (city)
    df = pd.read_sql_query (query, disk_engine)

    traces.append (Bar (x=df['ComplaintType'],
                        y=df.NumComplaints,
                        name=city.capitalize()))


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Next, as a quick test let's make a simple plot using the "baby names" data set from Lab 8.
Can you write Python code for it?

import pandas as pd

# Build a Pandas data frame
names = ['Bob','Jessica','Mary','John','Mel']
births = [968, 155, 77, 578, 973]
BabyDataSet = zip (names, births)
df = pd.DataFrame(data=BabyDataSet, columns=['Names', 'Births'])
df

# Plot, using `plotly`
from plotly.graph_objs import Bar

plot_data = [Bar (x=df.Names, y=df.Births)]
py.iplot (plot_data)


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Note that the above example prints the bottom (tail) of the data frame. You could have also asked for the query results in reverse (descending) order, by prefixing the ORDER BY attribute with a - (minus) symbol.
Can you write Python code for it?

query = '''
  SELECT Agency, COUNT(*) as NumComplaints
    FROM data
    GROUP BY Agency
    ORDER BY -NumComplaints
'''
df = pd.read_sql_query (query, disk_engine)
df.head ()


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Preview the data. This sample database has just a single table, named data. Let's query it and see how long it takes to read. To carry out the query, we will use the SQL reader built into pandas.
Can you write Python code for it?

import time

print ("Reading ...")
start_time = time.time ()

# Perform SQL query through the disk_engine connection.
# The return value is a pandas data frame.
df = pd.read_sql_query('SELECT * FROM data', disk_engine)

elapsed_time = time.time () - start_time
print ("==> Took %g seconds." % elapsed_time)

# Dump the first few rows
df.head()


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Renaming columns: AS operator. Sometimes you might want to rename a result column. For instance, the following query counts the number of complaints by "Agency," using the COUNT(*) function and GROUP BY clause, which we discussed in an earlier lab. If you wish to refer to the counts column of the resulting data frame, you can give it a more "friendly" name using the AS operator.
Can you write Python code for it?

query = '''
  SELECT Agency, COUNT(*) AS NumComplaints
    FROM data
    GROUP BY Agency
'''
df = pd.read_sql_query (query, disk_engine)
df.head()


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Segregate complaints by hour
Can you write Python code for it?

complaint_traces = {} # Each series in the graph will represent a complaint
complaint_traces['Other'] = {}

for hour in range(1, 24):
    hour_str = '0'+str(hour) if hour < 10 else str(hour)
    query = '''
      SELECT CreatedDate,
             ComplaintType,
             STRFTIME ('%H', CreatedDate) AS Hour,
             COUNT (*) AS NumComplaints
        FROM data
        WHERE Hour = "{}"
        GROUP BY ComplaintType
        ORDER BY -NumComplaints
    '''.format (hour_str)
    df = pd.read_sql_query (query, disk_engine)

    complaint_traces['Other'][hour] = sum (df.NumComplaints)

    # Grab the 7 most common complaints for that hour
    for i in range(7):
        complaint = df.get_value(i, 'ComplaintType')
        count = df.get_value(i, 'NumComplaints')
        complaint_traces['Other'][hour] -= count
        if complaint in complaint_traces:
            complaint_traces[complaint][hour] = count
        else:
            complaint_traces[complaint] = {hour: count}

traces = []
for complaint in complaint_traces:
    traces.append({
        'x': range(25),
        'y': [complaint_traces[complaint].get(i, None) for i in range(25)],
        'name': complaint,
        'type': 'bar'
    })

py.iplot({
    'data': traces,
    'layout': {
        'barmode': 'stack',
        'xaxis': {'title': 'Hour in Day'},
        'yaxis': {'title': 'Number of Complaints'},
        'title': 'The 7 Most Common 311 Complaints by Hour in a Day'
    }}, filename='311/most common complaints by hour')


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
Set membership: IN operator. Another common idiom is to ask for rows whose attributes fall within a set, for which you can use the IN operator.
Can you write Python code for it?

query = '''
  SELECT ComplaintType, Descriptor, Agency
    FROM data
    WHERE Agency IN ("NYPD", "DOB")
    LIMIT 10
'''

df = pd.read_sql_query (query, disk_engine)
df.head()


I am working on the file "10--nyc-whiners.ipynb".
The first task is:
You should notice two bits of funny behavior, namely, that cities are treated in a case-sensitive manner and that None appears as a city. (Presumably this setting occurs when a complaint is non-localized or the city is not otherwise specified.)
Case-insensitive grouping: COLLATE NOCASE. One way to carry out the preceding query in a case-insensitive way is to add a COLLATE NOCASE qualifier to the GROUP BY clause.
Let's filter out the 'None' cases as well, while we are at it.
Can you write Python code for it?

query = '''
  SELECT City, COUNT(*) AS NumComplaints
    FROM data
    WHERE City <> 'None'
    GROUP BY City COLLATE NOCASE
    ORDER BY -NumComplaints
    LIMIT 10
'''
df = pd.read_sql_query (query, disk_engine)
df


I am working on the file "10-Getting-at-Data.ipynb".
The first task is:
Break it down
So we can pass many attributes to the VSO, in this case we started with time

 vso.attrs.Time('2001/1/1','2001/1/2')

Start and end times for the query as strings, any date/time function can be understood by SunPy's parse_time function e.g. the datetime onjects we will look at later. Next we give it the instrument we want:

 vso.attrs.Instrument('eit')

You don't have to pass it an instrument, the client will find all available missions in the parameter you've defined if you like. Next, wavelength:

 vso.attrs.Wave(14.2*u.nm, 12.3*u.nm)

We pass it a min and max wavelength. This has to be an astropy units quantity (in SI for the love of coffee). If you don't you will get an error.
For a full list of attributes that vso can take use:
So we can use multiple instument queries and define get smaller sample times but narrowing down the query:
Can you write Python code for it?

qr = client.query(vso.attrs.Time('2001/1/1T12:00:00', '2001/1/2T13:00:00'), vso.attrs.Instrument('eit') | vso.attrs.Instrument('mdi'))
len(qr)


I am working on the file "10-Getting-at-Data.ipynb".
The first task is:
In this case the query is looking at a specific set of coordinates
Can you write Python code for it?

from astropy import coordinates
import astropy.units as u
c = coordinates.SkyCoord("05h35m17.3s -05d23m28s", frame='icrs')
r = 5 * u.arcminute
result = Simbad.query_region(c, radius=r)
result.pprint(show_unit=True, max_width=80, max_lines=5)


I am working on the file "10-Getting-at-Data.ipynb".
The first task is:
Notice that the HEK query is extremely similar to the VSO query style, with our attributes defined accordingly.
Instead of returning a list, HEK returns a list of dictionary objects. Each entry in the dictionary sis a pair of key-value pairs that exactly correspond to the parameters. We can return the key words using:
Can you write Python code for it?


I am working on the file "10-Getting-at-Data.ipynb".
The first task is:
This is your client object. This is effectively the intermediary between yourself and the treasure chest of solar data available. You query VSO, then VSO querys all data providers which fit the limiations you imposed during your search command. The VSO client also handles the particulars of dowloading the data onto your machiene.
Making a query
Lets kick off with an example, lets ask the veteran of solar imaging, SoHO for some EIS data, between the dates of between January 1st and 2nd, 2001,
Can you write Python code for it?

qr = client.query(vso.attrs.Time('2001/1/1', '2001/1/2'),
                  vso.attrs.Instrument('eit'))


I am working on the file "10-Getting-at-Data.ipynb".
The first task is:
This way we can avoid troublesome doubling up of results. We can do the same help(hek.attrs) command as VSO to fins out further options.
Aquiring data with AstroQuery
Astroquery supports a plethora of services, all of which follow roughly the same API (application program interface). In its simplest for the API involves queries based on coordinates or object names e.g. using SIMBAD:
Can you write Python code for it?

from astroquery.simbad import Simbad
result_table = Simbad.query_object("m1")
result_table.pprint(show_unit=True)


I am working on the file "10-Information-Based-Learning/HW10/ANSWERS.ipynb".
The first task is:
Sample output:
In [1]: find_H(cer.iloc[:20], 'ratingID')
Out[1]: 0.60984030471640038
Question 5
Write a function called "info_gain" which takes four arguments:

df, which is a pandas data frame
cname, which is the name of the target column (that should correspond to a categorical variable)
csplit, which is the name of a numeric column in df
threshold, which is a numeric value

and returns:

info_gain, the information gain you get in column cname by splitting the dataset on the threshold value in column csplit.
Can you write Python code for it?

#### play with code here #####

def info_gain(df, cname, csplit, thr):
    H0 = find_H(df, cname)
    o1 = df[df[csplit]<thr]
    o2 = df[df[csplit]>=thr]
    R1 = find_H(o1, cname)
    R2 = find_H(o2, cname)
    return H0 - len(o1)/len(df)*R1 - len(o2)/len(df)*R2

info_gain(cer.iloc[:20], 'ratingID', 'sugars', 7.0)


I am working on the file "10-Information-Based-Learning/HW10/ANSWERS.ipynb".
The first task is:
Sample output:
In [1]: get_corr_pairs(cer[['name','fat','sugars','rating']])
Out[1]: {'fat': ['rating'], 'rating': ['fat', 'sugars'], 'sugars': ['rating']}
Short explanation: the correlation between 'fat' and 'rating' is -0.409, 'sugars' and 'rating' is -0.760; the remaining correlations have magnitude < 0.3.
Question 3
Write a function called "sample_cereal" which takes two arguments:

df, which is a pandas data frame
kind, which is a string that can take value 'up' or 'down'

and returns:

a pandas data frame with balanced target class 'ratingID', using up sampling if kind='up' and downsampling if kind='down'.
Can you write Python code for it?

def sample_cereal(df, kind):
    id_counts = df.groupby('ratingID').ratingID.count()
    o1 = df[df.ratingID==id_counts.argmax()]
    o2 = df[df.ratingID==id_counts.argmin()]
    if (kind=='up'):
        return o1.append(o2.iloc[np.random.choice(len(o2), len(o1), replace=True)])
    elif (kind=='down'):
        return o2.append(o1.iloc[np.random.choice(len(o1), len(o2), replace=False)])
    else:
        print 'This kind of sampling is not recognized! Returning original DataFrame!'
        return df

sample_cereal(cer.ix[3:5,['name','mfr','type','calories','protein','ratingID']], 'up')

sample_cereal(cer.ix[3:5,['name','mfr','type','calories','protein','ratingID']], 'down')


I am working on the file "10-Information-Based-Learning/HW10/README_corrected.ipynb".
The first task is:
Question 1
Write a function called "get_corrs" which takes one argument:

df, which is a pandas data frame

and returns:

m, a correlation matrix for the numerical variables in df.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_corrected.ipynb".
The first task is:
Sample output:
In [1]: find_H(cer.iloc[:20], 'ratingID')
Out[1]: 0.60984030471640038
Question 5
Write a function called "info_gain" which takes four arguments:

df, which is a pandas data frame
cname, which is the name of the target column (that should correspond to a categorical variable)
csplit, which is the name of a numeric column in df
threshold, which is a numeric value

and returns:

info_gain, the information gain you get in column cname by splitting the dataset on the threshold value in column csplit.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_corrected.ipynb".
The first task is:
Sample output:
In [1]: get_corr_pairs(cer[['name','fat','sugars','rating']])
Out[1]: {'fat': ['rating'], 'rating': ['fat', 'sugars'], 'sugars': ['rating']}
Short explanation: the correlation between 'fat' and 'rating' is -0.409, 'sugars' and 'rating' is -0.760; the remaining correlations have magnitude < 0.3.
Question 3
Write a function called "sample_cereal" which takes two arguments:

df, which is a pandas data frame
kind, which is a string that can take value 'up' or 'down'

and returns:

a pandas data frame with balanced target class 'ratingID', using up sampling if kind='up' and downsampling if kind='down'.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_corrected.ipynb".
The first task is:
Sample output:
In [1]: get_corrs(cer[['name','calories','carbo','sugars']])
Out[1]:     calories    carbo       sugars
   calories 1.000000    0.250681    0.562340
      carbo    0.250681    1.000000    -0.331665
     sugars 0.562340   -0.331665     1.000000
Question 2
Write a function called "get_corr_pairs" which takes one argument:

df, which is a pandas data frame

and returns:

corr_pairs, a dictionary where keys are names of columns of df corresponding to numerical features, and values are arrays of names of columns whose correlation coefficient with the key has magnitude 0.3 or greater.

You can use your function from question 1 to get the correlation values.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_corrected.ipynb".
The first task is:
Sample output:
In [1]: sample_cereal(cer.ix[3:5,['name','mfr','type','calories','protein','ratingID']], 'up')
Out[1]:     name                mfr     type    calories    protein     ratingID
3   All-Bran with Extra Fiber   K   C        50           4          1
3   All-Bran with Extra Fiber   K   C        50           4          1
4   Almond Delight                R     C        110         2       0
5   Apple Cinnamon Cheerios       G     C        110         2       0
Short explanation: The input has only one positive sample and two negative samples; random sampling from a distribution of 1 can only return one possible result, so our up-sampling of the smaller class merely replicates the row for "All-Bran with Extra Fiber".
Question 4
Write a function called "find_H" which takes two arguments:

df, which is a pandas data frame
cname, which is the name of the target column (that should correspond to a categorical variable)

and returns:

H, the entropy in the column cname (use logarithm base 2)
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_orig.ipynb".
The first task is:
Sample output:
In [1]: find_H(cer.iloc[:20], 'ratingID')
Out[1]: 0.60984030471640038
Question 5
Write a function called "info_gain" which takes four arguments:

df, which is a pandas data frame
cname, which is the name of the target column (that should correspond to a categorical variable)
csplit, which is the name of a numeric column in df
threshold, which is a numeric value

and returns:

info_gain, the information gain you get in column cname by splitting the dataset on the threshold value in column csplit.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_orig.ipynb".
The first task is:
Sample output:
In [1]: get_corr_pairs(cer[['name','calories','fat','sugars','carbo']])
Out[1]: {'fat': ['rating'], 'rating': ['fat', 'sugars'], 'sugars': ['rating']}
Short explanation: the correlation between 'fat' and 'rating' is -0.409, 'sugars' and 'rating' is -0.760; the remaining correlations have magnitude < 0.3.
Question 3
Write a function called "sample_cereal" which takes two arguments:

df, which is a pandas data frame
kind, which is a string that can take value 'up' or 'down'

and returns:

a pandas data frame with balanced target class 'ratingID', using up sampling if kind='up' and downsampling if kind='down'.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_orig.ipynb".
The first task is:
Sample output:
In [1]: get_corrs(cer['name','calories','carbo','sugars'])
Out[1]:     calories    carbo       sugars
   calories 1.000000    0.250681    0.562340
      carbo    0.250681    1.000000    -0.331665
     sugars 0.562340   -0.331665     1.000000
Question 2
Write a function called "get_corr_pairs" which takes one argument:

df, which is a pandas data frame

and returns:

corr_pairs, a dictionary where keys are names of columns of df corresponding to numerical features, and values are arrays of names of columns whose correlation coefficient with the key has magnitude 0.3 or greater.

You can use your function from question 1 to get the correlation values.
Can you write Python code for it?


I am working on the file "10-Information-Based-Learning/HW10/README_orig.ipynb".
The first task is:
Sample output:
In [1]: sample_cereal(cer.ix[3:5,['name','mfr','type','calories','protein','ratingID']], 'up')
Out[1]:     name                mfr     type    calories    protein     ratingID
3   All-Bran with Extra Fiber   K   C        50           4          1
3   All-Bran with Extra Fiber   K   C        50           4          1
4   Almond Delight                R     C        110         2       0
5   Apple Cinnamon Cheerios       G     C        110         2       0
Short explanation: The input has only one positive sample and two negative samples; random sampling from a distribution of 1 can only return one possible result, so our up-sampling of the smaller class merely replicates the row for "All-Bran with Extra Fiber".
Question 4
Write a function called "find_H" which takes two arguments:

df, which is a pandas data frame
cname, which is the name of the target column (that should correspond to a categorical variable)

and returns:

H, the entropy in the column cname (use logarithm base 2)
Can you write Python code for it?


I am working on the file "10-Iterators.ipynb".
The first task is:
Any number of iterables can be zipped together, and if they are different lengths, the shortest will determine the length of the zip.
map and filter
The map iterator takes a function and applies it to the values in an iterator:
Can you write Python code for it?

# find the first 10 square numbers
square = lambda x: x ** 2
for val in map(square, range(10)):
    print(val, end=' ')


I am working on the file "10-Iterators.ipynb".
The first task is:
Had we not thrown-in a loop break here, it would go on happily counting until the process is manually interrupted or killed (using, for example, ctrl-C).
Useful Iterators
This iterator syntax is used nearly universally in Python built-in types as well as the more data science-specific objects we'll explore in later sections.
Here we'll cover some of the more useful iterators in the Python language:
enumerate
Often you need to iterate not only the values in an array, but also keep track of the index.
You might be tempted to do things this way:
Can you write Python code for it?

L = [2, 4, 6, 8, 10]
for i in range(len(L)):
    print(i, L[i])


I am working on the file "10-Iterators.ipynb".
The first task is:
Here we're going to dig a bit deeper.
It turns out that in Python 3, range is not a list, but is something called an iterator, and learning how it works is key to understanding a wide class of very useful Python functionality.
Iterating over lists
Iterators are perhaps most easily understood in the concrete case of iterating through a list.
Consider the following:
Can you write Python code for it?

for value in [2, 4, 6, 8, 10]:
    # do some operation
    print(value + 1, end=' ')


I am working on the file "10-Iterators.ipynb".
The first task is:
If range were to actually create that list of one trillion values, it would occupy tens of terabytes of machine memory: a waste, given the fact that we're ignoring all but the first 10 values!
In fact, there's no reason that iterators ever have to end at all!
Python's itertools library contains a count function that acts as an infinite range:
Can you write Python code for it?

from itertools import count

for i in count():
    if i >= 10:
        break
    print(i, end=', ')


I am working on the file "10-Iterators.ipynb".
The first task is:
It is this iterator object that provides the functionality required by the for loop.
The iter object is a container that gives you access to the next object for as long as it's valid, which can be seen with the built-in function next:
Can you write Python code for it?

I = iter([2, 4, 6, 8, 10])

print(next(I))

print(next(I))

print(next(I))


I am working on the file "10-Iterators.ipynb".
The first task is:
So, for example, we can get tricky and compress the map example from before into the following:
Can you write Python code for it?

print(*map(lambda x: x ** 2, range(10)))


I am working on the file "10-Iterators.ipynb".
The first task is:
The benefit of the iterator indirection is that the full list is never explicitly created!
We can see this by doing a range calculation that would overwhelm our system memory if we actually instantiated it (note that in Python 2, range creates a list, so running the following will not lead to good things!):
Can you write Python code for it?

N = 10 ** 12
for i in range(N):
    if i >= 10: break
    print(i, end=', ')


I am working on the file "10-Iterators.ipynb".
The first task is:
The map and filter functions, along with the reduce function (which lives in Python's functools module) are fundamental components of the functional programming style, which, while not a dominant programming style in the Python world, has its outspoken proponents (see, for example, the pytoolz library).
Iterators as function arguments
We saw in *args and **kwargs: Flexible Arguments. that *args and **kwargs can be used to pass sequences and dictionaries to functions.
It turns out that the *args syntax works not just with sequences, but with any iterator:
Can you write Python code for it?


I am working on the file "10-Iterators.ipynb".
The first task is:
This is the more "Pythonic" way to enumerate the indices and values in a list.
zip
Other times, you may have multiple lists that you want to iterate over simultaneously.
You could certainly iterate over the index as in the non-Pythonic example we looked at previously, but it is better to use the zip iterator, which zips together iterables:
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Adding volumes from HathiTrust
To build your own corpus, you first need to find the volumes you'd like to include in the HathiTrust Library. Alternately, you can access volumes from existing public HT collections, or use one of the sample datasets included below under the Sample datasets heading. To access extracted features from HathiTrust:

Install and configure the HT + HTRC mashup browser extension.
Once the extension is running, go to the HathiTrust Library, and search for the titles you want to include.
You can manually download extracted features one result at a time by simply choosing the Download Extracted Features link for any item in your search results. Save the .json.bz2 file or files and skip to the next section, Working with Extracted Features below to load them into your workspace.
If you plan to work with a large number of texts, you might choose instead to create a collection in HathiTrust, and then download the Extracted Features for the entire collection at once. This requires a valid CalNet ID.

To create a collection:

Login to HathiTrust
Change the HathiTrust search tab to Full-Text or go to the Advanced Full-Text search.

Check the boxes to the left of any search results you want to add to your collection (or select all), and use the Select Collection dropdown to Add Selected volumes to collections of your own design.

Choose My Collections from the top of the HathiTrust interface, choose your collection, and from the Download Metadata button/dropdown choose the TSV option.

Open the TSV file, and then delete all of the columns except for the first column, htitem_id. Delete the htitem_id header row as well and then save the file to your working directory.

Loading Extracted Features
Go to the directory where you plan to do your work.
Add a single volume
If you're planning to analyze only a few volumes you can use the following command, replacing {{volume_id}} with your own:
htid2rsync {{volume_id}} | rsync -azv --files-from=- data.sharc.hathitrust.org::features/ local-folder/
Add...
Can you write Python code for it?

!rm -rf local-folder/
!rm -rf local-folder/
!rm -rf data/coo*
!rm -rf data/mdp*
!rm -rf data/uc1*
download_output = !htid2rsync --f data/vol_ids_5.txt | rsync -azv --files-from=- data.sharc.hathitrust.org::features/ data/
download_output


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Challenge
Play around with this table and see what you can learn about the book:
Try typing vol. and then tab to see everything that's provided in the volume object:
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Let's take a closer look at the first volume:
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Let's try to pull out some more metadata about these titles, using the Volume object in FeatureReader. We'll get the HT URL, year, and page count for each volume.
Can you write Python code for it?

for vol in fr.volumes():
    print("URL: %s Year: %s Page count: %s " % (vol.handle_url, vol.year, vol.page_count))


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Now we'll feed these paths into the FeatureReader method which will create a FeatureReader object:
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
The tokens_per_page method will give us the words in the volume:
Can you write Python code for it?

tokens = vol.tokens_per_page()
tokens.head()


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
We can do this for every page and get a huge table!
Can you write Python code for it?

all_pages_meta = Table.from_df(pd.concat([p.tokenlist().reset_index() for p in pages]))
all_pages_meta.show(10)


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
We can easily plot the number of tokens across every page of the book
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
We can get a list of the tokens with the tokenlist method:
Can you write Python code for it?


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
We can now cycle through properties of the FeatureReader:
Can you write Python code for it?

for vol in fr.volumes():
    print(vol.id, vol.title, vol.author)
    print()


I am working on the file "10-Metadata/01-HTRC-Extracted Features.ipynb".
The first task is:
Working with Extracted Features
All of the examples of code below are taken directly, or adapted, from the Programming Historian tutorial or the FeatureReader's Readme.md file.
You'll notice, from the output above, that the content for each volume is stored in a compressed JSON file, in a rather lengthy file directory. We can import and initialize FeatureReader with file paths pointing to those JSON files (using the full paths from the output above). If you chose to work with your own volumes in the previous step you can edit the cell above and re-run the cells below.
First we'll get all the data filepaths from the output of our command above:
Can you write Python code for it?

suffix = '.json.bz2'
file_paths = ['data/' + path for path in download_output if path.endswith(suffix)]
file_paths


I am working on the file "10-Metadata/02-HTRC-Classification-Example.ipynb".
The first task is:
As in the previous notebooks, we'll construct FeatureReader objects for each corpus. The line below reads in path files we created to the downloaded data:
Can you write Python code for it?

paths = {}
subjects = ['poetry', 'scifi']
for subject in subjects:
    with open('data/' + subject + '_paths.txt', 'r') as f:
        paths[subject] = ['data/' + line[:len(line)-1] for line in f.readlines()]

poetry = FeatureReader(paths['poetry'])
scifi = FeatureReader(paths['scifi'])


I am working on the file "10-Metadata/02-HTRC-Classification-Example.ipynb".
The first task is:
Once we construct the global dictionary, we can fill the bag of words matrix with the word counts for each volume. Once we have this, we will use it to format the training data for our model.
Can you write Python code for it?

dtm = np.zeros((200, len(wordDict.keys())))

for i, vol in enumerate(volumes):
    tok_list = vol.tokenlist(pages=False)
    counts = list(tok_list['count'])
    tokens = tok_list.index.get_level_values('token')

    for token, count in zip(tokens, counts):
        try:
            index = wordDict[token]
            dtm[i, index] = count
        except:
            pass

X = dtm
y = np.zeros((200))
y[100:200] = 1


I am working on the file "10-Metadata/02-HTRC-Classification-Example.ipynb".
The first task is:
We can also get the most helpful features, or words, for each class. First we'll fit the model:
Can you write Python code for it?

model.fit(X, y)

feats = np.argsort(model.coef_[0])[:50]
top_scifi = [(list(feats).index(wordDict[w]) + 1, w) for w in wordDict.keys() if wordDict[w] in feats]
sorted(top_scifi)

feats = np.argsort(model.coef_[0])[-50:]
top_poetry = [(list(feats).index(wordDict[w]) + 1, w) for w in wordDict.keys() if wordDict[w] in feats]
sorted(top_poetry, key=lambda tup: tup[0])


I am working on the file "10-Metadata/03-Bonus-Moretti/03-Metadata.ipynb".
The first task is:
A Fortunate Formula
Can you write Python code for it?

def istheXofY(text):
    return re.search(r'the .* of .*', text.lower())!=None and len(text.split())<=4

print(istheXofY('The Castle of Otronto'))
print(istheXofY('The Castle in which there are some people of Otronto and other places'))

# Graph the frequency of "The X of Y" titles per decade
singlevol_tb['theXofY'] = singlevol_tb.apply(istheXofY, 'title')
singlevol_tb['decade'] = singlevol_tb['date']//10*10
singlevol_tb.group('decade', collect=np.mean).scatter('decade', 'theXofY mean')

# Create table containing only "The X of Y" titles
theXofY_tb = singlevol_tb.where('theXofY').drop('theXofY')

def gettheX(text):
    X = re.findall(r'(?<=the ).*(?= of )', text.lower())[0]
    return X

def gettheY(text):
    Y = re.findall(r'(?<= of ).*', text.lower())[0]
    return Y

print(gettheX('The Castle of Otronto'))
print(gettheY('The Castle of Otronto'))
print()
print(gettheX('The castle in which there are some people of Otronto and other places'))
print(gettheY('The castle in which there are some people of Otronto and other places'))

# Create new columns containing on the the Y and Y from each title
theXofY_tb['theX'] = theXofY_tb.apply(gettheX, 'title')
theXofY_tb['ofY'] = theXofY_tb.apply(gettheY, 'title')

theXofY_tb

from collections import Counter

Xs = Counter(theXofY_tb['theX'])
Ys = Counter(theXofY_tb['ofY'])

Xs.most_common(10)

Ys.most_common(10)

# EX. In Moretti's study, he gives examples of titles using the formula "The X of Y"
#     with lengths of up to seven words. If we tweak our function istheXofY()to allow
#     for longer titles, how does this change our findings? Why?


I am working on the file "10-Metadata/03-Bonus-Moretti/03-Metadata.ipynb".
The first task is:
Intro to Regex (Regular Expressions)
Can you write Python code for it?

import re

# Example from previous lesson
for line in open('lecture notes 09-22-15.txt'):
    for word in line.split():
        if word.endswith('ing'):
            print(word)

# Reproduced using regex
for line in open('lecture notes 09-22-15.txt'):
    for word in line.split():
        if re.search(r'ing$', word): # only change from above
            print(word)

# EX. Remove the "$" from the code above. How does it change the output? Why?

word = 'Having'
re.search(r'ing$', word)

word = 'Ideas'
re.search(r'ing$', word)

# assign list of words to variable, so we don't have to read in the file each time
with open('lecture notes 09-22-15.txt') as file_in:
    lec_notes = file_in.read()
word_list = lec_notes.split()

[word for word in word_list if re.search(r'^..t..$', word)]

# EX. What do you think the "^" and "." metacharacters do in the code?

[word for word in word_list if re.search(r'^a.*t', word)]

# EX. What do you think the "*" metacharacter does in the code?

poe = "While I nodded, nearly napping, suddenly there came a tapping,\
As of someone gently rapping, rapping at my chamber door."

re.findall(r'.apping', poe)

re.findall(r'.(?=apping)', poe)

re.findall(r"(?<=ly ).apping", poe)

re.findall(r"(?<=ly ).(?=apping)", poe)

# EX. Find a list of "-apping" words that are followed by a comma in the line from Poe
#     -- but make sure the comma doesn't appear in your list entries!


I am working on the file "10-Metadata/03-Bonus-Moretti/03-Metadata.ipynb".
The first task is:
Trends
Can you write Python code for it?

import numpy as np

singlevol_tb = singlevol_tb.select(['title', 'date'])

# Determine length of each title
title_tokens = [x.split() for x in singlevol_tb['title']]
title_length = [len(x) for x in title_tokens]
singlevol_tb['title_len'] = title_length

singlevol_tb

# Determine average title length per year
mean_table = singlevol_tb.group('date', collect=np.mean)

mean_table

mean_table.scatter('date','title_len mean')

# Does the pattern hold when we treat individual titles as data points?
singlevol_tb.scatter('date', 'title_len')

singlevol_tb.scatter('date', 'title_len', fit_line=True)

# EX. Moretti also produces graphs for the median and standard deviation
#     of title lengths by year. Create graphs that represent these data.


I am working on the file "10-generators.ipynb".
The first task is:
So, when the generator gets to the end of its iteration it raises an exception. As seen in previous sections, we could surround the next call with a try block to capture the StopIteration so that we can continue after it finishes. This is effectively what the for loop is doing.
We can now find all primes (less than or equal to 100, for example) that have the form $4 k - 1$ using
Can you write Python code for it?

for p in all_primes(100):
    if (1+p)%4 == 0:
        print("The prime {} is 4 * {} - 1.".format(p, int((1+p)/4)))


I am working on the file "10-generators.ipynb".
The first task is:
This code needs careful examination. First it defines the list of all prime numbers that it currently knows, primes (which is initially empty). Then it loops through all integers $n$ from $2$ to $N$ (ignoring $1$ as we know it's not prime).
Inside this loop it initially assumes that $n$ is prime. It then checks if any of the known primes exactly divides $n$ (n%p == 0 checks if $n \bmod p = 0$). As soon as it finds such a prime divisor it knows that $n$ is not prime it resets the assumption with this new knowledge, then breaks out of the loop. This statement stops the for p in primes loop early, as we don't need to look at later primes.
If no known prime ever divides $n$ then at the end of the for p in primes loop we will still have is_n_prime being True. In this case we must have $n$ being prime, so we add it to the list of known primes and return it.
It is precisely this point which makes the code above define a generator. We return the value of the prime number found

using the yield keyword, not the return keyword, and
we return the value as soon as it is known.

It is the use of the yield keyword that makes this function a generator.
This means that only the latest prime number is stored for return.
To use the iterator within a loop, we code it in the same way as with the range function:
Can you write Python code for it?

print("All prime numbers less than or equal to 20:")
for p in all_primes(20):
    print(p)


I am working on the file "10-generators.ipynb".
The first task is:
To see what the generator is actually doing, we can step through it one call at a time using the built in next function:
Can you write Python code for it?

a = all_primes(10)

next(a)

next(a)

next(a)

next(a)

next(a)


I am working on the file "10-generators.ipynb".
The first task is:
We see that the memory usage is growing very rapidly - as the list gets large it's growing as $N$.
Instead we can use the range function that yields one integer at a time:
Can you write Python code for it?

def naivesum(N):
  """
  Naively sum the first N integers
  """
  A = 0
  for i in range(N + 1):
      A += i
  return A

%memit naivesum(10**4)

%memit naivesum(10**5)

%memit naivesum(10**6)

%memit naivesum(10**7)

%memit naivesum(10**8)


I am working on the file "10-generators.ipynb".
The first task is:
We see that the memory usage is unchanged with $N$, making it practical to run much larger calculations.
Iterators
The range function is returning an iterator here. This is an object - a general thing - that represents a stream, or a sequence, of data. The iterator knows how to create the first element of the stream, and it knows how to get the next element. It does not, in general, need to know all of the elements at once.
As we've seen above this can save a lot of memory. It can also save time: the code does not need to construct all of the members of the sequence before starting, and it's quite possible you don't need all of them (think about the "Shortest published mathematical paper" exercise).
An iterator such as range is very useful, and there's a lot more useful ways to work with iterators in the itertools module. These functions that return iterators, such as range, are called generators, and it's useful to be able to make your own.
Making your own generators
Let's look at an example: finding all primes less than $N$ that can be written in the form $4 k - 1$, where $k$ is an integer.
We're going to need to calculate all prime numbers less than or equal to $N$. We could write a function that returns all these numbers as a list. However, if $N$ gets large then this will be expensive, both in time and memory. As we only need one number at a time, we can use a generator.
Can you write Python code for it?

def all_primes(N):
    """
    Return all primes less than or equal to N.

    Parameters
    ----------

    N : int
        Maximum number

    Returns
    -------

    prime : generator
        Prime numbers
    """

    primes = []
    for n in range(2, N+1):
        is_n_prime = True
        for p in primes:
            if n%p == 0:
                is_n_prime = False
                break
        if is_n_prime:
            primes.append(n)
            yield n


I am working on the file "10-generators.ipynb".
The first task is:
We will now see how much memory this uses:
Can you write Python code for it?

%load_ext memory_profiler

%memit naivesum_list(10**4)

%memit naivesum_list(10**5)

%memit naivesum_list(10**6)

%memit naivesum_list(10**7)

%memit naivesum_list(10**8)


I am working on the file "10-marr.ipynb".
The first task is:
En la modelación de créditos con cashflow se consideran dos tipos de costos:

Los puntos de descuento (dispoints) como porcentaje sobre el monto de la deuda. Estos son una forma de pago de intereses por anticipado con el fin de bajar la tasa de interés del crédito.

Los puntos de constitución (orgpoints) como porcentaje del monto de deuda. Son los costos de constitución del crédito y no son considerados como intereses.

Ya que los intereses de los créditos pueden descontarse como costos financieros, estos disminuyen el pago del impuesto de renta. Por consiguiente, en el análisis de los créditos debe tenerse en cuenta el beneficio por pago de intereses el cual equivale a los impuestos pagados por período multiplicados por la tasa del impuesto de renta. Ya que los puntos de descuento son intereses, estos se tienen en cuenta en el cálculo de este beneficio.
Can you write Python code for it?

## flujo de caja para el crédito antes de impuestos
tax_rate = cf.interest_rate(const_value=[30]*5, start=2018)
credito1.tocashflow(tax_rate=tax_rate)

## la tasa efectiva pagada por el crédito es
## aquella que hace el valor presente cero para
## el flujo de caja anterior (antes o después de
## impuestos)
credito1.true_rate(tax_rate = tax_rate)

## se almacenan los datos para este credito
monto[1]   = 2000
interes[1] = credito1.true_rate(tax_rate = tax_rate)

## préstamo 2.
## -------------------------------------------------------
##
credito2 = cf.fixed_rate_loan(amount    = 1000,     # monto
                              nrate     =   20,     # tasa de interés
                              start     = 2018,
                              grace     =    0,
                              life      =    4,     # número de cuotas
                              dispoints =    0.24)  # costos de constitución
credito2

credito2.tocashflow(tax_rate = tax_rate)

credito2.true_rate(tax_rate = tax_rate)

## se almacenan los datos para este credito
monto[2]   = 1000
interes[2] = credito2.true_rate(tax_rate = tax_rate)

## préstamo 3.
## -------------------------------------------------------
##
nrate = cf.interest_rate(const_value=[7]*5, start=2018)
credito3 = cf.bullet_loan(amount     = 5000,     # monto
                          nrate      =   nrate,  # tasa de interés
                          orgpoints  =    0.01,  # costos de originación
                          dispoints  =    0.20)  # puntos de descuento
credito3

credito3.tocashflow(tax_rate=tax_rate)

credito3.true_rate(tax_rate=tax_rate)

## se almacenan los datos de este crédito
monto[3]   = 5000
interes[3] = credito3.true_rate(tax_rate=tax_rate)

## montos
monto

## tasas
interes

## Costo ponderado del capital (WACC)
## -------------------------------------------------------------
## es el promdio ponderado de las tasas por
## el porcentaje de capital correspondiente a cada fuente
##
s    = sum(monto)           # capital total
wacc = sum([x*r/s for...


I am working on the file "10-notebook-solution.ipynb".
The first task is:
(b) Erstellen einer PCFG
Die aus den Daten extrahierten relativen Häufigkeiten sollen nun zur Erstellung einer probabilistischen kontextfreien Grammatik (PCFG)  genutzt werden.
Can you write Python code for it?

# {} are placeholders that are filled by the .format method
pcfg = """
S -> NP VP     [1.0]
VP -> V NP PP  [{}]
VP -> V NP     [{}]
NP -> DET N    [{}]
NP -> NP PP [{}]
PP -> P NP     [1.0]

DET -> "the"     [{}]
DET -> "a"       [{}]
N -> "boy"       [0.4]
N -> "woman"     [0.4]
N -> "telescope" [0.2]
V -> "saw"       [1.0]
P -> "with"      [1.0]
""".format(
    vp_with_pp, vp_without, np_without,
    np_with_pp, det_the, det_a
)
grammar = nltk.PCFG.fromstring(pcfg)

from IPython.display import display

parser = nltk.ViterbiParser(grammar)
for tree in parser.parse("the boy saw a woman with a telescope".split()):
    display(tree)


I am working on the file "10-notebook-solution.ipynb".
The first task is:
Die Ausgabe sollte sein:
True
True
False
False
False
Can you write Python code for it?

for p in pos_examples:
    print(classify(p))
for n in neg_examples:
    print(classify(n))


I am working on the file "10-notebook-solution.ipynb".
The first task is:
Parser-Erstellung:
Can you write Python code for it?

parser = StanfordParser(
    jar, model,
    model_path="edu/stanford/nlp/models/lexparser/germanPCFG.ser.gz"
)


I am working on the file "10-notebook-solution.ipynb".
The first task is:
Sie modelliert sehr einfache Sätze der Form SBJ saw OBJ mit optionaler Präpositionalphrase am Ende. Diese Präpositionalphrase kann entweder der näheren Bestimmung des Objekts oder der näheren Bestimmung der in der Verbalphrase ausgedrückten Handlung dienen.
Im folgenden sollen aus der Penn Treebank Wahrscheinlichkeiten für die einzelnen Regeln extrahiert werden, um dieser Ambiguität Herr zu werden.
(a) Relevante Konstruktionen erkennen
Nutzen das im NLTK enthaltene Sample der Penn Treebank (nach Installation unter nltk.corpus.treebank zu finden) zunächst zur Identifikation der für eine Disambiguierung nützlichen (Teil-)bäume der Penn Treebank.
Gegebenfalls müssen Sie die obige Grammatik den Strukturen, die Sie finden, anpassen.
Hinweis: Sie können sich bei der Analyse auf die jeweils 30 häufigsten Konstruktionen der Baumbank beschränken.
Can you write Python code for it?

from collections import defaultdict
from nltk.corpus import treebank

def find_relevant_constructions(lhs, only_with=None):
    lhs_nt = nltk.grammar.Nonterminal(lhs)
    should_filter = only_with is not None
    if should_filter:
        filter_by = list(map(nltk.grammar.Nonterminal, only_with))
        def passes_filter(tup):
            for f in filter_by:
                if f not in tup:
                    return False
            return True

    # iterate over all trees and all their productions
    # and count those which pass the filter
    counter = defaultdict(int)
    for tree in treebank.parsed_sents():
        for prod in tree.productions():
            if prod.lhs() == lhs_nt:
                # either we do not have to filter or the rhs has to pass the filter
                if not should_filter or passes_filter(prod.rhs()):
                    counter[prod.rhs()] += 1

    # sort right sides of productions according to frequency
    return [ (k, counter[k]) for k in sorted(counter.keys(), key=counter.__getitem__) ]

constructions = find_relevant_constructions('VP', only_with=["NP"])
constructions[-30:]


I am working on the file "10-notebook-solution.ipynb".
The first task is:
Wenn Sie sich die extrahierten Wahrscheinlichkeiten und das disambiguierte Ergebnis ansehen, überrascht Sie dann das Ergebnis der Syntaxanalyse?
Aufgabe 3     Weiterverarbeitung syntaktischer Analysen
In dieser Aufgabe sollen Sie die Ausgaben eines state-of-the-art-Parsers, nämlich des Stanford Parsers, weiterverarbeiten.
Mit dem Ziel, Sie erst einmal mit den typischen Strukturen einer solchen Aufgabe vertraut zu machen, sollen Sie in dieser Aufgabe lediglich entscheiden, ob die Eingabe einen Infinitivsatz mit Objekt enthält.
Zur Klarheit betrachten Sie die folgenden positiven und negativen Beispiele:
Can you write Python code for it?

pos_examples = [
    "Er beabsichtigt , den Kuchen ganz alleine zu essen .",
    "Er behauptet , ihn gesehen zu haben ."
]
neg_examples = [
    "Er glaubt , nach Hause zu fliegen .",
    "Zu fliegen ist schön .",
    "Er will gehen ."
]


I am working on the file "10-notebook-solution.ipynb".
The first task is:
Zählen Sie nun, wie oft die jeweiligen Konstruktionen in der Penn Treebank vorkommen und berechnen Sie die relativen Häufigkeiten als Approximation der Regelwahrscheinlichkeiten. Das Vorgehen wird in folgender Formel veranschaulicht:
$$P(V, N!P, P!P \mid V!P) = \dfrac{count(V!P \rightarrow V\:N!P\:P!P)}{count(V!P \rightarrow \setminus*)}$$
Can you write Python code for it?

# the following are numbers counted from iteratively executing the previous code cell
# with different parameters for find_relevant_constructions
vp_with_pp_frq = 170 + 34 + 24 + 11 + 4 + 4
vp_without_frq = 805 + 378 + 375 + 261 + 250 + 185
vp_with_pp = vp_with_pp_frq / (vp_with_pp_frq + vp_without_frq)
vp_without = vp_without_frq / (vp_with_pp_frq + vp_without_frq)

np_with_pp_frq = 2188
np_without_frq = 2020
np_with_pp = np_with_pp_frq / (np_with_pp_frq + np_without_frq)
np_without = np_without_frq / (np_with_pp_frq + np_without_frq)

det_the_frq = 4038
det_a_frq = 1874
det_the = det_the_frq / (det_the_frq + det_a_frq)
det_a = det_a_frq / (det_the_frq + det_a_frq)

(vp_with_pp, vp_without, np_with_pp, np_without, det_the, det_a)


I am working on the file "10-notebook.ipynb".
The first task is:
(b) Erstellen einer PCFG
Die aus den Daten extrahierten relativen Häufigkeiten sollen nun zur Erstellung einer probabilistischen kontextfreien Grammatik (PCFG)  genutzt werden.
Can you write Python code for it?

pcfg = """
S -> NP VP     [1.0]
VP -> V NP PP  [{}]
VP -> V NP     [{}]
NP -> DET N    [{}]
NP -> NP PP    [{}]
PP -> P NP     [1.0]

DET -> "the"     [{}]
DET -> "a"       [{}]
N -> "boy"       [0.4]
N -> "woman"     [0.4]
N -> "telescope" [0.2]
V -> "saw"       [1.0]
P -> "with"      [1.0]
""".format(
    vp_with_pp, vp_without, np_without,
    np_with_pp, det_the, det_a
)
grammar = nltk.PCFG.fromstring(pcfg)
print(grammar)

from IPython.display import display

parser = nltk.ViterbiParser(grammar)
for tree in parser.parse("the boy saw a woman with a telescope".split()):
    display(tree)


I am working on the file "10-notebook.ipynb".
The first task is:
Aufgabe 2     Von Daten zu Regelwahrscheinlichkeiten
Gegeben sei folgende kontextfreie Grammatik:
Can you write Python code for it?

cfg = """
S -> NP VP
VP -> V NP PP
VP -> V NP
NP -> DET N
NP -> NP PP
PP -> P NP

DET -> "the" | "a"
N -> "boy" | "woman" | "telescope"
V -> "saw"
P -> "with"
"""


I am working on the file "10-notebook.ipynb".
The first task is:
Die Ausgabe sollte sein:
True
True
False
False
False
Can you write Python code for it?


I am working on the file "10-notebook.ipynb".
The first task is:
Die folgende Code-Zelle sollte mehrmals mit unterschiedlichen Parametern für find_relevant_constructions ausgeführt werden. Verschaffen Sie sich so interaktiv die nötigen Informationen zur Lösung der Aufgabe!
Can you write Python code for it?

constructions = find_relevant_constructions('VP', only_with=["NP"])
constructions[-30:]


I am working on the file "10-notebook.ipynb".
The first task is:
Sie modelliert sehr einfache Sätze der Form SBJ saw OBJ mit optionaler Präpositionalphrase am Ende. Diese Präpositionalphrase kann entweder der näheren Bestimmung des Objekts oder der näheren Bestimmung der in der Verbalphrase ausgedrückten Handlung dienen.
Im folgenden sollen aus der Penn Treebank Wahrscheinlichkeiten für die einzelnen Regeln extrahiert werden, um dieser Ambiguität Herr zu werden.
(a) Relevante Konstruktionen erkennen
Nutzen das im NLTK enthaltene Sample der Penn Treebank (nach Installation unter nltk.corpus.treebank zu finden) zunächst zur Identifikation der für eine Disambiguierung nützlichen (Teil-)bäume der Penn Treebank.
Hinweis: Sie können sich bei der Analyse auf die jeweils 30 häufigsten Konstruktionen der Baumbank beschränken.
Can you write Python code for it?

from collections import defaultdict
from nltk.corpus import treebank

def find_relevant_constructions(lhs, only_with=None):
    lhs_nt = nltk.grammar.Nonterminal(lhs)
    should_filter = only_with is not None
    if should_filter:
        filter_by = list(map(nltk.grammar.Nonterminal, only_with))
        def passes_filter(tup):
            for f in filter_by:
                if f not in tup:
                    return False
            return True

    counter = defaultdict(int)
    ### TODO: zähle Produktionen in treebank mit lhs als linker Seite ###
    ### und einer rechten Seite, für die passes_filter True liefert ###

    return [ (k, counter[k]) for k in sorted(counter.keys(), key=counter.__getitem__) ]


I am working on the file "10-notebook.ipynb".
The first task is:
Wenn Sie sich die extrahierten Wahrscheinlichkeiten und das disambiguierte Ergebnis ansehen, überrascht Sie dann das Ergebnis der Syntaxanalyse?
Aufgabe 3     Weiterverarbeitung syntaktischer Analysen
In dieser Aufgabe sollen Sie die Ausgaben eines state-of-the-art-Parsers, nämlich des Stanford Parsers, weiterverarbeiten.
Mit dem Ziel, Sie erst einmal mit den typischen Strukturen einer solchen Aufgabe vertraut zu machen, sollen Sie in dieser Aufgabe lediglich entscheiden, ob die Eingabe einen Infinitivsatz mit Objekt enthält.
Zur Klarheit betrachten Sie die folgenden positiven und negativen Beispiele:
Can you write Python code for it?


I am working on the file "10-notebook.ipynb".
The first task is:
Zählen Sie nun, wie oft die jeweiligen Konstruktionen in der Penn Treebank vorkommen und berechnen Sie die relativen Häufigkeiten als Approximation der Regelwahrscheinlichkeiten. Das Vorgehen wird in folgender Formel veranschaulicht:
$$P(V, N!P, P!P \mid V!P) = \dfrac{count(V!P \rightarrow V\:N!P\:P!P)}{count(V!P \rightarrow \setminus*)}$$
Can you write Python code for it?

vp_with_pp_frq = 1
vp_without_frq = 1
vp_with_pp = vp_with_pp_frq / (vp_with_pp_frq + vp_without_frq)
vp_without = vp_without_frq / (vp_with_pp_frq + vp_without_frq)

np_with_pp_frq = 1
np_without_frq = 1
np_with_pp = np_with_pp_frq / (np_with_pp_frq + np_without_frq)
np_without = np_without_frq / (np_with_pp_frq + np_without_frq)

det_the_frq = 1
det_a_frq = 1
det_the = det_the_frq / (det_the_frq + det_a_frq)
det_a = det_a_frq / (det_the_frq + det_a_frq)

(vp_with_pp, vp_without, np_with_pp, np_without, det_the, det_a)


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/01. SciPy를 이용한 확률 분포 분석.ipynb".
The first task is:
누적 분포 함수
Can you write Python code for it?

xx = np.linspace(-8, 8, 100)
cdf = rv.cdf(xx)
plt.plot(xx, cdf)
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/01. SciPy를 이용한 확률 분포 분석.ipynb".
The first task is:
확률 분포 메서드
확률 분포 객체가 가지는 메서드는 다음과 같다.
| 메서드 | 기능 |
|-|-|
| pdf | 확률 밀도 함수  (probability density function) |
| pmf | 확률 질량 함수  (probability mass function) |
| cdf | 누적 분포 함수 (cumulative distribution function)  |
| rvs | 랜덤 샘플 생성 (random variable sampling)    |
| stats | 기술 통계 함수 (descriptive statistics)   |
| fit | 모수 추정 (parameter estimation)    |
메서드 사용법은 다음과 같다.
확률 밀도 함수
Can you write Python code for it?

xx = np.linspace(-8, 8, 100)
pdf = rv.pdf(xx)
plt.plot(xx, pdf)
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/02. 베르누이 확률 분포 (파이썬 버전).ipynb".
The first task is:
SciPy의 describe 명령을 쓰면 다음과 같이 계산할 수 있다.
Can you write Python code for it?


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/02. 베르누이 확률 분포 (파이썬 버전).ipynb".
The first task is:
pmf 메서드를 사용하면 확률 질량 함수(pmf: probability mass function)를 계산할 수 있다.
Can you write Python code for it?

xx = [0, 1]
plt.bar(xx, rv.pmf(xx), align="center")
plt.xlim(-1, 2)
plt.ylim(0, 1)
plt.xticks([0, 1], ["X=0", "X=1"])
plt.ylabel("P(x)")
plt.title("pmf of Bernoulli distribution")
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/02. 베르누이 확률 분포 (파이썬 버전).ipynb".
The first task is:
seaborn의 barplot 명령으로 시각화하면 다음과 같다.
Can you write Python code for it?

sns.barplot(x="value", y="ratio", hue="type", data=df)
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/02. 베르누이 확률 분포 (파이썬 버전).ipynb".
The first task is:
또는 Pandas의 Series객체로 바꾸어 describe 메서드를 써서 다음과 같이 계산한다.
Can you write Python code for it?


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/02. 베르누이 확률 분포 (파이썬 버전).ipynb".
The first task is:
시뮬레이션을 하려면 rvs 메서드를 사용한다.
Can you write Python code for it?


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/03. 이항 확률 분포 (파이썬 버전).ipynb".
The first task is:
pmf 메서드를 사용하면 확률 질량 함수(pmf: probability mass function)를 계산할 수 있다.
Can you write Python code for it?

xx = np.arange(N + 1)
plt.bar(xx, rv.pmf(xx), align="center")
plt.ylabel("P(x)")
plt.title("pmf of binomial distribution")
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/03. 이항 확률 분포 (파이썬 버전).ipynb".
The first task is:
시뮬레이션을 하려면 rvs 메서드를 사용한다.
Can you write Python code for it?

np.random.seed(0)
x = rv.rvs(100)
x

sns.countplot(x)
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/03. 이항 확률 분포 (파이썬 버전).ipynb".
The first task is:
이론적인 확률 분포와 샘플의 확률 분포를 동시에 나타내려면 다음과 같은 코드를 사용한다.
Can you write Python code for it?

y = np.bincount(x, minlength=N)/float(len(x))
df = pd.DataFrame({"theoretic": rv.pmf(xx), "simulation": y}).stack()
df = df.reset_index()
df.columns = ["value", "type", "ratio"]
df.pivot("value", "type", "ratio")

sns.barplot(x="value", y="ratio", hue="type", data=df)
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/04. 가우시안 정규 분포 (파이썬 버전).ipynb".
The first task is:
Q-Q 플롯
정규 분포는 여러가지 연속 확률 분포 중에서도 가장 유용한 특성을 지니며 널리 사용되는 확률 분포이다. 따라서 어떤 확률 변수의 분포가 정규 분포인지 아닌지 확인하는 것은 정규 분포 검정(normality test)은 가장 중요한 통계적 분석 중의 하나이다. 그러나 구체적인 정규 분포 검정을 사용하기에 앞서 시작적으로 간단하게 정규 분포를 확인하는 Q-Q 플롯을 사용할 수 있다.
Q-Q(Quantile-Quantile) 플롯은 분석하고자 하는 샘플의 분포과 정규 분포의 분포 형태를 비교하는 시각적 도구이다. Q-Q 플롯은 동일 분위수에 해당하는 정상 분포의 값과 주어진 분포의 값을 한 쌍으로 만들어 스캐터 플롯(scatter plot)으로 그린 것이다. Q-Q 플롯을 그리는 구체적인 방법은 다음과 같다.

대상 샘플을 크기에 따라 정렬(sort)한다.
각 샘플의 분위수(quantile number)를 구한다.
각 샘플의 분위수와 일치하는 분위수를 가지는 정규 분포 값을 구한다.
대상 샘플과 정규 분포 값을 하나의 쌍으로 생각하여 2차원 공간에 하나의 점(point)으로 그린다.
모든 샘플에 대해 2부터 4까지의 과정을 반복하여 스캐터 플롯과 유사한 형태의 플롯을 완성한다.
비교를 위한 45도 직선을 그린다.

SciPy 패키지의 stats 서브 패키지는 Q-Q 플롯을 계산하고 그리기 위한 probplot 명령을 제공한다.

http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.probplot.html

probplot은 기본적으로 인수로 보낸 데이터 샘플에 대한 Q-Q 정보만을 반환하고 챠트는 그리지 않는다. 만약 차트를 그리고 싶다면 plot 인수에 matplotlib.pylab 모듈 객체 혹은 Axes 클래스 객체를 넘겨주어야 한다.
정규 분포를 따르는 데이터 샘플을 Q-Q 플롯으로 그리면 다음과 같이 직선의 형태로 나타난다.
Can you write Python code for it?

np.random.seed(0)
x = np.random.randn(100)
plt.figure(figsize=(7,7))
sp.stats.probplot(x, plot=plt)
plt.axis("equal")
plt.show()


I am working on the file "10. 기초 확률론3 - 확률 분포 모형/04. 가우시안 정규 분포 (파이썬 버전).ipynb".
The first task is:
정규 분포를 따르지 않는 데이터 샘플을 Q-Q 플롯으로 그리면 다음과 같이 직선이 아닌 휘어진 형태로 나타난다.
Can you write Python code for it?

np.random.seed(0)
x = np.random.rand(100)
plt.figure(figsize=(7,7))
sp.stats.probplot(x, plot=plt)
plt.ylim(-0.5, 1.5)
plt.show()