I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
The Problem
Handwriting is messy. Can we decipher hand-written digits?
Load the MNIST Data
We'll adapt a package from GitHub that allows us to read the second dataset and display the images in it.
Can you write Python code for it?

# Initialize the mnist object
mnist = MNIST()
type(mnist)

# Get the testing images and labels
test_images, test_labels = mnist.load_testing()

type(test_images), type(test_labels)

mnist.display(test_images[4000])

# Get the training images and labels
train_images, train_labels = mnist.load_training()

mnist.display(train_images[3390])

# How many images and labels in the training and test datasets?
[len(d) for d in [train_images, train_labels, test_images, test_labels]]

# Size of the datasets
[np.array(d).shape for d in [train_images, train_labels, test_images, test_labels]]

# How many unique digits do we have handwriting examples for?
list(set(test_labels))

# Finally, extract the data as inputs to the neural network classifier.
# Scale the data and one-hot encode the labels
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder()

X_train = np.array(train_images)/255 # rescale to values between 0 and 1
# One-hot encode
y_train = enc.fit_transform(np.array(train_labels).reshape(-1,1)).toarray()

X_test = np.array(test_images)/255 # rescale
# One-hot encode
y_test = enc.fit_transform(np.array(test_labels).reshape(-1,1)).toarray()

y_train[2000]

# Check the one-hot encoding
mnist.display(X_train[2000])

# We need a multi-level classifier because we have to classify digits into 0 through 9 distinct classes.

from sklearn.neural_network import MLPClassifier
# Hidden layers are specified as follows
# (n1, ) means n1 units and 1 hidden layer
# (n1, n2) means n1 units in the first hidden layer and n2 units in the second hidden layer
# (n1, n2, n3) maeans n1 units in the first hidden layer, n2 units in the second hidden layer,
#    and n3 units in the third hidden layer
# Experiment with max_iter -- set it to 10, 50, 100, 200 to see how the neural network behaves
clf = MLPClassifier(solver='sgd', alpha=1e-5,
                    hidden_layer_sizes=(25, 25), random_state=1, verbose=False, max_iter=50)

clf.fit(X_train, y_train)

# How quickly is the classifier learning?
fig, ax =...


I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
The data set here is the one where we have two input measurements (suitably normalized) and one output. So it's a binary classifier that we need. We'll set up our neural network to reflect this structure. The nice thing is we can use the same hidden layer structure as before to do our work -- the classifier is smart enough to recognize and adjust for the differences in dimension of the inputs and the outputs.
Can you write Python code for it?

# We need a binary classifier because we have to classify results into 1 of 2 categories -- Accepted or Rejected.

# As a reminder, here is the classifier we've been using for the MNIST dataset
# **********************
#from sklearn.neural_network import MLPClassifier
# Hidden layers are specified as follows
# (n1, ) means n1 units and 1 hidden layer
# (n1, n2) means n1 units in the first hidden layer and n2 units in the second hidden layer
# (n1, n2, n3) maeans n1 units in the first hidden layer, n2 units in the second hidden layer,
#    and n3 units in the third hidden layer
# Experiment with max_iter -- set it to 10, 50, 100, 200 to see how the neural network behaves

# Here is our modified neural network classifier for our new dataset
# Note that the solver is now lbfgs instead of sdg; everything else is the same
clf2 = MLPClassifier(solver='lbfgs', alpha=1e-3,
                       hidden_layer_sizes=(25, 25), activation='logistic', random_state=1, verbose=False, max_iter=100)
# ***********************

# We'll now fit the same classifier to the new dataset
clf2.fit(X2_train, y2_train)

# How quickly is the classifier learning?
# NOTE: There is no loss curve in scikit learn for the lbfgs solver
fig, ax = plt.subplots(figsize=(8,5))
plt.plot(clf2.loss_curve_)

# How well does the classifer perform?
print("Training set score: %f" % clf2.score(X2_train, y2_train))
print("Test set score: %f" % clf2.score(X2_test, y2_test))

# Get the predictions for the test set
preds = clf2.predict(X2_test)

actual_vals = y2_test

clf2.coefs_[0]

clf2.intercepts_[0]


I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
Using a Neural Network on a Different Data Set
Let's use the neural network on the non-linear classification problem we tackled in the Non-Linear Logistic Regression notebook. Note that it's a small data set with only 94 samples total.
Can you write Python code for it?

# Unpickle the file
import cPickle as pickle
pkl_file_path = os.getcwd() + '/Data/ex2data2.pkl'
X2_train, X2_test, y2_train, y2_test = pickle.load( open( pkl_file_path, "rb" ) )

X2_train[0:3,:]

y2_train[0:3]


I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
We can see the kind of mistakes the classifier is making. Let's be systematic and figure out if there's a way in which the classifier is prone to making mistakes. One way to investigate this is to look at every digit in the test set and see how the classifier fails to identify that digit.
Can you write Python code for it?

# The classifier's failure patterns
combined_errors_labels = []
for i in idx_failures:
    pred = clf.predict(X_test[i].reshape(1,-1))
    pred_value = np.where(pred[0] == 1)[0]
    if pred_value.size == 0:
        pred_value = [10] #  NOTE: 10 means no prediction is made
    label = y_test[i]
    label_value = np.squeeze(np.where(label == 1)[0]) #np.squeeze to make into an integer
    combined_values = [pred_value, label_value]
    combined_errors_labels.append(combined_values)

# The classifier's success patterns
combined_success_labels = []
for i in idx_successes:
    pred = clf.predict(X_test[i].reshape(1,-1))
    pred_value = np.where(pred[0] == 1)[0]
    # This if clause will never fire...
    if pred_value.size == 0:
        pred_value = [10] #  NOTE: 10 means no prediction is made
    label = y_test[i]
    label_value = np.squeeze(np.where(label == 1)[0]) #np.squeeze to make into an integer
    combined_values = [pred_value, label_value]
    combined_success_labels.append(combined_values)

combined_success_labels[0:5]

# Get the count of successes by digit
# NOTE: This would be much easier if the data were in a Pandas dataframe. Alas...
# Create a list of indexed variables
correct_classified_0 = []
correct_classified_1 = []
correct_classified_2 = []
correct_classified_3 = []
correct_classified_4 = []
correct_classified_5 = []
correct_classified_6 = []
correct_classified_7 = []
correct_classified_8 = []
correct_classified_9 = []

outcomes = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

for i in range(len(outcomes)):
    eval('correct_classified_'+ outcomes[i]).append(combined_success_labels.count([[i], [i]]))

correct_classified_0

combined_errors_labels[0:5]

# NOTE: 10 means no prediction is made
from tabulate import tabulate
headers = ['Classfier Predicts', 'Actual Value']
print tabulate(combined_errors_labels[0:10], tablefmt='grid', headers=headers)

# When the actual value is 0, 1, 2, ..., 9 what are the different ways in which the classifier misclassifies?

# Create a list of indexed...


I am working on the file "Chapter02.ipynb".
The first task is:
3.2 Auto
Can you write Python code for it?

auto = pd.read_table('data/Auto',sep='\s+')
rows=np.sum(auto.values=='?',axis=1)
delete_rows = []
for idx,_ in enumerate(rows):
    if _!=0:
        delete_rows.append(idx)
auto=auto.drop(auto.index[delete_rows])

mpg = auto['mpg'].values
displacement = auto['displacement'].values
horsepower = auto['horsepower'].values
weight = auto['weight'].values
acceleration = auto['acceleration'].values
plt.subplot(2,2,1)
plt.scatter(displacement, mpg)
plt.subplot(2,2,2)
plt.scatter(weight, mpg)
plt.subplot(2,2,3)
plt.scatter(acceleration, mpg)
plt.subplot(2,2,4)
displacement = [float(item) for item in displacement]
plt.scatter(displacement, mpg)
plt.show()


I am working on the file "Chapter02.ipynb".
The first task is:
3.3 Boston
Can you write Python code for it?

boston = pd.read_csv('Boston.csv', index_col=0)
boston

crim = boston['crim'].values
zn = boston['zn'].values
plt.scatter(crim, zn)
plt.show()

indus=boston['indus'].values
plt.scatter(crim, indus)
plt.show()

nox = boston['nox'].values
plt.scatter(nox,crim,)
plt.show()

rm = boston['rm'].values
plt.scatter(rm,crim, )
plt.show()

age = boston['age'].values
plt.scatter(age,crim, )
plt.show()

dis = boston['dis'].values
plt.scatter(dis,crim)
plt.show()

rad = boston['rad'].values
plt.scatter(crim, rad)
plt.show()

tax = boston['tax'].values
plt.scatter(tax,crim, )
plt.show()

ptratio=boston['ptratio'].values
plt.scatter(ptratio,crim, )
plt.show()

b = boston['black'].values
plt.scatter(b,crim, )
plt.show()

lstat = boston['lstat'].values
plt.scatter(lstat,crim)
plt.show()

medv = boston['medv'].values
plt.scatter(medv,crim, )
plt.show()


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.distinct()
This method returns a list of distinct values in a specified column.
Can you write Python code for it?

distinct_gender = data_from_file_conv.map(lambda row: row[5]).distinct().collect()
distinct_gender


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.flatMap(...)
The .flatMap(...) method works similarly to .map(...) but returns a flattened results instead of a list.
Can you write Python code for it?

data_2014_flat = data_from_file_conv.flatMap(lambda row: (row[16], int(row[16]) + 1))
data_2014_flat.take(10)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.reduce(...)
Another action that processes your data, the .reduce(...) method reduces the elements of an RDD using a specified method.
Can you write Python code for it?

rdd1.map(lambda row: row[1]).reduce(lambda x, y: x + y)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.repartition(...)
Repartitioning the dataset changes the number of partitions the dataset is divided into.
Can you write Python code for it?

rdd1 = rdd1.repartition(4)

len(rdd1.glom().collect())


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.sample(...)
The .sample() method returns a randomized sample from the dataset.
Can you write Python code for it?

fraction = 0.1
data_sample = data_from_file_conv.sample(False, fraction, 666)

data_sample.take(1)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
.saveAsTextFile(...)
As the name suggests, the .saveAsTextFile() the RDD and saves it to text files: each partition to a separate file.
Can you write Python code for it?


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Actions
.take(...)
The method returns n top rows from a single data partition.
Can you write Python code for it?

data_first = data_from_file_conv.take(1)
data_first


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Another useful method is the .intersection(...) that returns the records that are equal in both RDDs.
Can you write Python code for it?

rdd5 = rdd1.intersection(rdd2)
rdd5.collect()


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
However, if you were to partition the data into 3 partitions, the result will be wrong.
Can you write Python code for it?

data_reduce = sc.parallelize([1, 2, .5, .1, 5, .2], 3)
data_reduce.reduce(lambda x, y: x / y)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
I we were to reduce the data in a manner that we would like to divide the current result by the subsequent one, we would expect a value of 10
Can you write Python code for it?

works = data_reduce.reduce(lambda x, y: x / y)
works


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
If we used .join(...) method instead we would have gotten only the values for 'a' and 'b' as these two values intersect between these two RDDs.
Can you write Python code for it?


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
If you want somewhat randomized records you can use .takeSample(...) instead.
Can you write Python code for it?

data_take_sampled = data_from_file_conv.takeSample(False, 1, 667)
data_take_sampled


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
If your dataset is in a form of a key-value you can use the .countByKey() method to get the counts of distinct keys.
Can you write Python code for it?


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
It has the same effect as the method below but does not require shifting the data to the driver.
Can you write Python code for it?

len(data_reduce.collect()) # WRONG -- DON'T DO THIS!


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Let's confirm that we really got 10% of all the records.
Can you write Python code for it?

print('Original dataset: {0}, sample: {1}'.format(data_from_file_conv.count(), data_sample.count()))


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Note, that to execute the code above you will have to change the path where the data is stored. The dataset can be downloaded from http://tomdrabas.com/data/VS14MORT.txt.gz
Schema
RDDs are schema-less data structures.
Can you write Python code for it?

data_heterogenous = sc.parallelize([('Ferrari', 'fast'), {'Porsche': 100000}, ['Spain','visited', 4504]]).collect()
data_heterogenous


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Now, instead of using lambda we will use the extractInformation(...) method to split and convert our dataset.
Can you write Python code for it?

data_from_file_conv = data_from_file.map(extractInformation)
data_from_file_conv.map(lambda row: row).take(1)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Reading from files
When you read from a text file, each row from the file forms an element of an RDD.
Can you write Python code for it?


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
Transformations
.map(...)
The method is applied to each element of the RDD: in the case for the data_from_file_conv dataset you can think of this as a transformation of each row.
Can you write Python code for it?

data_2014 = data_from_file_conv.map(lambda row: int(row[16]))
data_2014.take(10)


I am working on the file "Chapter02/LearningPySpark_Chapter02.ipynb".
The first task is:
User defined functions
You can create longer method to transform your data instead of using the lambda expression.
Can you write Python code for it?

def extractInformation(row):
    import re
    import numpy as np

    selected_indices = [
         2,4,5,6,7,9,10,11,12,13,14,15,16,17,18,
         19,21,22,23,24,25,27,28,29,30,32,33,34,
         36,37,38,39,40,41,42,43,44,45,46,47,48,
         49,50,51,52,53,54,55,56,58,60,61,62,63,
         64,65,66,67,68,69,70,71,72,73,74,75,76,
         77,78,79,81,82,83,84,85,87,89
    ]

    '''
        Input record schema
        schema: n-m (o) -- xxx
            n - position from
            m - position to
            o - number of characters
            xxx - description
        1. 1-19 (19) -- reserved positions
        2. 20 (1) -- resident status
        3. 21-60 (40) -- reserved positions
        4. 61-62 (2) -- education code (1989 revision)
        5. 63 (1) -- education code (2003 revision)
        6. 64 (1) -- education reporting flag
        7. 65-66 (2) -- month of death
        8. 67-68 (2) -- reserved positions
        9. 69 (1) -- sex
        10. 70 (1) -- age: 1-years, 2-months, 4-days, 5-hours, 6-minutes, 9-not stated
        11. 71-73 (3) -- number of units (years, months etc)
        12. 74 (1) -- age substitution flag (if the age reported in positions 70-74 is calculated using dates of birth and death)
        13. 75-76 (2) -- age recoded into 52 categories
        14. 77-78 (2) -- age recoded into 27 categories
        15. 79-80 (2) -- age recoded into 12 categories
        16. 81-82 (2) -- infant age recoded into 22 categories
        17. 83 (1) -- place of death
        18. 84 (1) -- marital status
        19. 85 (1) -- day of the week of death
        20. 86-101 (16) -- reserved positions
        21. 102-105 (4) -- current year
        22. 106 (1) -- injury at work
        23. 107 (1) -- manner of death
        24. 108 (1) -- manner of disposition
        25. 109 (1) -- autopsy
        26. 110-143 (34) -- reserved positions
        27. 144 (1) -- activity code
        28. 145 (1) -- place of injury
        29. 146-149 (4) -- ICD code
        30. 150-152 (3) -- 358 cause recode
        31....


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-02] MNISTのデータセットを用意します。
Can you write Python code for it?

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-03] ソフトマックス関数による確率 p の計算式を用意します。
Can you write Python code for it?

x = tf.placeholder(tf.float32, [None, 784])
w = tf.Variable(tf.zeros([784, 10]))
w0 = tf.Variable(tf.zeros([10]))
f = tf.matmul(x, w) + w0
p = tf.nn.softmax(f)


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-04] 誤差関数 loss とトレーニングアルゴリズム train_step を用意します。
Can you write Python code for it?

t = tf.placeholder(tf.float32, [None, 10])
loss = -tf.reduce_sum(t * tf.log(p))
train_step = tf.train.AdamOptimizer().minimize(loss)


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-05] 正解率 accuracy を定義します。
Can you write Python code for it?

correct_prediction = tf.equal(tf.argmax(p, 1), tf.argmax(t, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-06] セッションを用意して、Variableを初期化します。
Can you write Python code for it?

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())


sess = tf.Session()
sess.run(tf.initialize_all_variables())


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-07] パラメーターの最適化を2000回繰り返します。
1回の処理において、トレーニングセットから取り出した100個のデータを用いて、勾配降下法を適用します。
最終的に、テストセットに対して約92%の正解率が得られます。
Can you write Python code for it?

i = 0
for _ in range(2000):
    i += 1
    batch_xs, batch_ts = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, t: batch_ts})
    if i % 100 == 0:
        loss_val, acc_val = sess.run([loss, accuracy],
            feed_dict={x:mnist.test.images, t: mnist.test.labels})
        print ('Step: %d, Loss: %f, Accuracy: %f'
               % (i, loss_val, acc_val))


I am working on the file "Chapter02/MNIST softmax estimation.ipynb".
The first task is:
[MSE-08] この時点のパラメーターを用いて、テストセットに対する予測を表示します。
ここでは、「０」〜「９」の数字に対して、正解と不正解の例を３個ずつ表示します。
Can you write Python code for it?

images, labels = mnist.test.images, mnist.test.labels
p_val = sess.run(p, feed_dict={x:images, t: labels})

fig = plt.figure(figsize=(8,15))
for i in range(10):
    c = 1
    for (image, label, pred) in zip(images, labels, p_val):
        prediction, actual = np.argmax(pred), np.argmax(label)
        if prediction != i:
            continue
        if (c < 4 and i == actual) or (c >= 4 and i != actual):
            subplot = fig.add_subplot(10,6,i*6+c)
            subplot.set_xticks([])
            subplot.set_yticks([])
            subplot.set_title('%d / %d' % (prediction, actual))
            subplot.imshow(image.reshape((28,28)), vmin=0, vmax=1,
                           cmap=plt.cm.gray_r, interpolation="nearest")
            c += 1
            if c > 6:
                break


I am working on the file "Chapter03.ipynb".
The first task is:
2.1 Import Question

Is at least one of the predictors $X_1,X_2,\ldots,X_p$ useful in predicting the response
Do all the predictors help to explain $Y$,, or is only a subset of the predcictor useful.
How well does the model fit the data
Given a set of predictor values, what response value should we predict, and how accurate is our prediction

2.1.1 Is There a Relationship Between the Response and Predictors?

null hypothesis
$$H_0:\beta_1=\beta_2=\ldots=\beta_p=0$$

alternative hypothesis
$$H_a: \text{at least one}\quad \beta_j \quad\text{is non-zero}$$
F-statisitc
$$F=\frac{(TSS-RSS)/p}{RSS/(n-p-1)}$$

If there is no relationshio between the response and predictors, that F-statistic takes on a value close to 1. On ther other hand, if $H_a$ is true, we except F to be greater than 1.
2.1.2 Deciding on important variables
By trying out every possible combinations of variables

Forward selection.
backward selction.
mixed selection.

2.1.3 Model Fit
RSE and $R^2$ are the most common numerial measures of model fit. And $R^2$ value close to 1 indicates that the model explains a large portion of variance in the response variable.
2.1.4 Predictions
The coefficient estimates $\hat{\beta_0},\hat{\beta_1},\ldots,\hat{\beta_p}$ are estimates for $\beta_0, \beta_1,\ldots,\beta_p$. the inaccuracy in the coefficient estimate is related to the $reducibel error$, we can use confidence interval. So does $Y$.
3.4 Other Considerations
3.4.1 Predicators with only two levels
$$
x_i=
\begin{cases}
1 & \text{if ith person is female} \
0 & \text{if ith person is male} \
\end{cases}
$$
Use this variable as the predictor in the regression equation
$$
y_i = \beta_0+\beta_1x_i+\epsilon =
\begin{cases}
\beta_0+\beta_1+\epsilon & \text{if ith person is female} \
\beta_0 + \epsilon & \text{if ith person is male} \
\end{cases}
$$
3.4.2 Qualitative predictors with more than two levels
In this situation, we can create additional dummy variables.
3.5 Extension of Linear Model
Above all, we assumpt that the relationship between the...
Can you write Python code for it?

auto = pd.read_table('Auto',sep='\s+')
rows=np.sum(auto.values=='?',axis=1)
delete_rows = []
for idx,_ in enumerate(rows):
    if _!=0:
        delete_rows.append(idx)
auto=auto.drop(auto.index[delete_rows])
data = auto[['mpg','horsepower']]
horsepower= data['horsepower'].values.astype('float')
data['horsepower_2'] = horsepower * horsepower
data['beta_0'] = np.full(horsepower.shape,1.0)
# plot the scatter
plt.scatter(horsepower, auto['mpg'])
# calcault linear
X = data[['horsepower','beta_0']].values.astype('float')
y = data['mpg'].values.astype('float').reshape(X.shape[0],1)
beta_linear =inv(X.T.dot(X)).dot(X.T).dot(y)
X = data[['horsepower','horsepower_2','beta_0']].values.astype('float')
beta_linear2 = inv(X.T.dot(X)).dot(X.T).dot(y)
x = np.linspace(40,230, 500)
y_linear = x*beta_linear[0] + beta_linear[1]
y_linear2 = x*beta_linear2[0] + x*x*beta_linear2[1] +beta_linear2[2]
plt.plot(x, y_linear, c='b')
plt.plot(x, y_linear2, c='g')
plt.show()


I am working on the file "Chapter03.ipynb".
The first task is:
3.6 Potential Probelms
3.6.1 Non-linearity of the Data
Residual plots are a useful graphical tool for identiying non-linearity.
Can you write Python code for it?

horsepower = data['horsepower'].values.astype('float')
mpg = data['mpg'].values.astype('float')
residual_linear = mpg - (horsepower*beta_linear[0]+beta_linear[1])
plt.scatter(mpg, residual_linear)
plt.show()

residual_quadratic = mpg - (horsepower*beta_linear2[0]+horsepower*horsepower*beta_linear2[1]+beta_linear2[-1])
plt.scatter(mpg, residual_quadratic)
plt.show()


I am working on the file "Chapter03.ipynb".
The first task is:
3.6.2 Correlation of Error Terms
Why might correlations among the error terms occur? Such correlationsfrequently occur in the context of time series data
3.6.3 Non-constant Variance of Error Terms
Another important assumption of the linear regression model is that the error terms have a constant variance, $Var(\epsilon_i) = \sigma ^2$. The standard errors, confidence intervals, and hypothesis tests associated with the linear model rely upon this assumption.
3.6.4 Outlier
An outlier is a point for which $y_i$ is far from the value predicted by the model. Use residual plot can recognize the outliers.
3.6.5 High leverage
For simple linear regression
$$
h_i=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{\sum_{i=1}^n(x_{i'}-\bar{x})^2}
$$
Collinearity
ollinearity refers to the situation in which two or more predictor variables are closely related to one another. We  detect from correlation matrix.
4 K-Nearest Neighbors Regression
$$
\hat{f}(x_0)=\frac{1}{k}\sum_{x_i \in N_0}y_i
$$
where $y_i$s are k nearest neighbors points
5 Exercises
5.1 Auto Data
$$mpg = \beta_0 + \beta_1 \times horsepower$$
Can you write Python code for it?

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
auto = pd.read_table('Auto',sep='\s+')
rows=np.sum(auto.values=='?',axis=1)
delete_rows = []
for idx,_ in enumerate(rows):
    if _!=0:
        delete_rows.append(idx)
auto=auto.drop(auto.index[delete_rows])
horsepower= auto['horsepower'].values.astype('float')
auto['ones'] = np.full(horsepower.shape, 1.0)
X = auto[['horsepower','ones']].values.astype('float')
y = auto['mpg'].values.astype('float').reshape(X.shape[0],1)
beta_linear =inv(X.T.dot(X)).dot(X.T).dot(y)
print('β0 :',beta_linear[-1][0])
print('β1 :', beta_linear[0][0])

sample_num = len(y)
residual = np.power(X.dot(beta_linear)-y,2).sum()
sigma = np.sqrt(residual/(sample_num-2))
horsepower_98 = np.array([[98.0,1.0]])
mpg_98 = horsepower_98.dot(beta_linear)[0,0]
mpg_98_uppper_bound = mpg_98+2*sigma
mpg_98_lower_bound = mpg_98-2*sigma
print('predict value is %f when horsepower is 98'%mpg_98)
print('The range is [%f,%f]' %(mpg_98_lower_bound,mpg_98_uppper_bound))

mpg = auto['mpg'].values.astype('float')
plt.scatter(horsepower, mpg)
plt.show()

auto_cor=auto[['mpg','displacement','horsepower','weight','acceleration']]
auto_cor.corr()


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-02] MNISTのデータセットを用意します。
Can you write Python code for it?


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-03] 単層ニューラルネットワークを用いた確率 p の計算式を用意します。
Can you write Python code for it?

num_units = 1024

x = tf.placeholder(tf.float32, [None, 784])

w1 = tf.Variable(tf.truncated_normal([784, num_units]))
b1 = tf.Variable(tf.zeros([num_units]))
hidden1 = tf.nn.relu(tf.matmul(x, w1) + b1)

w0 = tf.Variable(tf.zeros([num_units, 10]))
b0 = tf.Variable(tf.zeros([10]))
p = tf.nn.softmax(tf.matmul(hidden1, w0) + b0)


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-04] 誤差関数 loss、トレーニングアルゴリズム train_step、正解率 accuracy を定義します。
Can you write Python code for it?

t = tf.placeholder(tf.float32, [None, 10])
loss = -tf.reduce_sum(t * tf.log(p))
train_step = tf.train.AdamOptimizer().minimize(loss)
correct_prediction = tf.equal(tf.argmax(p, 1), tf.argmax(t, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-05] セッションを用意して、Variableを初期化します。
Can you write Python code for it?


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-06] パラメーターの最適化を2000回繰り返します。
1回の処理において、トレーニングセットから取り出した100個のデータを用いて、勾配降下法を適用します。
最終的に、テストセットに対して約97%の正解率が得られます。
Can you write Python code for it?


I am working on the file "Chapter03/MNIST single layer network.ipynb".
The first task is:
[MSL-07] 最適化されたパラメーターを用いて、テストセットに対する予測を表示します。
ここでは、「０」〜「９」の数字に対して、正解と不正解の例を３個ずつ表示します。
Can you write Python code for it?


I am working on the file "Chapter04.ipynb".
The first task is:
1 Logistic Regression
1.1 Model
To make probability to fall between 0 and 1, we model the $p(X)$ using sigmod function: $f(x)=\frac{e^x}{1+e^x}$. So we extend the linear regression to form this:
$$p(X)=\frac{e^{\beta_0+\beta_1X}}{1+e^{\beta_0+\beta_1X}}$$
After a bit of manipulation, we conclude that:
$$\frac{p(X)}{1-p(X)}=e^{\beta_0+\beta_1X} \rightarrow log(\frac{p(X)}{1-p(X)}) = \beta_0+\beta_1X$$
We call log-odds or logit is linear in $X$.
1.2 Estimating the regression coefficients
The more general method of maximum likehood is preferred.
$$l(\beta_0,\beta_1)=\prod_{i:y_i=1}p(x_i)\prod_{i:y_{i'}=0}\big(1-p(x_{x'})\big)$$
1.3 Using Scikit learning
Can you write Python code for it?

X = default['balance'].values.astype('float')
X = X.reshape((len(X),1))
y = default['default'].values
y = [1 if item=='Yes' else 0 for item in y]
from sklearn.linear_model import LogisticRegression
lg = LogisticRegression()
lg.fit(X,y)
print(lg.coef_, lg.intercept_)


I am working on the file "Chapter04.ipynb".
The first task is:
2 Linear Discriminatnt Analysis
The comparsion of logistic regression and LDA:

The LR's parameters will be unstable when the clases are well-separated, while LDA is not.
if $n$ is small and the distribution of predictor $X$ is approximately normal in each of the class, the linear discrimiinat model is again more stable than the LR mdoel
LDA is more popular when we want to separate more than two classes.

2.1 Bayes's Theorem
Let $\pi_k$ represent the overall or pripr probability that a randomly chosen comes fro mthe $k$th class; Let $f_k(X)=Pr(X=x|Y=k)$ denote the density fucntion of $X$ for an observation that comes form the $k$th class.
the Bayes's theorem state that:
$$
Pr(Y=k|X=x)=\frac{\pi_kf_k(x)}{\sum_{l=1}^K\pi_lf_l(x)}
$$
We use the abbreviation $P_k(X)=Pr(Y=k|X)$ and refer to $p_k(x)$ as the posterior probability that an observation $X=x$ belongs to the $k$th class. We classifiy an observation to the class for which $p_k(X)$ is largest, has the lowest possible error rate out of the all classifiers.
2.2 Linear Discriminant Analysis for $p=1$
As we assume that $f_k(x)$ is normal or Gaussian. In the one-dimensional setting, the normal density takes the form:
$$
f_k(x)=\frac{1}{\sqrt{2\pi}\sigma_k}exp(-\frac{1}{2\sigma_k^2(x-\mu_k)^2}
$$
where $\mu_k$ and $\sigma_k^2$ are the mean and variance parametes for the $k$th class. Let us further assume that $\sigma_1^2=\ldots=\sigma_k^2$ which means all $K$ classes share the same variance.
Take the log of and rearrangeing the terms, it is not hard to show that this is the equivalent to assigning the observation to the class for which:
$$
\delta_k(x)=x\times \frac{\mu_k}{\sigma^2} -\frac{\mu_k^2}{2\sigma^2}+log(\pi_k)
$$
is largest. where
$$\hat{\mu_k}=\frac{1}{n_k}\sum_{i:y_i=k}x_i$$
$$\hat{\sigma^2}=\frac{1}{n-K}\sum_{k=1}^K\sum_{i:y_i=k}(x_i-\hat{\mu_k})^2$$
2.3 Linear Discriminant Analysis for $p>1$
Extending the LDA classifier to case of the mulitple predictors, we will assume that $X=(X_1, X_2,\ldots,X_p)$ is drawn from the Multivariate Gaussian...
Can you write Python code for it?

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.metrics import accuracy_score
incomes = default['income'].values.astype('float')
students = default['student'].values
students = np.array([1.0 if student=='Yes' else 0.0 for student in students])
defaults = default['default'].values
incomes = incomes.reshape((len(incomes),1))
students = students.reshape((len(students),1))
y = [1 if default=='Yes' else 0 for default in defaults]
X = np.hstack((incomes, students))
lda = LinearDiscriminantAnalysis()
lda.fit(X,y)
pred = lda.predict(X)
print(1-accuracy_score(pred, y))


I am working on the file "Chapter04.ipynb".
The first task is:
3 Quadratic Discriminant Analysis
Unlike the LDA, the QDA assumes that each class has its own convariance matrix. That is, it assume that an obserivation from the $k$th class is of the form $X \sim N(\mu_k, \Sigma_k)$, where $\Sigma_k$ is a convariance matrix for $k$th class. Under this assumption, the Bayes classifier assigns an observation $X=x$ to the class for which
$$
\delta_k=-\frac{1}{2}(x-\mu_k)^T\Sigma_k^{-1}(x-\mu_k) - \frac{1}{2}log|\Sigma_k|+log\pi_k
$$
is largest. The quantity $x$ appears as a quadratic function. That is where QDA gets its name.
3.1 Comparsion of LDA and QDA
LDA has the bettter variance but suffers from high bias, QDA is vice-versa.
Can you write Python code for it?

from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
qda = QuadraticDiscriminantAnalysis()
qda.fit(X,y)
pred = qda.predict(X)
print(1-accuracy_score(pred, y))


I am working on the file "Chapter04.ipynb".
The first task is:
4 Exercises
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt


I am working on the file "Chapter04/GridWorld/RunGridWorld.ipynb".
The first task is:
Now suppose the dynamics of state 13 are also changed, such that action down from state 13 takes the agent to the new state 15.
Can you write Python code for it?

# World configuration
world_size = (5, 4)
special_state = [([0, 0], [-1, -1], range(4)), ([3, 3], [-1, -1], range(4)), ([4, 0], [-1, -1], range(4)),
                 ([4, 2], [-1, -1], range(4)), ([4, 3], [-1, -1], range(4)), ([4, 1], [3, 1], [0]),
                 ([4, 1], [4, 1], [1]), ([4, 1], [3, 0], [2]), ([4, 1], [3, 2], [3])]

# Initialize grid world
world = GridWorld(world_size, special_state)

# Policy configuration
policy = RandomPolicy(1)

# Iteration
iteration = 0
diffs = []
while not (world.diff < 1e-4):
    world.step(policy)
    iteration += 1
    diffs.append(world.diff)

# Show value matrix
world.show_value(3, 5)


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-02] MNISTのデータセットを用意します。
Can you write Python code for it?

mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-03] フィルターに対応する Variable を用意して、入力データにフィルターとプーリング層を適用する計算式を定義します。
Can you write Python code for it?

num_filters = 16

x = tf.placeholder(tf.float32, [None, 784])
x_image = tf.reshape(x, [-1,28,28,1])

W_conv = tf.Variable(tf.truncated_normal([5,5,1,num_filters], stddev=0.1))
h_conv = tf.nn.conv2d(x_image, W_conv,
                      strides=[1,1,1,1], padding='SAME')
h_pool =tf.nn.max_pool(h_conv, ksize=[1,2,2,1],
                       strides=[1,2,2,1], padding='SAME')


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-05] 誤差関数 loss、トレーニングアルゴリズム train_step、正解率 accuracy を定義します。
Can you write Python code for it?

t = tf.placeholder(tf.float32, [None, 10])
loss = -tf.reduce_sum(t * tf.log(p))
train_step = tf.train.AdamOptimizer(0.0005).minimize(loss)
correct_prediction = tf.equal(tf.argmax(p, 1), tf.argmax(t, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-07] 畳込みフィルターの値と、最初の9個分の画像データに対して、畳み込みフィルターとプーリング層を適用した結果を取得します。
Can you write Python code for it?

filter_vals, conv_vals, pool_vals = sess.run(
    [W_conv, h_conv, h_pool], feed_dict={x:mnist.test.images[:9]})


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-08] 畳込みフィルターを適用した結果を画像として表示します。
畳込みフィルターを適用した後は、ピクセル値が負の値をとることもあるため、背景（ピクセル値 0）の部分が白にならない点に注意してください。
Can you write Python code for it?

fig = plt.figure(figsize=(10,num_filters+1))

for i in range(num_filters):
    subplot = fig.add_subplot(num_filters+1, 10, 10*(i+1)+1)
    subplot.set_xticks([])
    subplot.set_yticks([])
    subplot.imshow(filter_vals[:,:,0,i],
                   cmap=plt.cm.gray_r, interpolation='nearest')

for i in range(9):
    subplot = fig.add_subplot(num_filters+1, 10, i+2)
    subplot.set_xticks([])
    subplot.set_yticks([])
    subplot.set_title('%d' % np.argmax(mnist.test.labels[i]))
    subplot.imshow(mnist.test.images[i].reshape((28,28)),
                   vmin=0, vmax=1,
                   cmap=plt.cm.gray_r, interpolation='nearest')

    for f in range(num_filters):
        subplot = fig.add_subplot(num_filters+1, 10, 10*(f+1)+i+2)
        subplot.set_xticks([])
        subplot.set_yticks([])
        subplot.imshow(conv_vals[i,:,:,f],
                       cmap=plt.cm.gray_r, interpolation='nearest')


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-09] 同じく、畳込みフィルターとプーリング層を適用した結果を画像として表示します。
Can you write Python code for it?

fig = plt.figure(figsize=(10,num_filters+1))

for i in range(num_filters):
    subplot = fig.add_subplot(num_filters+1, 10, 10*(i+1)+1)
    subplot.set_xticks([])
    subplot.set_yticks([])
    subplot.imshow(filter_vals[:,:,0,i],
                   cmap=plt.cm.gray_r, interpolation='nearest')

for i in range(9):
    subplot = fig.add_subplot(num_filters+1, 10, i+2)
    subplot.set_xticks([])
    subplot.set_yticks([])
    subplot.set_title('%d' % np.argmax(mnist.test.labels[i]))
    subplot.imshow(mnist.test.images[i].reshape((28,28)),
                   vmin=0, vmax=1,
                   cmap=plt.cm.gray_r, interpolation='nearest')

    for f in range(num_filters):
        subplot = fig.add_subplot(num_filters+1, 10, 10*(f+1)+i+2)
        subplot.set_xticks([])
        subplot.set_yticks([])
        subplot.imshow(pool_vals[i,:,:,f],
                       cmap=plt.cm.gray_r, interpolation='nearest')


I am working on the file "Chapter04/MNIST dynamic filter result.ipynb".
The first task is:
[MDR-10] 正しく分類できなかったいくつかのデータについて、それぞれの文字である確率を確認します。
Can you write Python code for it?

fig = plt.figure(figsize=(12,10))
c=0
for (image, label) in zip(mnist.test.images,
                          mnist.test.labels):
    p_val = sess.run(p, feed_dict={x:[image]})
    pred = p_val[0]
    prediction, actual = np.argmax(pred), np.argmax(label)
    if prediction == actual:
        continue
    subplot = fig.add_subplot(5,4,c*2+1)
    subplot.set_xticks([])
    subplot.set_yticks([])
    subplot.set_title('%d / %d' % (prediction, actual))
    subplot.imshow(image.reshape((28,28)), vmin=0, vmax=1,
                   cmap=plt.cm.gray_r, interpolation="nearest")
    subplot = fig.add_subplot(5,4,c*2+2)
    subplot.set_xticks(range(10))
    subplot.set_xlim(-0.5,9.5)
    subplot.set_ylim(0,1)
    subplot.bar(range(10), pred, align='center')
    c += 1
    if c == 10:
        break


I am working on the file "Chapter05.ipynb".
The first task is:
5.1.3 scikit-learn の主成分分析
Can you write Python code for it?

from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, classifier, resolution=0.02):
    # マーカーとカラーマップの用意
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # 決定領域のプロット
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    # グリッドポイントの生成
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    # 各特徴量を1次元配列に変換して予測を実行
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    # 予測結果を元のグリッドポイントのデータサイズに変換
    Z = Z.reshape(xx1.shape)
    # グリッドポイントの等高線のプロット
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    # 軸の範囲の設定
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    # クラスごとにサンプルをプロット
    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y==cl, 0], y=X[y==cl, 1], alpha=0.8, c=cmap(idx), marker=markers[idx], label=cl)

from sklearn.linear_model import LogisticRegression
from sklearn.decomposition import PCA
# 主成分数を指定して、PCAのインスタンスを生成
pca = PCA(n_components=2)
# ロジスティック回帰のインスタンスを生成
lr = LogisticRegression()
# トレーニングデータやテストデータをPCAに適合させる
X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)
# トレーニングデータをロジスティック回帰に適合させる
lr.fit(X_train_pca, y_train)
# 決定境界をプロット
plot_decision_regions(X_train_pca, y_train, classifier=lr)
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.legend(loc='lower left')
plt.show()

# 決定境界をプロット
plot_decision_regions(X_test_pca, y_test, classifier=lr)
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.legend(loc='lower left')
plt.show()

pca = PCA(n_components=None)
X_train_pca = pca.fit_transform(X_train_std)
# 分散説明率を計算
pca.explained_variance_ratio_


I am working on the file "Chapter05.ipynb".
The first task is:
5.2 線形判別分析による教師ありデータ圧縮

d次元のデータセットを標準化する(dは特徴量の個数)
クラスごとにd次元の平均ベクトルを計算する
クラス間変動行列SBと、クラス内変動行列SWを生成する
行列 SW^-1 SB の固有ベクトルと対応する固有値を計算する
d x k次元の変換行列Wを生成するために、最も大きいk個の固有値に対応するk個の固有ベクトルを選択する
変換行列Wを使ってサンプルを新しい特徴部分空間へ射影する

5.2.1 変動行列を計算する

クラス内変動行列(within-class scatter matrix)
クラス間変動行列(between-class scatter matrix)
Can you write Python code for it?

# 1. d次元のデータセットを標準化する(dは特徴量の個数)
# X_train_std, X_test_std は作成済

# 2. クラスごとにd次元の平均ベクトルを計算する
np.set_printoptions(precision=4)
mean_vecs = []
for label in range(1, 4):
    mean_vecs.append(np.mean(X_train_std[y_train==label], axis=0))
    print('MV {}:, {}\n'.format(label, mean_vecs[label - 1]))

# 3. クラス間変動行列SBと、クラス内変動行列SWを生成する
d = 13 # 特徴量の個数
# クラス内変動行列 SW
S_W = np.zeros((d, d)) # 13 x 13 で値がすべて0の行列を生成
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.zeros((d, d))
    for row in X_train_std[y_train == label]:
        row, mv = row.reshape(d, 1), mv.reshape(d, 1)
        class_scatter += (row - mv).dot((row - mv).T)

    S_W += class_scatter

print('Within-class scatter matrix: {}x{}'.format(S_W.shape[0], S_W.shape[1]))

# クラスラベルの一様に分散していない
print('Class label distribution: {}'.format(np.bincount(y_train)[1:]))

d = 13
# クラス内変動行列 SW
S_W = np.zeros((d, d))
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.cov(X_train_std[y_train == label].T)
    S_W += class_scatter

print('Scaled within-class scatter matrix: {}x{}'.format(S_W.shape[0], S_W.shape[1]))

# クラス間変動行列SB
mean_overall = np.mean(X_train_std, axis=0)
d = 13
S_B = np.zeros((d, d))
for i, mean_vec in enumerate(mean_vecs):
    n = X_train[y_train==i + 1, :].shape[0]
    mean_vec = mean_vec.reshape(d, 1)
    mean_overall = mean_overall.reshape(d, 1)
    S_B += n * (mean_vec - mean_overall).dot((mean_vec - mean_overall).T)

print('Between-class scatter matrix: {}x{}'.format(S_B.shape[0], S_B.shape[1]))


I am working on the file "Chapter05.ipynb".
The first task is:
5.2.4 scikit-learn による LDA
Can you write Python code for it?

if Version(sklearn_version) < '0.18':
    from sklearn.lda import LDA
else:
    from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# 次元数を指定して、LDAのインスタンスを生成
lda = LDA(n_components=2)
X_train_lda = lda.fit_transform(X_train_std, y_train)

from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr.fit(X_train_lda, y_train)
# 決定境界をプロット
plot_decision_regions(X_train_lda, y_train, classifier=lr)
plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower left')
plt.show()

X_test_lda = lda.transform(X_test_std)
plot_decision_regions(X_test_lda, y_test, classifier=lr)
plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower left')
plt.show()


I am working on the file "Chapter05.ipynb".
The first task is:
5.3 カーネル主成分分析を使った非線形写像

カーネル化したPCA(kernel PCA)

5.3.1 カーネル関数とカーネルトリック
5.3.2 Python でカーネル主成分分析を実装する
Can you write Python code for it?

from scipy.spatial.distance import pdist, squareform
from scipy import exp
from scipy.linalg import eigh
import numpy as np

def rbf_kernel_pca(X, gamma, n_components):
    """
    RBPカーネルPCAの実装

    パラメータ
    ----------
    X: [NumPy ndarray], shape = [n_samples, n_features]

    gamma: float
        RBFカーネルのチューニングパラメータ

    n_components: int
        返される主成分の個数

    戻り値
    ------
    X_pc: [NumPy ndarray], shape = [n_samples, n_features]
        射影されたデータセット
    """
    # M x N 次元のデータセットでペアごとの平方ユークリッド距離を計算
    sq_dists = pdist(X, 'sqeuclidean')

    # ペアごとの距離を正方行列に変換
    mat_sq_dists = squareform(sq_dists)

    # 対象カーネル行列を計算
    K = exp(-gamma * mat_sq_dists)

    # カーネル行列を中心窩
    N = K.shape[0]
    one_n = np.ones((N, N)) / N
    K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)

    # 中心化されたカーネル行列から固有値を取得
    # numpy.eigh はそれらをソート順に返す
    eigvals, eigvecs = eigh(K)

    # 上位k個の固有ベクトル(射影されたサンプル)を収集
    X_pc = np.column_stack((eigvecs[:, -i] for i in range(1, n_components + 1)))

    return X_pc


I am working on the file "Chapter05.ipynb".
The first task is:
5.3.2 新しい特徴空間にサンプルを射影する
Can you write Python code for it?

# 標準化したトレーニングデータに変換行列をかける
X_train_lda = X_train_std.dot(w)

colors = ['r', 'b', 'g']
markers = ['s', 'x', 'o']
# クラスラベル、点の色、点の種類の組み合わせからなるリストを生成してプロット
for label, c, m in zip(np.unique(y_train), colors, markers):
    plt.scatter(X_train_lda[y_train==label, 0] * -1, X_train_lda[y_train==label, 1] * -1, c=c, label=label, marker=m)

plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower left')
plt.show()


I am working on the file "Chapter05.ipynb".
The first task is:
例1: 半月形の分離
Can you write Python code for it?

from sklearn.datasets import make_moons
import matplotlib.pyplot as plt

# 2つの半月形データを作成
X, y = make_moons(n_samples=100, random_state=123)
plt.scatter(X[y==0, 0], X[y==0, 1], color='red', marker='^', alpha=0.5)
plt.scatter(X[y==1, 0], X[y==1, 1], color='blue', marker='o', alpha=0.5)
plt.show()

# 標準のPCAを使ってみる
from sklearn.decomposition import PCA
scikit_pca = PCA(n_components=2)
X_spca = scikit_pca.fit_transform(X)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
ax[0].scatter(X_spca[y==0, 0], X_spca[y==0, 1], color='red', marker='^', alpha=0.5)
ax[0].scatter(X_spca[y==1, 0], X_spca[y==1, 1], color='blue', marker='o', alpha=0.5)
# 2番目のグラフ領域に散布図をプロット
ax[1].scatter(X_spca[y==0, 0], np.zeros((50, 1)) + 0.02, color='red', marker='^', alpha=0.5)
ax[1].scatter(X_spca[y==1, 0], np.zeros((50, 1)) - 0.02, color='blue', marker='o', alpha=0.5)
ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')
plt.show()

from matplotlib.ticker import FormatStrFormatter
# カーネルPCA関数を使う
X_kpca = rbf_kernel_pca(X, gamma=15, n_components=2)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
ax[0].scatter(X_kpca[y==0, 0], X_spca[y==0, 1], color='red', marker='^', alpha=0.5)
ax[0].scatter(X_kpca[y==1, 0], X_spca[y==1, 1], color='blue', marker='o', alpha=0.5)
# 2番目のグラフ領域に散布図をプロット
ax[1].scatter(X_kpca[y==0, 0], np.zeros((50, 1)) + 0.02, color='red', marker='^', alpha=0.5)
ax[1].scatter(X_kpca[y==1, 0], np.zeros((50, 1)) - 0.02, color='blue', marker='o', alpha=0.5)
ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')
ax[0].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
ax[1].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
plt.show()


I am working on the file "Chapter05.ipynb".
The first task is:
例2: 同心円の分離
Can you write Python code for it?

from sklearn.datasets import make_circles
import matplotlib.pyplot as plt

# 同心円用のデータを作成してプロット
X, y = make_circles(n_samples=1000, random_state=123, noise=0.1, factor=0.2)
plt.scatter(X[y==0, 0], X[y==0, 1], color='red', marker='^', alpha=0.5)
plt.scatter(X[y==1, 0], X[y==1, 1], color='blue', marker='o', alpha=0.5)
plt.show()

# 標準のPCAを使ってみる
from sklearn.decomposition import PCA
scikit_pca = PCA(n_components=2)
X_spca = scikit_pca.fit_transform(X)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
ax[0].scatter(X_spca[y==0, 0], X_spca[y==0, 1], color='red', marker='^', alpha=0.5)
ax[0].scatter(X_spca[y==1, 0], X_spca[y==1, 1], color='blue', marker='o', alpha=0.5)
# 2番目のグラフ領域に散布図をプロット
ax[1].scatter(X_spca[y==0, 0], np.zeros((500, 1)) + 0.02, color='red', marker='^', alpha=0.5)
ax[1].scatter(X_spca[y==1, 0], np.zeros((500, 1)) - 0.02, color='blue', marker='o', alpha=0.5)
ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')
plt.show()

# カーネルPCA関数を使う
X_kpca = rbf_kernel_pca(X, gamma=15, n_components=2)
fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
ax[0].scatter(X_kpca[y==0, 0], X_spca[y==0, 1], color='red', marker='^', alpha=0.5)
ax[0].scatter(X_kpca[y==1, 0], X_spca[y==1, 1], color='blue', marker='o', alpha=0.5)
# 2番目のグラフ領域に散布図をプロット
ax[1].scatter(X_kpca[y==0, 0], np.zeros((500, 1)) + 0.02, color='red', marker='^', alpha=0.5)
ax[1].scatter(X_kpca[y==1, 0], np.zeros((500, 1)) - 0.02, color='blue', marker='o', alpha=0.5)
ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')
plt.show()


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Correct the features related to the number of smoked cigarettes.
Can you write Python code for it?

births_transformed = births_trimmed \
    .withColumn('CIG_BEFORE', correct_cig('CIG_BEFORE'))\
    .withColumn('CIG_1_TRI', correct_cig('CIG_1_TRI'))\
    .withColumn('CIG_2_TRI', correct_cig('CIG_2_TRI'))\
    .withColumn('CIG_3_TRI', correct_cig('CIG_3_TRI'))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Correlations
Correlations between our features.
Can you write Python code for it?

corrs = st.Statistics.corr(numeric_rdd)

for i, el in enumerate(corrs > 0.5):
    correlated = [
        (numeric_cols[j], corrs[i][j])
        for j, e in enumerate(el)
        if e == 1.0 and j != i]

    if len(correlated) > 0:
        for e in correlated:
            print('{0}-to-{1}: {2:.2f}' \
                  .format(numeric_cols[i], e[0], e[1]))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Create the final dataset
Create an RDD of LabeledPoints
We will use a hashing trick to encode the 'BIRTH_PLACE' feature.
Can you write Python code for it?

import pyspark.mllib.feature as ft
import pyspark.mllib.regression as reg

hashing = ft.HashingTF(7)

births_hashed = births_transformed \
    .rdd \
    .map(lambda row: [
            list(hashing.transform(row[1]).toArray())
                if col == 'BIRTH_PLACE'
                else row[i]
            for i, col
            in enumerate(features_to_keep)]) \
    .map(lambda row: [[e] if type(e) == int else e
                      for e in row]) \
    .map(lambda row: [item for sublist in row
                      for item in sublist]) \
    .map(lambda row: reg.LabeledPoint(
            row[0],
            ln.Vectors.dense(row[1:]))
        )


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Figure out which Yes/No/Unknown features are.
Can you write Python code for it?

cols = [(col.name, col.dataType) for col in births_trimmed.schema]

YNU_cols = []

for i, s in enumerate(cols):
    if s[1] == typ.StringType():
        dis = births.select(s[0]) \
            .distinct() \
            .rdd \
            .map(lambda row: row[0]) \
            .collect()

        if 'Y' in dis:
            YNU_cols.append(s[0])


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
For the categorical variables we will calculate the frequencies of their values.
Can you write Python code for it?

categorical_cols = [e for e in births_transformed.columns
                    if e not in numeric_cols]

categorical_rdd = births_transformed\
                       .select(categorical_cols)\
                       .rdd \
                       .map(lambda row: [e for e in row])

for i, col in enumerate(categorical_cols):
    agg = categorical_rdd \
        .groupBy(lambda row: row[i]) \
        .map(lambda row: (row[0], len(row[1])))

    print(col, sorted(agg.collect(),
                      key=lambda el: el[1],
                      reverse=True))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Get to know your data
Descriptive statistics
We will use the colStats(...) method.
Can you write Python code for it?

import pyspark.mllib.stat as st
import numpy as np

numeric_cols = ['MOTHER_AGE_YEARS','FATHER_COMBINED_AGE',
                'CIG_BEFORE','CIG_1_TRI','CIG_2_TRI','CIG_3_TRI',
                'MOTHER_HEIGHT_IN','MOTHER_PRE_WEIGHT',
                'MOTHER_DELIVERY_WEIGHT','MOTHER_WEIGHT_GAIN'
               ]

numeric_rdd = births_transformed\
                       .select(numeric_cols)\
                       .rdd \
                       .map(lambda row: [e for e in row])

mllib_stats = st.Statistics.colStats(numeric_rdd)

for col, m, v in zip(numeric_cols,
                     mllib_stats.mean(),
                     mllib_stats.variance()):
    print('{0}: \t{1:.2f} \t {2:.2f}'.format(col, m, np.sqrt(v)))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Let's check if we got it correctly.
Can you write Python code for it?


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Let's now use the model to predict the classes for our testing set.
Can you write Python code for it?

LR_results = (
        births_test.map(lambda row: row.label) \
        .zip(LR_Model \
             .predict(births_test\
                      .map(lambda row: row.features)))
    ).map(lambda row: (row[0], row[1] * 1.0))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Let's see how the logistic regression would perform with reduced number of features.
Can you write Python code for it?

LR_Model_2 = LogisticRegressionWithLBFGS \
    .train(topFeatures_train, iterations=10)

LR_results_2 = (
        topFeatures_test.map(lambda row: row.label) \
        .zip(LR_Model_2 \
             .predict(topFeatures_test \
                      .map(lambda row: row.features)))
    ).map(lambda row: (row[0], row[1] * 1.0))

LR_evaluation_2 = ev.BinaryClassificationMetrics(LR_results_2)

print('Area under PR: {0:.2f}' \
      .format(LR_evaluation_2.areaUnderPR))
print('Area under ROC: {0:.2f}' \
      .format(LR_evaluation_2.areaUnderROC))
LR_evaluation_2.unpersist()


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Let's see how well our model did.
Can you write Python code for it?

RF_results = (
        topFeatures_test.map(lambda row: row.label) \
        .zip(RF_model \
             .predict(topFeatures_test \
                      .map(lambda row: row.features)))
    )

RF_evaluation = ev.BinaryClassificationMetrics(RF_results)

print('Area under PR: {0:.2f}' \
      .format(RF_evaluation.areaUnderPR))
print('Area under ROC: {0:.2f}' \
      .format(RF_evaluation.areaUnderROC))
RF_evaluation.unpersist()


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Predicting infant survival
Logistic regression in Spark
MLLib used to provide a logistic regression model estimated using a stochastic gradient descent (SGD) algorithm. This model has been deprecated in Spark 2.0 in favor of the LogisticRegressionWithLBFGS model.
Can you write Python code for it?

from pyspark.mllib.classification \
    import LogisticRegressionWithLBFGS

LR_Model = LogisticRegressionWithLBFGS \
    .train(births_train, iterations=10)


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Random Forest in Spark
We are now ready to build the random forest model.
Can you write Python code for it?

from pyspark.mllib.tree import RandomForest

RF_model = RandomForest \
    .trainClassifier(data=topFeatures_train,
                     numClasses=2,
                     categoricalFeaturesInfo={},
                     numTrees=6,
                     featureSubsetStrategy='all',
                     seed=666)


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Selecting only the most predictable features
MLLib allows us to select the most predictable features using a Chi-Square selector.
Can you write Python code for it?

selector = ft.ChiSqSelector(4).fit(births_train)

topFeatures_train = (
        births_train.map(lambda row: row.label) \
        .zip(selector \
             .transform(births_train \
                        .map(lambda row: row.features)))
    ).map(lambda row: reg.LabeledPoint(row[0], row[1]))

topFeatures_test = (
        births_test.map(lambda row: row.label) \
        .zip(selector \
             .transform(births_test \
                        .map(lambda row: row.features)))
    ).map(lambda row: reg.LabeledPoint(row[0], row[1]))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Specify our recode dictionary.
Can you write Python code for it?


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
Statistical testing
Run a Chi-square test to determine if there are significant differences for categorical variables.
Can you write Python code for it?

import pyspark.mllib.linalg as ln

for cat in categorical_cols[1:]:
    agg = births_transformed \
        .groupby('INFANT_ALIVE_AT_REPORT') \
        .pivot(cat) \
        .count()

    agg_rdd = agg \
        .rdd\
        .map(lambda row: (row[1:])) \
        .flatMap(lambda row:
                 [0 if e == None else e for e in row]) \
        .collect()

    row_length = len(agg.collect()[0]) - 1
    agg = ln.Matrices.dense(row_length, 2, agg_rdd)

    test = st.Statistics.chiSqTest(agg)
    print(cat, round(test.pValue, 4))


I am working on the file "Chapter05/LearningPySpark_Chapter05.ipynb".
The first task is:
We can drop most of highly correlated features.
Can you write Python code for it?

features_to_keep = [
    'INFANT_ALIVE_AT_REPORT',
    'BIRTH_PLACE',
    'MOTHER_AGE_YEARS',
    'FATHER_COMBINED_AGE',
    'CIG_1_TRI',
    'MOTHER_HEIGHT_IN',
    'MOTHER_PRE_WEIGHT',
    'DIABETES_PRE',
    'DIABETES_GEST',
    'HYP_TENS_PRE',
    'HYP_TENS_GEST',
    'PREV_BIRTH_PRETERM'
]
births_transformed = births_transformed.select([e for e in features_to_keep])


I am working on the file "Chapter06.ipynb".
The first task is:
6.1.2 パイプラインで変換器と推定器を結合する
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

# スケーリング(標準化)、主成分分析(PCA)、ロジスティック回帰をパイプラインで連結する
pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca', PCA(n_components=2)),
                    ('clf', LogisticRegression(random_state=1))
                   ])
pipe_lr.fit(X_train, y_train)
print('Test Accuracy: {:.3f}'.format(pipe_lr.score(X_test, y_test)))


I am working on the file "Chapter06.ipynb".
The first task is:
6.2 k分割交差検証を使ったモデルの性能の評価

ホールドアウト法(holdout method)
k分割交差検証(k-fold corss-validation)

6.2.1 ホールドアウト法

トレーニングデータセットをトレーニングデータセットと検証データセットに分割
検証データセットでの性能により、モデルを選択する

6.2.2 k分割交差検証

標準的に用いられるkの値は10
比較的小さなトレーニングデータセットの場合→分割数(k)を増やす
大きなデータセットの場合→分割数(k)を減らす

かなり小さいデータセットの場合→サンプルの個数と同じにする(k=n)

改善したものに、層化k分割交差検証(stratified k-fold corss-validation)がある

クラスの比率が均等でないときに、評価のバイアスとバリアンスが改善される
Can you write Python code for it?

import numpy as np
from sklearn.cross_validation import StratifiedKFold

# 分割元データ、分割数、乱数生成器の状態を指定し、層化k分割交差検証のインスタンスを生成
kfold = StratifiedKFold(y=y_train, n_folds=10, random_state=1)
scores = []

for k, (train, test) in enumerate(kfold):
    pipe_lr.fit(X_train[train], y_train[train])
    score = pipe_lr.score(X_train[test], y_train[test])
    scores.append(score)
    print('Fold: {}, Class dist.: {}, Acc: {:.3f}'.format(k + 1, np.bincount(y_train[train]), score))

# 正解率の平均と標準偏差を出力
print('CV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores), np.std(scores)))

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import cross_val_score
else:
    from sklearn.model_selection import cross_val_score

# 交差検証の cross_val_score 関数でモデルの正解率を算出
scores = cross_val_score(estimator=pipe_lr, X=X_train, y=y_train, cv=10, n_jobs=1)
print('CV accuracy scores: {}'.format(scores))
print('CV accuracy: {:.3f} +/- {:.3f}'.format(np.mean(scores), np.std(scores)))


I am working on the file "Chapter06.ipynb".
The first task is:
6.3 学習曲線と検証曲線によるアルゴリズムの診断

学習曲線(learning curve)
検証曲線(validation curve)

6.3.1 学習曲線を使ってバイアスとバリアンスの問題を診断する
Can you write Python code for it?

import matplotlib.pyplot as plt
if Version(sklearn_version) < '0.18':
    from sklearn.learning_curve import learning_curve
else:
    from sklearn.model_selection import learning_curve

pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('clf', LogisticRegression(penalty='l2', random_state=0))
                   ])
# learning_curve 関数で交差検証による正解率を算出
train_sizes, train_scores, test_scores = learning_curve(estimator=pipe_lr,
                                                         X=X_train, y=y_train,
                                                         train_sizes=np.linspace(0.1, 1.0, 10),
                                                         cv=10, n_jobs=1)
display(train_sizes)

# 平均と標準偏差を求める
train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(train_sizes, train_mean, color='blue', marker='o', markersize=5, label='training accuracy')
# fill_between 関数で平均±標準偏差の幅を塗りつぶす
plt.fill_between(train_sizes, train_mean + train_std, train_mean - train_std, alpha=0.15, color='blue')

plt.plot(train_sizes, test_mean, color='green', linestyle='--', marker='s', markersize=5, label='validation accuracy')
plt.fill_between(train_sizes, test_mean + test_std, test_mean - test_std, alpha=0.15, color='green')

plt.grid()
plt.xlabel('Number of training samples')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')
plt.ylim([0.8, 1.0])
plt.show()


I am working on the file "Chapter06.ipynb".
The first task is:
6.3.2 検証曲線を使って過学習と学習不足を明らかにする

たとえばロジスティック回帰の逆正則化パラメータCを変化させる
Can you write Python code for it?

if Version(sklearn_version) < '0.18':
    from sklearn.learning_curve import validation_curve
else:
    from sklearn.model_selection import validation_curve

param_range = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
# validation_curve関数によりモデルのパラメータを変化させ、交差検証による正解率を算出
# clf__C は LogisticRegression オブジェクトのパラメータ
train_scores, test_scores = validation_curve(estimator=pipe_lr,
                                             X=X_train, y=y_train,
                                             param_name='clf__C',
                                             param_range=param_range,
                                             cv=10)

# 平均と標準偏差を求める
train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(param_range, train_mean, color='blue', marker='o', markersize=5, label='training accuracy')
plt.fill_between(param_range, train_mean + train_std, train_mean - train_std, alpha=0.15, color='blue')

plt.plot(param_range, test_mean, color='green', linestyle='--', marker='s', markersize=5, label='validation accuracy')
plt.fill_between(param_range, test_mean + test_std, test_mean - test_std, alpha=0.15, color='green')

plt.grid()
plt.xscale('log')
plt.xlabel('Parameter C')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')
plt.ylim([0.8, 1.0])
plt.show()


I am working on the file "Chapter06.ipynb".
The first task is:
6.4 グリッドサーチによる機械学習モデルのチューニング

グリッドサーチ(grid search)
ハイパーパラメーター(hyperparameter)
例: ロジスティック回帰の正則化パラメータ
例: 決定木の深さパラメータ

6.4.1 グリッドサーチを使ってハイパーパラメータをチューニングする
Can you write Python code for it?

from sklearn.svm import SVC
if Version(sklearn_version) < '0.18':
    from sklearn.grid_search import GridSearchCV
else:
    from sklearn.model_selection import GridSearchCV

pipe_svc = Pipeline([('scl', StandardScaler()),
                     ('clf', SVC(random_state=1))
    ])
param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
param_grid = [{'clf__C': param_range, 'clf__kernel': ['linear']},
              {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['rbf']}
             ]
# ハイパーパラメータ値のリストparam_gridを指定し、グリッドサーチを行うGridSearchCVクラスのインスタンス化
gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring='accuracy', cv=10, n_jobs=-1)
gs = gs.fit(X_train, y_train)
# モデルの最良スコアを出力
print(gs.best_score_)
# 最良スコアとなるパラメータ値を出力
print(gs.best_params_)

# 選択されたモデルの性能を評価する
clf = gs.best_estimator_
display(clf)
clf.fit(X_train, y_train)
print('Test accuracy: {:.3f}'.format(clf.score(X_test, y_test)))

for params, mean_score, socres in gs.grid_scores_:
    print('{:.3f} (+/-{:.3f}) for {}'.format(mean_score, scores.std(), params))


I am working on the file "Chapter06.ipynb".
The first task is:
6.4.2 入れ子式の交差検証によるアルゴリズムの選択

入れ子式の交差検証(nested corss-validation)
Can you write Python code for it?

gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring='accuracy', cv=2, n_jobs=-1)
scores = cross_val_score(gs, X_train, y_train, scoring='accuracy', cv=5)
print('CV accuracy: {:.3f} +/-{:.3f}'.format(np.mean(scores), np.std(scores)))

from sklearn.tree import DecisionTreeClassifier
# ハイパーパラメーター値として決定木の深さパラメータを指定し、
# グリッドサーチを行う GridSearchCV クラスをインスタンス化
estimator = DecisionTreeClassifier(random_state=0)
param_grid = [{'max_depth': [1, 2, 3, 4, 5, 6, 7, None]}]
gs = GridSearchCV(estimator=estimator, param_grid=param_grid, scoring='accuracy', cv=2)

scores = cross_val_score(gs, X_train, y_train, scoring='accuracy', cv=5)
print('CV accuracy: {:.3f} +/-{:.3f}'.format(np.mean(scores), np.std(scores)))


I am working on the file "Chapter06.ipynb".
The first task is:
6.5 さまざまな性能評価指標

適合率(precision)
再現率(recall)
F1 スコア(F1-score)

6.5.1 混同行列を解釈する

混同行列(cofusion matrix)
真陽性(true positive)
真陰性(true negative)
偽陽性(false positive)
偽陰性(false negative)
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
pipe_svc.fit(X_train, y_train)
y_pred = pipe_svc.predict(X_test)
# テストと予測データから混同行列を生成
confmat = confusion_matrix(y_true=y_test, y_pred=y_pred)
print(confmat)

# 図のサイズを指定
fig, ax = plt.subplots(figsize=(2.5, 2.5))
# matshow 関数で行列からヒートマップを描画
ax.matshow(confmat, cmap=plt.cm.Blues, alpha=0.3)
for i in range(confmat.shape[0]):
    for j in range(confmat.shape[1]):
        ax.text(x=j, y=i, s=confmat[i, j], va='center', ha='center')
plt.xlabel('predicted label')
plt.ylabel('true label')
plt.show()


I am working on the file "Chapter06.ipynb".
The first task is:
6.5.2 分類モデルの適合率と再現率を最適化する

誤分類率(ERR)
正解率(ACC)
真陽性率(TPR)
偽陽性率(FPR)
適合率(PRE: precision)
再現率(REC: recall)
Can you write Python code for it?

# 適合率、再現率、F1 スコアを出力
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
print('Precision: {:.3f}'.format(precision_score(y_true=y_test, y_pred=y_pred)))
print('Recall: {:.3f}'.format(recall_score(y_true=y_test, y_pred=y_pred)))
print('F1: {:.3f}'.format(f1_score(y_true=y_test, y_pred=y_pred)))

# カスタムの性能指標を出力
from sklearn.metrics import make_scorer, f1_score
scorer = make_scorer(f1_score, pos_label=0)
param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
param_grid = [{'clf__C': param_range, 'clf__kernel': ['linear']},
              {'clf__C': param_range, 'clf__gamma': param_range, 'clf__kernel': ['rbf']}
             ]
gs = GridSearchCV(estimator=pipe_svc, param_grid=param_grid, scoring=scorer, cv=10)
gs = gs.fit(X_train, y_train)
# モデルの最良スコアを出力
print(gs.best_score_)
# 最良スコアとなるパラメータ値を出力
print(gs.best_params_)


I am working on the file "Chapter06.ipynb".
The first task is:
6.5.3 ROC曲線をプロットする

受信者操作特性(Receiver Operator Characteristic: ROC)曲線
曲線下面積(Area Under the Curve: AUC)
Can you write Python code for it?

from sklearn.metrics import roc_curve, auc
from scipy import interp
# スケーリング、主成分分析、ロジスティック回帰を指定して、Pipelineを作る
pipe_lr = Pipeline([('scl', StandardScaler()),
                    ('pca', PCA(n_components=2)),
                    ('clf', LogisticRegression(penalty='l2', random_state=0, C=100.0))
                   ])
# 2つの特徴量を抽出
X_train2 = X_train[:, [4, 14]]
# 層化k分割交差検証イテレータを表す StratifiedKFold クラスをインスタンス化
cv = StratifiedKFold(y_train, n_folds=3, random_state=1)
fig = plt.figure(figsize=(7, 5))
mean_tpr = 0.0

# 0から1までの間に100個の要素を生成
mean_fpr = np.linspace(0, 1, 100)
all_tpr = []

for i, (train, test) in enumerate(cv):
    # predict_probaメソッドで確率を予測、fitメソッドでモデルに適合させる
    probas = pipe_lr.fit(X_train2[train], y_train[train]).predict_proba(X_train2[test])
    # roc_curve関数でROC曲線の性能を計算してプロット
    fpr, tpr, thresholds = roc_curve(y_train[test], probas[:, 1], pos_label=1)
    # FPR(X軸)とTPR(Y軸)を線形補完
    mean_tpr += interp(mean_fpr, fpr, tpr)
    mean_tpr[0] = 0.0
    # 曲線下面積(AUC)を計算
    roc_auc = auc(fpr, tpr)
    plt.plot(fpr, tpr, lw=1, label='ROC fold {:d} (area = {:.2f})'.format(i + 1, roc_auc))

# 当て推量をプロット
plt.plot([0, 1], [0, 1], linestyle='--', color=(0.6, 0.6, 0.6), label='random guessing')

# FPR, TPR, ROC AUC それぞれの平均を計算してプロット
mean_tpr /= len(cv)
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
plt.plot(mean_fpr, mean_tpr, 'k--', label='mean ROC (area = {:.2f})'.format(mean_auc), lw=2)

# 完全に予測が正解したときのROC曲線をプロット
plt.plot([0, 0, 1], [0, 1, 1], lw=2, linestyle=':', color='black', label='perfect performance')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('false positive rate')
plt.ylabel('true positive rate')
plt.title('Receiver Operator Characteristic')
plt.legend(loc='lower right')
plt.show()

# 予測値のAUCと正解率を計算して出力
pipe_lr = pipe_lr.fit(X_train2, y_train)
y_pred2 = pipe_lr.predict(X_test[:, [4, 14]])
from sklearn.metrics import roc_auc_score, accuracy_score
print('ROC AUC: {:.3f}'.format(roc_auc_score(y_true=y_test, y_score=y_pred2)))
print('Accuracy:...


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Classification
We will now use the RandomForestClassfier to model the chances of survival for an infant.
First, we need to cast the label feature to DoubleType.
Can you write Python code for it?

import pyspark.sql.functions as func

births = births.withColumn(
    'INFANT_ALIVE_AT_REPORT',
    func.col('INFANT_ALIVE_AT_REPORT').cast(typ.DoubleType())
)

births_train, births_test = births \
    .randomSplit([0.7, 0.3], seed=666)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Clustering
In this example we will use k-means model to find similarities in the births data.
Can you write Python code for it?

import pyspark.ml.clustering as clus

kmeans = clus.KMeans(k = 5,
    featuresCol='features')

pipeline = Pipeline(stages=[
        encoder,
        featuresCreator,
        kmeans]
)

model = pipeline.fit(births_train)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Create an estimator
In this example we will (once again) us the Logistic Regression model.
Can you write Python code for it?


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Create transformers
Can you write Python code for it?

import pyspark.ml.feature as ft

births = births \
    .withColumn(       'BIRTH_PLACE_INT',
                births['BIRTH_PLACE'] \
                    .cast(typ.IntegerType()))


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Finally, again, we put it all together into a Pipeline.
Can you write Python code for it?

pipeline = Pipeline(stages=[
        featuresCreator,
        selector,
        regressor])

weightGain = pipeline.fit(births_train)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
First, we will once again use the RegexTokenizer and the StopWordsRemover models.
Can you write Python code for it?

tokenizer = ft.RegexTokenizer(
    inputCol='documents',
    outputCol='input_arr',
    pattern='\s+|[,.\"]')

stopwords = ft.StopWordsRemover(
    inputCol=tokenizer.getOutputCol(),
    outputCol='input_stop')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Fit the model
Conventiently, DataFrame API has the .randomSplit(...) method.
Can you write Python code for it?

births_train, births_test = births \
    .randomSplit([0.7, 0.3], seed=666)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Having created the weightGain model, let's see if it performs well on our testing data.
Can you write Python code for it?

evaluator = ev.RegressionEvaluator(
    predictionCol="prediction",
    labelCol='MOTHER_WEIGHT_GAIN')

print(evaluator.evaluate(
     weightGain.transform(births_test),
    {evaluator.metricName: 'r2'}))


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Having done this, we are ready to find the optimal combination of parameters for our model.
Can you write Python code for it?


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Having estimated the model, let's see if we can find some differences between clusters.
Can you write Python code for it?

test = model.transform(births_test)

test \
    .groupBy('prediction') \
    .agg({
        '*': 'count',
        'MOTHER_HEIGHT_IN': 'avg'
    }).collect()


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Here's what the test_model looks like.
Can you write Python code for it?


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
In order to predict the weight gain we will use the gradient boosted trees regressor.
Can you write Python code for it?

import pyspark.ml.regression as reg

regressor = reg.GBTRegressor(
    maxIter=15,
    maxDepth=3,
    labelCol='MOTHER_WEIGHT_GAIN')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Let's now see how the RandomForestClassifier model performs compared to the LogisticRegression.
Can you write Python code for it?

evaluator = ev.BinaryClassificationEvaluator(
    labelCol='INFANT_ALIVE_AT_REPORT')
print(evaluator.evaluate(test,
    {evaluator.metricName: "areaUnderROC"}))
print(evaluator.evaluate(test,
    {evaluator.metricName: "areaUnderPR"}))


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Let's see what we got.
Can you write Python code for it?

data_discretized = discretizer.fit(data).transform(data)

data_discretized \
    .groupby('discretized')\
    .mean('continuous_var')\
    .sort('discretized')\
    .collect()


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Model performance
Obviously, we would like to now test how well our model did.
Can you write Python code for it?

import pyspark.ml.evaluation as ev

evaluator = ev.BinaryClassificationEvaluator(
    rawPredictionCol='probability',
    labelCol='INFANT_ALIVE_AT_REPORT')

print(evaluator.evaluate(test_model,
     {evaluator.metricName: 'areaUnderROC'}))
print(evaluator.evaluate(test_model, {evaluator.metricName: 'areaUnderPR'}))


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Next let's specify our model and the list of parameters we want to loop through.
Can you write Python code for it?

logistic = cl.LogisticRegression(
    labelCol='INFANT_ALIVE_AT_REPORT')

grid = tune.ParamGridBuilder() \
    .addGrid(logistic.maxIter,
             [2, 10, 50]) \
    .addGrid(logistic.regParam,
             [0.01, 0.05, 0.3]) \
    .build()


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Now run our pipeline and estimate our model.
Can you write Python code for it?

model = pipeline.fit(births_train)
test_model = model.transform(births_test)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Now that we have the pipeline we follow in the very similar fashion as before.
Can you write Python code for it?

data_ngram = pipeline \
    .fit(text_data) \
    .transform(text_data)

data_ngram.select('nGrams').take(1)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Once loaded, let's create the model.
Can you write Python code for it?

logistic = cl.LogisticRegression(
    maxIter=10,
    regParam=0.01,
    labelCol='INFANT_ALIVE_AT_REPORT')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Regression
In this section we will try to predict the MOTHER_WEIGHT_GAIN.
Can you write Python code for it?

features = ['MOTHER_AGE_YEARS','MOTHER_HEIGHT_IN',
            'MOTHER_PRE_WEIGHT','DIABETES_PRE',
            'DIABETES_GEST','HYP_TENS_PRE',
            'HYP_TENS_GEST', 'PREV_BIRTH_PRETERM',
            'CIG_BEFORE','CIG_1_TRI', 'CIG_2_TRI',
            'CIG_3_TRI'
           ]


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Saving the model
PySpark allows you to save the Pipeline definition for later use.
Can you write Python code for it?

pipelinePath = './infant_oneHotEncoder_Logistic_Pipeline'
pipeline.write().overwrite().save(pipelinePath)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
So, you can load it up later and use straight away to .fit(...) and predict.
Can you write Python code for it?

loadedPipeline = Pipeline.load(pipelinePath)
loadedPipeline \
    .fit(births_train)\
    .transform(births_test)\
    .take(1)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Standardizing continuous variables
Create a vector representation of our continuous variable (as it is only a single float)
Can you write Python code for it?

vectorizer = ft.VectorAssembler(
    inputCols=['continuous_var'],
    outputCol= 'continuous_vec')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
That's it. We got our n-grams and we can then use them in further NLP processing.
Discretize continuous variables
It is sometimes useful to band the values into discrete buckets.
Can you write Python code for it?

import numpy as np

x = np.arange(0, 100)
x = x / 100.0 * np.pi * 4
y = x * np.sin(x / 1.764) + 20.1234

schema = typ.StructType([
    typ.StructField('continuous_var',
                    typ.DoubleType(),
                    False
   )
])

data = spark.createDataFrame([[float(e), ] for e in y], schema=schema)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
The TrainValidationSplit object gets created in the same fashion as the CrossValidator model.
Can you write Python code for it?

tvs = tune.TrainValidationSplit(
    estimator=logistic,
    estimatorParamMaps=grid,
    evaluator=evaluator
)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
The cvModel will return the best model estimated. We can now use it to see if it performed better than our previous model.
Can you write Python code for it?

data_train = data_transformer \
    .transform(births_test)
results = cvModel.transform(data_train)

print(evaluator.evaluate(results,
     {evaluator.metricName: 'areaUnderROC'}))
print(evaluator.evaluate(results,
     {evaluator.metricName: 'areaUnderPR'}))


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
The output of the tokenizer looks similar to this.
Can you write Python code for it?

tok = tokenizer \
    .transform(text_data) \
    .select('input_arr')

tok.take(1)


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Use the QuantileDiscretizer model to split our continuous variable into 5 buckets (see the numBuckets parameter).
Can you write Python code for it?

discretizer = ft.QuantileDiscretizer(
    numBuckets=5,
    inputCol='continuous_var',
    outputCol='discretized')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
Use the StopWordsRemover(...).
Can you write Python code for it?

stopwords = ft.StopWordsRemover(
    inputCol=tokenizer.getOutputCol(),
    outputCol='input_stop')


I am working on the file "Chapter06/LearningPySpark_Chapter06.ipynb".
The first task is:
What parameters has the best model? The answer is a little bit convoluted but here's how you can extract it.
Can you write Python code for it?

results = [
    (
        [
            {key.name: paramValue}
            for key, paramValue
            in zip(
                params.keys(),
                params.values())
        ], metric
    )
    for params, metric
    in zip(
        cvModel.getEstimatorParamMaps(),
        cvModel.avgMetrics
    )
]

sorted(results,
       key=lambda el: el[1],
       reverse=True)[0]


I am working on the file "Chapter08.ipynb".
The first task is:
8.2 BoWモデルの紹介

BoW(Bag-of-Words)

ドキュメントの集合全体から、たとえば単語という一意なトークン(token)からなる語彙(vocabulary)を作成する

各ドキュメントでの各単語の出現回数を含んだ特徴ベクトルを構築する

疎ベクトル(sparse vector)

8.2.1 単語を特徴ベクトルに変換する

生の出現頻度(raw term frequencies): tf(t, d)
Can you write Python code for it?

import numpy as np
from sklearn.feature_extraction.text import CountVectorizer

count = CountVectorizer()
docs = np.array([
        'The sun is shining',
        'The weather is sweet',
        'The sun is shining, the weather is sweet, and one and one is two'])
bag = count.fit_transform(docs)

# 語彙の中身を出力
print(count.vocabulary_)
# 特徴ベクトルを出力
print(bag.toarray())


I am working on the file "Chapter08.ipynb".
The first task is:
8.2.4 ドキュメントをトークン化する

トークン化(tokenize)
Porterステミング(Porter stemming)アルゴリズム: 単語を原形に変換する
ストップワードの除去(stop-word removal)
Can you write Python code for it?

def tokenizer(text):
    return text.split()

tokenizer('runners like running and thus they run')

# Porterステミングを行う
from nltk.stem.porter import PorterStemmer
porter = PorterStemmer()

def tokenizer_porter(text):
    return [porter.stem(word) for word in text.split()]

tokenizer_porter('runners like running and thus they run')

# ストップワードをダウンロードする
import nltk
nltk.download('stopwords')

from nltk.corpus import stopwords
stop = stopwords.words('english')
[w for w in tokenizer_porter('a runner likes running and runs a lot')[-10:] if w not in stop]


I am working on the file "Chapter08.ipynb".
The first task is:
8.2.5 ドキュメントを分類するロジスティック回帰モデルのトレーニング
Can you write Python code for it?

X_train = df.loc[:25000, 'review'].values
y_train = df.loc[:25000, 'sentiment'].values
X_test = df.loc[25000:, 'review'].values
y_test = df.loc[25000:, 'sentiment'].values

from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
if Version(sklearn_version) < '0.18':
    from sklearn.grid_search import GridSearchCV
else:
    from sklearn.model_selection import GridSearchCV

tfidf = TfidfVectorizer(strip_accents=None,
                        lowercase=False,
                        preprocessor=None)
param_grid = [{'vect__ngram_range': [(1, 1)],
               'vect__stop_words': [stop, None],
               'vect__tokenizer': [tokenizer, tokenizer_porter],
               'clf__penalty': ['l1', 'l2'],
               'clf__C': [1.0, 10.0, 100.0]},
              {'vect__ngram_range': [(1, 1)],
               'vect__stop_words': [stop, None],
               'vect__tokenizer': [tokenizer, tokenizer_porter],
               'vect__use_idf':[False],
               'vect__norm':[None],
               'clf__penalty': ['l1', 'l2'],
               'clf__C': [1.0, 10.0, 100.0]},
             ]
lr_tfidf = Pipeline([('vect', tfidf),
                     ('clf', LogisticRegression(random_state=0))])
gs_lr_tfidf = GridSearchCV(lr_tfidf, param_grid,
                           scoring='accuracy',
                           cv=5,
                           verbose=1,
                           n_jobs=-1)
gs_lr_tfidf.fit(X_train, y_train)

print('Best parameter set: %s ' % gs_lr_tfidf.best_params_)
print('CV Accuracy: %.3f' % gs_lr_tfidf.best_score_)

clf = gs_lr_tfidf.best_estimator_
print('Test Accuracy: %.3f' % clf.score(X_test, y_test))


I am working on the file "Chapter08.ipynb".
The first task is:
8.3 さらに大規模なデータの処理: オンラインアルゴリズムとアウトオブコア学習

アウトオブコア学習(out-of-core learning)
Can you write Python code for it?

import numpy as np
import re
from nltk.corpus import stopwords
stop = stopwords.words('english')

def tokenizer(text):
    text = re.sub('<[^>]*>', '', text)
    emoticons = re.findall('(?::|;|=)(?:-)?(?:\)|\(|D|P)', text.lower())
    text = re.sub('[\W]+', ' ', text.lower()) + ' '.join(emoticons).replace('-', '')
    tokenized = [w for w in text.split() if w not in stop]
    return tokenized

def stream_docs(path):
    with open(path, 'r', encoding='utf-8') as csv:
        next(csv)
        for line in csv:
            text, label = line[:-3], int(line[-2])
            yield text, label

next(stream_docs(path='./movie_data.csv'))

def get_minibatch(doc_stream, size):
    docs, y = [], []
    try:
        for _ in range(size):
            text, label = next(doc_stream)
            docs.append(text)
            y.append(label)
    except StopIteration:
        return None, None
    return docs, y

from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.linear_model import SGDClassifier

vect = HashingVectorizer(decode_error='ignore',
                         n_features=2**21,
                         preprocessor=None,
                         tokenizer=tokenizer)
clf = SGDClassifier(loss='log', random_state=1, n_iter=1)
doc_stream = stream_docs(path='./movie_data.csv')

import pyprind
pbar = pyprind.ProgBar(45)
classes = np.array([0, 1])
for _ in range(45):
    X_train, y_train = get_minibatch(doc_stream, size=1000)
    if not X_train:
        break
    X_train = vect.transform(X_train)
    clf.partial_fit(X_train, y_train, classes=classes)
    pbar.update()

X_test, y_test = get_minibatch(doc_stream, size=5000)
X_test = vect.transform(X_test)
print('Accuracy: %.3f' % clf.score(X_test, y_test))

clf = clf.partial_fit(X_test, y_test)


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Abstract data
Working with NumPy array
Let's create a simple NumPy array: we first load NumPy and then create a matrix with two rows and three columns.
Can you write Python code for it?

import numpy as np

simpleArray = np.array([
        [1,2,3],
        [4,5,6]
    ])


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Blaze can also read directly from a GZipped archives.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
If you want to retrieve columns you have to transpose your DataShape.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
If you want to retrieve the first column you can use indexing.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
If you wanted to log-transform the Stop_year you need to
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
In order to peek inside the structure you can either use .peek() method
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
In order to save traffic data for year 2013 you can call odo like this
Can you write Python code for it?

odo.odo(traffic[traffic.Stop_year == 2013],
    '../Data/Years/TrafficViolations_2013.csv.gz')


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Interacting with MongoDB database
Reading from MongoDB is very similar to reading from a PostGRE SQL or SQLite databases.
Can you write Python code for it?

traffic_mongo = bl.Data('mongodb://localhost:27017/packt::traffic')
traffic_mongo.head(2)


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Joins
We first select all the traffic violations by violation type (the violation object) and the traffic violations involving belts (the belts object).
Can you write Python code for it?

violation = traffic[
    ['Stop_month','Stop_day','Stop_year',
     'Stop_hr','Stop_min','Stop_sec','Violation_Type']]

belts = traffic[
    ['Stop_month','Stop_day','Stop_year',
     'Stop_hr','Stop_min','Stop_sec','Belts']]


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Next, we create a DataFrame.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Now that we have an array we can abstract it with Blaze's Data structure.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Now, we join the two objects on the six date and time columns.
Can you write Python code for it?

violation_belts = bl.join(violation, belts,
      ['Stop_month','Stop_day','Stop_year',
       'Stop_hr','Stop_min','Stop_sec'])


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Once we have the full dataset in place, let's check how many traffic violations involved belts and what sort of punishment was issued to the driver.
Can you write Python code for it?

bl.by(violation_belts[['Violation_Type', 'Belts']],
      Violation_count=violation_belts.Belts.count()
).sort('Violation_count', ascending=False)


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Reading data from the SQLite database should be trivial by now.
Can you write Python code for it?

traffic_sqlt = bl.Data('sqlite:///traffic_local.sqlite::traffic2016')
traffic_sqlt.head(2)


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Reducing data
Some reduction methods are also available, like .mean() (that calculates the average), .std (that calculates standard deviation) or .max() (that returns the maximum from the list).
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
Since we already have an existing dataset traffic, we can reuse the schema by calling traffic.dshape and specify our transformations directly on it.
Can you write Python code for it?

traffic_s = bl.symbol('traffic', traffic.dshape)
traffic_2013 = traffic_s[traffic_s['Stop_year'] == 2013][
    ['Stop_year', 'Arrest_Type','Color', 'Charge']
]


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
To compare we get exactly the same data let's retrieve the first two records from each structure.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
To present how this works, let's read our dataset into pandas' DataFrame.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
We can subtract 2000 from the Stop_year column as we do not need a greater detail.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
You can also pass a list of lists or a list of NumPy arrays.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
You can calculate the age of the car (in years) at the time when the violation occured
Can you write Python code for it?

traffic = bl.transform(traffic,
             Age_of_car = traffic.Stop_year - traffic.Year)

traffic.head(2)[['Stop_year', 'Year', 'Age_of_car']]


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
You can now pass the dataset directly to traffic_2013 object and perform the computation using the .compute(...) method of Blaze.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
You can now retrieve the data simply by calling the column by it's name; let's retrieve column 'b'
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
You can retrieve data in the same manner as with the DataShape created from the NumPy array.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
and transform it into a DataShape.
Can you write Python code for it?


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
or indexing; this allows to select more than one column at a time
Can you write Python code for it?

(traffic[['Location', 'Year', 'Accident', 'Fatal', 'Alcohol']]
    .head(2))


I am working on the file "Chapter09/LearningPySpark_Chapter09.ipynb".
The first task is:
or use (familiar to those of you versed in pandas' syntax) the .head(...) method
Can you write Python code for it?


I am working on the file "Chapter1/Chapter1.ipynb".
The first task is:
1.4.2 Code Management with Git
See the links from the 2014 Physics 231 website.
1.5.4 Fetching and Displaying SDSS Spectra
Reproduce Figure 1.2 showing a sample SDSS spectrum:
Can you write Python code for it?

"""
SDSS Spectrum Example
---------------------
Figure 1.2.

An example of an SDSS spectrum (the specific flux plotted as a function of
wavelength) loaded from the SDSS SQL server in real time using Python tools
provided here (this spectrum is uniquely described by SDSS parameters
plate=1615, fiber=513, and mjd=53166).
"""
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
from matplotlib import pyplot as plt
from astroML.datasets import fetch_sdss_spectrum

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Fetch single spectrum
plate = 1615
mjd = 53166
fiber = 513

spec = fetch_sdss_spectrum(plate, mjd, fiber)

#------------------------------------------------------------
# Plot the resulting spectrum
fig, ax = plt.subplots(figsize=(5, 3.75))
ax.plot(spec.wavelength(), spec.spectrum, '-k', lw=1)

ax.set_xlim(3000, 10000)
ax.set_ylim(25, 300)

ax.set_xlabel(r'$\lambda {(\rm \AA)}$')
ax.set_ylabel('Flux')
ax.set_title('Plate = %(plate)i, MJD = %(mjd)i, Fiber = %(fiber)i' % locals())

plt.show()


I am working on the file "Chapter1/Chapter1.ipynb".
The first task is:
1.6.1 Plotting Two-Dimensional Representations of Large Data Sets
Reproduce Figure 1.9 showing g-r vs r-i for SDSS stripe-82 standard stars as a scatter plot with contours overlayed:
Can you write Python code for it?

"""
SDSS Stripe 82 Standard Stars
-----------------------------
Figure 1.9.

Scatter plot with contours over dense regions.This is a color-color diagram
of the entire set of SDSS Stripe 82 standard stars; cf. figure 1.6.
"""
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
from matplotlib import pyplot as plt

from astroML.plotting import scatter_contour
from astroML.datasets import fetch_sdss_S82standards

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Fetch the Stripe 82 standard star catalog

data = fetch_sdss_S82standards()

g = data['mmu_g']
r = data['mmu_r']
i = data['mmu_i']

#------------------------------------------------------------
# plot the results
fig, ax = plt.subplots(figsize=(5, 3.75))
scatter_contour(g - r, r - i, threshold=200, log_counts=True, ax=ax,
                histogram2d_args=dict(bins=40),
                plot_args=dict(marker=',', linestyle='none', color='black'),
                contour_args=dict(cmap=plt.cm.bone))

ax.set_xlabel(r'${\rm g - r}$')
ax.set_ylabel(r'${\rm r - i}$')

ax.set_xlim(-0.6, 2.5)
ax.set_ylim(-0.6, 2.5)

plt.show()


I am working on the file "Chapter1/Chapter1.ipynb".
The first task is:
1.6.3 Plotting Representations of Data on the Sky
Reproduce Figure 1.15 showing the WMAP7 raw temperature map using healpix with nside=512 (~3.1Mpix):
Can you write Python code for it?

"""
Example of HealPix pixellization
--------------------------------
Figure 1.15.

The top panel shows HEALPix pixels in nested order. The 12 fundamental sky
divisions can be seen, as well as the hierarchical nature of the smaller
pixels. This shows a pixelization with nside = 4, that is, each of the 12
large regions has 4 x 4 pixels, for a total of 192 pixels. The lower panel
shows a seven-year co-add of raw WMAP data, plotted using the HEALPix
projection using the HealPy package. This particular realization has
nside = 512, for a total of 3,145,728 pixels. The pixels are roughly
6.8 arcminutes on a side.
"""
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general

from __future__ import print_function

import numpy as np
from matplotlib import pyplot as plt
# warning: due to a bug in healpy, importing it before pylab can cause
#  a segmentation fault in some circumstances.
import healpy as hp

from astroML.datasets import fetch_wmap_temperatures

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=True)

#------------------------------------------------------------
# Next plot the wmap pixellization
wmap_unmasked = fetch_wmap_temperatures(masked=False)

# plot the unmasked map
fig = plt.figure(2, figsize=(10, 7.5))
hp.mollview(wmap_unmasked, min=-1, max=1, title='Raw WMAP data',
            unit=r'$\Delta$T (mK)', fig=2)

plt.show()