I am working on the file "NeuralNetworks.ipynb".
The first task is:
Well that fell apart quickly! We're at roughly the point where neural networks faded from popularity in the 90s. Perceptrons generated intense interest because they were biologically inspired and could be applied generically to any supervised learning problem. However they weren't extensible to more realistic problems, and for supervised learning there were techniques such as support vector machines that provided better performance and avoided the explosion in training time seen for large perceptrons.
Recent interest in neural networks surged in 2012 when a team using a deep convolutional neural network aceived record results classifying objects in the ImageNet data set. Some examples of the types of classification performed on the dataset are shown below.
This is clearly much more sophisticated than our basic perceptron. "Deep" networks consist of tens of layers with thousands of neurons. These large networks have become usabel thanks to two breakthroughs: the use of sparse layers and the power of graphics processing units (GPUs).
Many image processing tasks involve convolving an image with a 2-dimensional kernel as shown below.

The sparse layers or convolutional layers in a deep network contain a large number of hidden nodes but very few synapses. The sparseness arises from the relatively small size of a typical convolution kernel (15x15 is a large kernel), so a hidden node representing one output of the convolution is connected to only a few input nodes. Compare this the our previous perceptron, in which every hidden node was connected to every input node.
Even though the total number of connections is greatly reduced in the sparse layers, the total number of nodes and connections in a modern deep network is still enormous. Luckily, training these networks turns out to be a great task for GPU acceleration! Serious work using neural networks is almost always done usign specialized GPU-accelerated platforms.
The Keras framework provides a Python environment for CNN development. Keras uses the TensorFlow...
Can you write Python code for it?

from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils

#Create a model
model = Sequential()

#Use two sparse layers to learn useful, translation-invariant features
model.add(Convolution2D(32,7,7,border_mode='valid',input_shape=(32,32,1)))
model.add(Activation('relu'))
model.add(Convolution2D(32,5,5))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.25))
model.add(Flatten())

#Add dense layers to do the actual classification
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop',metrics=['accuracy'])
model.summary()

#Keras has some particular requirements for data formats...
dataX = images_ex.reshape(images_ex.shape[0],images_ex.shape[1],images_ex.shape[2],1)
dataY = np_utils.to_categorical(digits.target)

#Train the model. We get a summary of performance after each training epoch
model.fit(dataX, dataY, validation_split=0.1, batch_size=128, nb_epoch=10)


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Check out your predictions
Here, use the test data to view how well your network is modeling the data. If something is completely wrong here, make sure each step in your network is implemented correctly.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(8,4))

mean, std = scaled_features['cnt']
predictions = network.run(test_features).T*std + mean
ax.plot(predictions[0], label='Prediction')
ax.plot((test_targets['cnt']*std + mean).values, label='Data')
ax.set_xlim(right=len(predictions))
ax.legend()

dates = pd.to_datetime(rides.ix[test_data.index]['dteday'])
dates = dates.apply(lambda d: d.strftime('%b %d'))
ax.set_xticks(np.arange(len(dates))[12::24])
_ = ax.set_xticklabels(dates[12::24], rotation=45)


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Checking out the data
This dataset has the number of riders for each hour of each day from January 1 2011 to December 31 2012. The number of riders is split between casual and registered, summed up in the cnt column. You can see the first few rows of the data above.
Below is a plot showing the number of bike riders over the first 10 days or so in the data set. (Some days don't have exactly 24 entries in the data set, so it's not exactly 10 days.) You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.
Can you write Python code for it?


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Load and prepare the data
A critical step in working with neural networks is preparing the data correctly. Variables on different scales make it difficult for the network to efficiently learn the correct weights. Below, we've written the code to load and prepare the data. You'll learn more about this soon!
Can you write Python code for it?

data_path = 'Bike-Sharing-Dataset/hour.csv'

rides = pd.read_csv(data_path)

rides.head()


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Scaling target variables
To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.
The scaling factors are saved so we can go backwards when we use the network for predictions.
Can you write Python code for it?

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Splitting the data into training, testing, and validation sets
We'll save the data for the last approximately 21 days to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.
Can you write Python code for it?

# Save data for approximately the last 21 days
test_data = data[-21*24:]

# Now remove the test data from the data set
data = data[:-21*24]

# Separate the data into features and targets
target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Training the network
Here you'll set the hyperparameters for the network. The strategy here is to find hyperparameters such that the error on the training set is low, but you're not overfitting to the data. If you train the network too long or have too many hidden nodes, it can become overly specific to the training set and will fail to generalize to the validation set. That is, the loss on the validation set will start increasing as the training set loss drops.
You'll also be using a method know as Stochastic Gradient Descent (SGD) to train the network. The idea is that for each training pass, you grab a random sample of the data instead of using the whole data set. You use many more training passes than with normal gradient descent, but each pass is much faster. This ends up training the network more efficiently. You'll learn more about SGD later.
Choose the number of iterations
This is the number of batches of samples from the training data we'll use to train the network. The more iterations you use, the better the model will fit the data. However, if you use too many iterations, then the model with not generalize well to other data, this is called overfitting. You want to find a number here where the network has a low training loss, and the validation loss is at a minimum. As you start overfitting, you'll see the training loss continue to decrease while the validation loss starts to increase.
Choose the learning rate
This scales the size of weight updates. If this is too big, the weights tend to explode and the network fails to fit the data. A good choice to start at is 0.1. If the network has problems fitting the data, try reducing the learning rate. Note that the lower the learning rate, the smaller the steps are in the weight updates and the longer it takes for the neural network to converge.
Choose the number of hidden nodes
The more hidden nodes you have, the more accurate predictions the model will make. Try a few different numbers and see how it affects the performance. You can look at the losses...
Can you write Python code for it?

train_features.shape[1]

import sys

### Set the hyperparameters here ###
iterations = 5000
learning_rate = 0.7
hidden_nodes = 8
output_nodes = 1

N_i = train_features.shape[1]
network = NeuralNetwork(N_i, hidden_nodes, output_nodes, learning_rate)

losses = {'train':[], 'validation':[]}
for ii in range(iterations):
    # Go through a random batch of 128 records from the training data set
    batch = np.random.choice(train_features.index, size=128)
    X, y = train_features.ix[batch].values, train_targets.ix[batch]['cnt']

    network.train(X, y)

    # Printing out the training progress
    train_loss = MSE(network.run(train_features).T, train_targets['cnt'].values)
    val_loss = MSE(network.run(val_features).T, val_targets['cnt'].values)
    sys.stdout.write("\rProgress: {:2.1f}".format(100 * ii/float(iterations)) \
                     + "% ... Training loss: " + str(train_loss)[:5] \
                     + " ... Validation loss: " + str(val_loss)[:5])
    sys.stdout.flush()

    losses['train'].append(train_loss)
    losses['validation'].append(val_loss)

print("Train loss:"); print(train_loss)
print("Val loss:"); print(val_loss)

plt.plot(losses['train'], label='Training loss')
plt.plot(losses['validation'], label='Validation loss')
plt.legend()
_ = plt.ylim()


I am working on the file "Neural_network_by_hand_GDP.ipynb".
The first task is:
Unit tests
Run these unit tests to check the correctness of your network implementation. This will help you be sure your network was implemented correctly before you starting trying to train it. These tests must all be successful to pass the project.
Can you write Python code for it?

import unittest

inputs = np.array([[0.5, -0.2, 0.1]])
targets = np.array([[0.4]])
test_w_i_h = np.array([[0.1, -0.2],
                       [0.4, 0.5],
                       [-0.3, 0.2]])
test_w_h_o = np.array([[0.3],
                       [-0.1]])

class TestMethods(unittest.TestCase):

    ##########
    # Unit tests for data loading
    ##########

    def test_data_path(self):
        # Test that file path to dataset has been unaltered
        self.assertTrue(data_path.lower() == 'bike-sharing-dataset/hour.csv')

    def test_data_loaded(self):
        # Test that data frame loaded
        self.assertTrue(isinstance(rides, pd.DataFrame))

    ##########
    # Unit tests for network functionality
    ##########

    def test_activation(self):
        network = NeuralNetwork(3, 2, 1, 0.5)
        # Test that the activation function is a sigmoid
        self.assertTrue(np.all(network.activation_function(0.5) == 1/(1+np.exp(-0.5))))

    def test_train(self):
        # Test that weights are updated correctly on training
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

        network.train(inputs, targets)
        self.assertTrue(np.allclose(network.weights_hidden_to_output,
                                    np.array([[ 0.37275328],
                                              [-0.03172939]])))
        self.assertTrue(np.allclose(network.weights_input_to_hidden,
                                    np.array([[ 0.10562014, -0.20185996],
                                              [0.39775194, 0.50074398],
                                              [-0.29887597, 0.19962801]])))

    def test_run(self):
        # Test correctness of run method
        network = NeuralNetwork(3, 2, 1, 0.5)
        network.weights_input_to_hidden = test_w_i_h.copy()
        network.weights_hidden_to_output = test_w_h_o.copy()

       ...


I am working on the file "Neurônio Sigmoid.ipynb".
The first task is:
Exercícios
Can you write Python code for it?

x, y = df[['idade', 'salario']].values, df.comprou.values.reshape(-1,1)

print(x.shape, y.shape)

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter3D(x[:,0], x[:,1], y, c=y.ravel())

minmax = MinMaxScaler(feature_range=(-1,1))
x = minmax.fit_transform(x.astype(np.float64))

print(x.min(), x.max())

clf_sk = LogisticRegression(C=1e15)
clf_sk.fit(x, y.ravel())

print(clf_sk.coef_, clf_sk.intercept_)
print(clf_sk.score(x, y))


I am working on the file "Neurônio Sigmoid.ipynb".
The first task is:
Introdução
A Regressão Logística, apesar do nome, é uma técnica utilizada para fazer classificação binária. Nesse caso, ao invés de prever um valor contínuo, a nossa saída é composta de apenas dois valores: 0 ou 1, em geral. Para fazer a regressão logística, utilizamos como função de ativação a função conhecida como sigmoid. Tal função, é descrita pela seguinte fórmula:
$$\widehat{y} = \frac{1}{1+e^{-z}} = \frac{e^z}{1+e^z}$$
No caso de redes neurais, em geral consideramos $z(w,b) = xw^T + b$.
Função de Custo
A função de custo da regressão logística é chamada de entropia cruzada (do inglês, cross-entropy) e é definida pela seguinte fórmula:
$$J(z) = -\frac{1}{N}\sum_{i}^N y_i\log(\widehat{y}_i) + (1-y_i)\log(1-\widehat{y}_i)$$
Onde $N$ é quantidade de amostras e $y_i$ representa o valor da $i$-ésima amostra (0 ou 1). Lembrando que $\widehat{y}_i$ é agora calculada agora utilizando a função sigmoid, como mostrado na seção anterior.
Repare também que:

quando $y_i = 0$, o primeiro termo anula-se (pois $y_i = 0$). Logo, vamos considerar os dois casos extremos para $\widehat{y}_i = 0$ no segundo termo da equação ($(1-y_i)\log(1-\widehat{y}_i)$):
quando $\widehat{y}_i = 0$, temos que o $\log(1-\widehat{y}_i) = \log(1) = 0$. Logo, o nosso custo $J = 0$. Repare que isso faz todo sentido, pois $y_i = 0$ e $\widehat{y}_i = 0$.
quando $\widehat{y}_i = 1$, temos que o $\log(1-\widehat{y}_i) = \log(0) = \infty$. Agora, o nosso custo $J = \infty$. Ou seja, quanto mais diferente são $y_i$ e $\widehat{y}_i$, maior o nosso custo.

quando $y_i = 1$, o segundo termo anula-se (pois $(1-y_i) = 0$). Novamente, vamos considerar os dois casos extremos para $\widehat{y}_i = 0$, só que agora no primeiro termo da equação ($y_i\log(\widehat{y}_i)$):
quando $\widehat{y}_i = 0$, temos que o $\log(\widehat{y}_i) = \infty$. Logo, o nosso custo $J = \infty$. Novamente, como $y_i$ e $\widehat{y}_i$ são bem diferentes, o custo tende a aumentar.
quando $\widehat{y}_i = 1$, temos que $\log(\widehat{y}_i) = \log(1) = 0$. Agora, o nosso custo...
Can you write Python code for it?

df = pd.read_csv('data/anuncios.csv')
print(df.shape)
df.head()

x, y = df.idade.values.reshape(-1,1), df.comprou.values.reshape(-1,1)

print(x.shape, y.shape)

plt.scatter(x, y, c=y, cmap='bwr')
plt.xlabel('idade')
plt.ylabel('comprou?')

minmax = MinMaxScaler(feature_range=(-1,1))
x = minmax.fit_transform(x.astype(np.float64))

print(x.min(), x.max())


I am working on the file "Neurônio Sigmoid.ipynb".
The first task is:
Numpy
Can you write Python code for it?

D = x.shape[1]
w = 2*np.random.random((1, D))-1 # [1x2]
b = 2*np.random.random()-1       # [1x1]

learning_rate = 1.0 # <- tente estimar a learning rate

for step in range(1): # <- tente estimar a #epochs
    # calcule a saida do neuronio sigmoid
    z =
    y_pred =

    error = y - y_pred     # [400x1]

    w = w + learning_rate*np.dot(error.T, x)
    b = b + learning_rate*error.sum()

    if step%100 == 0:
        # implemente a entropia cruzada (1 linhas)
        cost =
        print('step {0}: {1}'.format(step, cost))

print('w: ', w)
print('b: ', b)

x1 = np.linspace(x[:, 0].min(), x[:, 0].max())
x2 = np.linspace(x[:, 1].min(), x[:, 1].max())
x1_mesh, x2_mesh = np.meshgrid(x1, x2)
x1_mesh = x1_mesh.reshape(-1, 1)
x2_mesh = x2_mesh.reshape(-1, 1)

x_mesh = np.hstack((x1_mesh, x2_mesh))
y_pred = sigmoid(np.dot(x_mesh, w.T) + b)

fig = plt.figure(figsize=(8, 6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter3D(x[:,0], x[:,1], y, c=y.ravel())
ax.plot_trisurf(x1_mesh.ravel(), x2_mesh.ravel(), y_pred.ravel(), alpha=0.3, shade=False)

print('Acurácia pelo Scikit-learn: {:.2f}%'.format(clf_sk.score(x, y)*100))

y_pred = np.round(sigmoid(np.dot(x, w.T) + b))
print('Acurária pela nossa implementação: {:.2f}%'.format(accuracy_score(y, y_pred)*100))


I am working on the file "Neurônio Sigmoid.ipynb".
The first task is:
Vamos utilizar o sklearn como gabarito para nossa implementação. Entretanto, como a Regressão Logística do sklearn faz uma regularização L2 automaticamente, temos de definir $C=10^{15}$ para "anular" a regularização. O parâmetro $C$ define a inversa da força da regularização (ver documentação). Logo, quanto menor for o $C$, maior será a regularização e menores serão os valores dos pesos e bias.
Can you write Python code for it?

clf_sk = LogisticRegression(C=1e15)
clf_sk.fit(x, y.ravel())

print(clf_sk.coef_, clf_sk.intercept_)
print(clf_sk.score(x, y))

x_test = np.linspace(x.min(), x.max(), 100).reshape(-1,1)
y_sk = clf_sk.predict_proba(x_test)

plt.scatter(x, y, c=y, cmap='bwr')
plt.plot(x_test, y_sk[:,1], color='black')
plt.xlabel('idade')
plt.ylabel('comprou?')

# implemente a função sigmoid aqui


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
Both pandas and Numpy use the concept of dtypes as data types, and if no arguments are specified, date_time will take on an object dtype.
Can you write Python code for it?

nrg.dtypes

# https://docs.python.org/3/library/functions.html#type
# https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iat.html
type(nrg.iat[0,0])


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
However, our hourly costs depend on the time of day.
If you use a loop to do the conditional calculation, you are not using pandas the way it was intended.
For the rest of this tutorial, you'll start with a sub-optimal solution and work your way up to a Pythonic approach that leverages the full power of pandas.
Take a look at a loop approach and see how it performs using our timing decorator.
Can you write Python code for it?

# Create a function to apply the appropriate rate to the given hour:
def apply_rate(kwh, hour):
    """
    Calculates the cost of electricity for a given hour.
    """
    if 0 <= hour < 7:
        rate = 12
    elif 7 <= hour <= 17:
        rate = 20
    elif 17 <= hour <= 24:
        rate = 28
    else:
        # +1 for error handling:
        raise ValueError(f'Invalid datetime entry: {hour}')
    return rate * kwh


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
Now for a computationally expensive and non-Pythonic loop:
Can you write Python code for it?

# Not the best way:
@timeit(repeat=2, number = 10)
def apply_rate_loop(nrg):
    """
    Calculate the costs using a loop, and modify `nrg` dataframe in place.
    """
    energy_cost_list = []
    for i in range(len(nrg)):
        # Get electricity used and the corresponding rate.
        energy_used = nrg.iloc[i]['energy_kwh']
        hour = nrg.iloc[i]['date_time'].hour
        energy_cost = apply_rate(energy_used, hour)
        energy_cost_list.append(energy_cost)
    nrg['cost_cents'] = energy_cost_list

apply_rate_loop(nrg)


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
One easily overlooked detail is that the datetimes in the energy_consumption.csv file are not in ISO 8601 format.
You need YYYY-MM-DD HH:MM.
If you don’t specify a format, Pandas will use the dateutil package to convert each string to a date.
Conversely, if the raw datetime data is already in ISO 8601 format, pandas can immediately take a fast route to parsing the dates.
This is one reason why being explicit about the format is so beneficial here.
Another option is to pass the infer_datetime_format=True parameter, but it generally pays to be explicit.
Also, remember that pandas' read_csv() method allows you to parse dates as part of the file I/O using the parse_dates, infer_datetime_format, and date_parser parameters.
Simple Looping Over Pandas Data
Now that dates and times are in a tidy format, we can begin calculating electricity costs.
Cost varies by hour, so a cost factor is conditionally applied for each hour of the day:
|    Usage    | Cents per kWh  |   Time Range   |
|-------------|----------------|----------------|
| Peak        | 28             | 17:00 to 24:00 |
| Shoulder    | 20             | 7:00 to 17:00  |
| Off-Peak    | 12             | 0:00 to 7:00   |
If costs were a flat rate of 28 cents per kilowatt hour every hour, we could just do this:
Can you write Python code for it?

nrg['cost_cents'] = nrg['energy_kwh'] * 28; nrg.head()


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
The rows contains the electricity used in each hour for a one year period.
Each row indicates the usage for the hour starting at the specified time, so 1/1/13 0:00 indicates the usage for the first hour of January 1st.
Working with datetime data
Let's take a closer look at our data:
Can you write Python code for it?


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
This will be an issue with any column that can't neatly fit into a single data type.
Working with dates as strings is also an inefficient use of memory and programmer time (not to mention patience).
This exercise will work with time series data, and the date_time column will be formatted as an array of datetime objects called a pandas.Timestamp.
Can you write Python code for it?

nrg['date_time'] = pd.to_datetime(nrg['date_time'])
# https://stackoverflow.com/questions/29206612/difference-between-data-type-datetime64ns-and-m8ns
nrg['date_time'].dtype


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
Time for a timing decorator
The code above is pretty straightforward, but how fast does it run?
Let's find out by using a timing decorator called @timeit (an homage to Python's timeit).
This decorator behaves like timeit.repeat(), but it also allows you to return the result of the function itself as well as get the average runtime from multiple trials.
When you create a function and place the @timeit decorator above it, the function will be timed every time it is called.
Keep in mind that the decorator runs an outer and an inner loop.
Can you write Python code for it?

from timer import timeit

@timeit(repeat=3, number=10)
def convert_with_format(nrg, column_name):
    return pd.to_datetime(nrg[column_name], format='%d/%m/%y %H:%M')

nrg['date_time'] = convert_with_format(nrg, 'date_time')


I am working on the file "NevadaDashboard/pythonic_pandas.ipynb".
The first task is:
You can consider the above to be an “antipattern” in pandas for several reasons.
First, initialize a list in which the outputs will be recorded.
Second, use the opaque object range(0, len(df)) to loop through nrg, then apply apply_rate(), and append the result to a list used to make the new DataFrame column.
Third, chained indexing with df.iloc[i]['date_time'] may lead to unintended results.
Each of these increase the time cost of the calculations.
On my machine, this loop took about 3 seconds for 8760 rows of data.
Next, you’ll look at some improved solutions for iteration over Pandas structures.
Looping with .itertuples() and .iterrows()
Instead of looping through a range of objects, you can use generator methods that yield one row at a time.
.itertuples() yields a namedtuple() for each row, with the row’s index value as the first element of the tuple.
A namedtuple() is a data structure from Python’s collections module that behaves like a Python tuple but has fields accessible by attribute lookup.
.iterrows() yields pairs (tuples) of (index, Series) for each row in the DataFrame.
While .itertuples() tends to be a bit faster, let’s focus on pandas and use .iterrows() in this example.
Can you write Python code for it?

@timeit(repeat=2, number=10)
def apply_rate_iterrows(nrg):
    energy_cost_list = []
    for index, row in nrg.iterrows():
        energy_used = row['energy_kwh']
        hour = row['date_time'].hour
        energy_cost = apply_rate(energy_used, hour)
        energy_cost_list.append(energy_cost)
    nrg['cost_cents'] = energy_cost_list

apply_rate_iterrows(nrg)


I am working on the file "New York Restaurants Demo.ipynb".
The first task is:
Brunel visualization will be used to map the latitude and longitude to a New York state map.  Colors represent the number of violations as noted in the key.
Can you write Python code for it?

import brunel
nyvPan = nyvDF.toPandas()
%brunel map ('NY') + data('nyvPan') x(lon) y(lat) color(Violations) tooltip(FACILITY)


I am working on the file "New York Restaurants Demo.ipynb".
The first task is:
Now, a Spark dataframe 'nyvDF' will be created using SQL that will contain the restaurant name (FACILITY), latitude, longitude and violations.  Note that the latitude and longitude are combined in the final column (Location1) of the retrieved data.  They will be extracted separately using regular expressions in the SQL.  The results are ordered by number of violations in descending order and the top 10 are displayed.
Can you write Python code for it?

query = """
select
    FACILITY,
    trim(regexp_extract(location1, '(\\\()(.*),(.*)(\\\))',2)) as lat,
    trim(regexp_extract(location1, '(\\\()(.*),(.*)(\\\))',3)) as lon,
    cast(`TOTAL # CRITICAL VIOLATIONS` as int) as Violations
from nyrDF
order by Violations desc
limit 1000
"""
nyvDF = sqlContext.sql(query)
nyvDF.show(10)


I am working on the file "New York Restaurants Demo.ipynb".
The first task is:
One of the many key strengths of Watson Studio is the ability to easily search and quickly learn about various topics.  For example, to find articles, tutorials or notebooks on Brunel, click on the 'link' icon on the top right hand corner of this web page ('Find Resources in the Commuity').  A side palette will appear where you can enter 'Brunel' or other topics of interest.  Related articles, tutorials, notebooks, data cards will appear.
Pixiedust provides charting and visualization.  It is an open source Python library that works as an add-on to Jupyter notebooks to improve the user experience of working with data.  Please execute the next cell for a tabular view of the data.
Can you write Python code for it?

from pixiedust.display import *
display(nyvDF)


I am working on the file "New York Restaurants Demo.ipynb".
The first task is:
The csv (comma separated values) data will be read into a Pandas dataframe (nyr) and the first 5 records are displayed using the 'head()' method.<br>
Can you write Python code for it?

import pandas as pd
nyr = pd.read_csv(nyrdata)
nyr.head()


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Alphabet checks
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Alphanumeric checks
Can you write Python code for it?

'total'.isalnum()

'abc123'.isalnum()

'1+1'.isalnum()


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Alternative ways of using string format
Can you write Python code for it?

'I have a {food_item} and a {drink_item} with me'.format(drink_item='soda', food_item='sandwich')

'The {animal} has the following attributes: {attributes}'.format(animal='dog', attributes=['lazy', 'loyal'])


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Formatting strings using the format method - new style
Can you write Python code for it?

'Hello {} {}, it is a great {} to meet you at {}'.format('Mr.', 'Jones', 'pleasure', 5)

'Hello {} {}, it is a great {} to meet you at {} o\' clock'.format('Sir', 'Arthur', 'honor', 9)


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Numeric checks
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String Immutability
Can you write Python code for it?

# strings are immutable hence assignment throws error
s[0] = 'X'

print('Original String id:', id(s))
# creates a new string
s = 'X' + s[1:]
print(s)
print('New String id:', id(s))


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String formatting
Simple string formatting expressions - old style
Can you write Python code for it?

'Hello %s' %('Python!')

'Hello %s %s' %('World!', 'How are you?')


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String indexing and slicing
Can you write Python code for it?

# creating a string
s = 'PYTHON'
s, type(s)


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String operations
Different ways of String concatenation
Can you write Python code for it?

'Hello 😊' + ' and welcome ' + 'to Python 🐍!'

'Hello 😊' ' and welcome ' 'to Python 🐍!'

# concatenation of variables and literals

s1 = 'Python 💻!'
'Hello 😊 ' + s1

'Hello 😊 ' s1

# some more ways of concatenating strings

s2 = '--🐍Python🐍--'
s2 * 5

s1 + s2

(s1 + s2)*3

# concatenating several strings together in parentheses

s3 = ('This '
      'is another way '
      'to concatenate '
      'several strings!')
s3

# checking for substrings in a string

'way' in s3

'python' in s3

# computing total length of the string

len(s3)


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String replace
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String slicing
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
String slicing with offsets
Can you write Python code for it?

s[::1]  # no offset

s[::2]  # print every 2nd character in string


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
Useful String methods
Case conversions
Can you write Python code for it?

s = 'python is great'

s.capitalize()

s.upper()

s.title()


I am working on the file "New-Second-Edition/Ch02 - Python for Natural Language Processing/Ch02b - Working with Text Data.ipynb".
The first task is:
stringliteral   ::=  [stringprefix](shortstring | longstring)
stringprefix    ::=  "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR"
                     | "b" | "B" | "br" | "Br" | "bR" | "BR"
shortstring     ::=  "'" shortstringitem* "'" | '"' shortstringitem* '"'
longstring      ::=  "'''" longstringitem* "'''" | '"""' longstringitem* '"""'
shortstringitem ::=  shortstringchar | escapeseq
longstringitem  ::=  longstringchar | escapeseq
shortstringchar ::=  <any source character except "\" or newline or the quote>
longstringchar  ::=  <any source character except "\">
escapeseq       ::=  "\" <any ASCII character>
String types
Can you write Python code for it?

new_string = "This is a String"  # storing a string

print('ID:', id(new_string))  # shows the object identifier (address)
print('Type:', type(new_string))  # shows the object type
print('Value:', new_string)  # shows the object value

# simple string
simple_string = 'Hello!' + " I'm a simple string"
print(simple_string)

# multi-line string, note the \n (newline) escape character automatically created
multi_line_string = """Hello I'm
a multi-line
string!"""

multi_line_string

print(multi_line_string)

# Normal string with escape sequences leading to a wrong file path!
escaped_string = "C:\the_folder\new_dir\file.txt"
print(escaped_string)  # will cause errors if we try to open a file here

# raw string keeping the backslashes in its normal form
raw_string = r'C:\the_folder\new_dir\file.txt'
print(raw_string)

# unicode string literals
string_with_unicode = 'H\u00e8llo!'
print(string_with_unicode)

more_unicode = 'I love Pizza 🍕!  Shall we book a cab 🚕 to get pizza?'
print(more_unicode)

print(string_with_unicode + '\n' + more_unicode)

' '.join([string_with_unicode, more_unicode])

more_unicode[::-1]  # reverses the string


I am working on the file "New-Second-Edition/Ch03 - Processing and Understanding Text/Ch03c - BONUS - Text Parsing with Stanford CoreNLP.ipynb".
The first task is:
We can see the nested hierarchical structure of the constituents in the preceding output as compared to the flat structure in shallow parsing. Refer to the Penn Treebank reference as needed to lookup other tags.
Constituency Parsing with Stanford CoreNLP
You may have seen in the above messages that they are deprecating the old Stanford Parsers in favor of the more active Stanford Core NLP Project. It might even get removed after nltk version 3.4 so best to stay updated.
Details: https://github.com/nltk/nltk/issues/1839
Step by Step Tutorial here: https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK
Sadly a lot of things have changed in the process so we need to do some extra effort to make it work!
Get CoreNLP from here
After you download, go to the folder and spin up a terminal and start the Core NLP Server locally
E:\> java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,ssplit,pos,lemma,ner,parse,depparse -status_port 9000 -port 9000 -timeout 15000
If it runs successfully you should see the following messages on the terminal
E:\stanford\stanford-corenlp-full-2018-02-27>java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,ssplit,pos,lemma,ner,parse,depparse -status_port 9000 -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
[main] INFO CoreNLP -     Threads: 4
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[main] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator - No tokenizer type provided. Defaulting to PTBTokenizer.
[main] INFO...
Can you write Python code for it?

from nltk.parse import CoreNLPParser

cnp = CoreNLPParser()
cnp

result = list(cnp.raw_parse(sentence))[0]
print(result)

result

result.pretty_print()


I am working on the file "New-Second-Edition/Ch06 - Text Summarization and Topic Models/Ch06c - Topic Modeling with scikit-learn.ipynb".
The first task is:
Basic Text Wrangling
Can you write Python code for it?

%%time
import nltk

stop_words = nltk.corpus.stopwords.words('english')
wtk = nltk.tokenize.RegexpTokenizer(r'\w+')
wnl = nltk.stem.wordnet.WordNetLemmatizer()

def normalize_corpus(papers):
    norm_papers = []
    for paper in papers:
        paper = paper.lower()
        paper_tokens = [token.strip() for token in wtk.tokenize(paper)]
        paper_tokens = [wnl.lemmatize(token) for token in paper_tokens if not token.isnumeric()]
        paper_tokens = [token for token in paper_tokens if len(token) > 1]
        paper_tokens = [token for token in paper_tokens if token not in stop_words]
        paper_tokens = list(filter(None, paper_tokens))
        if paper_tokens:
            norm_papers.append(paper_tokens)

    return norm_papers

norm_papers = normalize_corpus(papers)
print(len(norm_papers))


I am working on the file "New-Second-Edition/Ch06 - Text Summarization and Topic Models/Ch06c - Topic Modeling with scikit-learn.ipynb".
The first task is:
Persisting Model and Transformers
This is just for visualizing the topics in the other notebook (since PyLDAViz expands the notebook size)
Can you write Python code for it?

import dill

with open('nmf_model.pkl', 'wb') as f:
    dill.dump(nmf_model, f)
with open('cv_features.pkl', 'wb') as f:
    dill.dump(cv_features, f)
with open('cv.pkl', 'wb') as f:
    dill.dump(cv, f)


I am working on the file "New-Second-Edition/Ch06 - Text Summarization and Topic Models/Ch06c - Topic Modeling with scikit-learn.ipynb".
The first task is:
Predicting Topics for New Research Papers
Can you write Python code for it?

import glob
# papers manually downloaded from NIPS 16
# https://papers.nips.cc/book/advances-in-neural-information-processing-systems-29-2016

new_paper_files = glob.glob('./test_data/nips16*.txt')
new_papers = []
for fn in new_paper_files:
    with open(fn, encoding='utf-8', errors='ignore', mode='r+') as f:
        data = f.read()
        new_papers.append(data)

print('Total New Papers:', len(new_papers))

norm_new_papers = normalize_corpus(new_papers)
cv_new_features = cv.transform(norm_new_papers)
cv_new_features.shape

topic_predictions = nmf_model.transform(cv_new_features)
best_topics = [[(topic, round(sc, 3))
                    for topic, sc in sorted(enumerate(topic_predictions[i]),
                                            key=lambda row: -row[1])[:2]]
                        for i in range(len(topic_predictions))]
best_topics

results_df = pd.DataFrame()
results_df['Papers'] = range(1, len(new_papers)+1)
results_df['Dominant Topics'] = [[topic_num+1 for topic_num, sc in item] for item in best_topics]
res = results_df.set_index(['Papers'])['Dominant Topics'].apply(pd.Series).stack().reset_index(level=1, drop=True)
results_df = pd.DataFrame({'Dominant Topics': res.values}, index=res.index)
results_df['Topic Score'] = [topic_sc for topic_list in
                                        [[round(sc*100, 2)
                                              for topic_num, sc in item]
                                                 for item in best_topics]
                                    for topic_sc in topic_list]

results_df['Topic Desc'] = [topics_df.iloc[t-1]['Terms per Topic'] for t in results_df['Dominant Topics'].values]
results_df['Paper Desc'] = [new_papers[i-1][:200] for i in results_df.index.values]

results_df


I am working on the file "New-Second-Edition/Ch06 - Text Summarization and Topic Models/Ch06c - Topic Modeling with scikit-learn.ipynb".
The first task is:
Text Representation with Feature Engineering
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(min_df=20, max_df=0.6, ngram_range=(1,2),
                     token_pattern=None, tokenizer=lambda doc: doc,
                     preprocessor=lambda doc: doc)
cv_features = cv.fit_transform(norm_papers)
cv_features.shape

vocabulary = np.array(cv.get_feature_names())
print('Total Vocabulary Size:', len(vocabulary))


I am working on the file "New-Second-Edition/Ch06 - Text Summarization and Topic Models/Ch06c - Topic Modeling with scikit-learn.ipynb".
The first task is:
Topic Models with Non-Negative Matrix Factorization (NMF)
Can you write Python code for it?

%%time
from sklearn.decomposition import NMF

nmf_model = NMF(n_components=TOTAL_TOPICS, solver='cd', max_iter=500,
                random_state=42, alpha=.1, l1_ratio=.85)
document_topics = nmf_model.fit_transform(cv_features)

topic_terms = nmf_model.components_
topic_key_term_idxs = np.argsort(-np.absolute(topic_terms), axis=1)[:, :top_terms]
topic_keyterms = vocabulary[topic_key_term_idxs]
topics = [', '.join(topic) for topic in topic_keyterms]
pd.set_option('display.max_colwidth', -1)
topics_df = pd.DataFrame(topics,
                         columns = ['Terms per Topic'],
                         index=['Topic'+str(t) for t in range(1, TOTAL_TOPICS+1)])
topics_df

pd.options.display.float_format = '{:,.3f}'.format
dt_df = pd.DataFrame(document_topics,
                     columns=['T'+str(i) for i in range(1, TOTAL_TOPICS+1)])
dt_df.head(10)

pd.options.display.float_format = '{:,.5f}'.format
pd.set_option('display.max_colwidth', 200)

max_score_topics = dt_df.max(axis=0)
dominant_topics = max_score_topics.index
term_score = max_score_topics.values
document_numbers = [dt_df[dt_df[t] == max_score_topics.loc[t]].index[0]
                       for t in dominant_topics]
documents = [papers[i] for i in document_numbers]

results_df = pd.DataFrame({'Dominant Topic': dominant_topics, 'Max Score': term_score,
                          'Paper Num': document_numbers, 'Topic': topics_df['Terms per Topic'],
                          'Paper Name': documents})
results_df


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
BM25 Similarity
Can you write Python code for it?

"""
Data:
-----
.. data:: PARAM_K1 - Free smoothing parameter for BM25.
.. data:: PARAM_B - Free smoothing parameter for BM25.
.. data:: EPSILON - Constant used for negative idf of document in corpus.
"""

import math
from six import iteritems
from six.moves import xrange

PARAM_K1 = 2.5
PARAM_B = 0.85
EPSILON = 0.2

class BM25(object):
    """Implementation of Best Matching 25 ranking function.
    Attributes
    ----------
    corpus_size : int
        Size of corpus (number of documents).
    avgdl : float
        Average length of document in `corpus`.
    corpus : list of list of str
        Corpus of documents.
    f : list of dicts of int
        Dictionary with terms frequencies for each document in `corpus`. Words used as keys and frequencies as values.
    df : dict
        Dictionary with terms frequencies for whole `corpus`. Words used as keys and frequencies as values.
    idf : dict
        Dictionary with inversed terms frequencies for whole `corpus`. Words used as keys and frequencies as values.
    doc_len : list of int
        List of document lengths.
    """

    def __init__(self, corpus):
        """
        Parameters
        ----------
        corpus : list of list of str
            Given corpus.
        """
        self.corpus_size = len(corpus)
        self.avgdl = sum(float(len(x)) for x in corpus) / self.corpus_size
        self.corpus = corpus
        self.f = []
        self.df = {}
        self.idf = {}
        self.doc_len = []
        self.initialize()

    def initialize(self):
        """Calculates frequencies of terms in documents and in corpus. Also computes inverse document frequencies."""
        for document in self.corpus:
            frequencies = {}
            self.doc_len.append(len(document))
            for word in document:
                if word not in frequencies:
                    frequencies[word] = 0
                frequencies[word] += 1
            self.f.append(frequencies)

            for word, freq in iteritems(frequencies):
                if...


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Build a movie recommender function to recommend top 5 similar movies for any movie
The movie title, movie title list and document similarity matrix dataframe will be given as inputs to the function
Can you write Python code for it?

def movie_recommender(movie_title, movies=movies_list, doc_sims=doc_sim_df):
    # find movie id
    movie_idx = np.where(movies == movie_title)[0][0]
    # get movie similarities
    movie_similarities = doc_sims.iloc[movie_idx].values
    # get top 5 similar movie IDs
    similar_movie_idxs = np.argsort(-movie_similarities)[1:6]
    # get top 5 movies
    similar_movies = movies[similar_movie_idxs]
    # return the top 5 movies
    return similar_movies


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Extract TF-IDF Features
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

tf = TfidfVectorizer(ngram_range=(1, 2), min_df=2)
tfidf_matrix = tf.fit_transform(norm_corpus)
tfidf_matrix.shape


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Get List of Movie Titles
Can you write Python code for it?

movies_list = df['title'].values
movies_list, movies_list.shape


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Get Popular Movie Recommendations
Can you write Python code for it?

for movie in popular_movies:
    print('Movie:', movie)
    print('Top 5 recommended Movies:', movie_recommender(movie_title=movie))
    print()


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Get movie similarities
Can you write Python code for it?

movie_similarities = doc_sim_df.iloc[movie_idx].values
movie_similarities


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Get top 5 similar movies
Can you write Python code for it?

similar_movies = movies_list[similar_movie_idxs]
similar_movies


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07b - Movie Recommender with Document Similarity.ipynb".
The first task is:
Sort Dataset by Popular Movies
Can you write Python code for it?

pop_movies = df.sort_values(by='popularity', ascending=False)
pop_movies.head()

popular_movies = ['Minions', 'Interstellar', 'Deadpool', 'Jurassic World', 'Pirates of the Caribbean: The Curse of the Black Pearl',
              'Dawn of the Planet of the Apes', 'The Hunger Games: Mockingjay - Part 1', 'Terminator Genisys',
              'Captain America: Civil War', 'The Dark Knight', 'The Martian', 'Batman v Superman: Dawn of Justice',
              'Pulp Fiction', 'The Godfather', 'The Shawshank Redemption', 'The Lord of the Rings: The Fellowship of the Ring',
              'Harry Potter and the Chamber of Secrets', 'Star Wars', 'The Hobbit: The Battle of the Five Armies',
              'Iron Man']


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07c - Document Clustering.ipynb".
The first task is:
Calculate Linkage Matrix using Cosine Similarity
Can you write Python code for it?

def ward_hierarchical_clustering(feature_matrix):

    cosine_distance = 1 - cosine_similarity(feature_matrix)
    linkage_matrix = ward(cosine_distance)
    return linkage_matrix


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07c - Document Clustering.ipynb".
The first task is:
Cluster Movies using K-Means
Can you write Python code for it?

from sklearn.cluster import KMeans

NUM_CLUSTERS = 6
km = KMeans(n_clusters=NUM_CLUSTERS, max_iter=10000, n_init=50, random_state=42).fit(cv_matrix)
km

from collections import Counter

Counter(km.labels_)

df['kmeans_cluster'] = km.labels_

movie_clusters = (df[['title', 'kmeans_cluster', 'popularity']]
                  .sort_values(by=['kmeans_cluster', 'popularity'],
                               ascending=False)
                  .groupby('kmeans_cluster').head(20))
movie_clusters = movie_clusters.copy(deep=True)

feature_names = cv.get_feature_names()
topn_features = 15
ordered_centroids = km.cluster_centers_.argsort()[:, ::-1]

# get key features for each cluster
# get movies belonging to each cluster
for cluster_num in range(NUM_CLUSTERS):
    key_features = [feature_names[index]
                        for index in ordered_centroids[cluster_num, :topn_features]]
    movies = movie_clusters[movie_clusters['kmeans_cluster'] == cluster_num]['title'].values.tolist()
    print('CLUSTER #'+str(cluster_num+1))
    print('Key Features:', key_features)
    print('Popular Movies:', movies)
    print('-'*80)

from sklearn.metrics.pairwise import cosine_similarity

cosine_sim_features = cosine_similarity(cv_matrix)
km = KMeans(n_clusters=NUM_CLUSTERS, max_iter=10000, n_init=50, random_state=42).fit(cosine_sim_features)
Counter(km.labels_)

df['kmeans_cluster'] = km.labels_

movie_clusters = (df[['title', 'kmeans_cluster', 'popularity']]
                  .sort_values(by=['kmeans_cluster', 'popularity'],
                               ascending=False)
                  .groupby('kmeans_cluster').head(20))
movie_clusters = movie_clusters.copy(deep=True)

# get movies belonging to each cluster
for cluster_num in range(NUM_CLUSTERS):
    movies = movie_clusters[movie_clusters['kmeans_cluster'] == cluster_num]['title'].values.tolist()
    print('CLUSTER #'+str(cluster_num+1))
    print('Popular Movies:', movies)
    print('-'*80)


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07c - Document Clustering.ipynb".
The first task is:
Hierarchical Clustering
So far, we were successfull in clustering movies using K-Means. But is there any further level of understanding we can extract from this dataset in an unsupervised manner?
Hierarchical Clustering to the rescue. K-Means helped us understand similarities amongst movies, with hierarchical clustering we can aim at understanding abstract or higher level concepts which are common across groups of movies. There are primarily two ways in which hierarchical clustering can be performed:

Divisive : The algorithm begins with every element in one big generic cluster and then goes on dividing them into specific clusters in a recursive manner.
Agglomerative : In this case, the algorithm starts by placing every element into a cluster of its own and then goes on merging them into more general clusters in a recursive manner (till they all merge into one big cluster).

For this tutorial, we will work with Ward clustering algorithm. Ward clustering is an agglomerative clustering method, i.e. at each stage, the pair of clusters with minimum between-cluster distance (or wcss) are merged.
To work with Ward Clustering Algorithm, we perform the following steps:
-  Prepare a cosine distance matrix
-  Calclate a linkage_matrix
-  Plot the hierarchical structure as a dendrogram.
Can you write Python code for it?

from scipy.cluster.hierarchy import ward, dendrogram
from sklearn.metrics.pairwise import cosine_similarity


I am working on the file "New-Second-Edition/Ch07 - Text Similarity and Clustering/Ch07c - Document Clustering.ipynb".
The first task is:
Plot Hierarchical Structure as a Dendrogram
Can you write Python code for it?

def plot_hierarchical_clusters(linkage_matrix, movie_data, p=100, figure_size=(8,12)):
    # set size
    fig, ax = plt.subplots(figsize=figure_size)
    movie_titles = movie_data['title'].values.tolist()
    # plot dendrogram
    R = dendrogram(linkage_matrix, orientation="left", labels=movie_titles,
                    truncate_mode='lastp',
                    p=p,
                    no_plot=True)
    temp = {R["leaves"][ii]: movie_titles[ii] for ii in range(len(R["leaves"]))}
    def llf(xx):
        return "{}".format(temp[xx])
    ax = dendrogram(
            linkage_matrix,
            truncate_mode='lastp',
            orientation="left",
            p=p,
            leaf_label_func=llf,
            leaf_font_size=10.,
            )
    plt.tick_params(axis= 'x',
                    which='both',
                    bottom='off',
                    top='off',
                    labelbottom='off')
    plt.tight_layout()
    plt.savefig('movie_hierachical_clusters.png', dpi=200)

linkage_matrix = ward_hierarchical_clustering(cv_matrix)

plot_hierarchical_clusters(linkage_matrix,
                           p=100,
                           movie_data=df,
                           figure_size=(12, 14))


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08b - Named Entity Recognition.ipynb".
The first task is:
NER with Stanford CoreNLP
NLTK is slowly deprecating the old Stanford Parsers in favor of the more active Stanford Core NLP Project. It might even get removed after nltk version 3.4 so best to stay updated.
Details: https://github.com/nltk/nltk/issues/1839
Step by Step Tutorial here: https://github.com/nltk/nltk/wiki/Stanford-CoreNLP-API-in-NLTK
Sadly a lot of things have changed in the process so we need to do some extra effort to make it work!
Get CoreNLP from here
After you download, go to the folder and spin up a terminal and start the Core NLP Server locally
E:\> java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,ssplit,pos,lemma,ner,parse,depparse -status_port 9000 -port 9000 -timeout 15000
If it runs successfully you should see the following messages on the terminal
E:\stanford\stanford-corenlp-full-2018-02-27>java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,ssplit,pos,lemma,ner,parse,depparse -status_port 9000 -port 9000 -timeout 15000
[main] INFO CoreNLP - --- StanfordCoreNLPServer#main() called ---
[main] INFO CoreNLP - setting default constituency parser
[main] INFO CoreNLP - warning: cannot find edu/stanford/nlp/models/srparser/englishSR.ser.gz
[main] INFO CoreNLP - using: edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz instead
[main] INFO CoreNLP - to use shift reduce parser download English models jar from:
[main] INFO CoreNLP - http://stanfordnlp.github.io/CoreNLP/download.html
[main] INFO CoreNLP -     Threads: 4
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[main] INFO edu.stanford.nlp.pipeline.TokenizerAnnotator - No tokenizer type provided. Defaulting to PTBTokenizer.
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[main] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[main] INFO edu.stanford.nlp.tagger.maxent.MaxentTagger - Loading POS tagger from...
Can you write Python code for it?

from nltk.parse import CoreNLPParser

ner_tagger = CoreNLPParser(url='http://localhost:9000', tagtype='ner')
ner_tagger

import nltk

tags = list(ner_tagger.raw_tag_sents(nltk.sent_tokenize(text)))
tags = [sublist[0] for sublist in tags]
tags = [word_tag for sublist in tags for word_tag in sublist]
print(tags)

named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in tags:
    if tag != 'O':
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None

print(named_entities)

c = Counter([item[1] for item in named_entities])
c.most_common()


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08b - Named Entity Recognition.ipynb".
The first task is:
NER with Stanford NLP
Stanford’s Named Entity Recognizer is based on an implementation of linear chain Conditional Random Field (CRF) sequence models.
Prerequisites: Download the official Stanford NER Tagger from here, which seems to work quite well. You can try out a later version by going to this website
This model is only trained on instances of PERSON, ORGANIZATION and LOCATION types. The model is exposed through nltk wrappers.
Can you write Python code for it?

import os
from nltk.tag import StanfordNERTagger

JAVA_PATH = r'C:\Program Files\Java\jre1.8.0_192\bin\java.exe'
os.environ['JAVAHOME'] = JAVA_PATH

STANFORD_CLASSIFIER_PATH = 'E:/stanford/stanford-ner-2014-08-27/classifiers/english.all.3class.distsim.crf.ser.gz'
STANFORD_NER_JAR_PATH = 'E:/stanford/stanford-ner-2014-08-27/stanford-ner.jar'

sn = StanfordNERTagger(STANFORD_CLASSIFIER_PATH,
                       path_to_jar=STANFORD_NER_JAR_PATH)
sn

text_enc = text.encode('ascii', errors='ignore').decode('utf-8')
ner_tagged = sn.tag(text_enc.split())
print(ner_tagged)

named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in ner_tagged:
    if tag != 'O':
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None

print(named_entities)

c = Counter([item[1] for item in named_entities])
c.most_common()


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08b - Named Entity Recognition.ipynb".
The first task is:
Spacy offers fast NER tagger based on a number of techniques. The exact algorithm hasn't been talked about in much detail but the documentation marks it as <font color=blue> "The exact algorithm is a pastiche of well-known methods, and is not currently described in any single publication " </font>
The entities identified by spacy NER tagger are as shown in the following table (details here: spacy_documentation)
Can you write Python code for it?

named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in ner_tagged:
    if tag:
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None

print(named_entities)

from collections import Counter
c = Counter([item[1] for item in named_entities])
c.most_common()


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Building Models with sklearn-crfsuite
sklearn-crfsuite is a thin CRFsuite (python-crfsuite) wrapper which provides scikit-learn-compatible sklearn_crfsuite.CRF estimator: you can use e.g. scikit-learn model selection utilities (cross-validation, hyperparameter optimization) with it, or save/load CRF models using joblib.
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Conditional Random Fields
As mentioned above, NER belongs to sequence modeling class of problems. There are different algorithms to tackle sequence modeling, CRF or Conditional Random Fields are one such example. CRFs are proven to perform extremely well on NER and related domains. In this notebook, we will attempt at developing our own NER based on CRFs.

Question: What is a CRF and how does it work?
Wikipedia :  CRF is an undirected graphical model whose nodes can be divided into exactly two disjoint sets $X$ and $Y$, the observed and output variables, respectively; the conditional distribution $p(Y|X)$ is then modeled.
For more details, checkout the paper Conditional Random Fields: Probabilistic Models
for Segmenting and Labeling Sequence Data
Prepare Data
CRF trains upon sequence of input data to learn transitions from one state (label) to another.
To enable such an algorithm, we need to define features which take into account different transitions.
In the function word2features() below, we transform each word into a feature dictionary depicting the following attributes or features:

lower case of word
suffix containing last 3 characters
suffix containing last 2 characters
flags to determine upper-case, title-case, numeric data and POS tag

We also attach attributes related to previous and next words or tags to determine beginning of sentence (BOS) or end of sentence (EOS)
Can you write Python code for it?

def word2features(sent, i):
    word = sent[i][0]
    postag = sent[i][1]

    features = {
        'bias': 1.0,
        'word.lower()': word.lower(),
        'word[-3:]': word[-3:],
        'word[-2:]': word[-2:],
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
        'postag': postag,
        'postag[:2]': postag[:2],
    }
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:word.isupper()': word1.isupper(),
            '-1:postag': postag1,
            '-1:postag[:2]': postag1[:2],
        })
    else:
        features['BOS'] = True

    if i < len(sent)-1:
        word1 = sent[i+1][0]
        postag1 = sent[i+1][1]
        features.update({
            '+1:word.lower()': word1.lower(),
            '+1:word.istitle()': word1.istitle(),
            '+1:word.isupper()': word1.isupper(),
            '+1:postag': postag1,
            '+1:postag[:2]': postag1[:2],
        })
    else:
        features['EOS'] = True

    return features

def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]

def sent2labels(sent):
    return [label for token, postag, label in sent]

agg_func = lambda s: [(w, p, t) for w, p, t in zip(s['Word'].values.tolist(),
                                                   s['POS'].values.tolist(),
                                                   s['Tag'].values.tolist())]

grouped_df = df.groupby('Sentence #').apply(agg_func)

print(grouped_df[grouped_df.index == 'Sentence: 1'].values)

grouped_df.shape

sentences = [s for s in grouped_df]
sentences[0]

sent2features(sentences[0][5:7])

sent2labels(sentences[0][5:7])


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Pipeline Step 3

Use the CRF Model crf to predict on the features
Can you write Python code for it?

labels = crf.predict(features)
doc_labels = labels[0]
doc_labels[10:20]


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Pipeline Step 4

Combine text tokens with NER Tags
Retrieve relevant named entities from NER Tags
Can you write Python code for it?

text_ner = [(token, tag) for token, tag in zip(text_tokens, doc_labels)]
print(text_ner)

named_entities = []
temp_entity_name = ''
temp_named_entity = None
for term, tag in text_ner:
    if tag != 'O':
        temp_entity_name = ' '.join([temp_entity_name, term]).strip()
        temp_named_entity = (temp_entity_name, tag)
    else:
        if temp_named_entity:
            named_entities.append(temp_named_entity)
            temp_entity_name = ''
            temp_named_entity = None

import pandas as pd

pd.DataFrame(named_entities, columns=['Entity', 'Tag'])


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Train the model!
Train the model using the default configurations mentioned in the sklearn-crfsuite API docs

algorithm: the training algorithm. We use L-BFGS for gradient descent for optimization and getting model parameters
c1: Coefficient for Lasso (L1) regularization
c2: Coefficient for Ridge (L2) regularization
all_possible_transitions: Specify whether CRFsuite generates transition features that do not even occur in the training data

Note: If the model is taking too long to train, you can load up the pre-trained model using the code after the training cells and use that for predictions.
Can you write Python code for it?

import sklearn_crfsuite

crf = sklearn_crfsuite.CRF(algorithm='lbfgs',
                           c1=0.1,
                           c2=0.1,
                           max_iterations=100,
                           all_possible_transitions=True,
                           verbose=True)

crf.fit(X_train, y_train)


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
Use the following to load our pre-trained model if training above takes a lot of time
Can you write Python code for it?

from sklearn.externals import joblib

#joblib.dump(crf, 'ner_model.pkl')

crf = joblib.load('ner_model.pkl')


I am working on the file "New-Second-Edition/Ch08 - Semantic Analysis/Ch08c - Building NER Taggers.ipynb".
The first task is:
We have intentially left out the Others tag to understand the performance of model on the remaining tags. The above evaluation statistics showcase a model which seems to have learnt the transitions quite well giving us an overall F1-score of 85%!
We can achieve even better results by fine tuning the feature engineering step along with hyper-parameter tuning.
End-to-End NER Tagger with trained NER Model
There is no fun (or value!) if we cannot use our model to tag new sentences in the future assuming we would want to put this model in production. Let's try and build an end-to-end workflow to perform NER Tagging on our sample document. First we perform NER tagging with SpaCy to remind you how it looks like.
Prepare Sample Document
Can you write Python code for it?

import re

text = """Three more countries have joined an “international grand committee” of parliaments, adding to calls for
Facebook’s boss, Mark Zuckerberg, to give evidence on misinformation to the coalition. Brazil, Latvia and Singapore
bring the total to eight different parliaments across the world, with plans to send representatives to London on 27
November with the intention of hearing from Zuckerberg. Since the Cambridge Analytica scandal broke, the Facebook chief
has only appeared in front of two legislatures: the American Senate and House of Representatives, and the European parliament.
Facebook has consistently rebuffed attempts from others, including the UK and Canadian parliaments, to hear from Zuckerberg.
He added that an article in the New York Times on Thursday, in which the paper alleged a pattern of behaviour from Facebook
to “delay, deny and deflect” negative news stories, “raises further questions about how recent data breaches were allegedly
dealt with within Facebook.”
"""

text = re.sub(r'\n', '', text)
text


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Build Deep Learning Model with Universal Sentence Encoder
Can you write Python code for it?

embedding_feature = hub.text_embedding_column(
    key='sentence',
    module_spec="https://tfhub.dev/google/universal-sentence-encoder/2",
    trainable=False)

dnn = tf.estimator.DNNClassifier(
          hidden_units=[512, 128],
          feature_columns=[embedding_feature],
          n_classes=2,
          activation_fn=tf.nn.relu,
          dropout=0.1,
          optimizer=tf.train.AdagradOptimizer(learning_rate=0.005))


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Build a Generic Model Trainer on any Input Sentence Encoder
Can you write Python code for it?

import time

TOTAL_STEPS = 1500
STEP_SIZE = 500

my_checkpointing_config = tf.estimator.RunConfig(
    keep_checkpoint_max = 2,       # Retain the 2 most recent checkpoints.
)

def train_and_evaluate_with_sentence_encoder(hub_module, train_module=False, path=''):
    embedding_feature = hub.text_embedding_column(
        key='sentence', module_spec=hub_module, trainable=train_module)

    print()
    print('='*100)
    print('Training with', hub_module)
    print('Trainable is:', train_module)
    print('='*100)

    dnn = tf.estimator.DNNClassifier(
            hidden_units=[512, 128],
            feature_columns=[embedding_feature],
            n_classes=2,
            activation_fn=tf.nn.relu,
            dropout=0.1,
            optimizer=tf.train.AdagradOptimizer(learning_rate=0.005),
            model_dir=path,
            config=my_checkpointing_config)

    for step in range(0, TOTAL_STEPS+1, STEP_SIZE):
        print('-'*100)
        print('Training for step =', step)
        start_time = time.time()
        dnn.train(input_fn=train_input_fn, steps=STEP_SIZE)
        elapsed_time = time.time() - start_time
        print('Train Time (s):', elapsed_time)
        print('Eval Metrics (Train):', dnn.evaluate(input_fn=predict_train_input_fn))
        print('Eval Metrics (Validation):', dnn.evaluate(input_fn=predict_val_input_fn))

    train_eval_result = dnn.evaluate(input_fn=predict_train_input_fn)
    test_eval_result = dnn.evaluate(input_fn=predict_test_input_fn)

    return {
      "Model Dir": dnn.model_dir,
      "Training Accuracy": train_eval_result["accuracy"],
      "Test Accuracy": test_eval_result["accuracy"],
      "Training AUC": train_eval_result["auc"],
      "Test AUC": test_eval_result["auc"],
      "Training Precision": train_eval_result["precision"],
      "Test Precision": test_eval_result["precision"],
      "Training Recall": train_eval_result["recall"],
      "Test Recall": test_eval_result["recall"]
    }


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Build train, validation and test datasets
Can you write Python code for it?

reviews = dataset['review'].values
sentiments = dataset['sentiment'].values

train_reviews = reviews[:30000]
train_sentiments = sentiments[:30000]

val_reviews = reviews[30000:35000]
val_sentiments = sentiments[30000:35000]

test_reviews = reviews[35000:]
test_sentiments = sentiments[35000:]
train_reviews.shape, val_reviews.shape, test_reviews.shape


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Check if GPU is available for use!
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Load and View Dataset
Can you write Python code for it?

dataset = pd.read_csv('movie_reviews.csv.bz2', compression='bz2')
dataset.info()

dataset['sentiment'] = [1 if sentiment == 'positive' else 0 for sentiment in dataset['sentiment'].values]
dataset.head()


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Load up Dependencies
Can you write Python code for it?

import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import pandas as pd


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Model Evaluation
Can you write Python code for it?


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Model Evaluations
Can you write Python code for it?

results_df = pd.DataFrame.from_dict(results, orient="index")
results_df

best_model_dir = results_df[results_df['Test Accuracy'] == results_df['Test Accuracy'].max()]['Model Dir'].values[0]
best_model_dir

embedding_feature = hub.text_embedding_column(
        key='sentence', module_spec="https://tfhub.dev/google/universal-sentence-encoder/2", trainable=True)

dnn = tf.estimator.DNNClassifier(
            hidden_units=[512, 128],
            feature_columns=[embedding_feature],
            n_classes=2,
            activation_fn=tf.nn.relu,
            dropout=0.1,
            optimizer=tf.train.AdagradOptimizer(learning_rate=0.005),
            model_dir=best_model_dir)
dnn

def get_predictions(estimator, input_fn):
    return [x["class_ids"][0] for x in estimator.predict(input_fn=input_fn)]

predictions = get_predictions(estimator=dnn, input_fn=predict_test_input_fn)
predictions[:10]

!pip install seaborn

import seaborn as sns
import matplotlib.pyplot as plt

%matplotlib inline

with tf.Session() as session:
    cm = tf.confusion_matrix(test_sentiments, predictions).eval()

LABELS = ['negative', 'positive']
sns.heatmap(cm, annot=True, xticklabels=LABELS, yticklabels=LABELS, fmt='g')
xl = plt.xlabel("Predicted")
yl = plt.ylabel("Actuals")

from sklearn.metrics import classification_report

print(classification_report(y_true=test_sentiments, y_pred=predictions, target_names=LABELS))


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Model Training
Can you write Python code for it?

tf.logging.set_verbosity(tf.logging.ERROR)
import time

TOTAL_STEPS = 1500
STEP_SIZE = 100
for step in range(0, TOTAL_STEPS+1, STEP_SIZE):
    print()
    print('-'*100)
    print('Training for step =', step)
    start_time = time.time()
    dnn.train(input_fn=train_input_fn, steps=STEP_SIZE)
    elapsed_time = time.time() - start_time
    print('Train Time (s):', elapsed_time)
    print('Eval Metrics (Train):', dnn.evaluate(input_fn=predict_train_input_fn))
    print('Eval Metrics (Validation):', dnn.evaluate(input_fn=predict_val_input_fn))


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Train Deep Learning Models on difference Sentence Encoders

NNLM - pre-trained and fine-tuning
USE - pre-trained and fine-tuning
Can you write Python code for it?

tf.logging.set_verbosity(tf.logging.ERROR)

results = {}

results["nnlm-en-dim128"] = train_and_evaluate_with_sentence_encoder(
    "https://tfhub.dev/google/nnlm-en-dim128/1", path='/storage/models/nnlm-en-dim128_f/')

results["nnlm-en-dim128-with-training"] = train_and_evaluate_with_sentence_encoder(
    "https://tfhub.dev/google/nnlm-en-dim128/1", train_module=True, path='/storage/models/nnlm-en-dim128_t/')

results["use-512"] = train_and_evaluate_with_sentence_encoder(
    "https://tfhub.dev/google/universal-sentence-encoder/2", path='/storage/models/use-512_f/')

results["use-512-with-training"] = train_and_evaluate_with_sentence_encoder(
    "https://tfhub.dev/google/universal-sentence-encoder/2", train_module=True, path='/storage/models/use-512_t/')


I am working on the file "New-Second-Edition/Ch10 - The Promise of Deep Learning/Ch10a - Deep Transfer Learning for NLP - Text Classification with Universal Embeddings.ipynb".
The first task is:
Train for approx 12 epochs
Can you write Python code for it?


I am working on the file "NewPyx.ipynb".
The first task is:

Fig 2. Emod by D. Koski

The E mod tetravolume is (√2/8)/ØØØ or .041731
In the figure above the E mod is further subdivided by additional intersecting planes creating what we might call Phe, Phi, Pho, Phum submodules.  This subdivision of the E is discussed in more detail elsewhere.  The planes that cut the E3 per LCD trianges (derived from the 31 great circles of the spinning icosahedron), define a way to cut an E lying sideways on the E3's external face.
The E module proper, as depicted, has three blue orthogonal lengths:
* short_face_diag = 1/(Ø*Ø) = 0.381966
* long_face_diag = 1/Ø = 0.618033
* radius = 1.000000
It then has 2 red lengths, serving as hypotenuses:
* long_hypot = √(5-√5)/2 = 1.175570
* short_hypot = √(5-2√5) = .726542
... and finally, a yellow length, also an hypotenuse:
* yellow = (Ø^-1)(√3) = 1.070466
Can you write Python code for it?

short_face_diag = 1/(Ø*Ø)
long_face_diag = 1/Ø
radius = 1.000000
long_hypot = sqrt((5-sqrt(5))/2)
short_hypot = sqrt(5-2*sqrt(5))
yellow = sqrt(3)/Ø


I am working on the file "NewPyx.ipynb".
The first task is:

Fig 3. Same Volume as Emod, by D. Koski

We may follow essentially the same workflow with this alternative tetrahedron which turns out to have the same volume as the E module.  Note the division by 8 in the final step.

Short Blue = 1/Ø
Long Blue = 2/Ø
Yellow = √3/Ø
Purple = √3/Ø
Green = √2/Ø
Red = √((25-11√5)/2)
Can you write Python code for it?

purple = yellow = (3**0.5)/Ø
short_blue = 1/Ø
long_blue = 2/Ø
green = sqrt(2)/Ø
red = sqrt((25-11*sqrt(5))/2)

print("phi", Ø)
print("Purple, Yellow:", purple)
print("    Short Blue:", short_blue)
print("     Long Blue:", long_blue)
print("         Green:", green)
print("           Red:", red)

ivm_vol = tetravolume.Tetrahedron(red, yellow, green, long_blue, short_blue, purple).ivm_volume()

ivm_vol/8


I am working on the file "NewPyx.ipynb".
The first task is:
Below, we convert these to Python and feed them to our volume computer, which sets D=1 i.e. our dimensions are twice as big (assuming R=1).  So the final step involves taking 1/8th the computed volume.
Can you write Python code for it?

print("Blue:")
print("   short_face_diag:", short_face_diag)
print("    long_face_diag:", long_face_diag)
print("            radius:", radius)
print("Red:")
print("        long_hypot:", long_hypot)
print("       short_hypot:", short_hypot)
print()
print("            yellow:", yellow)

ivm_vol = tetravolume.Tetrahedron(radius, long_face_diag, short_face_diag,
                                 long_hypot, short_hypot, yellow).ivm_volume()
ivm_vol/8


I am working on the file "NewPyx.ipynb".
The first task is:
So far, we've depending on the built-in float type for our computations, but lets remember our 3rd party assets, in addition to decimal.Decimal in the Standard Library.
The computations below run through some of the same volumes already talked about, such as the E-module.
Can you write Python code for it?

import gmpy2

gmpy2.get_context().precision=300
Ø = (1 + gmpy2.root(5, 2))/2
vol_scale_factor = 1.5
rad_scale_factor = gmpy2.root(vol_scale_factor, 3)

h = (Ø / gmpy2.root(2,2)) * (1/rad_scale_factor)
print("RT5 radius: {:42.40f}".format(h))
S3 = gmpy2.root(9/8, 2)
SuperRT_vol = S3 * 20
print("SuperRT:   {:42.40f}".format( SuperRT_vol))
Emod_RT_vol = SuperRT_vol * (1/Ø)**3
print("5+ RT:      {:42.40f}".format( Emod_RT_vol ))
Tmod_RT_vol = gmpy2.mpfr(float(5.0))
print("5  RT:      {:42.40f}".format( Tmod_RT_vol ))
print("Emod:       {:42.40f}".format( Emod_RT_vol/120 ))
print("Tmod:       {:42.40f}".format( Tmod_RT_vol/120 ))


I am working on the file "New_Table.ipynb".
The first task is:
Reading Tables, compare them and remove the rows not in common.
Can you write Python code for it?

Tab3=Table.read("Tab3_totbin.fit")
Tab2=Table.read('Tab2_totbin.fit')
Tab4=Table.read('Tab4_totbin.fit')

Tab2.remove_rows([75,206])   #remove the row of CIDA11 B and the row not present in table 4 (J04290068+2755033)
Tab3.remove_rows([66,76,134,146])
#remove the rows not present in table 2 (J04263055+2443558, J04335245+2612548, J04361030+2159364)
#and in table 4 (J04290068+2755033)
Tab4.remove_rows([8,10,31,34,76,94,103,110,118,125,126,174,180,195,204,220])
#removing close pairs star (MHO2B,MHO3B, V892TAUB,DFTAUB,ZZTAUB,V710TAUB,GGTAUAb,UZTauEb,
#V807TauBa,V807TauBb,J04403979+2519061B,CoKuTau/4B,DQTauB,St34B,CIDA11B)

name=Tab2['Name']  #sources name
lF=Tab2['l_F1_3']  # sign for upper limit
F1_3=Tab2["F1_3"]  # mm Flux at 1.3mm
eF1_3=Tab2['e_F1_3'] #error on mm Flux
SpT=Tab2["SpT"]    # Stellar Spectral Type
logM=Tab3["logM_"]   # Stellar host mass
b_logMlo=Tab3["b_logM_"]    #low value of range of confidence (1sigma)
b_logMup=Tab3["b_logm__lc"]  #upper value of range of confidence (1sigma)
logL=Tab4['logL_']    #Stellar host Luminosity
#logT=Tab4['logT_']    #Stellar host Temperature

logM2=Tab3["logM_2"]   # Stellar host mass
b_logMlo2=Tab3["b_logM_2"]    #low value of range of confidence (1sigma)
b_logMup2=Tab3["b_logm_2_lc"]  #upper value of range of confidence (1sigma)

logM3=Tab3["logM_3"]   # Stellar host mass
b_logMlo3=Tab3["b_logM_3"]    #low value of range of confidence (1sigma)
b_logMup3=Tab3["b_logm_3_lc"]  #upper value of range of confidence (1sigma)

    ##### Error on upper limits
nnan=np.where(np.isnan(eF1_3))
eF1_3[nnan]=F1_3[nnan]/3.    #considering 1sigma
#eF1_3[nnan]=1e-20


I am working on the file "New_Table.ipynb".
The first task is:
Write the new table in a new fit file, and read it
Can you write Python code for it?

T.write('Table.fit', format='fits', overwrite='True')
S.write('Table_Mass.fit', format='fits', overwrite='True')
New=Table.read("Table.fit")
#New.info()
#New.colnames
New.show_in_browser()


I am working on the file "News Categorization MNB.ipynb".
The first task is:
Categories are literal labels, but it is better for machine learning algorithms just to work with numbers, so we will encode them using LabelEncoder, which encode labels with value between 0 and n_classes-1.
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
ncategories = encoder.fit_transform(categories)


I am working on the file "News Categorization MNB.ipynb".
The first task is:
Function head gives us the first 5 items in a column (or the first 5 rows in the DataFrame)
Can you write Python code for it?


I am working on the file "News Categorization MNB.ipynb".
The first task is:
In order to make the training process easier, scikit-learn provides a Pipeline class that behaves like a compound classifier. The first step should be to tokenize and count the number of occurrence of each word that appears into the news'titles. For that, we will use the CountVectorizer class. Then we will transform the counters to a tf-idf representation using TfidfTransformer class. The last step creates the Naive Bayes classifier
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline

print('Training...')

text_clf = Pipeline([('vect', CountVectorizer()),
                     ('tfidf', TfidfTransformer()),
                     ('clf', MultinomialNB()),
                     ])


I am working on the file "News Categorization MNB.ipynb".
The first task is:
Now we can procede to apply the classifier to the test set and calculate the predicted values
Can you write Python code for it?

print('Predicting...')
predicted = text_clf.predict(X_test)


I am working on the file "News Categorization MNB.ipynb".
The first task is:
Now we should split our data into two sets:
1. a training set (70%) used to discover potentially predictive relationships, and
2. a test set (30%) used to evaluate whether the discovered relationships hold and to assess the strength and utility of a predictive relationship.
Samples should be first shuffled and then split into a pair of train and test sets. Make sure you permute (shuffle) your training data before fitting the model.
Can you write Python code for it?

Ntrain = int(N * 0.7)
from sklearn.utils import shuffle
titles, ncategories = shuffle(titles, ncategories, random_state=0)

X_train = titles[:Ntrain]
print('X_train.shape',X_train.shape)
y_train = ncategories[:Ntrain]
print('y_train.shape',y_train.shape)
X_test = titles[Ntrain:]
print('X_test.shape',X_test.shape)
y_test = ncategories[Ntrain:]
print('y_test.shape',y_test.shape)


I am working on the file "News Categorization MNB.ipynb".
The first task is:
This way we can refer to pandas by its alias 'pd'. Let's import news aggregator data via Pandas
Can you write Python code for it?


I am working on the file "News Categorization MNB.ipynb".
The first task is:
We want to predict the category of a news article based only on its title. Class LabelEncoder allows to encode labels with values between 0 and n_classes-1.
Can you write Python code for it?

from sklearn.preprocessing import LabelEncoder

encoder = LabelEncoder()
y = encoder.fit_transform(news['CATEGORY'])
print(y[:5])

categories = news['CATEGORY']
titles = news['TITLE']
N = len(titles)
print('Number of news',N)

labels = list(set(categories))
print('possible categories',labels)

for l in labels:
    print('number of ',l,' news',len(news.loc[news['CATEGORY'] == l]))


I am working on the file "News Categorization MNB.ipynb".
The first task is:
sklearn.metrics module includes score functions, performance metrics, and pairwise metrics and distance computations.
accuracy_score: computes subset accuracy; used to compare set of predicted labels for a sample to the corresponding set of true labels
Can you write Python code for it?

from sklearn import metrics

print('accuracy_score',metrics.accuracy_score(y_test,predicted))
print('Reporting...')


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
As we can see, the entertainment (e) category is the biggest one, which is more than three times bigger than health (m) category. In second place we have business (b) and technology (t), which are more than two times bigger than health category.
Splitting the data
Now we should split our data into two sets:
1. a training set (70%) used to discover potentially predictive relationships, and
2. a test set (30%) used to evaluate whether the discovered relationships hold and to assess the strength and utility of a predictive relationship.
Before splitting it, the data should be first permuted. Shuffle is a method included in scikit-learn library which allows to do random permutations of collections. Then data could be splitted into a pair of train and test sets.
Can you write Python code for it?

from sklearn.utils import shuffle # Shuffle arrays in a consistent way

X_train = []
y_train = []
X_test = []
y_test = []

def split_data():
    global titles, categories
    global X_train, y_train, X_test, y_test,labels
    N = len(titles)
    Ntrain = int(N * 0.7)
    # Let's shuffle the data
    titles, categories = shuffle(titles, categories, random_state=0)
    X_train = titles[:Ntrain]
    y_train = categories[:Ntrain]
    X_test = titles[Ntrain:]
    y_test = categories[Ntrain:]

%time split_data()


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
Let's print a plot for the confussion matrix.
Can you write Python code for it?

import itertools
import matplotlib.pyplot as plt

def plot_confusion_matrix(cm, classes,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, '{:5.2f}'.format(cm[i, j]),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.colorbar()
    plt.show()

plot_confusion_matrix(cm, labels, title='Confusion matrix')


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
Let's see how long it takes to import the data by %time magic command.
Can you write Python code for it?


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
Now, we have to initialize some variables that will be used. They will be used to collect the news titles, its categories, as well as a list of the different possible categories (without repetitions).
Can you write Python code for it?

titles = [] # list of news titles
categories = [] # list of news categories
labels = [] # list of different categories (without repetitions)
nlabels = 4 # number of different categories
lnews = [] # list of dictionaries with two fields: one for the news and
            # the other for its category


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
The code for this section will be organized in two functions: one which imports the data and the other which counts the news in each category, its percentage and plots it.
Can you write Python code for it?

def import_data():
    global titles, labels, categories
    # importing news aggregator data via Pandas (Python Data Analysis Library)
    news = pd.read_csv("uci-news-aggregator.csv")
    # function 'head' shows the first 5 items in a column (or
    # the first 5 rows in the DataFrame)
    print(news.head())
    categories = news['CATEGORY']
    titles = news['TITLE']
    labels = sorted(list(set(categories)))


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
Time required to split data is 1.28 seconds. Now let's analyze the proportion of news categories in the training set.
Can you write Python code for it?


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
To compare the predicted labels to the corresponding set of true labels we will use the method accuracy_score from scikit-learn, which gives an accuracy over 0.92
Can you write Python code for it?


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
Train and test the classifier
In order to train and test the classifier, the first step should be to tokenize and count the number of occurrence of each word that appears into the news'titles using for that CountVectorizer class. Each term found is assigned a unique integer index.
Then the counters will be transformed to a TF-IDF representation using TfidfTransformer class. The last step creates the Multinomial Naive Bayes classifier.
In order to make the training process easier, scikit-learn provides a Pipeline class that behaves like a compound classifier.
The metrics module allows to calculate score functions, performance metrics and pairwise metrics and distance computations. F1-score can be interpreted as a weighted average of the precision and recall.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn import metrics
import numpy as np
import pprint

# lmats = [] # list of confussion matrix
nrows = nlabels
ncols = nlabels
# conf_mat_sum = np.zeros((nrows, ncols))
# f1_acum = [] # list of f1-score

def train_test():
    global X_train, y_train, X_test, y_test, labels
    #lmats, \
     #       conf_mat_sum, f1_acum, ncategories
    text_clf = Pipeline([('vect', CountVectorizer()),
                         ('tfidf', TfidfTransformer()),
                         ('clf', MultinomialNB()),
                         ])
    text_clf = text_clf.fit(X_train, y_train)
    predicted = text_clf.predict(X_test)
    return predicted

%time predicted = train_test()


I am working on the file "News_Categorization_MNB 10-oct-2017.ipynb".
The first task is:
We can see that, although the metrics (precision, recall and f1-score) in average give us 0.92, the results for category e (entertainment) are even better.
Confusion matrix allows to detect if a classification algorithm is confusing two or more classes if you have an unequal number of observations in each class as in this case. An ideal classifier with 100% accuracy would produce a pure diagonal matrix which would have all the points predicted in their correct class. In case of class imbalance, confusion matrix normalization by class support size (number of elements in each class) can be interesting in order to have a visual interpretation of which class is being misclassified.
Can you write Python code for it?

mat = metrics.confusion_matrix(y_test, predicted,labels=labels)
cm = mat.astype('float') / mat.sum(axis=1)[:, np.newaxis]
cm


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
Let's print a plot for the confussion matrix.
Can you write Python code for it?


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
Let's see how long it takes to import the data by %time magic command.
Can you write Python code for it?


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
Let's show a pie plot with the proportion of news by category.
Can you write Python code for it?

import pylab as pl # useful for drawing graphics

def categories_pie_plot(cont,tit):
    global labels
    sizes = [cont[l] for l in labels]
    pl.pie(sizes, explode=(0, 0, 0, 0), labels=labels,
        autopct='%1.1f%%', shadow=True, startangle=90)
    pl.title(tit)
    pl.show()

categories_pie_plot(cont,"Plotting categories")


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
Now, we have to initialize some variables that will be used. They will be used to collect the news titles, its categories, as well as a list of the different possible categories (without repetitions).
Can you write Python code for it?


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
The time to import the dat was 3.54 seconds. Let's analyze how many news we have from the different categories and its percentage. We will use the class Counter from the collections library, which keeps track of how many values contains a collection. Then we will tabulate the different categories and its percentage via a DataFrame.
Can you write Python code for it?

from collections import Counter

def count_data(labels,categories):
    c = Counter(categories)
    cont = dict(c)
    # total number of news
    tot = sum(list(cont.values()))
    d = {
        "category" : labels,
        "news" : [cont[l] for l in labels],
        "percent" : [cont[l]/tot for l in labels]
    }

    print(pd.DataFrame(d))
    print("total \t",tot)

    return cont

cont = count_data(labels,categories)


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
Train and test the classifier
In order to train and test the classifier, the first step should be to tokenize and count the number of occurrence of each word that appears into the news'titles using for that CountVectorizer class. Each term found is assigned a unique integer index.
Then the counters will be transformed to a TF-IDF representation using TfidfTransformer class. The last step creates the Multinomial Naive Bayes classifier.
In order to make the training process easier, scikit-learn provides a Pipeline class that behaves like a compound classifier.
The metrics module allows to calculate score functions, performance metrics and pairwise metrics and distance computations. F1-score can be interpreted as a weighted average of the precision and recall.
Can you write Python code for it?


I am working on the file "News_Categorization_MNB.ipynb".
The first task is:
We can see that, although the metrics (precision, recall and f1-score) in average give us 0.92, the results for category e (entertainment) are even better.
Confusion matrix allows to detect if a classification algorithm is confusing two or more classes if you have an unequal number of observations in each class as in this case. An ideal classifier with 100% accuracy would produce a pure diagonal matrix which would have all the points predicted in their correct class. In case of class imbalance, confusion matrix normalization by class support size (number of elements in each class) can be interesting in order to have a visual interpretation of which class is being misclassified.
Can you write Python code for it?


I am working on the file "Newton Method.ipynb".
The first task is:
Quadratic approximation at x=a when a>1 or a<-1
In this case, the curvature at a is inverted and the point will move farther away from 0.
Can you write Python code for it?

# z is the approximation at x=2
a = 1.2
x = np.arange(-4, 4, 0.02)
z = f(a) + (x-a)*f_d(a) + (x-a)*(x-a)*f_d_d(a)/2
plt.plot(x, y, x, z)
#plt.axis([1, 3, 0, 0.5])
plt.show()


I am working on the file "Newton Method.ipynb".
The first task is:
The Taylor series (quadratic) approximation to the function at x=a is $f(a) + (x-a) f^{'}(a)+ (x-a)^2 f^{"}(a)/2$
$f^{'}(x) = -x e^{-x^2/2}$ and $f^{"}(x) = e^{-x^2/2} (x^2 - 1)$

$x-a = \Delta x = -f^{'}(a)/f^{"}(a) = \frac{a}{a^2-1}$, gives the maximum of the quadratic approximation.
Can you write Python code for it?

def f(x):
    return np.exp(-x*x/2)

# first derivative
def f_d(x):
    return -x * f(x)

# second derivative
def f_d_d(x):
    return (x*x-1) * f(x)


I am working on the file "Newton Method.ipynb".
The first task is:
Where is the value $1\sqrt 2$ coming from??
we have $x_n = a + \Delta x = a + \frac{a}{a^2-1} = \frac{a^3}{a^2-1}$
Let's plot the difference between magnitudes of previous value a and the next value $x_n$.
It is 0 at $1/\sqrt 2$
Can you write Python code for it?

def g(x):
    return np.abs(x) - np.abs(x**3/(x*x - 1.0))

p = np.arange(0,0.73, 0.02)
q = g(p)
plt.plot(p, q)
plt.show()
print g(0.5)


I am working on the file "Newton.ipynb".
The first task is:
Når stopper vi? Vi kan sjekke for x3:
Can you write Python code for it?


I am working on the file "Newton.ipynb".
The first task is:
Vi ser en løsning nær $4$, så vi velger $x_0=4$.
Vi deriverer og finner at $f'(x)=1/x-\sin(x)$.
Can you write Python code for it?


I am working on the file "Newton.ipynb".
The first task is:
Vi skal tilnærme løsningen av $\ln(x)+cos(x)-1=0$.
Can you write Python code for it?

def f(x):
    return log(x) + cos(x) - 1


I am working on the file "Newton.ipynb".
The first task is:
Vi tegner for å finne $x_0$:
Can you write Python code for it?

x = linspace(1,20,200)
y = f(x)
plot(x,y, lw=2)
plot([1,20],[0,0], lw=2, color='k')
ax = gca()
mpld3.display()


I am working on the file "Ng's Machine Learning Exercise 6.ipynb".
The first task is:
For the first experiment we'll use C=1 and see how it performs.
Can you write Python code for it?