I am working on the file "L2 - Nets.ipynb".
The first task is:
Выведем на экран обобщения наших чисел, которые получились в результате работы автоэнкодера, а затем проверим, насколько после полученного обобщения повысится качество классификации, используя поверх модели построенную ранее.
Can you write Python code for it?

import matplotlib.pyplot as plt

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
    ax = plt.subplot(2, n, i + 1)
    plt.imshow(X_train[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    ax = plt.subplot(2, n, i + 1 + n)
    plt.imshow(decoded_imgs[i].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()

np.random.seed(1337)  # for reproducibility

model = Sequential()
model.add(Dense(512,input_shape=(784,)))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dense(512))
model.add(Activation('tanh'))
model.add(Dropout(0.1))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer=Adadelta(), metrics=['accuracy'])

history = model.fit(decoded_imgs, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    verbose=0, validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, verbose=0)
print('Test accuracy:', score[1])


I am working on the file "L2 - Nets.ipynb".
The first task is:
Заметим, что по сравнению с однослойной моделью качество выросло значительно — примерно на 5,2 процента.
Зависимость от размерности первого линейного преобразования
Входной вектор имеет размер $l\cdot 785$, для корректности умножения вертикальный размер матрицы первого линейного преобразования равен 785, тогда, если её ширина $w$, то при перемножении получим матрицу $l\cdot w$. Таким образом, зависимость от размерности матрицы линейная, график зависимости точности от размерности приведён ниже.
Can you write Python code for it?

acc=[]
for layer_size in range(1,12):

    np.random.seed(1337)  # for reproducibility

    model = Sequential()
    model.add(Dense(2**layer_size, input_shape=(784,)))
    model.add(Activation('relu'))
    model.add(Dense(10))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

    history = model.fit(X_train, Y_train,
                        batch_size=batch_size, nb_epoch=nb_epoch,
                        verbose=0, validation_data=(X_test, Y_test))
    score = model.evaluate(X_test, Y_test, verbose=0)
    acc.append(score[1])

ax=plt.gca()
plt.plot(list((2**i for i in range (1,12))),acc)
ax.set_xscale("log",basex=2)
plt.show()


I am working on the file "L2 - Nets.ipynb".
The first task is:
Как видим на графике, точность (к тому же, и Q, так как мы штрафуем некорректные классификации) с отличными от 0 коэффициентами регуляризации лишь понижается, притом значительно. Вполне возможно, что для конкретной архитектуры и данных выигрыша стратегия использования регуляризации в целом не даёт.
2. Полносвязные нейронные сети
Можно задаться вопросом, а что будет, если поверх модели $Wx$ применить еще одну матрицу, а затем уже $\text{softmax}$? Ответ следующий, дополнительное умножение на матрицу слева $W_2\cdot(W_1x)$ ничего не даст, в итоге это та же самая модель $Wx$, где $W=W_2 W_1$. А вот, если между матрицами добавить третий "слой", то может получиться очень интерсная штука.
Давайте рассмотрим пример на множестве бинарных функций, $x \in {0, 1}^{2}$. Пусть есть модель
$$y = \theta(wx + b),$$
где $\theta$ - функция Хевисайда.
Рассмотрим такие $w$ и $b$, чтобы $y$ реализовала операторы and, or, not.

and: w=(1,1),b=-1,5.
Получаем $x_1+x_2-1,5$, и функция Хевисайда принимает значение 1, если $x_1+x_2\geq 1,5$, так как $x \in {0, 1}^{2}$,
то значение 1 получается только при векторе $(1,1)$.

or: $w=(1,1),b=-0,5$. Получаем $x_1+x_2-0,5$, и функция Хевисайда принимает значение 1, если $x_1+x_2\geq 0,5$, так как $x \in {0, 1}^{2}$, то значение 1 получается при векторах $(1,1)$, $(0,1)$, $(1,0)$.
not: $w=(-1),b=0,5$. Получаем $-x+0,5$, функция Хевисайда принимает значение 1, если $-x\geq -0,5 \Leftrightarrow x\leq 0,5$, то есть при $x=0$.
Существует функция XOR(x,y), которая принимает значение 1 при $x,y=(0,1)$ и $(1,0)$. Чтобы доказать линейную неразделимость множества элементов, на которых функция принимает значение 1 (а функция Хевисайда от скалярного произведения возвращает 1 в случае положительного скалярного произведения, то есть случая нахождения обеих точек по одну сторону от разделяющей прямой), составим систему неравенств:
$$\begin{array}{rcl} 0w_1+0w_2+b< 0 \Leftrightarrow b < 0 \ 0w_1+1w_2+b \geq 0 \Leftrightarrow b \geq -w_2 \ 1w_1+0w_2+b \geq 0 \Leftrightarrow b \geq -w_1 \ 1w_1+1w_2+b < 0...
Can you write Python code for it?

batch_size = 128
nb_epoch = 50

np.random.seed(1337)  # for reproducibility

model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu'))
model.add(Dense(10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer=SGD(), metrics=['accuracy'])

history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    verbose=0, validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])


I am working on the file "L2 - Nets.ipynb".
The first task is:
Получили точность 90,96% на отложенной выборке. Посмотрим, получится ли её улучшить позже.
Давайте кратко каснемся темы регуляризации. Как было рассказано на семинаре, подходы есть разные. Мы же остановимся на модификации оптимизируемого функционала.
$$\arg\min_W -\frac{1}{\mathcal{l}}\sum_{j=0}^{\mathcal{l}}\sum_{i=0}^{C} [y_j = i] \cdot \ln(p_i(x_j W)) + \lambda_1 L_1(W) + \lambda_2 L_2(W)$$

$L_1(W) = sum_{i,j} |w_{i,j}|$ - пытается занулить бесполезные признаки
$L_2(W) = sum_{i,j} w_{i,j}^2$ - не дает параметрам быть слишком большими.

Заметим, что в случае L1-нормы градиентом является просто матрица знаков компонент матрицы, L2-нормы — матрица удвоенных компонент.
Can you write Python code for it?

def sm_single_layer_train_gd_reg(w,X,Y, learn_rate,l1coeff,l2coeff, iter_num):
    for i in range(iter_num):
        prod=X@ w
        prod-=np.amax(prod,axis=1,keepdims=True)
        sm=np.exp(prod)/np.sum(np.exp(prod),axis=1,keepdims=True)
        g=(X.T @ (sm-Y))/X.shape[0]+l1coeff*np.sign(w)+l2coeff*2*w
        w -= learn_rate * g
    return w

reg_coeffs=np.arange(0,1,5e-2)
xgrid, ygrid = np.meshgrid(reg_coeffs, reg_coeffs)
zgrid=np.empty_like(xgrid)
for i in range(len(reg_coeffs)):
    for j in range(len(reg_coeffs)):
        w_start=np.zeros((X_train_single.shape[1],10))
        w_res=sm_single_layer_train_gd_reg(w_start,X_train_single,Y_train,1,reg_coeffs[i],reg_coeffs[j],10)
        zgrid[i][j]=calculate_acc(w_res,X_test_single,y_test)

plt.figure()
plt.xlabel('L1 coeff')
plt.ylabel('L2 coeff')
cont=plt.contourf(xgrid, ygrid, zgrid,1000, cmap=plt.cm.jet)
cbar = plt.colorbar(cont)
cbar.ax.set_ylabel('Accuracy')
plt.show()


I am working on the file "L2 - Nets.ipynb".
The first task is:
Результат даже ухудшился. Вывод можно сделать один: одну и ту же архитектуру во всех случаях использовать не стоит.
Can you write Python code for it?

layers = [
    Dense(encoding_dim, input_shape=(784,)),
    Activation('relu')
]

autoencoder = [Dense(784, input_shape=(encoding_dim,))]

model = Sequential(layers + autoencoder)
model.compile(loss='mean_squared_error',optimizer=Adadelta())

model.fit(X_train, X_train,batch_size=batch_size, nb_epoch=nb_epoch,verbose=0)
imgs = model.predict(X_train)

layers[0].trainable=False
layers[1].trainable=False
layers.append(Dense(512,input_shape=(encoding_dim,)))
layers.append(Activation('relu'))
layers.append(Dropout(0.2))
layers.append(Dense(512))
layers.append(Activation('relu'))
layers.append(Dense(512))
layers.append(Activation('tanh'))
layers.append(Dropout(0.1))
layers.append(Dense(10))
layers.append(Activation('softmax'))

model = Sequential(layers)
model.compile(loss='categorical_crossentropy',optimizer=Adadelta(),metrics=['accuracy'])

model.fit(X_train, Y_train,batch_size=batch_size,nb_epoch=nb_epoch, verbose=1,validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])


I am working on the file "L2 - Nets.ipynb".
The first task is:
Явную зависимость от числа слоём мы вывести не можем, однако при 7 слоях достигается оптимальная точность. Стоит, однако, заметить, что при ином виде оптимизации график выглядит иначе:
Can you write Python code for it?

acc=[]
for i in range(10):
    np.random.seed(1337)  # for reproducibility

    model = Sequential()
    model.add(Dense(512, input_shape=(784,)))
    model.add(Activation('relu'))
    for layer in range(i):
        model.add(Dense(512))
        model.add(Activation('relu'))
    model.add(Dense(10))
    model.add(Activation('softmax'))

    model.compile(loss='categorical_crossentropy', optimizer=RMSprop(), metrics=['accuracy'])

    history = model.fit(X_train, Y_train,
                        batch_size=batch_size, nb_epoch=nb_epoch,
                        verbose=0, validation_data=(X_test, Y_test))
    score = model.evaluate(X_test, Y_test, verbose=0)
    acc.append(score[1])

plt.plot(list(range(1,11)),acc)
plt.show()


I am working on the file "L2 - Preprocessing by NLTK.ipynb".
The first task is:
Collocations and n-grams
Collocations are good for getting a quick glimpse of what a text is about. Collocations(num = VAL) returns multi-word expressions that commonly co-occur. Notice that is not necessarily related to the frequency of the words.
Can you write Python code for it?


I am working on the file "L2 - Preprocessing by NLTK.ipynb".
The first task is:
Examining the text
Can you write Python code for it?

word = 'swallow'
doc.concordance(word, width=100, lines=30)

word = 'live'
doc.concordance(word, width=100, lines=30)

word = 'lived'
doc.concordance(word, width=100, lines=30)

word = 'CASTLE'
doc.concordance(word, width=100, lines=30)


I am working on the file "L2 - Preprocessing by NLTK.ipynb".
The first task is:
Note that the tokenization algorithm is pretty smart in not splitting experimental-interference into two tokens.
Can you write Python code for it?

stemmed_tokens = [porter.stem(tk) for tk in tokens]
print("   ".join(stemmed_tokens))

wnl = nltk.WordNetLemmatizer()
lemmatized_tokens = [wnl.lemmatize(tk) for tk in tokens]
print("   ".join(lemmatized_tokens))


I am working on the file "L2 - Preprocessing by NLTK.ipynb".
The first task is:
You probably wonder by proposes above is not lemmatized to propose. This is because lemmatize method has a default optional parameter pos = 'n' (i.e., treating the proposes as a noun). If we specify the correct POS tag ('v' for verb), the output will be correct.
Can you write Python code for it?

wnl.lemmatize('proposes', 'v')

wnl.lemmatize('is', 'v')


I am working on the file "L3 - POS Tagging.ipynb".
The first task is:
Various algorithms can be used to perform POS tagging. In general, the accuracy is pretty high (state-of-the-art can reach approximately 97%). However, there are still incorrect tags. We demonstrate this below.
Can you write Python code for it?

truths = [[(u'Pierre', u'NNP'), (u'Vinken', u'NNP'), (u',', u','), (u'61', u'CD'),
            (u'years', u'NNS'), (u'old', u'JJ'), (u',', u','), (u'will', u'MD'),
            (u'join', u'VB'), (u'the', u'DT'), (u'board', u'NN'), (u'as', u'IN'),
            (u'a', u'DT'), (u'nonexecutive', u'JJ'), (u'director', u'NN'),
            (u'Nov.', u'NNP'), (u'29', u'CD'), (u'.', u'.')],
        [(u'Mr.', u'NNP'), (u'Vinken', u'NNP'), (u'is', u'VBZ'), (u'chairman', u'NN'),
            (u'of', u'IN'), (u'Elsevier', u'NNP'), (u'N.V.', u'NNP'), (u',', u','),
            (u'the', u'DT'), (u'Dutch', u'NNP'), (u'publishing', u'VBG'),
            (u'group', u'NN'), (u'.', u'.'), (u'Rudolph', u'NNP'), (u'Agnew', u'NNP'),
            (u',', u','), (u'55', u'CD'), (u'years', u'NNS'), (u'old', u'JJ'),
            (u'and', u'CC'), (u'former', u'JJ'), (u'chairman', u'NN'), (u'of', u'IN'),
            (u'Consolidated', u'NNP'), (u'Gold', u'NNP'), (u'Fields', u'NNP'),
            (u'PLC', u'NNP'), (u',', u','), (u'was', u'VBD'), (u'named', u'VBN'),
            (u'a', u'DT'), (u'nonexecutive', u'JJ'), (u'director', u'NN'), (u'of', u'IN'),
            (u'this', u'DT'), (u'British', u'JJ'), (u'industrial', u'JJ'),
            (u'conglomerate', u'NN'), (u'.', u'.')],
        [(u'A', u'DT'), (u'form', u'NN'),
            (u'of', u'IN'), (u'asbestos', u'NN'), (u'once', u'RB'), (u'used', u'VBN'),
            (u'to', u'TO'), (u'make', u'VB'), (u'Kent', u'NNP'), (u'cigarette', u'NN'),
            (u'filters', u'NNS'), (u'has', u'VBZ'), (u'caused', u'VBN'), (u'a', u'DT'),
            (u'high', u'JJ'), (u'percentage', u'NN'), (u'of', u'IN'),
            (u'cancer', u'NN'), (u'deaths', u'NNS'),
            (u'among', u'IN'), (u'a', u'DT'), (u'group', u'NN'), (u'of', u'IN'),
            (u'workers', u'NNS'), (u'exposed', u'VBN'), (u'to', u'TO'), (u'it', u'PRP'),
            (u'more', u'RBR'), (u'than', u'IN'), (u'30', u'CD'), (u'years', u'NNS'),
            (u'ago', u'IN'), (u',', u','), (u'researchers', u'NNS'),
            (u'reported',...


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Binning
Equal-width Partitioning
We use the table with all null values filled
Can you write Python code for it?

pd.cut(df['Price'],5).head() # equally partition Price into 5 bins

# We could label the bins and add new column
df['Bin'] = pd.cut(df['Price'],5,labels=["Very Low","Low","Medium","High","Very High"])
df.head()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Equal-depth Partitioining
Can you write Python code for it?

pd.qcut(df['Price'],5).head() # Note the difference from the Equal-width Partitioning case

# Let's check the depth of each bin
df['Bin'] = pd.qcut(df['Price'],5,labels=["Very Low","Low","Medium","High","Very High"])
df.groupby('Bin').size()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Finding and Handling Missing Data
Can you write Python code for it?

# find rows that Price is null
df[pd.isnull(df['Price'])]

index_with_null = df[pd.isnull(df['Price'])].index
index_with_null


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Import data
Data is generated from Australia Bureau of statistics, some cells are removed (set to NaN) manually in order to serve this notebook.
Can you write Python code for it?

df = pd.read_csv('./asset/Median Price of Established House Transfers.txt', sep='\t') # row 3 has a null value
df.head()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Normalization
Can you write Python code for it?

df = pd.read_csv('./asset/Median Price of Established House.txt', sep='\t')
df.head()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Obviously, none of the above solutions are appropriate.
A better way to deal with the null value is to replace them with the mean value of the prices of the corresponding city over the whole year.
Can you write Python code for it?

df["Price"] = df.groupby("City").transform(lambda x: x.fillna(x.mean()))
df.ix[index_with_null]


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Robust Scaler
Can you write Python code for it?

from sklearn import preprocessing

min_max_scaler = preprocessing.RobustScaler()
x_scaled = min_max_scaler.fit_transform(df[df.columns[1:5]]) # we need to remove the first column

df_robust = pd.DataFrame(x_scaled)
df_robust.insert(0, 'City', df.City)
df_robust


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Smoothing
Smoothing by Bin Means
Can you write Python code for it?

df.head()

df['Price-Smoothing-mean'] = df.groupby('Bin')['Price'].transform('mean')
df.head()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Smoothing by Bin Max
Can you write Python code for it?

df['Price-Smoothing-max'] = df.groupby('Bin')['Price'].transform('max')
df.head()


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
Standard Scaler
Can you write Python code for it?

from sklearn import preprocessing

min_max_scaler = preprocessing.StandardScaler()
x_scaled = min_max_scaler.fit_transform(df[df.columns[1:5]]) # we need to remove the first column

df_standard = pd.DataFrame(x_scaled)
df_standard.insert(0, 'City', df.City)
df_standard


I am working on the file "L3 - Preprocessing.ipynb".
The first task is:
We can specify a value (e.g., 0) to replace those null values, through the fillna() method
Can you write Python code for it?

df2 = df.fillna(0) # price value of row 3 is set to 0.0
df2.ix[index_with_null]
# df2.index in index_with_null


I am working on the file "L4 - Multiway Merge.ipynb".
The first task is:
Exercise
Implement the merge-way merge function multiway_merge(data, k, pagesize), where data is an unordered list of integers, k is the maximum number of runs to merge simultaneously, and pagesize is the maximum number of integers in the initial runs.
Can you write Python code for it?

def multiway_merge(data, k, pagesize):
    p('{}-way merge, initial page size = {}'.format(k, pagesize))
    ###
    ### INSERT YOUR OWN CODE
    ###


I am working on the file "L4 - Multiway Merge.ipynb".
The first task is:
If you have implemented it correctly, if you run the code below, you should see the following output.
Can you write Python code for it?

import random

random.seed(1999)
n = 23
data = [random.randint(1, 100) for _ in range(n)]

sorted_data = multiway_merge(data, 3, 2)
print(sorted_data)
print()

sorted_data = multiway_merge(data, 2, 2)
print(sorted_data)


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Although we cannot remember what happened on these 7 days (can you?), it seems the predictions are probably correct. E.g.,
* the first instance is predicted as 'high' and it is a summer day, without rain or even cloud;
* the last instance is predicted as 'low' as there was heavy rainfall, as Rain = 61.0 (mm). Note that the model didn't use the rain feature at all!
Visualize of the Decision Tree
You need to install pydotplus (use pip install pydotplus) and graphviz on your computer to get the following code run
Can you write Python code for it?

import pydotplus
import sys

# sys.path.append("C:\\Program Files (x86)\\Graphviz2.38\bin")

dotfile = StringIO()
export_graphviz(dtree, out_file = dotfile, feature_names = X_train.columns)
graph = pydotplus.graph_from_dot_data(dotfile.getvalue())
Image(graph.create_png())


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Apparently the above tree is overfitting. One way to deal with it is to change the maximum height of the decision tree.
Can you write Python code for it?

def experiment(train, test, features, depth=5):
    X_train = train[features]
    y_train = train.Sun_level
    clf = DecisionTreeClassifier(criterion = "gini", max_depth = depth, random_state = RANDOM_SEED)
    dtree = clf.fit(X_train, y_train)
    err_training = dtree.score(X_train,y_train)
    X_test = test[features]
    y_test = test.Sun_level
    err_testing = dtree.score(X_test,y_test)
    err_diff = err_training - err_testing
    print('{}, {}, {}'.format(err_training, err_testing, err_diff))
    return err_training, err_testing

def evaluate(features, repeat_times = 10, depth = 5):
    print('features: {}'.format(features))
    print('max_depth: {}\n'.format(depth))
    total_err_training = 0
    total_err_testing = 0
    for i in range(repeat_times):
        train, test = train_test_split(data, test_size = 0.2, random_state = RANDOM_SEED + i)
        err_training, err_testing = experiment(train, test, features, depth)
        total_err_training += err_training
        total_err_testing += err_testing
        print('==============')
    print('avg. training error:\t{}'.format(total_err_training/repeat_times))
    print('avg. testing error:\t{}'.format(total_err_testing/repeat_times))
    print('avg. difference:\t{}'.format((total_err_training - total_err_testing)/repeat_times))
    print('============================')

feature_list = ['CLD_at_9am', 'CLD_at_3pm', 'RH_at_3pm', 'RH_at_9am', 'Temps_min', 'Temps_diff', 'Month']
evaluate(feature_list, 10, 6)

evaluate(feature_list, 10, 5)

evaluate(feature_list, 10, 4)

evaluate(feature_list, 10, 3)


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Build Decision Tree
We preserve 80% of the data as training data, and the rest will be used to tune the classifier.
Can you write Python code for it?

train, test = train_test_split(data, test_size = 0.2)


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Evaluations and Investigations of the Trained Decision Tree
Firstly let's see the accuracy of the decision tree on the training dataset
Can you write Python code for it?

dtree.score(X_train,y_train) # output the accuracy of the trained decision tree


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
If you have domain knowledge, you will know that the difference between max and min temperature might be highly correlated to sun hours. Let us add such column.
Can you write Python code for it?

data['Temps_diff'] = data['Temps_max'] - data['Temps_min']
corr(data)


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Import Data
Below is the Daily Weather Observations of Sydney, New South Wales between Aug 2015 and Aug 2016.
Can you write Python code for it?

data = pd.read_csv('./asset/Daily_Weather_Observations.csv', sep=',')
print(data.shape)
data.head()


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Let's see the accuracy on the testing dataset
Can you write Python code for it?


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Predict the missing Sun_level value
Can you write Python code for it?

X = data[feature_list]
y = data.Sun_level
clf = DecisionTreeClassifier(criterion = "gini", max_depth = 4, random_state = RANDOM_SEED)
dtree = clf.fit(X, y)

data_missing_sun_hours['Temps_diff'] = data_missing_sun_hours['Temps_max'] - data_missing_sun_hours['Temps_min']

X_pred = data_missing_sun_hours[feature_list]
data_missing_sun_hours['Sun_level_pred'] = dtree.predict(X_pred)
data_missing_sun_hours


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Preprocessing (Handling Missing Values)
From time to time, observations will not be available, for a variety of reasons. We need to handle missing values before training a classifier.
Can you write Python code for it?

# output all rows with missing values
data[data.isnull().any(axis=1)]


I am working on the file "L5 - Decision Tree.ipynb".
The first task is:
Select Feature(s)
Can you write Python code for it?

def corr(data):
    c = data.corr()
    df = c.Sun_level_num.to_frame()
    df['abs'] = abs(df['Sun_level_num'])
    df.sort_values(by = 'abs', ascending=False)
    print(df.sort_values(by = 'abs', ascending=False))

# we need to convert labels (string) into numeric to get the correlation
labels = [0,1,2]
data['Sun_level_num'] = pd.cut(data.Sun_hours, [-1,5,10,25], labels=labels)
data[['Sun_level_num']] = data[['Sun_level_num']].apply(pd.to_numeric)
corr(data)
# c = data.corr()
# c.sort_values(by='Sun_level_num', ascending=True)['Sun_level_num']


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
CV on Naive Bayes Classifier
Can you write Python code for it?

train_error_total = 0
test_error_total = 0
for train, test in kf:
    train_X = X.iloc[train]
    test_X = X.iloc[test]
    train_y = y.iloc[train]
    test_y = y.iloc[test]
    train_error, test_error = test_Gaussian_NB(train_X, train_y, test_X, test_y)
    train_error_total += train_error
    test_error_total += test_error

print('===================')
print('avg. training error (Gaussian NB):\t{}'.format(train_error_total/n_folds))
print('avg. testing error (Gaussian NB):\t{}'.format(test_error_total/n_folds))


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
Cross Validation
We use K-fold cross validation to get a reliable estimate of how well a model performs on unseen data (i.e., the generalization error). This helps to (1) determine which model works well, or (2) how to set the values for the hyper parameters for a model. We will see an example of the latter usage in finding the best $k$ for kNN classifiers.
Can you write Python code for it?

n_folds = 10
kf = KFold(n=len(X), n_folds=n_folds, shuffle=True, random_state=42)

def test_Gaussian_NB(train_X, train_y, test_X, test_y, debug_flag = False):
    gnb = GaussianNB()
    gnb.fit(train_X, train_y)
    train_error = gnb.score(train_X, train_y)
    test_error = gnb.score(test_X, test_y)
    if debug_flag:
        print('=============')
        print('training error:\t{}'.format(train_error))
        print('testing error:\t{}'.format(test_error))
    return train_error, test_error

def test_KNN(train_X, train_y, test_X, test_y, n_neighbors=3, weights='uniform', p=2, debug_flag = False):
    neigh = KNeighborsClassifier(n_neighbors=n_neighbors, weights=weights, p=p)
    neigh.fit(train_X, train_y)
    train_error = neigh.score(train_X, train_y)
    test_error = neigh.score(test_X, test_y)
    if debug_flag:
        print('=============')
        print('training error:\t{}'.format(train_error))
        print('testing error:\t{}'.format(test_error))
    return train_error, test_error


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
Data Preprocessing
We use the same data resource as in L5, i.e., Daily Weather Observations of Sydney, New South Wales between Aug 2015 and Aug 2016.
We will handle the missing values, and category Sun_hours into three levels: High(>10), Med(>5 and <=10), and Low(<=5) as we did in L5.
Column Meanings
| Heading         | Meaning                                                  | Units               |
|-----------------|----------------------------------------------------------|---------------------|
| Day             | Day of the week                                          | first two letters   |
| Temps_min       | Minimum temperature in the 24 hours to 9am.              | degrees Celsius     |
| Temps_max       | Maximum temperature in the 24 hours from 9am.            | degrees Celsius     |
| Rain            | Precipitation (rainfall) in the 24 hours to 9am.         | millimetres         |
| Evap            | Class A pan evaporation in the 24 hours to 9am           | millimetres         |
| Sun_hours       | Bright sunshine in the 24 hours to midnight              | hours               |
| Max_wind_dir    | Direction of strongest gust in the 24 hours to midnight  | 16 compass points   |
| Max_wind_spd    | Speed of strongest wind gust in the 24 hours to midnight | kilometres per hour |
| Max_wind_time   | Time of strongest wind gust                              | local time hh:mm    |
| Temp_at_9am     | Temperature at 9 am                                      | degrees Celsius     |
| RH_at_9am       | Relative humidity at 9 am                                | percent             |
| CLD_at_9am      | Fraction of sky obscured by cloud at 9 am                | eighths             |
| Wind_dir_at_9am | Wind direction averaged over 10 minutes prior to 9 am    | compass points      |
| Wind_spd_at_9am | Wind speed averaged over 10 minutes prior to 9 am        | kilometres per hour |
| MSLP_at_9am     | Atmospheric pressure reduced to mean sea level at 9 am   | hectopascals        |
| Temp_at_3pm     |...
Can you write Python code for it?

data = pd.read_csv('./asset/Daily_Weather_Observations.csv', sep=',')
data_missing_sun_hours = data[pd.isnull(data['Sun_hours'])]
data = data[pd.notnull(data['Sun_hours'])]

labels = ['Low','Med','High']
data['Sun_level'] = pd.cut(data.Sun_hours, [-1,5,10,25], labels=labels)

data = data.dropna(subset = ['CLD_at_9am', 'Max_wind_dir', 'Max_wind_spd', 'Max_wind_dir'])
bitmap1 = data.Evap.notnull()
bitmap2 = bitmap1.shift(1)
bitmap2[0] = True
data = data[bitmap1 & bitmap2]

data['Temps_diff'] = data['Temps_max'] - data['Temps_min']

print(data.shape)


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
KNN Classifier
Before classification, we need to normalize the data first.
Can you write Python code for it?

min_max_scaler = preprocessing.MinMaxScaler()
X_scaled = min_max_scaler.fit_transform(X)
X_scaled.shape


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
The following parameters affects the performance (accuracy) of the classifier
 * n_neighbors: Number of neighbors to use
 * weights: weight function of the points. Possible values are:
   * ‘uniform’ : uniform weights.
   * ‘distance’ : weight points by the inverse of their distance.
   * [callable] : a user-defined weight function.
 * metric: The distance metric to use. The default metric is minkowski.
 * p: Power parameter for the Minkowski metric (assume we use the default metric). Usual choices are:
   * p = 2: Euclidean Distance
   * p = 1: Manhattan Distance
Can you write Python code for it?

neigh = KNeighborsClassifier(n_neighbors=3,weights='uniform',p=2)
neigh.fit(X_scaled, y)
neigh.score(X_scaled, y)


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
We also need to normalize data, but be aware that we need to make sure that testing data and training data are normalized in the same way (e.g., use min_max_scaler.transform())
Can you write Python code for it?

data_missing_sun_hours['Temps_diff'] = data_missing_sun_hours['Temps_max'] - data_missing_sun_hours['Temps_min']
test_data = data_missing_sun_hours[feature_list]
test_data_scaled = min_max_scaler.transform(test_data)
test_data


I am working on the file "L6 - GaussianNB, KNN, and Cross-Validation.ipynb".
The first task is:
We use CLD_at_9am, CLD_at_3pm, RH_at_9am, RH_at_3pm, and Temps_diff as features.
Can you write Python code for it?

feature_list = ['CLD_at_9am', 'CLD_at_3pm', 'RH_at_9am', 'RH_at_3pm', 'Temps_diff']


I am working on the file "L7 - Multinomial and Bernoulli Naive Bayes Classifiers.ipynb".
The first task is:
Bernoulli Naive Bayes Classifier
Note that since we have the count matrix for words in the vocabulary, we can binarize it to be used in a Bernoulli NB classifier.
Can you write Python code for it?

nb = BernoulliNB(alpha=1, binarize=0.0) # every value >0.0 will be binarized to 1
nb.fit(X, y)
nb.score(X, y)

def test_Bernoulli_NB(train_X, train_y, test_X, test_y, alpha=1, debug = False):
    nb = BernoulliNB(alpha=alpha)
    nb.fit(train_X, train_y)
    train_error = nb.score(train_X, train_y)
    test_error = nb.score(test_X, test_y)
    if debug:
        print('training error:\t{}'.format(train_error))
        print('testing error:\t{}'.format(test_error))
    return train_error, test_error

def cv_BernoulliNB(alpha = 1.0):
    train_error_total = 0
    test_error_total = 0
    for train, test in kf:
        train_X = X[train]
        test_X = X[test]
        train_y = y[train]
        test_y = y[test]
        train_error, test_error = test_Bernoulli_NB(train_X, train_y, test_X, test_y, alpha)
        train_error_total += train_error
        test_error_total += test_error
    return train_error_total/n_folds, test_error_total/n_folds
#     print('===================')
#     print('avg. training error:\t{}'.format(train_error_total/n_folds))
#     print('avg. testing error:\t{}'.format(test_error_total/n_folds))

def cv_plot_BernoulliNB():
    cv_res = []
    rng = sorted([0.5] + list(range(0, 10)))
    for i in rng:
        train_error, test_error = cv_BernoulliNB(i)
        cv_res.append([i, train_error, test_error])
    cv_res_arr = np.array(cv_res)
    plt.figure(figsize=(16,9))
    plt.title('Error vs. alpha')
    plot_train, = plt.plot(cv_res_arr[:,0], cv_res_arr[:,1], label='training')
    plot_test, = plt.plot(cv_res_arr[:,0], cv_res_arr[:,2], label='testing')
    plt.legend(handles=[plot_train, plot_test])
    plt.ylim((min(min(cv_res_arr[:,1]), min(cv_res_arr[:,2])) - 0.01, max(max(cv_res_arr[:,1]), max(cv_res_arr[:,2]))+0.01))
    plt.xticks(rng)

cv_plot_BernoulliNB()


I am working on the file "L7 - Multinomial and Bernoulli Naive Bayes Classifiers.ipynb".
The first task is:
Build Multinomial Naive Bayes Classifier
alpha is the additive (Laplace/Lidstone) smoothing parameter. Its default value is 1.0 -- we will see that this works pretty well for our problem later.
Can you write Python code for it?

nb = MultinomialNB(alpha=1)
nb.fit(X, y)
nb.score(X, y)


I am working on the file "L7 - Multinomial and Bernoulli Naive Bayes Classifiers.ipynb".
The first task is:
Let's use cross validation to explore how alpha affects the performance of the classifier
Can you write Python code for it?

n_folds = 10
kf = KFold(n=X.shape[0], n_folds=n_folds, shuffle=True, random_state=42)

def test_Multinomial_NB(train_X, train_y, test_X, test_y, alpha=1, debug = False):
    nb = MultinomialNB(alpha=alpha)
    nb.fit(train_X, train_y)
    train_error = nb.score(train_X, train_y)
    test_error = nb.score(test_X, test_y)
    if debug:
        print('training error:\t{}'.format(train_error))
        print('testing error:\t{}'.format(test_error))
    return train_error, test_error

def cv_MultinomialNB(alpha = 1.0):
    train_error_total = 0
    test_error_total = 0
    for train, test in kf:
        train_X = X[train]
        test_X = X[test]
        train_y = y[train]
        test_y = y[test]
        train_error, test_error = test_Multinomial_NB(train_X, train_y, test_X, test_y, alpha)
        train_error_total += train_error
        test_error_total += test_error
    return train_error_total/n_folds, test_error_total/n_folds
#     print('===================')
#     print('avg. training error:\t{}'.format(train_error_total/n_folds))
#     print('avg. testing error:\t{}'.format(test_error_total/n_folds))

def cv_plot_MultinomialNB():
    cv_res = []
    rng = sorted([0.5] + list(range(0, 10)))
    for i in rng:
        train_error, test_error = cv_MultinomialNB(i)
        cv_res.append([i, train_error, test_error])
    cv_res_arr = np.array(cv_res)
    plt.figure(figsize=(16,9))
    plt.title('Error vs. alpha')
    plot_train, = plt.plot(cv_res_arr[:,0], cv_res_arr[:,1], label='training')
    plot_test, = plt.plot(cv_res_arr[:,0], cv_res_arr[:,2], label='testing')
    plt.legend(handles=[plot_train, plot_test])
    plt.ylim((min(min(cv_res_arr[:,1]), min(cv_res_arr[:,2])) - 0.01, max(max(cv_res_arr[:,1]), max(cv_res_arr[:,2]))+0.01))
    plt.xticks(rng)

cv_plot_MultinomialNB()


I am working on the file "L7 - Multinomial and Bernoulli Naive Bayes Classifiers.ipynb".
The first task is:
Use the first SMS (i.e., X[0]) to show how to get the feature names and values.
Can you write Python code for it?

arr = X[0].toarray()
for i in range(len(arr[0])):
    if arr[0][i] > 0:
        print('{}:{}'.format(vectorizer.feature_names_[i], arr[0][i]))


I am working on the file "L7 - Multinomial and Bernoulli Naive Bayes Classifiers.ipynb".
The first task is:
We then trainsform the features into a sparse martix.
Can you write Python code for it?

encoder = LabelEncoder()
vectorizer = DictVectorizer(dtype=float, sparse=True)
X, y = list(zip(*features_and_labels))
X = vectorizer.fit_transform(X)
    # print(X)
    # print(vectorizer.feature_names_)
y = encoder.fit_transform(y)

X


I am working on the file "L7126/Classification.ipynb".
The first task is:
Did it work? Were you able to view the reconstructed images?
Next, we are going preprocess our data from the TensorFlow record file and create batches for training. What should the outputs be for this function? What variables in read_and_decode to you want to feed into a neural network?
You can look here for a hint.
Can you write Python code for it?

IMAGE_HEIGHT = 32
IMAGE_WIDTH = 32
BATCH_SIZE = 32
NUM_CLASSES=2
NUM_EXAMPLES_PER_EPOCH_FOR_TRAIN=264#len(img_list)
def read_and_decode(filename_queue):

    reader = tf.TFRecordReader()

    _, serialized_example = reader.read(filename_queue)

    features = tf.parse_single_example(
      serialized_example,
      # Defaults are not specified since both keys are required.
      features={
        'height': tf.FixedLenFeature([], tf.int64),
        'width': tf.FixedLenFeature([], tf.int64),
        'depth': tf.FixedLenFeature([], tf.int64),
        'label':tf.FixedLenFeature([], tf.int64),
        'image_raw': tf.FixedLenFeature([], tf.string)
        })

    # Convert from a scalar string tensor to a uint8 tensor

    image = tf.decode_raw(features['image_raw'], tf.uint8)
    height = tf.cast(features['height'], tf.int32)
    width = tf.cast(features['width'], tf.int32)
    label = tf.cast(features['label'], tf.int32)
    image_shape = tf.stack([height, width, 3])
    #image.set_shape(image_shape)

    image = tf.reshape(image, image_shape)

    #Randomly cropping image
    random_crop_image=tf.random_crop(image, size=[IMAGE_HEIGHT,IMAGE_WIDTH,3], seed=None, name=None)

    min_after_dequeue = 500
    capacity = min_after_dequeue + 3 * BATCH_SIZE
    shuffled_img, shuffled_label = tf.train.shuffle_batch([random_crop_image, label],
                                                 batch_size=BATCH_SIZE ,
                                                 capacity=capacity,
                                                 num_threads=4,
                                                 min_after_dequeue=min_after_dequeue)

    return shuffled_img,shuffled_label


I am working on the file "L7126/Classification.ipynb".
The first task is:
Now you are ready to start training. Before you train this network with the data, modify the code below so that the network is tested with the training data on a cadence of 50 iterations. If you want a hint look here.
Please note this training may take 5 minutes or more to complete.
Can you recall all of the work we did to prepare our data and configure the network?
Can you write Python code for it?

tf.get_variable_scope().reuse_variables()
init_op = tf.group(tf.global_variables_initializer(),
                   tf.local_variables_initializer())

########################## Calculate Loss and Accuracy ####################
softmax_linear=cnn_model(tf.to_float(img))
cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y_,
        logits=softmax_linear, name='cross_entropy_per_example')
#Loss
cross_entropy_mean = tf.reduce_mean(cross_entropy, name='cross_entropy')

train_step = tf.train.GradientDescentOptimizer(learning_rate=0.00001).minimize(cross_entropy) #weight updater
correct_prediction=tf.equal(tf.cast(tf.argmax(softmax_linear,1),tf.int32),y_)
accuracy=tf.reduce_mean(tf.cast(correct_prediction,tf.float32))

#Add ops to save and restore all the variables
saver = tf.train.Saver()

with tf.Session() as sess:

        sess.run(init_op)
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(coord=coord)

        for i in xrange(500):
                image,label=sess.run([images,labels])
                _,loss=sess.run([train_step,cross_entropy_mean],
                                  feed_dict={img:image, y_:label})

                if i % 50 == 0:
                    val_image,val_label=sess.run([val_images,val_labels])
                    ################# FIXME #################
                    # Calculate the accruacy of the network using the validation images and label
                    acc2=sess.run([accuracy],
                        feed_dict={img:val_image, y_:val_label})
                    print("iteration ",i, "loss ", loss, 'val accuracy', acc2)

        # Save the variables to disk.
        save_path = saver.save(sess, "my_first_cnn")
        print("Model saved in file: %s" % save_path)
        coord.request_stop()
        coord.join(threads)
        sess.close()


I am working on the file "L7126/Classification.ipynb".
The first task is:
The next cell contains the CNN we are going to use. For this example, we are applying the CIFAR network. This network is comprised of convolutional, pooling, normalization, ReLU activation, and fully connected layers. There is more than one place to view examples of this CNN, including the TensorFlow documentation as well as Caffe.
For the model function below, cnn_model, what should the output be?
Need a hint? Look here.
Can you write Python code for it?

def cnn_model(img):
    #CIFAR10 Network
    conv1 = tf.nn.conv2d(img, variables_dict['conv1_weights'], [1, 1, 1, 1], padding='SAME')
    act_conv1=tf.nn.relu(tf.nn.bias_add(conv1,variables_dict['conv1_biases']))

    ####################### Pooling Layer ##################
    pool1=tf.nn.max_pool(act_conv1,ksize=[1,3,3,1],strides=[1,2,2,1],padding='SAME')
    norm1 = tf.nn.lrn(pool1, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75)
    ###################### Conv Layer 2 ####################

    conv2=tf.nn.conv2d(norm1, variables_dict['conv2_weights'], [1, 1, 1, 1], padding='SAME')
    act_conv2=tf.nn.relu(tf.nn.bias_add(conv2,variables_dict['conv2_biases']))

    norm2 = tf.nn.lrn(conv2, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75, name='norm2')

    ########################## Pooling Layer ######################

    pool2 = tf.nn.max_pool(norm2, ksize=[1, 3, 3, 1],strides=[1, 2, 2, 1], padding='SAME', name='pool2')

    ######################### Reshape ############################
    reshape = tf.reshape(pool2, [BATCH_SIZE, -1])
    dim = reshape.get_shape()[1].value #dim==pool2[0]*pool[1]*#featuremaps==4096
    linear1 = tf.nn.relu(tf.matmul(reshape, variables_dict['linear1_weights']) + variables_dict['linear1_biases'],
                                 name="linear1")
    linear2 = tf.nn.relu(tf.matmul(linear1, variables_dict['linear2_weights']) + variables_dict['linear2_biases'],
                                name="linear2")
    softmax_linear = tf.add(tf.matmul(linear2, variables_dict['linear3_weights']),variables_dict['linear3_biases'],
                                name="softmax_output")
    return softmax_linear


I am working on the file "L7126/Classification.ipynb".
The first task is:
There are many things we could do to improve this accuracy. Can you name three?
Can you write Python code for it?

#Free our GPU memory before proceeding to the next part of the lab
import os
os._exit(00)


I am working on the file "L7126/Classification.ipynb".
The first task is:
We can look a data from the last batch processed by our network using the image variable from our training or the val_image variable from the validation set. To view an image change the item parameter to a value within the domain of the batch, 0 -BATCH_SIZE-1.
If you would like to see an example look here.
Can you write Python code for it?

#Let's take a look at some of the data from the batch.

item=0
plt.imshow(image[item, :, :, :])
if label[item]==1:
    print("This is part of a Vanh Gogh painting")
else:
    print("This is not part of a Vanh Gogh painting")


I am working on the file "L7126/Classification.ipynb".
The first task is:
When training varialbles are used to hold and update network parameters. The variable_dict is a dictionary of all the variables for the CIFAR CNN, cnn_model.
Can you write Python code for it?

tf.reset_default_graph()

variables_dict = {
    "conv1_weights":tf.Variable(tf.truncated_normal([5, 5, 3, 64],stddev=1.0),name="conv1_weights"),
    "conv1_biases": tf.Variable(tf.constant(0.1, shape=[64]), name="conv1_biases"),
    "conv2_weights":tf.Variable(tf.random_normal([5, 5, 64, 64],stddev=1.0),name="conv2_weights"),
    "conv2_biases": tf.Variable(tf.constant(0.1,shape=[64]), name="conv2_biases"),
    "linear1_weights":tf.Variable(tf.truncated_normal([4096,384],stddev=1.0),name="linear1_weights"),
    "linear1_biases": tf.Variable(tf.constant(0.1,shape=[384]), name="linear1_biases"),
    "linear2_weights":tf.Variable(tf.truncated_normal([384,192],stddev=1.0),name="linear2_weights"),
    "linear2_biases": tf.Variable(tf.constant(0.1,shape=[192]), name="linear2_biases"),
    "linear3_weights":tf.Variable(tf.truncated_normal([192,NUM_CLASSES],stddev=1.0),name="linear2_weights"),
    "linear3_biases": tf.Variable(tf.constant(0.1,shape=[NUM_CLASSES]), name="linear2_biases"),
}

filename_queue = tf.train.string_input_producer([tfrecord_filename], num_epochs=200)
val_filename_queue = tf.train.string_input_producer([test_tfrecord_filename], num_epochs=200)

# tf placeholders
img = tf.placeholder(tf.float32,[BATCH_SIZE,IMAGE_HEIGHT,IMAGE_WIDTH,3])
y_=tf.placeholder(tf.int32,[BATCH_SIZE])

images, labels = read_and_decode(filename_queue)
val_images, val_labels=read_and_decode(val_filename_queue)


I am working on the file "L7126/MSCOCO_Captioning.ipynb".
The first task is:
How can you look at feature maps from the first convolutional layer? Look here if you need a hint.
Can you write Python code for it?

print(ENDPTS['vgg_16/conv1/conv1_1'][0].shape)
FEATUREMAPID=0
print('input image and feature map from conv1')
plt.subplot(1,2,1)
plt.imshow(resize_img)
plt.subplot(1,2,2)
plt.imshow(ENDPTS['vgg_16/conv1/conv1_1'][0][:,:,FEATUREMAPID])


I am working on the file "L7126/MSCOCO_Captioning.ipynb".
The first task is:
In the cell above we created three lists, one for the image_id, feature map. and caption. To verify that the indices of each list are aligned, display the image id and caption for one image.
Can you write Python code for it?

STRING='%012d' % image_id_key[0][0]
exp_image=ndimage.imread(TRAIN_IMAGE_PATH+'COCO_train2014_'+STRING+'.jpg')
plt.imshow(exp_image)
print('image_id ',image_id_key[:5])
print('the captions for this image ')
print(caption_to_id[:5])

num_steps=20
######################################################################
##Create a list of all of the sentences.
DatasetWordList=[]
for dataset_caption in caption_to_id:
        DatasetWordList+=str(dataset_caption).split()
#Determine number of distint words
distintwords=collections.Counter(DatasetWordList)
#Order words
count_pairs = sorted(distintwords.items(), key=lambda x: (-x[1], x[0])) #ascending order
words, occurence = list(zip(*count_pairs))
#DictionaryLength=occurence.index(4) #index for words that occur 4 times or less
words=['PAD','UNK','EOS']+list(words)#[:DictionaryLength])
word_to_id=dict(zip(words, range(len(words))))
#####################  Tokenize Sentence #######################
Tokenized=[]
for full_words in caption_to_id:
        EmbeddedSentence=[word_to_id[word] for word in full_words.split() if word in word_to_id]+[word_to_id['EOS']]
        #Pad sentences that are shorter than the number of steps
        if len(EmbeddedSentence)<num_steps:
            b=[word_to_id['PAD']]*num_steps
            b[:len(EmbeddedSentence)]=EmbeddedSentence
        if len(EmbeddedSentence)>num_steps:
            b=EmbeddedSentence[:num_steps]
        if len(b)==EmbeddedSentence:
            b=EmeddedSentence
        #b=[word_to_id['UNK'] if x>=DictionaryLength else x for x in b] #turn all words used 4 times or less to 'UNK'
        #print(b)
        Tokenized+=[b]

print("Number of words in this dictionary ", len(words))

#Tokenized Sentences
Tokenized[::2000]


I am working on the file "L7126/MSCOCO_Captioning.ipynb".
The first task is:
Questions
[1] Can the show_next_predicted_word function be used for deployment?
Probably not. Can you think of any reason why? Each predicted word is based on the previous ground truth word. In a deployment scenario, we will only have the feature map from our input image.
[2] Can you load your saved network and use it to generate a caption from a validation image?
The validation images are stored in /data/mscoco/val2014. A npy file of the feature vectors is stored /data/mscoco/val_vgg_16_fc7_100.npy. For a hint on how to add this look here.
[3] Do you need to calculate the loss or cost when only performing inference?
[4] Do you use dropout when performing inference?
Can you write Python code for it?

##Load and test our test set
val_load=np.load('/data/mscoco/val_vgg_16_fc7_100.npy').tolist()
val_ids=val_load.keys()

#Create 3 lists image_id, feature maps, and captions.
val_id_key=[]
val_map_to_id=[]
val_caption_to_id=[]
for observed_image in val_ids:
    val_id_key.append([observed_image])
    val_map_to_id.append(val_load[observed_image])

print('number of images ',len(val_ids))
print('number of captions ',len(val_map_to_id))


I am working on the file "L7126/MSCOCO_Captioning.ipynb".
The first task is:
The next cell contains functions for queuing our data and the RNN model. What should the output for each function be? If you need a hint look here.
Can you write Python code for it?

def data_queue(caption_input,feature_vector,batch_size,):

    train_input_queue = tf.train.slice_input_producer(
                    [caption_input, np.asarray(feature_vector)],num_epochs=10000,
                                    shuffle=True) #False before

    ##Set our train data and label input shape for the queue

    TrainingInputs=train_input_queue[0]
    FeatureVectors=train_input_queue[1]
    TrainingInputs.set_shape([num_steps])
    FeatureVectors.set_shape([len(feature_vector[0])]) #fc7 is 4096
    min_after_dequeue=1000000
    capacity = min_after_dequeue + 3 * batch_size
    #input_x, target_y
    tokenized_caption, input_feature_map = tf.train.batch([TrainingInputs, FeatureVectors],
                                                 batch_size=batch_size,
                                                 capacity=capacity,
                                                 num_threads=6)
    return tokenized_caption,input_feature_map

def rnn_model(Xconcat,input_keep_prob,output_keep_prob,num_layers,num_hidden):
#Create a multilayer RNN
#reuse=False for training but reuse=True for sharing
    layer_cell=[]
    for _ in range(num_layers):
        lstm_cell = tf.contrib.rnn.LSTMCell(num_units=num_hidden, state_is_tuple=True)
        lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell,
                                          input_keep_prob=input_keep_prob,
                                          output_keep_prob=output_keep_prob)
        layer_cell.append(lstm_cell)

    cell = tf.contrib.rnn.MultiRNNCell(layer_cell, state_is_tuple=True)
    outputs, last_states = tf.contrib.rnn.static_rnn(
        cell=cell,
        dtype=tf.float32,
        inputs=tf.unstack(Xconcat))

    output_reshape=tf.reshape(outputs, [batch_size*(num_steps),num_hidden]) #[12==batch_size*num_steps,num_hidden==12]
    pred=tf.matmul(output_reshape, variables_dict["weights_mscoco"]) +variables_dict["biases_mscoco"]
    return...


I am working on the file "L7126/MSCOCO_Captioning.ipynb".
The first task is:
We can use the function below to estimate how well the network is able to predict the next word in the caption. You can evaluate a single image and its caption from the last batch using the index of the batch. If you need a hint look here.
Please note that depending on the status of the neural network at the time it was saved, incomplete, incoherent, and sometimes inappropriate captions could be generated.
Can you write Python code for it?

def show_next_predicted_word(batch_id,batch_size,id_of_image,target_caption,predicted_caption,words,PATH):
        Target=[words[ind] for ind in target_caption[batch_id]]
        Prediction_Tokenized=np.argmax(predicted_caption[batch_id::batch_size],1)
        Prediction=[words[ind] for ind in Prediction_Tokenized]
        STRING2='%012d' % id_of_image
        img=ndimage.imread(PATH+STRING2+'.jpg')
        return Target,Prediction,img,STRING2

#You can change the batch id to a number between [0 , batch_size-1]
batch_id=0
image_id_for_predicted_caption=[x for x in range(len(Tokenized)) if target_caption[batch_id].tolist()== Tokenized[x]][0]

t,p,input_img,string_out=show_next_predicted_word(batch_id,batch_size,image_id_key[image_id_for_predicted_caption][0]
                                         ,target_caption,y_pred,words,TRAIN_IMAGE_PATH+'COCO_train2014_')
print('Caption')
print(t)
print('Predicted Words')
print(p)
plt.imshow(input_img)


I am working on the file "L7126/RecurrentNeuralNetwork.ipynb".
The first task is:
Compare the loss from our single and multi-layer RNN.
Can you write Python code for it?

## import matplotlib.pyplot as plt
plt.plot(plot_loss,'r.')
plt.plot(plot_loss2,'b--')
plt.legend(["1-Layer LSTM","Multi-Layer LSTM"])
plt.ylabel("loss")
plt.xlabel("iteration")
plt.title("Loss During Training")
plt.show()


I am working on the file "L7126/RecurrentNeuralNetwork.ipynb".
The first task is:
Create a single layer RNN with LSTMs and train it with a toy dataset.
Can you write Python code for it?

plot_loss=[]
num_hidden=24
num_steps=6
dict_length=8
batch_size=2
tf.reset_default_graph()

## Make Variables
variables_dict = {
    "weights1":tf.Variable(tf.truncated_normal([num_hidden,dict_length],stddev=1.0,dtype=tf.float32),name="weights1"),
    "biases1": tf.Variable(tf.truncated_normal([dict_length],stddev=1.0,dtype=tf.float32), name="biases1")}

# Create input data
small_dict=['EOS','i','will','walk','the','dog','cat','run']
X=np.array([[1,2,7,4,5,0],[1,2,3,4,6,0]],dtype=np.int32)
X_one_hot=tf.nn.embedding_lookup(np.identity(dict_length), X) #[batch,num_steps,dictionary_length][2,6,7]
y=np.zeros((batch_size,num_steps),dtype=np.int32)
y[:,:-1]=X[:,1:]
y_one_hot=tf.unstack(tf.nn.embedding_lookup(np.identity(dict_length), y),num_steps,1) #[batch,num_steps,dictionary_length][2,6,7]

y_target_reshape=tf.reshape(y_one_hot,[batch_size*num_steps,dict_length])

#Create our LSTM
cell = tf.contrib.rnn.LSTMCell(num_units=num_hidden, state_is_tuple=True)

outputs, last_states = tf.contrib.rnn.static_rnn(
    cell=cell,
    dtype=tf.float32,
    inputs=tf.unstack(tf.to_float(X_one_hot),num_steps,1))

output_reshape=tf.reshape(outputs, [batch_size*num_steps,num_hidden]) #[12==batch_size*num_steps,num_hidden==12]
pred=tf.matmul(output_reshape, variables_dict["weights1"]) +variables_dict["biases1"]
# Define loss and optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y_target_reshape))
optimizer = tf.train.AdamOptimizer(0.01).minimize(cost)

init_op = tf.group(tf.global_variables_initializer(),
                   tf.local_variables_initializer())

with tf.Session() as sess:

        sess.run(init_op)
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(coord=coord)
        for i in range(300):
            loss,_,y_target,y_pred,output=sess.run([cost,optimizer,y_target_reshape,pred,outputs])
            plot_loss.append([loss])

            if i% 25 ==0:
                print("iteration: ",i," loss: ",loss)
              ...


I am working on the file "L7126/RecurrentNeuralNetwork.ipynb".
The first task is:
Does the single word prediction cell above represent a deployment scenario?
No, the ground truth is being propagated through the network. The  predicted caption above represents a single predicted word based on the previous ground truth.
In the classification part we had a function for our network. If you were going to make a function for this network, what would you include?  What would your inputs and outputs be?
Can you write Python code for it?


I am working on the file "L7126/RecurrentNeuralNetwork.ipynb".
The first task is:
Train an RNN with MSCOCO Captions
Now we are going to try a slightly more complicated example. Let's use the Microsoft Common Objects in Context (MSCOCO) image captions to train an RNN. The cell below shows one way to read, format, and feed the data into TensorFlow. First, we will read the caption file, then we will remove the punctuation, and then train. Due to time constraints, we won't use the full dataset for this training. However, it would be easy to change this and train with more or the entire dataset. Can you see an easy way to do this?
Can you write Python code for it?

num_steps=20
## Read Training files
with open("/data/mscoco/captions_train2014.json") as data_file:
         data=json.load(data_file)

TotalNumberofCaptions=len(data['annotations'])

sentences=[]

##Create a list of all of the sentences.
for i in range(TotalNumberofCaptions):
        sentences+=[re.sub('[^A-Za-z0-9]+',' ',data['annotations'][i]['caption']).lower()]

TotalWordList=[]
for i in range(TotalNumberofCaptions):
        TotalWordList+=re.sub('[^A-Za-z0-9]+',' ',data['annotations'][i]['caption']).lower().split()

#Determine number of distint words
distintwords=collections.Counter(TotalWordList)
#Order words
count_pairs = sorted(distintwords.items(), key=lambda x: (-x[1], x[0])) #ascending order
words, occurence = list(zip(*count_pairs))
DictionaryLength=occurence.index(4) #index for words that occur 4 times or less
words=['PAD','UNK','EOS']+list(words[:DictionaryLength])
word_to_id=dict(zip(words, range(len(words))))
#Tokenize Sentence
Tokenized=[]
for full_words in sentences:
        EmbeddedSentence=[word_to_id[word] for word in full_words.split() if word in word_to_id]+[word_to_id['EOS']]
        #Pad sentences that are shorter than the number of steps
        if len(EmbeddedSentence)<num_steps:
            b=[word_to_id['PAD']]*num_steps
            b[:len(EmbeddedSentence)]=EmbeddedSentence
        if len(EmbeddedSentence)>num_steps:
            b=EmbeddedSentence[:num_steps]
        if len(b)==EmbeddedSentence:
            b=EmeddedSentence
        b=[word_to_id['UNK'] if x>=DictionaryLength else x for x in b] #turn all words used 4 times or less to 'UNK'
        #print(b)
        Tokenized+=[b]

print(sentences[:10])
print(Tokenized[:10])

############################################# Parameters #####################################################

num_hidden=256
num_steps=20
dict_length=len(words)
batch_size=4
num_layers=1

## Create labels
Label=[]
for caption in Tokenized:
   ...


I am working on the file "L8 - Hierarchical Clustering.ipynb".
The first task is:
As you can see, the two red points are quite close to each other.
You can try the rest iterations by yourself.
Plotting a Dendrogram
A dendrogram is a visualization in form of a tree showing the order and distances of merges during the hierarchical clustering.
Can you write Python code for it?

plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z,
    leaf_rotation=90.,
    leaf_font_size=9.,
)
plt.show()


I am working on the file "L8 - Hierarchical Clustering.ipynb".
The first task is:
Double click the figure to zoom in.
y-axis tells the distance in each merge (that's why the blue line is "tall")
We can also plot a truncated dendrogram, which only shows part of the tree. For more details please refer to the scipy page.
Can you write Python code for it?

plt.title('Hierarchical Clustering Dendrogram (truncated)')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z,
    truncate_mode='lastp',  # show only the last p merged clusters,
    p=12,  # show only the last p merged clusters
    show_leaf_counts=True,  # numbers with brackets are counts, without brackets are indics
    leaf_rotation=90.,
    leaf_font_size=12.,
)
plt.show()


I am working on the file "L8 - Hierarchical Clustering.ipynb".
The first task is:
Hierarchical Clustering - Complete Linkage
You can try with Complete Linkage by yourself.
Can you write Python code for it?

Z_complete = linkage(X, 'complete')

plt.figure(figsize=(25, 10))
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('sample index')
plt.ylabel('distance')
dendrogram(
    Z_complete,
    leaf_rotation=90.,
    leaf_font_size=9.,
)
plt.show()


I am working on the file "L8 - Hierarchical Clustering.ipynb".
The first task is:
Hierarchical Clustering - Single Linkage
Can you write Python code for it?


I am working on the file "L8 - Hierarchical Clustering.ipynb".
The first task is:
Z[i] will tell us which clusters were merged in the i-th iteration. The format is [idx1, idx2, dist, sample_count]. For example:
Can you write Python code for it?


I am working on the file "L8 - KMeans Clustering.ipynb".
The first task is:
Convergence of KMeans
Can you write Python code for it?

init_array = np.asarray([[-10, -10], [-9, -10], [-10, -9], [-9, -9]])

for i in range(10):
    #kmeans = KMeans(n_clusters = 4, max_iter = i+1, n_init = 1, init='k-means++', random_state=42)
    #kmeans = KMeans(n_clusters = 4, max_iter = i+1, n_init = 1, init='random', random_state=42)
    kmeans = KMeans(n_clusters = 4, max_iter = i+1, n_init = 1, init=init_array, random_state=42)
    kmeans.fit(data)
    kmeans_plot(data, kmeans)
    print('Sum of distances to the centroids: {}'.format(kmeans.inertia_))


I am working on the file "L8 - KMeans Clustering.ipynb".
The first task is:
KMeans Clustering
Can you write Python code for it?

# plot the kmeans results
def kmeans_plot(data, kmeans):
    h = .02
    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.figure(1)
    plt.clf()
    plt.imshow(Z, interpolation='nearest',
               extent=(xx.min(), xx.max(), yy.min(), yy.max()),
               cmap=plt.cm.Paired,
               aspect='auto', origin='lower')
    plt.plot(data[:, 0], data[:, 1], 'k.', markersize=2)

    centroids = kmeans.cluster_centers_ # get the centroids
    plt.scatter(centroids[:, 0], centroids[:, 1],
                marker='x', s=169, linewidths=3,
                color='w', zorder=10)
    plt.xlim(x_min, x_max)
    plt.ylim(y_min, y_max)
    plt.xticks(())
    plt.yticks(())
    plt.show()

## n_clusters: Number of clusters
## max_iter: Maximum number of iterations of the k-means algorithm for a single run.
## n_init: Number of time the k-means algorithm will be run with different centroid seeds.
## init: How to select initial centers
kmeans = KMeans(n_clusters = 4, max_iter = 100, n_init = 10, init='k-means++', random_state=42)
kmeans.fit(data)
kmeans_plot(data, kmeans)
print('Sum of distances to the centroids: {}'.format(kmeans.inertia_))


I am working on the file "L8 - KMeans Clustering.ipynb".
The first task is:
KMeans with different number of clusters
Can you write Python code for it?

d = []
for i in range(30):
    kmeans = KMeans(n_clusters = i+1, max_iter = 300, n_init = 20, init='k-means++', random_state=42)
    kmeans.fit(data)
    kmeans_plot(data, kmeans)
    print('Sum of distances to the centers: {}'.format(kmeans.inertia_))
    d.append([i+1, kmeans.inertia_])
d = np.asarray(d)

plt.plot(d[:,0], d[:,1])


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Adding idf
The above vectors in vec_list only keep track of term frequencies (tfs). Now we make each element $w_{d, t} = tf(d, t) \cdot idf(t)$.
Note that the textbook version of idf will have 0 if df == ndocs. Therefore, we follow the implementation in Lucene
(https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/search/similarities/TFIDFSimilarity.html) which gives:
$$idf(t) = 1 + \log(\frac{ndocs}{df + 1})$$
BTW, on the above page, you can see the detailes and justifications of all major deviations of Lucene's implementation from the standard VSM. You will always need to do this to tackle new or specific problems you are facing when you join the workforce, :p. Therefore, a deep understanding of the principles behind the knowledge you learned in the uni is more important.
Can you write Python code for it?

from math import log
def idf(cnt, ndocs):
    # here we use ln(). The base of the log does not matter much.
    return 1.0 + log(ndocs/(cnt+1))

ndocs = len(doc_list)
# voc = [ (v, [b[v] for b in bags]) for v in vocabulary] # if you want to see the individual counts
voc = [(v, sum([b[v] for b in bags])) for v in vocabulary]
pprint(voc)


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Look at the matrix m before normalization.
* m[0][6] = 1.405. This is because $tf(d_0, 'linda') = 1$, and $idf('linda') = 1.4054651081081644$.
* m[0][8] = 0.828. This is because $tf(d_0, 'me') = 1.693$, and $idf('me') = 0.4891743762340093$.
Measuring the Similarity of Two Documents
We use the cosine to measure the similarity of two weighted vectors.
Can you write Python code for it?

import numpy as np

def normalize(v):
    norm = np.linalg.norm(v)
    if norm == 0:
       return v
    return v / norm

def cosine(u, v):
    a = normalize(u)
    b = normalize(v)
    return np.inner(a, b)

cosine(doc_term_matrix_tfidf[0], doc_term_matrix_tfidf[1])

cosine(doc_term_matrix_tfidf[0], doc_term_matrix_tfidf[2])

cosine(doc_term_matrix_tfidf[1], doc_term_matrix_tfidf[2])


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Now we can look at the final weighted vectors of the documents.
We choose to implement it based on matrix operations supported by numpy. E.g., to obtained ${a_i b_i}{i=1}^n$ from two vector $\vec{A} = {a_i}{i=1}^n$ and $\vec{B} = {b_i}_{i=1}^n$, we first construct a diagonal matrix $\mathbf{D_A}$ whose diagonal elements are $a_i$, then we can obtain the desired result by the standard matrix multiplication of $\vec{B} \mathbf{D_A}$.
Can you write Python code for it?

import numpy as np

def build_idf_matrix(idf_vector):
    idf_mat = np.zeros((len(idf_vector), len(idf_vector)))
    np.fill_diagonal(idf_mat, idf_vector)
    return idf_mat

idf_matrix = build_idf_matrix([v[1] for v in voc])
cprint(idf_matrix)

import math
def l2_normalizer(vec):
    denom = np.sum([el**2 for el in vec])
    return [(el / math.sqrt(denom)) for el in vec]

doc_term_matrix_tfidf = []

#performing tf-idf matrix multiplication
for vec in tf_vec_list:
    doc_term_matrix_tfidf.append(np.dot(vec, idf_matrix))
cprint(np.matrix(doc_term_matrix_tfidf)) # np.matrix() just to make it easier to look at

#normalizing
print('\nAfter normalization ...')
doc_term_matrix_tfidf_l2 = []
for tf_vector in doc_term_matrix_tfidf:
    doc_term_matrix_tfidf_l2.append(l2_normalizer(tf_vector))

cprint(np.matrix(doc_term_matrix_tfidf_l2))


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Now we construct the vocabulary and the doc-term matrix.
Can you write Python code for it?

import itertools

vocabulary = sorted(list(set(itertools.chain(*[list(b) for b in bags]))))
vocabulary

def print_vocabulary_vertically(voc, leading_str = '', spacing=2, align=1):
    # align = 0: align top; otherwise, align bottom
    max_len = max([len(v) for v in voc])
    for i in range(max_len):
        if align == 0:
            line = [v[i] if i < len(v) else ' ' for v in voc]
        else:
            line = [' ' if i < max_len - len(v) else v[i-max_len] for v in voc]
        print('{}{}'.format(leading_str, (' '*spacing).join(line)))

print_vocabulary_vertically(vocabulary, align=0)

for doc in doc_list:
    print(doc)
print()
vec_list = []
print_vocabulary_vertically(vocabulary, leading_str=' ')
print('-'*70)
for bag in bags:
    vec = [bag[v] for v in vocabulary] # Counter['non-existing-key'] = 0
    vec_list.append(vec)
    print(vec)


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Now we transform raw tf to tf by $1 + \log(tf)$.
Can you write Python code for it?

from math import log
def normalize_tf(tf):
    if tf == 0:
        return 0.0
    else:
        return 1.0 + log(tf)

tf_vec_list = []
for vec in vec_list:
    tf_vec_list.append([normalize_tf(val) for val in vec])

print_vocabulary_vertically(vocabulary, leading_str='   ', spacing=6)
print('-'*80)
cprint(np.matrix(tf_vec_list))


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
Scoring Documents
Can you write Python code for it?

for i, vec in enumerate(vec_list):
    print('score of doc[{}] = {}'.format(i, cosine(doc_term_matrix_tfidf[i], vec_q)))

def topk(query, k):
    v_q = encode(query, vocabulary)
    out = []
    for i, vec in enumerate(vec_list):
        out.append((i, cosine(doc_term_matrix_tfidf[i], v_q)))
    answer = sorted(out, key=lambda r: r[1], reverse=True)[:k]
    return answer

topk('loves baseball', 3)


I am working on the file "L8 - tf-idf.ipynb".
The first task is:
We will eventually record in voc the idf values. You can look at the raw df value and the resulting idf values from the outputs of the above the below cells.
Can you write Python code for it?

voc = [(v, idf( sum([b[v] for b in bags]), ndocs)) for v in vocabulary]
idf_dict = dict(voc)
pprint(voc)
print(idf_dict['he'])


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
A partir de este punto tenemos los datos listos para empezar hacer un estudio semántico de los mismos sin que las anomalías de los mismos nos puedan interferir.
Correlación entre las variables
El primer estudio que vamos a hacer con los datos va a ser evaluar como de relacionados están unos con otros. Esto nos permitirá evaluar cuales de los atributos pueden ser determinantes para el problema que plantea estos datos.
Para ello, generaremos la matriz de correlación lineal gracias a la función corr() del objeto dataset de panda:
Can you write Python code for it?

correlation=dataset.corr() #Correlation Matrix
correlation


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
A partir de este punto ya tenemos un conjunto de datos normalizado con el que operar.
Deteccion de outliers
A la hora de tratar cualquier conjunto de datos es común observar valores que parecen que rompen la norma y que se salen de la distribución. En los algoritmos de machine learninng, este tipo de valores pueden desfavorecedores pues a tenerlos en cuenta pueden generar problemas y llegar a estados espúreos.
Por ello, debemos intentar localizarlos y tratarlos. No debemos olvidar nunca la semántica de los datos que tratamos. Para poder identificar outliers, una buena herramienta son las gráficas de caja:
Can you write Python code for it?

dataset.boxplot(column='Age')

dataset.boxplot(column='Total_Bilirubin')

dataset.boxplot(column='Direct_Bilirubin')

dataset.boxplot(column='Alkaline_Phosphotase')

dataset.boxplot(column='Alamine_Aminotransferase')

dataset.boxplot(column='Aspartate_Aminotransferase')

dataset.boxplot(column='Total_Protiens')

dataset.boxplot(column='Albumin')

dataset.boxplot(column='Albumin_and_Globulin_Ratio')


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Cambiando valores nominales
Como se puede observar desde un primer momento, existe una variable nominal denominada Gender relacionada con el sexo. Para facilitar los cálculos posteriores, vamos a reemplazarla cambiando:
* Male: 0
* Female: 1
Can you write Python code for it?

dataset.replace("Male",0, True)
dataset.replace("Female",1,True)
dataset


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Datos de estadisticos de las variables
En este apartado vamos a desarrollar un breve estudio estadístico para los datos proporcionados. Para ello utilizaremos el método describe del objeto DataSet de la librería panda. Este método genéra un cálculo estadístico básico en el que nos describe por cada variable valores tales como la media, la desviación estandar y los percentiles 25%, 50% y 75% respecto al valor máximo.
Can you write Python code for it?

print(dataset.describe()) #Descripción de los datos

#### Obtenemos contéo de sexo
print('\nPacientes masculinos (0) y femeninos (1):')
print(pd.value_counts(dataset['Gender'].values))
#### Obtenemos contéo de la clase
print('\nPacientes enfermos (1) y sanos (2):')
print(pd.value_counts(dataset['Dataset'].values))


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Listamos cabeceras de los datos
Can you write Python code for it?

# Header
header = []
for row in dataset:
    header.append(row)
header


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Observando estos datos podemos sacar conclusiones tales como:
* Tratamos con un conjunto de 583 pacientes
* Referente a la edad de los pacientes:
  * La edad media de los pacientes es de 44 ± 16 años
  * El paciente más joven tiene la edad de 4 años.
  * El paciente más anciano tiene 90 años.
  * El grueso de los pacientes están entre el percentíl 50 y 75, lo que indica que están por encima de los 45 años.
* Referente al sexo de los pacientes:
  * Tenemos un total de 441 hombres
  * Tenemos un total de 142 mujeres
* De estos pacientes:
  * 416 están enfermos
  * 167 están sanos
Normalizando variables
Para saber si debemos normalizar algúna variable, debemos ver la distribución que estas siguen. Para ello, podemos realizar un histograma de cada una de las variables y ver gráficamente las variables que parezcan seguir una distribución normal.
Can you write Python code for it?

#### Histogram Matrix Plot

plt.figure()
dataset.hist(xlabelsize=0.5, ylabelsize=0.2,figsize=(10,10))
plt.xlabel("Data")
plt.show()


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Obteniendo los datos del csv
Descripcion de las variables
Tenemos una base de datos médica con los resultados de análisis sanguineos de varios pacientes y una clasificación de su estado de salud, 1 para los enfermos y 2 para los sanos. En primer lugar tenemos los datos del paciente, su edad y su género. A continuación se nos dan los resultados del total de bilirrubina y bilirrubina directa, la bilirrubina es un pigmento amarillo que se encuentra en la bilis (liquido generado por el hígado), la bilirrubina total es la suma de la bilirrubina directa o conjugada y la bilirrubina indirecta.
 Despues se proporciona los valores de la fosfatasa alcalina, que es una enzima que se desplaza a través de el sistema sanguíneo, pero que se acumula más que nada en el hígado, la bilis, riñones y el sistema intestinal, es útil para detectar enfermedades óseas o hepáticas.

Después están las transaminasas, la alaninoamino transferasa (ALT o GPT) y la aspartato aminotransferasa (AST o GOT), que son enzimas que se encuentran en el interior de las células de órganos como el hígado, el corazón, los riñones o los músculos, y desempeñan una importante función en el metabolismo. Cuando un análisis de sangre detecta niveles elevados de estas moléculas puede indicar que existe una lesión de las células hepáticas.

Las últimas tres variables son las proteinas totales, la albumina y el ratio de albumina y globulina. El examen de proteína total mide la cantidad total de dos clases de proteínas encontradas en la porción líquida de la sangre: albúmina y globulina.

La albúmina es una proteína plasmática cuya función más importante es el mantenimiento de la presión oncótica, es decir  ayuda a impedir que se escape líquido fuera de los vasos sanguíneos,  y la capacidad de transporte de hormonas, medicamentos,etc.

Las globulinas son útiles en la lucha contra las infecciones y mejorar el proceso de coagulación de la sangre. También sirven como portadora de la hormona y el transporte de las hormonas a diferentes partes...
Can you write Python code for it?

#Path para linux
path = '../Recursos/indian_liver_patient.csv'

#Path para Windows
#path = '..\Recursos\indian_liver_patient.csv'
dataset = pd.read_csv('../Recursos/indian_liver_patient.csv',delimiter=',',header=0)
dataset


I am working on the file "LAB1/src/Practica1.ipynb".
The first task is:
Si representamos esta matriz como un grafico donde podamos verlo más visualmente:
Can you write Python code for it?

# Display the correlation matrix with a specified figure number and a bluescale
# colormap
plt.figure()
plt.matshow(correlation, fignum=1, cmap=plt.cm.Blues)
plt.ylabel("Attribute Index")
plt.show()

#### Scatter Matrix Plot

plt.figure()
from pandas.tools.plotting import scatter_matrix
scatter_matrix(dataset, alpha=0.3, figsize=(20, 20), diagonal='kde')
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Chi-Square
Ahora aplicamos Chi-Square para seleccionar las variables informativas
Can you write Python code for it?

chi = FS.chi2(X = data_X, y = data["Type"])[0]
print(chi)
indice_chi=np.argsort(chi)[::-1]
print(indice_chi)
print(names[indice_chi])

plt.figure()

plt.scatter(data[data.Type==1].Proline,data[data.Type==1].Color_Intensity, color='red')
plt.scatter(data[data.Type==2].Proline,data[data.Type==2].Color_Intensity, color='blue')
plt.scatter(data[data.Type==0].Proline,data[data.Type==0].Color_Intensity, color='green')
plt.title('Good Predictor Variables Chi-Square \n Proline vs Color_Intensity')
plt.xlabel('Proline')
plt.ylabel('Color_Intensity')
plt.legend(['A','B','C'])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Dibujamos la proyección en las dos primeras componentes principales
Can you write Python code for it?

pca = PCA(n_components=2)
X_pca = pd.DataFrame(pca.fit_transform(data_X))

pca_A = X_pca[data_y == 0]
pca_B = X_pca[data_y == 1]
pca_C = X_pca[data_y == 2]

#plot
plt.scatter(x = pca_A[0], y = pca_A[1], c="blue")
plt.scatter(x = pca_B[0], y = pca_B[1], c="turquoise")
plt.scatter(x = pca_C[0], y = pca_C[1], c="darkorange")
plt.xlabel("First Component")
plt.ylabel("Second Component")
plt.legend(["A","B","C"])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
LDA with normalization
Can you write Python code for it?

lda = LDA(n_components=2)
lda.fit(X_scaled,data_y)
print("Porcentaje explicado:", lda.explained_variance_ratio_)

X_lda = pd.DataFrame(lda.fit_transform(data_X, data_y))

# Dividimos en los 3 tipos para ponerles diferentes colores
lda_A = X_lda[data_y == 0]
lda_B = X_lda[data_y == 1]
lda_C = X_lda[data_y == 2]

#plot
plt.scatter(x = lda_A[0], y = lda_A[1], c="blue")
plt.scatter(x = lda_B[0], y = lda_B[1], c="turquoise")
plt.scatter(x = lda_C[0], y = lda_C[1], c="darkorange")
plt.xlabel("First LDA Component")
plt.ylabel("Second LDA Component")
plt.legend(["A","B","C"],loc="lower right")
plt.title("LDA with normalization")
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Linear Discriminant Analysis (LDA)
Linear Discriminant Analysis
A classifier with a linear decision boundary, generated by fitting class conditional densities to the data and using Bayes’ rule.
The model fits a Gaussian density to each class.
The fitted model can also be used to reduce the dimensionality of the input by projecting it to the most discriminative directions.
Can you write Python code for it?

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Mutual Information
Can you write Python code for it?

mi = FS.mutual_info_classif(data_X, data_y)
print(mi)
data_X.head(0)

names=data_X.axes[1]
names

indice=np.argsort(mi)[::-1]
print(indice)
print(names[indice])

plt.figure(figsize=(8,6))
plt.subplot(121)

plt.scatter(data[data.Type==1].Flavanoids,data[data.Type==1].Color_Intensity, color='red')
plt.scatter(data[data.Type==2].Flavanoids,data[data.Type==2].Color_Intensity, color='blue')
plt.scatter(data[data.Type==0].Flavanoids,data[data.Type==0].Color_Intensity, color='green')
plt.title('Good Predictor Variables \n Flavanoids vs Color_Intensity')
plt.xlabel('Flavanoids')
plt.ylabel('Color_Intensity')
plt.legend(['A','B','C'])

plt.subplot(122)

plt.scatter(data[data.Type==1].Ash,data[data.Type==1].Nonflavanoid_Phenols, color='red')
plt.scatter(data[data.Type==2].Ash,data[data.Type==2].Nonflavanoid_Phenols, color='blue')
plt.scatter(data[data.Type==0].Ash,data[data.Type==0].Nonflavanoid_Phenols, color='green')
plt.title('Ash vs Nonflavanoid_Phenols')
plt.xlabel('Ash')
plt.ylabel('Nonflavanoid_Phenols')
plt.legend(['A','B','C'])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
PCA with Normalization
Can you write Python code for it?

from sklearn import preprocessing
X_scaled = preprocessing.scale(data_X)

pca = PCA()
pca.fit(X_scaled)
plt.plot(pca.explained_variance_)
plt.ylabel("eigenvalues")
plt.xlabel("position")
plt.show()
print ("Eigenvalues\n",pca.explained_variance_)

# Percentage of variance explained for each components
print('\nExplained variance ratio (first two components):\n %s'
      % str(pca.explained_variance_ratio_))

pca = PCA(n_components=2)
X_pca = pd.DataFrame(pca.fit_transform(X_scaled))

pca_A = X_pca[data_y == 'A']
pca_B = X_pca[data_y == 'B']
pca_C = X_pca[data_y == 'C']

#plot
plt.scatter(x = pca_A[0], y = pca_A[1], c="blue")
plt.scatter(x = pca_B[0], y = pca_B[1], c="turquoise")
plt.scatter(x = pca_C[0], y = pca_C[1], c="darkorange")
plt.xlabel("First Component")
plt.ylabel("Second Component")
plt.legend(["A","B","C"])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
PCA without normalization
Can you write Python code for it?

pca = PCA()
pca.fit(data_X)
plt.plot(pca.explained_variance_)
plt.ylabel("eigenvalues")
plt.xlabel("position")
plt.show()
print ("Eigenvalues\n",pca.explained_variance_)
# Percentage of variance explained for each components
print('\nExplained variance ratio (first two components):\n %s'
      % str(pca.explained_variance_ratio_))


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Se sustituye la columna Type por un valor categórico
Can you write Python code for it?

data["Type"] = pd.Categorical.from_array(data["Type"]).codes
data["Type"].replace("A",0)
data["Type"].replace("B",1)
data["Type"].replace("C",2)
data.head()

data.describe()


I am working on the file "LAB2/Recursos/Red_Dimension.ipynb".
The first task is:
Separamos la columna target del resto de variables predictoras
Can you write Python code for it?

data_y = data["Type"]
data_X = data.drop("Type", 1)
data_X.head()


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
Chi-Square
Ahora aplicamos Chi-Square para seleccionar las variables informativas
Can you write Python code for it?


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
Dibujamos la proyección en las dos primeras componentes principales
Can you write Python code for it?


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
LDA with normalization
Can you write Python code for it?


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
Mutual Information
Can you write Python code for it?

mi = FS.mutual_info_classif(data_X, data_y)
print(mi)
data_X.head(0)

names=data_X.axes[1]
names

indice=np.argsort(mi)[::-1]
print(indice)
print(names[indice])

mi_regr = FS.mutual_info_regression(data_X, data_y)
indice_regr=np.argsort(mi_regr)[::-1]
print(indice_regr)
print(names[indice_regr])

plt.figure(figsize=(8,6))
plt.subplot(121)

plt.scatter(data[data.Type==1].Flavanoids,data[data.Type==1].Color_Intensity, color='red')
plt.scatter(data[data.Type==2].Flavanoids,data[data.Type==2].Color_Intensity, color='blue')
plt.scatter(data[data.Type==0].Flavanoids,data[data.Type==0].Color_Intensity, color='green')
plt.title('Good Predictor Variables \n Flavanoids vs Color_Intensity')
plt.xlabel('Flavanoids')
plt.ylabel('Color_Intensity')
plt.legend(['A','B','C'])

plt.subplot(122)

plt.scatter(data[data.Type==1].Ash,data[data.Type==1].Nonflavanoid_Phenols, color='red')
plt.scatter(data[data.Type==2].Ash,data[data.Type==2].Nonflavanoid_Phenols, color='blue')
plt.scatter(data[data.Type==0].Ash,data[data.Type==0].Nonflavanoid_Phenols, color='green')
plt.title('Ash vs Nonflavanoid_Phenols')
plt.xlabel('Ash')
plt.ylabel('Nonflavanoid_Phenols')
plt.legend(['A','B','C'])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
PCA with Normalization
Can you write Python code for it?

from sklearn import preprocessing
X_scaled = preprocessing.scale(data_X)

pca = PCA()
pca.fit(X_scaled)
plt.plot(pca.explained_variance_)
plt.ylabel("eigenvalues")
plt.xlabel("position")
plt.show()
print ("Eigenvalues\n",pca.explained_variance_)

# Percentage of variance explained for each components
print('\nExplained variance ratio (first two components):\n %s'
      % str(pca.explained_variance_ratio_))

pca = PCA(n_components=2)
X_pca = pd.DataFrame(pca.fit_transform(X_scaled))

pca_A = X_pca[data_y == 0]
pca_B = X_pca[data_y == 1]
pca_C = X_pca[data_y == 2]

#plot
plt.scatter(x = pca_A[0], y = pca_A[1], c="blue")
plt.scatter(x = pca_B[0], y = pca_B[1], c="turquoise")
plt.scatter(x = pca_C[0], y = pca_C[1], c="darkorange")
plt.xlabel("First Component")
plt.ylabel("Second Component")
plt.legend(["A","B","C"])
plt.show()


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
PCA without normalization
Can you write Python code for it?


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
Principal Component Analysis (PCA)
Can you write Python code for it?

from sklearn.decomposition.pca import PCA


I am working on the file "LAB2/Recursos/Red_Dimension1.ipynb".
The first task is:
Separamos la columna target del resto de variables predictoras
Can you write Python code for it?


I am working on the file "LAB2/src/Practica2.ipynb".
The first task is:
Como se ve en las graficas, aunque las primeras dos variables (Direct_Bilirubin y Age) no separan bien los elementos, algo de esperar por los resultados tan bajos obtenidos, si estan mejor separadas que con dos de las tres ultimas variables (Total_Protiens y Albumin)
Can you write Python code for it?

def specificity(y_true, y_pred):
    tn, fp, fn, tp = metrics.confusion_matrix(y_true, y_pred).ravel()
    return tn/(tn+fp)

#Con todas las variables 10
modelo_lr = LogisticRegression()
modelo_lr.fit(X=train_X,y=train_Y)

predicion = modelo_lr.predict(test_X)
print('Regresion logistica con todas las variables\n')
print(f'\tprecision_score={metrics.precision_score(y_true=test_Y, y_pred=predicion)}')
print(f'\trecall_score={metrics.recall_score(y_true=test_Y, y_pred=predicion)}')
print(f'\taccuracy_score={metrics.accuracy_score(y_true=test_Y, y_pred=predicion)}')
print(f'\tspecificity_score={specificity(y_true=test_Y, y_pred=predicion)}')

train_X_copia=train_X.copy()
test_X_copia=test_X.copy()
allvar="";
#quitando variables
nvar=9
for i in regr_var[:0:-1]:
    allvar=allvar+i+", "
    train_X_copia=train_X_copia.drop(i,1)
    test_X_copia=test_X_copia.drop(i,1)
    modelo_lr = LogisticRegression()
    modelo_lr.fit(X=train_X_copia,y=train_Y)

    print('Con',nvar," variables")
    nvar=nvar-1
    print('Quitando \n'+allvar+"\n")
    predicion = modelo_lr.predict(test_X_copia)

    print(f'\tprecision_score={metrics.precision_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\trecall_score={metrics.recall_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\taccuracy_score={metrics.accuracy_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\tspecificity_score={specificity(y_true=test_Y, y_pred=predicion)}')


I am working on the file "LAB2/src/Practica2.ipynb".
The first task is:
En el caso de realizar la media vemos que los valores no varian tanto de una ejecucion a otra:

ej1:3 5 0 6 2 4 9 1 8 7

ej2:3 0 5 6 2 4 9 8 1 7

ej3:3 0 5 6 2 4 9 8 1 7

ej4:3 5 6 0 2 4 9 8 1 7

ej5:5 6 3 0 2 4 9 8 1 7

Y nos a partir de aqui vamos a usar los valores de ejecución 2 y 3 que son los mismos.
Can you write Python code for it?

indice_regr = [ 3, 0, 5, 6, 2, 4, 9, 8, 1, 7]
regr_var=names[indice_regr]
regr_var

plt.figure(figsize=(8,6))
plt.subplot(121)

plt.scatter(dataset[dataset.Dataset==0].Direct_Bilirubin,dataset[dataset.Dataset==0].Age, color='red')
plt.scatter(dataset[dataset.Dataset==1].Direct_Bilirubin,dataset[dataset.Dataset==1].Age, color='blue')
plt.title('Good Predictor Variables \n Direct_Bilirubin vs Age')
plt.xlabel('Direct_Bilirubin')
plt.ylabel('Age')
plt.legend(['Enfermos','Sanos'])

plt.subplot(122)

plt.scatter(dataset[dataset.Dataset==0].Total_Protiens,dataset[dataset.Dataset==0].Albumin, color='red')
plt.scatter(dataset[dataset.Dataset==1].Total_Protiens,dataset[dataset.Dataset==1].Albumin, color='blue')
plt.title('Good Predictor Variables \n Total_Protiens vs Albumin')
plt.xlabel('Total_Protiens')
plt.ylabel('Albumin')
plt.legend(['Sick','Healthy'])
plt.show()


I am working on the file "LAB2/src/Practica2.ipynb".
The first task is:
Según hemos visto con las pruebas anteriores, podemos ver como alcanzamos la mejor precisión quitando las variables Albumin_and_Globulin_Ratio, Albumin, Total_Protiens, Aspartate_Aminotransferase, Alamine_Aminotransferase, Alkaline_Phosphotase. Hasta ahora ha sido el mejor resultado que hemos obtenido entre todos los métodos.
LDA
Al contrario de PCA, LDA es invariante a escala y por tanto no tenemos por que trabajara con los datos normalizados.
Can you write Python code for it?

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA()
lda.fit(train_X,train_Y)

print("Porcentaje explicado:", lda.explained_variance_ratio_)

ldaRes = lda.explained_variance_ratio_
indice_lda=np.argsort(ldaRes)[::-1]
print(indice_lda)
print(names[indice_lda])
pca_var=names[indice_lda]

train_X_copia=train_X.copy()
test_X_copia=test_X.copy()
allvar="";
nvar=9

#quitando variables
for i in pca_var[:0:-1]:
    allvar=allvar+i+", "
    train_X_copia=train_X_copia.drop(i,1)
    test_X_copia=test_X_copia.drop(i,1)
    modelo_lr = LogisticRegression()
    modelo_lr.fit(X=train_X_copia,y=train_Y)
    print('Con',nvar," variables")
    nvar=nvar-1
    print('Quitando \n'+allvar+'\n')
    predicion = modelo_lr.predict(test_X_copia)
    print(f'\tprecision_score={metrics.precision_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\trecall_score={metrics.recall_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\taccuracy_score={metrics.accuracy_score(y_true=test_Y, y_pred=predicion)}')
    print(f'\tspecificity_score={specificity(y_true=test_Y, y_pred=predicion)}')


I am working on the file "LAB3/Recursos/practica_cnn.ipynb".
The first task is:
Analysis of CNN weights
Can you write Python code for it?

weights = model.get_weights()
print(np.shape(weights))
for i in range(len(weights)):
    print('shape of weights[%d]: ' % i, np.shape(weights[i]))

weights[0][:,:,0,0]

# kernels de la primera capa

nfilters = weights[0].shape[3]
lado = int(np.ceil(np.sqrt(nfilters)))
plt.subplots(lado,lado,figsize = (12, 15))

ma = abs(weights[0]).max()

for i in range(nfilters):
    kernel = weights[0][:,:,0,i]
    plt.subplot(lado,lado,i+1)
    plt.imshow(kernel, vmin=-ma, vmax=ma, cmap='bwr')
    plt.title('kernel %d' % i)


I am working on the file "LAB3/Recursos/practica_cnn.ipynb".
The first task is:
Partition and reformat the data
We want to partition the data into training and validation datasets and then reform the class as a one-hot so it fits to the model. We also perform a simple normalization of the image data so the range is between -1 and 1 instead of -3000 to 2000
Can you write Python code for it?

from sklearn.model_selection import train_test_split
from keras.utils.np_utils import to_categorical

X_vec = (np.expand_dims(all_slices,-1) - np.mean(all_slices))/np.std(all_slices)

y_vec = to_categorical(all_classes)
X_train, X_test, y_train, y_test = train_test_split(X_vec, y_vec,
                                                   train_size = 0.75,
                                                   random_state = 1,
                                                   stratify = all_classes)


I am working on the file "LAB3/Recursos/practica_cnn.ipynb".
The first task is:
Results obtained by the CNN model
Can you write Python code for it?

score_tr = model.evaluate(X_train, y_train, verbose=0)
print('Train loss    :', score_tr[0])
print('Train accuracy:', score_tr[1])

score_te = model.evaluate(X_test, y_test, verbose=0)
print('Test loss     :', score_te[0])
print('Test accuracy :', score_te[1])

y_pred_proba = model.predict(X_test)
y_pred = np.argmax(y_pred_proba,1)
print('')
print(classification_report(np.argmax(y_test,1), y_pred))

fpr, tpr, thresholds = roc_curve(np.argmax(y_test, 1), y_pred_proba[:,1])
fig, ax1 = plt.subplots(1,1)
ax1.plot(fpr, tpr, 'r-.', label = 'CNN (%2.2f)' % auc(fpr, tpr))
ax1.set_xlabel('False Positive Rate')
ax1.set_ylabel('True Positive Rate')
ax1.plot(fpr, fpr, 'b-', label = 'Random Guess')
ax1.legend()
plt.show()

model.summary()
#SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg'))


I am working on the file "LAB3/Recursos/practica_cnn.ipynb".
The first task is:
Standard model
Can you write Python code for it?

X_tr = np.zeros((np.shape(X_train)[0], np.shape(X_train)[1]*np.shape(X_train)[2]))
for i in range(np.shape(X_train)[0]):
    X_tr[i] = X_train[i].flatten()
X_te = np.zeros((np.shape(X_test)[0], np.shape(X_test)[1]*np.shape(X_test)[2]))
for i in range(np.shape(X_test)[0]):
    X_te[i] = X_test[i].flatten()

print("X_train shape:", np.shape(X_train))
print("X_tr shape:   ", np.shape(X_tr))
print("X_test shape: ", np.shape(X_test))
print("X_te shape:   ", np.shape(X_te))

from sklearn.linear_model import LogisticRegression

model2 = LogisticRegression()

model2.fit(X_tr, np.argmax(y_train,1))

print('Train accuracy:', model2.score(X_tr, np.argmax(y_train,1)))
print('Test accuracy :', model2.score(X_te, np.argmax(y_test,1)))

from sklearn.metrics import classification_report, roc_curve, auc

print('')
print(classification_report(np.argmax(y_test,1),
                            model2.predict(X_te)))

fpr, tpr, thresholds = roc_curve(np.argmax(y_test, 1), model2.predict_proba(X_te)[:,1])
fig, ax1 = plt.subplots(1,1)
ax1.plot(fpr, tpr, 'r-.', label = 'Simple model (%2.2f)' % auc(fpr, tpr))
ax1.set_xlabel('False Positive Rate')
ax1.set_ylabel('True Positive Rate')
ax1.plot(fpr, fpr, 'b-', label = 'Random Guess')
ax1.legend()
plt.show()


I am working on the file "LAB3/src/LAB3.ipynb".
The first task is:
Datos de partida:
¿Cuántas clases hay? ¿Cuántos ejemplos de cada?
Hay dos clases en nuestro grupo de datos, muestras malignas y muestras benignas, que se representan el las imagenes de la celda 4.
Tenemos 6691 muestras de las cuales 4165 (62.25%) benignas y 2526 (37,75%) malignas.
Can you write Python code for it?

print ('benign=>', (all_classes<0.5).sum())
print ('malignant=>', (all_classes>0.5).sum())


I am working on the file "LAB3/src/LAB3.ipynb".
The first task is:
Modelo CNN básico
Can you write Python code for it?

from keras.models import Sequential, load_model
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D
from keras import backend as K
from keras.losses import binary_crossentropy
from keras import optimizers

from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

print("X_train shape:", np.shape(X_train))
print("X_test shape: ", np.shape(X_test))
print("y_train shape:", np.shape(y_train))
print("y_test shape: ", np.shape(y_test))


I am working on the file "LAB3/src/LAB3.ipynb".
The first task is:
¿Cuál es el acierto mínimo esperado?
//////El acierto mínomo de esperado será que clasifique todas las muestras en una sola clase...
¿Cómo se normalizan las imágenes de entrada? ¿se te ocurre una opción mejor? Si es así, ¿cuál? Prúebala
Los datos se han normalizan usando la siguiente sentencia:
X_vec = (np.expand_dims(all_slices,-1) - np.mean(all_slices))/np.std(all_slices)
Es decir usa la normalización de Z-core: $x'=\frac{x-\mu}{\sigma}$, que normaliza los datos a valores entre -1 y 1, útil en casos de que los datos sigan una distribución normal, otra posible forma de normalizar sería user la normalización a escala decimal, $x'=\frac{x}{10^j}$ donde $max(x')<1$
Can you write Python code for it?

X_vec = (np.expand_dims(all_slices,-1) /(10**4))
(X_vec>1).sum()

from sklearn.model_selection import train_test_split
from keras.utils.np_utils import to_categorical

X_vec = (np.expand_dims(all_slices,-1) - np.mean(all_slices))/np.std(all_slices)

y_vec = to_categorical(all_classes)
X_train, X_test, y_train, y_test = train_test_split(X_vec, y_vec,
                                                   train_size = 0.75,
                                                   random_state = 1,
                                                   stratify = all_classes)

X_tr = np.zeros((np.shape(X_train)[0], np.shape(X_train)[1]*np.shape(X_train)[2]))
for i in range(np.shape(X_train)[0]):
    X_tr[i] = X_train[i].flatten()
X_te = np.zeros((np.shape(X_test)[0], np.shape(X_test)[1]*np.shape(X_test)[2]))
for i in range(np.shape(X_test)[0]):
    X_te[i] = X_test[i].flatten()

print("X_train shape:", np.shape(X_train))
print("X_tr shape:   ", np.shape(X_tr))
print("X_test shape: ", np.shape(X_test))
print("X_te shape:   ", np.shape(X_te))

from sklearn.linear_model import LogisticRegression

model2 = LogisticRegression()

model2.fit(X_tr, np.argmax(y_train,1))


I am working on the file "LAB3/src/LAB3.ipynb".
The first task is:
¿Cuál es el formato de las imágenes de entrada (dimensiones)? ¿Por qué es necesario hacer la operación flatten en dichas imágenes para aplicar la regresión logística?
Los datos de entrada tiene una dimension de 64X64, al hacer la operación flatten los transformamos en dimension 4096x1. La función de regresión logistica espera matrices de un máximo de dos dimensiones ,nº de ejemplox4096 (nº de variables), mientras tal y como tenemos los datos, tenemos matrices de 4 dimensiones nº de ejemplox64x64x1
Modelo simple
¿Cuántos parámetros libres tiene la regresión logística?
Es 4096, por la dimesión de entrada +1 por el sesgo= 4097
¿qué resultados da tras el entrenamiento? ¿produce sobreajuste?
A continuación se muestran los resultados de precesión para los datos de train y de test, como vemos hay una diferencia notable entre la precisión de los datos de train y la de los datos de test, siendo mejor el valor en el primer caso. Esto nos indica que el entrenamiento de la regresión logistica está produciendo sobreajuste.
Can you write Python code for it?

print('Train accuracy:', model2.score(X_tr, np.argmax(y_train,1)))
print('Test accuracy :', model2.score(X_te, np.argmax(y_test,1)))


I am working on the file "LAB3/src/LAB3.ipynb".
The first task is:
¿qué ocurre si la curva ROC se calcula con los datos de entrenamiento? ¿sale mejor? ¿por qué?
Como vemos, en la curva ROC de los datos de entrenamiento, es excelente (es 1), por lo tanto si sale mucho mejor, esto se debe a que como está sobreajustando los datos de entrenemiento, es decir se los aprende de memoria, posteriormente los clasifica casi perfectamente, dejando una curva ROC óptima
Can you write Python code for it?

from sklearn.metrics import classification_report, roc_curve, auc

print('')
print(classification_report(np.argmax(y_test,1),
                            model2.predict(X_te)))

fpr, tpr, thresholds = roc_curve(np.argmax(y_train, 1), model2.predict_proba(X_tr)[:,1])
fig, ax1 = plt.subplots(1,1)
ax1.plot(fpr, tpr, 'r-.', label = 'Simple model (%2.2f)' % auc(fpr, tpr))
ax1.set_xlabel('False Positive Rate')
ax1.set_ylabel('True Positive Rate')
ax1.plot(fpr, fpr, 'b-', label = 'Random Guess')
ax1.legend()
plt.show()