I am working on the file "CSW/csw_unidata.ipynb".
The first task is:
The csw object created from CatalogueServiceWeb did not fetched anything yet.
It is the method getrecords2 that uses the filter for the search. However, even though there is a maxrecords option, the search is always limited by the server side and there is the need to iterate over multiple calls of getrecords2 to actually retrieve all records.
The get_csw_records does exactly that.
Can you write Python code for it?

def get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000):
    """Iterate `maxrecords`/`pagesize` times until the requested value in
    `maxrecords` is reached.
    """
    from owslib.fes import SortBy, SortProperty
    # Iterate over sorted results.
    sortby = SortBy([SortProperty('dc:title', 'ASC')])
    csw_records = {}
    startposition = 0
    nextrecord = getattr(csw, 'results', 1)
    while nextrecord != 0:
        csw.getrecords2(constraints=filter_list, startposition=startposition,
                        maxrecords=pagesize, sortby=sortby)
        csw_records.update(csw.records)
        if csw.results['nextrecord'] == 0:
            break
        startposition += pagesize + 1  # Last one is included.
        if startposition >= maxrecords:
            break
    csw.records.update(csw_records)

get_csw_records(csw, filter_list, pagesize=10, maxrecords=1000)

records = '\n'.join(csw.records.keys())
print('Found {} records.\n'.format(len(csw.records.keys())))
for key, value in list(csw.records.items()):
    print('[{}]\n{}\n'.format(value.title, key))

csw.request

#write to JSON for use in TerriaJS
csw_request = '"{}": {}"'.format('getRecordsTemplate',str(csw.request,'utf-8'))

import io
import json
with io.open('query.json', 'a', encoding='utf-8') as f:
            f.write(json.dumps(csw_request, ensure_ascii=False))
            f.write('\n')


I am working on the file "CSW/csw_unidata.ipynb".
The first task is:
With these 3 elements it is possible to assemble a OGC Filter Encoding (FE) using the owslib.fes* module.
* OWSLib is a Python package for client programming with Open Geospatial Consortium (OGC) web service (hence OWS) interface standards, and their related content models.
Can you write Python code for it?

from owslib import fes
from ioos_tools.ioos import fes_date_filter

kw = dict(wildCard='*', escapeChar='\\',
          singleChar='?', propertyname='apiso:AnyText')

or_filt = fes.Or([fes.PropertyIsLike(literal=('*%s*' % val), **kw)
                  for val in model_names])

kw = dict(wildCard='*', escapeChar='\\',
          singleChar='?', propertyname='apiso:ServiceType')

serviceType = fes.PropertyIsLike(literal=('*%s*' % service_type), **kw)

begin, end = fes_date_filter(start, stop)
bbox_crs = fes.BBox(bbox, crs=crs)

filter_list = [
    fes.And(
        [
            bbox_crs,  # bounding box
            begin, end,  # start and end date
            or_filt,  # or conditions (CF variable names)
            serviceType  # search only for datasets that have WMS services
        ]
    )
]

from owslib.csw import CatalogueServiceWeb

endpoint = 'https://data.ioos.us/csw'

csw = CatalogueServiceWeb(endpoint, timeout=60)


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Hmmm..... In the query above, we only get 10 records, even though we specified maxrecords=100.
What's up with that?
Turns out the CSW service specified a MaxRecordDefault that cannot be exceeded.
For example, checking: https://dev-catalog.ioos.us/csw?request=GetCapabilities&service=CSW we find:
<ows:Constraint name="MaxRecordDefault">
    <ows:Value>10</ows:Value>
</ows:Constraint>
So we need to loop the getrecords request, incrementing the startposition:
Can you write Python code for it?

from owslib.fes import SortBy, SortProperty
pagesize = 10
sort_property = 'dc:title'  # a supported queryable of the CSW
sort_order = 'ASC'  # should be 'ASC' or 'DESC' (ascending or descending)
maxrecords = 50
sortby = SortBy([SortProperty(sort_property, sort_order)])

startposition = 0
while True:
    print 'getting records %d to %d' % (startposition, startposition+pagesize)
    csw.getrecords2(constraints=filter_list,
                    startposition=startposition, maxrecords=pagesize, sortby=sortby)
    for rec,item in csw.records.iteritems():
        print(item.title)
    print
    if csw.results['nextrecord'] == 0:
        break
    startposition += pagesize
    if startposition >= maxrecords:
        break


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Lets see what the full XML record looks like
Can you write Python code for it?


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Now add a NOT filter to eliminate some entries
Can you write Python code for it?

kw = dict(wildCard='*', escapeChar='\\',
          singleChar='?', propertyname='apiso:AnyText')

not_filt = fes.Not([fes.PropertyIsLike(literal='*Waikiki*', **kw)])

filter_list = [fes.And([filter1, filter2, service_filt, bbox_filter, start, stop, not_filt])]

startposition = 0
while True:
    print 'getting records %d to %d' % (startposition, startposition+pagesize)
    csw.getrecords2(constraints=filter_list,
                    startposition=startposition, maxrecords=pagesize, sortby=sortby)
    for rec,item in csw.records.iteritems():
        print(item.title)
    print
    if csw.results['nextrecord'] == 0:
        break
    startposition += pagesize
    if startposition >= maxrecords:
        break


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Now add time contraints.  Here we first define a function that will return records if any data in the records overlaps the specified time period
Can you write Python code for it?

def dateRange(start_date='1900-01-01',stop_date='2100-01-01',constraint='overlaps'):
    if constraint == 'overlaps':
        start = fes.PropertyIsLessThanOrEqualTo(propertyname='apiso:TempExtent_begin', literal=stop_date)
        stop = fes.PropertyIsGreaterThanOrEqualTo(propertyname='apiso:TempExtent_end', literal=start_date)
    elif constraint == 'within':
        start = fes.PropertyIsGreaterThanOrEqualTo(propertyname='apiso:TempExtent_begin', literal=start_date)
        stop = fes.PropertyIsLessThanOrEqualTo(propertyname='apiso:TempExtent_end', literal=stop_date)
    return start,stop

import datetime as dt
# 2014 recent
jd_start = dt.datetime(1988,1,1)
jd_stop = dt.datetime(1988,3,1)

# 2011
#jd_start = dt.datetime(2013,4,20)
#jd_stop = dt.datetime(2013,4,24)

# ... or relative to now
jd_now = dt.datetime.utcnow()
jd_start = jd_now - dt.timedelta(days=3)
jd_stop = jd_now + dt.timedelta(days=3)

start_date = jd_start.strftime('%Y-%m-%d %H:00')
stop_date  = jd_stop.strftime('%Y-%m-%d %H:00')

jd_start = dt.datetime.strptime(start_date,'%Y-%m-%d %H:%M')
jd_stop = dt.datetime.strptime(stop_date,'%Y-%m-%d %H:%M')

print(start_date,'to',stop_date)
start,stop = dateRange(start_date,stop_date)

filter_list = [fes.And([filter1, filter2, service_filt, bbox_filter, start, stop])]

startposition = 0
while True:
    print 'getting records %d to %d' % (startposition, startposition+pagesize)
    csw.getrecords2(constraints=filter_list,
                    startposition=startposition, maxrecords=pagesize, sortby=sortby)
    for rec,item in csw.records.iteritems():
        print(item.title)
    print
    if csw.results['nextrecord'] == 0:
        break
    startposition += pagesize
    if startposition >= maxrecords:
        break


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Okay, now lets add another query filter and add it to the first one
Can you write Python code for it?

val = 'CariCOOS'
#val = '0115145'
filter2 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [fes.And([filter1, filter2])]

startposition = 0
maxrecords = 50
while True:
    print 'getting records %d to %d' % (startposition, startposition+pagesize)
    csw.getrecords2(constraints=filter_list,
                    startposition=startposition, maxrecords=pagesize, sortby=sortby)
    for rec,item in csw.records.iteritems():
        print(item.title)
    print
    if csw.results['nextrecord'] == 0:
        break
    startposition += pagesize
    if startposition >= maxrecords:
        break

choice=np.random.choice(list(csw.records.keys()))
print(csw.records[choice].title)
csw.records[choice].references


I am working on the file "CSW/data.gov_pycsw.ipynb".
The first task is:
Specify a CSW endpoint.  You can test if it's working with a getCapabilities request:
<endpoint>?request=GetCapabilities&service=CSW
for example:
http://catalog.data.gov/csw-all?service=CSW&version=2.0.2&request=GetCapabilities
Can you write Python code for it?

#endpoint = 'http://catalog.data.gov/csw-all'  #granule level production catalog
#endpoint = 'https://data.ioos.us/csw'
endpoint = 'https://dev-catalog.ioos.us/csw'
#endpoint = 'http://geoport.whoi.edu/csw'
#endpoint = 'http://www.ngdc.noaa.gov/geoportal/csw'
csw = CatalogueServiceWeb(endpoint,timeout=60)
print csw.version

val = 'sea_water_salinity'
#val = 'NODC'
filter1 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [ filter1 ]
csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')
print len(csw.records.keys())
for rec in list(csw.records.keys()):
    print csw.records[rec].title


I am working on the file "CSW/data.ioos.us-pycsw.ipynb".
The first task is:
Now let's print out the references (service endpoints) to see what types of services are available
Can you write Python code for it?

choice=np.random.choice(list(csw.records.keys()))
print(csw.records[choice].title)
csw.records[choice].references

csw.records[choice].xml


I am working on the file "CSW/data.ioos.us-pycsw.ipynb".
The first task is:
Search first for records containing the two text strings
Can you write Python code for it?

val = 'COAWST'
filter1 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [ filter1 ]

val = 'USGS'
filter2 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [fes.And([filter1, filter2])]

endpoint = 'http://gamone.whoi.edu/csw'

csw = CatalogueServiceWeb(endpoint,timeout=60,version="2.0.2")
print csw.version
csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')
print len(csw.records.keys())
for rec in list(csw.records.keys()):
    print csw.records[rec].title

endpoint = 'http://gamone.whoi.edu/csw/'

csw = CatalogueServiceWeb(endpoint,timeout=60,version="2.0.2")
print csw.version
csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')
print len(csw.records.keys())
for rec in list(csw.records.keys()):
    print csw.records[rec].title

val = 'G1SST'
filter1 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [ filter1 ]

val = 'GHRSST'
filter2 = fes.PropertyIsLike(propertyname='apiso:AnyText',literal=('*%s*' % val),
                        escapeChar='\\',wildCard='*',singleChar='?')
filter_list = [fes.And([filter1, filter2])]

endpoint = 'https://ngdc.noaa.gov/geoportal/csw'

csw = CatalogueServiceWeb(endpoint,timeout=60)
print csw.version
csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')
print len(csw.records.keys())
for rec in list(csw.records.keys()):
    print csw.records[rec].title

endpoint = 'https://dev-catalog.ioos.us/csw'

csw = CatalogueServiceWeb(endpoint,timeout=60)
print csw.version
csw.getrecords2(constraints=filter_list,maxrecords=100,esn='full')
print len(csw.records.keys())
for rec in list(csw.records.keys()):
    print csw.records[rec].title

def fes_date_filter(start, stop, constraint='overlaps'):
    """
    Take...


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Deep Learning
Why do we want Deep Neural Networks?
On the (Small) Number of Atoms in the Universe
On the number of digital pictures
Let's switch from Go positions to digital pictures. There is an art project to display every possible picture. Surely that would take a long time, because there must be many possible pictures. But how many?
We will assume the color model known as True Color, in which each pixel can be one of 224 ≅ 17 million distinct colors. The digital camera shown below left has 12 million pixels, and we'll also consider much smaller pictures: the array below middle, with 300 pixels, and the array below right with just 12 pixels; shown are some of the possible pictures:

12M pixels
300 pixels
12 pixels

Quiz: Which of these produces a number of pictures similar to the number of atoms in the universe?
**Answer: An array of n pixels produces (17 million)^n different pictures. (17 million)^12 ≅ 10^86, so the tiny 12-pixel array produces a million times more pictures than the number of atoms in the universe!
How about the 300 pixel array? It can produce 10^2167 pictures. You may think the number of atoms in the universe is big, but that's just peanuts to the number of pictures in a 300-pixel array. And 12M pixels? 10^86696638 pictures. Fuggedaboutit!
So the number of possible pictures is really, really, really big. And the number of atoms in the universe is looking relatively small, at least as a number of combinations.
A Deeper MLP
Next we build a two-layer MLP with the same number of hidden nodes, half in each layer.
Can you write Python code for it?

from keras.models import Sequential
nb_layers = 2
mlp2 = Sequential()
# add hidden layers
for i in range(nb_layers):
    mlp2.add(Dense(output_dim=nb_hidden//nb_layers, input_dim=nb_input if i==0 else nb_hidden//nb_layers, init='uniform'))
    mlp2.add(Activation('sigmoid'))
# add output layer
mlp2.add(Dense(output_dim=nb_output, input_dim=nb_hidden//nb_layers, init='uniform'))
mlp2.add(Activation('softmax'))

mlp2.compile(loss='categorical_crossentropy', optimizer='SGD', metrics=["accuracy"])

mlp2.fit(X_train, y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1)


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Did you notice anything about the accuracy? Let's train it some more.
Can you write Python code for it?


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
First we define the "architecture" of the network
Can you write Python code for it?

from keras.models import Sequential
from keras.layers.core import Dense, Activation
mlp = Sequential()
mlp.add(Dense(output_dim=nb_hidden, input_dim=nb_input, init='uniform'))
mlp.add(Activation('sigmoid'))
mlp.add(Dense(output_dim=nb_output, input_dim=nb_hidden, init='uniform'))
mlp.add(Activation('softmax'))


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
For example, let's inspect the first 3 labels:
Can you write Python code for it?

print('labels_train[:3]:', labels_train[:3])
print('y_train[:3]', y_train[:3])


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Next we train the model on our training data. Watch the loss, which is the objective function which we are minimising, and the estimated accuracy of the model.
Can you write Python code for it?

mlp.fit(X_train, y_train,
        batch_size=batch_size, nb_epoch=nb_epoch,
        verbose=1)


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Once the model is trained, we can evaluate its performance on the test data.
Can you write Python code for it?

mlp.evaluate(X_test, y_test)

def draw_mlp_prediction(j):
    plot_mnist_digit(to_images(X_test)[j])
    prediction = mlp.predict_classes(X_test[j:j+1], verbose=False)[0]
    print(j, ':', '\tpredict:', prediction, '\tactual:', labels_test[j])
interact(draw_mlp_prediction, j=(0, len(X_test)-1))

plot_10_by_10_images(images_test, figsize=(10,10))


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Simple Multi-Layer Perceptron (MLP)
The simplest kind of Artificial Neural Network is as Multi-Layer Perceptron (MLP) with a single hidden layer.
Can you write Python code for it?

# Neural Network Architecture Parameters
nb_input = 784
nb_hidden = 512
nb_output = 10
# Training Parameters
nb_epoch = 1
batch_size = 128


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
Stacked Autoencoder
Hinton 2006 (local)
Can you write Python code for it?

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout

class StackedAutoencoder(object):

    def __init__(self, layers, mode='autoencoder',
                 activation='sigmoid', init='uniform', final_activation='softmax',
                 dropout=0.2, optimizer='SGD', metrics=["accuracy"]):
        self.layers = layers
        self.mode = mode
        self.activation = activation
        self.final_activation = final_activation
        self.init = init
        self.dropout = dropout
        self.optimizer = optimizer
        self.metrics = metrics
        self._model = None

        self.build()
        self.compile()

    def _add_layer(self, model, i, is_encoder):
        if is_encoder:
            input_dim, output_dim = self.layers[i], self.layers[i+1]
            activation = self.final_activation if i==len(self.layers)-2 else self.activation
        else:
            input_dim, output_dim = self.layers[i+1], self.layers[i]
            activation = self.activation
        model.add(Dense(output_dim=output_dim,
                        input_dim=input_dim,
                        init=self.init))
        model.add(Activation(activation))

    def build(self):
        self.encoder = Sequential()
        self.decoder = Sequential()
        self.autoencoder = Sequential()
        for i in range(len(self.layers)-1):
            self._add_layer(self.encoder, i, True)
            self._add_layer(self.autoencoder, i, True)
            #if i<len(self.layers)-2:
            #    self.autoencoder.add(Dropout(self.dropout))

        # Note that the decoder layers are in reverse order
        for i in reversed(range(len(self.layers)-1)):
            self._add_layer(self.decoder, i, False)
            self._add_layer(self.autoencoder, i, False)

    def compile(self):
        print("Compiling the encoder ...")
        self.encoder.compile(loss='categorical_crossentropy', optimizer=self.optimizer, metrics=self.metrics)
        print("Compiling...


I am working on the file "CTPUG Talk 2016-05-14.ipynb".
The first task is:
then we compile it. This takes the symbolic computational graph of the model and compiles it an efficient implementation which can then be used to train and evaluate the model.
Note that we have to specify what loss/objective function we want to use as well which optimisation algorithm to use. SGD stands for Stochastic Gradient Descent.
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Hello_word_pycuda.ipynb".
The first task is:
Celem naszego programu będzie pomnożenie wszystkich elementów tej tablicy przez zadaną liczbę.
Tablica a znajduje się w pamięci systemu gospodarza. Aby wykonać operacje na niej z pomocą GPU musimy zaalokować pamięć na urządzeniu GPU:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Hello_word_pycuda.ipynb".
The first task is:
Grids & Blocks
W CUDA wątki można uruchamiać w blokach (blocks). Należy zapamiętać regułe:
wszystkie wątki na jednym bloku zawsze będą uruchamiane na tym samym multiprocesorze
Co za tym idzie, będą miały do dyspozycji tą samą pamięc typu "shared memory". Maksymalna liczba wątków w jednym bloku jest z reguły ograniczona do 512 lub 1024 w zależności od typu urządzenia GPU. Warto zauważyć, że jest to liczba zdecydowanie większa od liczby procesorów skalarnych w na jednym multiprocesorze.

Jeżeli chcemy odpalic np. milion wątków, to musimy wykorzystać tzw. grid bloków. Czyli odpalając jądro obliczeniowe specyfikujemy ile potrzeujemy wątków i ile jaki chcemy mieć rozmiar bloku. Chcąc mieć np. $128\times 64$ wątki, rozsądnym jest odpalenie jądra z rozmiarem gridu $128$ i rozmiarem bloku $64$.
Ponadto, ponieważ często operujemy siatkami reprezentujemy dwu i trój-wymiarowe pola, CUDA ma wbudowany mechanizm, który ułatwia posługiwaniem się dwu i trój-wymiarowymi siatkami.
Poniższy przykład pokazuje jak zbudować grid składający się z dwóch bloków, po pięc wątków.
Ćwiczenie.
Zastąp w kodzie  if(threadIdx.x==1) przez:

if(blockIdx.x==1)
if(idx==1)

i sprawdź działanie.
Can you write Python code for it?

import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    __global__ void kernel(float *a)
    {
      int idx = threadIdx.x + blockDim.x*blockIdx.x;

      if(threadIdx.x==0)
          a[idx] += 1.0f;
      a[idx] += 10.0f * idx;
    }
    """)

a = np.zeros(10).astype(np.float32)
func = mod.get_function("kernel")
print (np.linspace(0,9,10))
print ("----------------")
print (a)
func(cuda.InOut(a),block=(5,1,1),grid=(2,1,1))
print (a)


I am working on the file "CUDA/iCSE_PR_Hello_word_pycuda.ipynb".
The first task is:
Uwaga printf - działa tylko z konsoli!
Can you write Python code for it?

%%writefile cuda_printf.py
import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    #include <stdio.h>

    __global__ void printf_test()
    {
      printf("GONZO %d.%d.%d\\n", threadIdx.x, threadIdx.y, threadIdx.z);
    }
    """)

func = mod.get_function("printf_test")
func(block=(2,2,1))


I am working on the file "CUDA/iCSE_PR_Hello_word_pycuda.ipynb".
The first task is:
W tablicy a_gpu powinniśmy mieć teraz
Can you write Python code for it?

print( a)
cuda.memcpy_dtoh(a, a_gpu)
print( a)


I am working on the file "CUDA/iCSE_PR_Hello_word_pycuda.ipynb".
The first task is:
gpuarray
Moduł gpuarray umożliwia bardzo zwarty zapis operacji na wektorach na GPU. Kluczową komendą jest gpuarray.to_gpu(.., która pozwala skopiować wektor numpy do urządzenia GPU, zwracając uchwyt do danych na GPU. Pewne operacje można wykonać automatycznie na GPU po prostu wpisując wzór arytmetyczny.
Can you write Python code for it?

import pycuda.gpuarray as gpuarray

a = np.linspace(1,16,16).astype(np.float32)
a_gpu = gpuarray.to_gpu(a.astype(np.float32))
a_doubled = (a_gpu*2).get()

print (a_gpu)
print (a_doubled)


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
CPU
Aby się przekonać, czy GPU rzeczywiście wykonał powyższe operacje szybciej niż CPU, sprawdźmy jak szybko wykona się kod skompilowany przez cythona. Cython daje wyniki nie gorsze od czystego "C".
Can you write Python code for it?

%load_ext Cython

%%cython

cdef extern from "stdlib.h":
    int RAND_MAX

from libc.stdlib cimport rand
from libc.math cimport pow

def cpu_rand():

    cdef double a,b
    cdef long n=0
    for i in range(5000000):
        a = float(rand())/RAND_MAX
        b = float(rand())/RAND_MAX
        if pow(a-0.5,2.0)+pow(b-0.5,2.0)-0.25<0:
            n=n+1

    return n

%%time
cpu_rand()


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
Generacja liczb losowych na urządzeniu
Funkcja curand pozwala nam skorzystać z wbudowanego generatora liczb losowych w CUDA. Możemy z jej pomocą zapełnić całą pamięć liczbami losowymi z przedziału $(0,1)$. Sprawdźmy najpiew ile mamy na GPU pamięci:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
Mamy więc ok 30s na CPU i ok 300ms na GPU - czyli ok 100x !
Należy jednak zauważyć, że na GPU wykonywaliśmy alokację wszystkich liczb w pamięci a potem ich równoległą redukcję. Jest to niezbyt optymalne i można by się spodziewać dalszego wzrostu względnej wydajności jesli by porównywać dokładnie te same algorytmy.
Zobaczmy jeszcze czy na CPU $\pi=3.14$:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
Ponieważ chcemy zapisywać liczny w pojedynczej prezycji, to mamy możliwość zaalokowania pamięci na:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
Wykres
... warto uważać ile punktów się rysuje...
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
Zadanie
Poeksperymentuj ze zwalnianiem pamięci wykonując np. takie operacje:
del a
print cuda.mem_get_info()[0]/1024**2
del b
print cuda.mem_get_info()[0]/1024**2
Can you write Python code for it?

a.get().shape

%%time
for i in range(10):
    a1 = a.get()

cuda.mem_get_info()[0]/1024.**3

%%time

pole = float(krnl(a, b).get())/(N/2)
ctx.synchronize()

print( "OK")

print (4*pole,"Pi z %d milionów losowań!"%(N/2/1e6))
print (np.pi)


I am working on the file "CUDA/iCSE_PR_Pi_on_pyCUDA.ipynb".
The first task is:
liczb 4 bajtowych.
Can you write Python code for it?

%%time
print(cuda.mem_get_info()[0]/1024**2)
N = 500000000
a = curand((N//2,))
b = curand((N//2,))
ctx.synchronize()
print(cuda.mem_get_info()[0]/1024**2)


I am working on the file "CUDA/iCSE_PR_Rownanie_Logistyczne.ipynb".
The first task is:
Algorytm z pętlą wewnątrz jądra CUDA
Napiszmy teraz algorytm, który będzie iterował równanie Niter razy wewnątrz jednego wywołania jądra CUDA.
Can you write Python code for it?

import pycuda.gpuarray as gpuarray

from pycuda.curandom import rand as curand
from pycuda.compiler import SourceModule
import pycuda.driver as cuda

try:
    ctx.pop()
    ctx.detach()
except:
    print( "No CTX!")

cuda.init()
device = cuda.Device(0)
ctx = device.make_context()

mod = SourceModule("""
    __global__ void logistic_iterations(float *a,float *x,int Niter)
    {

      int idx = threadIdx.x + blockDim.x*blockIdx.x;
      float a_ = a[idx];
      float x_ = x[idx];
      int i;
      for (i=0;i<Niter;i++){

          x_ = a_*x_*(1-x_);
      }

      x[idx] = x_;
    }
    """)
logistic_iterations = mod.get_function("logistic_iterations")

block_size=128
Nx = 10240
Na = 1024*2
blocks = Nx*Na//block_size

a = np.linspace(3.255,4,Na).astype(np.float32)
a = np.repeat(a,Nx)

a_gpu = gpuarray.to_gpu(a)
x_gpu = curand((Na*Nx,))

%%time
logistic_iterations(a_gpu,x_gpu, np.int32(10000),block=(block_size,1,1), grid=(blocks,1,1))
ctx.synchronize()

a,x = a_gpu.get(),x_gpu.get()

plt.figure(num=1, figsize=(9, 8))
every = 100
plt.plot(a[::every],x[::every],'.',markersize=1,alpha=1)
plt.plot([3.83,3.83],[0,1])

H, xedges, yedges = np.histogram2d(a,x,bins=(1024,1024))

plt.figure(num=1, figsize=(10,10))

plt.imshow(1-np.log(H.T+5e-1),origin='lower',cmap='gray')


I am working on the file "CUDA/iCSE_PR_Rownanie_Logistyczne.ipynb".
The first task is:
Jądro Elementwise
Zdefiniujemy sobie jądro, które dla wektora stanów początkowych, element po  elementcie wykona iteracje rówania logistycznego. Ponieważ będziemy chcieli wykonać powyższe iteracje dla różnych parametrów $a$, zdefiniujemy nasze jądro tak by brało zarówno wektor wartości paramteru $a$ jak i wektor wartości początkowych. Ponieważ będziemy mieli tą samą wartość parametru $a$ dla wielu wartości początkowych to wykorzystamy użyteczną w tym przypadku funkcję numpy:
a =  np.repeat(a,Nx)
Can you write Python code for it?

import numpy as np
Nx = 1024
Na = 1024

a = np.linspace(3.255,4,Na).astype(np.float32)
a =  np.repeat(a,Nx)

a_gpu = gpuarray.to_gpu(a)
x_gpu = curand((Na*Nx,))

from pycuda.elementwise import ElementwiseKernel
iterate = ElementwiseKernel(
        "float *a, float *x",
        "x[i] = a[i]*x[i]*(1.0f-x[i])",
        "iterate")

%%time
Niter = 1000
for i in range(Niter):
    iterate(a_gpu,x_gpu)
ctx.synchronize()
a,x = a_gpu.get(),x_gpu.get()

plt.figure(num=1, figsize=(10, 6))

every = 10
plt.plot(a[::every],x[::every],'.',markersize=1)
plt.plot([3.83,3.83],[0,1])


I am working on the file "CUDA/iCSE_PR_Rownanie_Logistyczne.ipynb".
The first task is:
Porównanie z wersją CPU
Dla porównania napiszemy prosty program, który oblicza iteracje równania logistycznego na CPU. Zatosujemy język cython, który umożliwia automatyczne skompilowanie funkcji do wydajnego kodu, którego wydajność jest porównywalna z kodem napisanym w języku C lub podobnym.
W wyniku działania programu widzimy, że nasze jądro wykonuje obliczenia znacznie szybciej.
Can you write Python code for it?

%load_ext Cython

%%cython
def logistic_cpu(double a = 3.56994):
    cdef double x

    cdef int i
    x = 0.1
    for i in range(1000*1024*1024):
        x = a*x*(1.0-x)

    return x

%%time
logistic_cpu(1.235)

print("OK")


I am working on the file "CUDA/iCSE_PR_SDE.ipynb".
The first task is:
Dane referencyjne dla walidacji
W tablicy hist_ref znajdują się dane referencyjne dla celów walidacji. Możemy sprawdzić czy program działa tak jak ten w pracy referencyjnej:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_SDE.ipynb".
The first task is:
Mając gotowe jądro, można wykonac testowe uruchomienie:
Can you write Python code for it?

import time

x  = np.zeros(N,dtype=np.float32)
v  = np.ones(N,dtype=np.float32)
rng_state = np.array(np.random.randint(1,2147483648,size=N),dtype=np.uint32)

x_g = gpuarray.to_gpu(x)
v_g = gpuarray.to_gpu(v)
rng_state_g = gpuarray.to_gpu(rng_state)

start = time.time()
for i in range(0,200000,spp):
    t = i * 2.0 * np.pi /omega /spp;
    SDE(x_g, v_g, rng_state_g, np.float32(t), block=(block_size,1,1), grid=(blocks,1))

ctx.synchronize()
elapsed = (time.time() - start)
x=x_g.get()
print (elapsed,N/1e6, 200000*N/elapsed/1024.**3,"Giter/sek")


I am working on the file "CUDA/iCSE_PR_SDE.ipynb".
The first task is:
W pewnych bardziej zaawansowanych przypadkach, można zastosować system szablonów np. mako templates (w projekcie http://sailfish.us.edu.pl).
Struktura kernela
Jądro:
__global__ void SDE(float *cx,float *cv,unsigned int *rng_state, float ct)

jest funkcją CUDA typu __global__, jako parametry przyjmuje tablice cx i cv, będące zmiennymi stanu układu dwóch równań różniczkowch:
$$ \dot x = v$$
$$ \dot v = -2\pi \cos(2.0\pi x) + A \cos(\omega t) + F - \gamma v$$
Ponadto w wywołaniu przekazujemy czas (przez wartość) oraz wskaźnik do stanu generatora liczb losowych na GPU.
Funkje dostępne dla jądra z GPU to:
generator liczb losowych o rozkładzie jednostajnym:
 __device__ float rng_uni(unsigned int *state)

transformacja Boxa-Mullera:
 __device__ void bm_trans(float& u1, float& u2)

i wreszczcie funkcja obliczająca prawe strony układu równań:
 __device__ inline void diffEq(float &nx, float &nv, float x, float v, float t)

Zauważmy, że dla poprawienia wydajności, każde wywołanie kernela, powoduje wielokrotne (określone przez parametr spp) wykonanie pętli iteracyjnej.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import pycuda.gpuarray as gpuarray

from pycuda.curandom import rand as curand
from pycuda.compiler import SourceModule
import pycuda.driver as cuda

cuda.init()
device = cuda.Device(0)
ctx = device.make_context()
print (device.name(), device.compute_capability(),device.total_memory()/1024.**3,"GB")

blocks = 2**11
block_size = 2**8
N = blocks*block_size

omega = 4.9
spp = 100
dt = 2.0*np.pi/omega/spp
pars = {'samples':spp,'N':N,'dt':dt,'gam':0.9,'d0':0.001,'omega':omega,'force':0.1,'amp':4.2}

rng_src = """
#define PI 3.14159265358979f
/*
 * Return a uniformly distributed random number from the
 * [0;1] range.
 */
__device__ float rng_uni(unsigned int *state)
{
	unsigned int x = *state;

	x = x ^ (x >> 13);
	x = x ^ (x << 17);
	x = x ^ (x >> 5);

	*state = x;

	return x / 4294967296.0f;
}
/*
 * Generate two normal variates given two uniform variates.
 */
__device__ void bm_trans(float& u1, float& u2)
{
	float r = sqrtf(-2.0f * logf(u1));
	float phi = 2.0f * PI * u2;
	u1 = r * cosf(phi);
	u2 = r * sinf(phi);
}

"""

src = """
  __device__ inline void diffEq(float &nx, float &nv, float x, float v, float t)
{{
	nx = v;
	nv = -2.0f * PI * cosf(2.0f * PI * x) + {amp}f * cosf({omega}f * t) + {force}f - {gam}f * v;
}}

__global__ void SDE(float *cx,float *cv,unsigned int *rng_state, float ct)
    {{
      int idx = blockDim.x*blockIdx.x + threadIdx.x;
      float n1, n2;
      unsigned int lrng_state;
      float xim, vim, xt1, vt1, xt2, vt2,t,x,v;
      lrng_state = rng_state[idx];
      t = ct;
      x = cx[idx];
	  v = cv[idx];

      for (int i=1;i<={samples};i++)  {{
      	n1 = rng_uni(&lrng_state);
		n2 = rng_uni(&lrng_state);
		bm_trans(n1, n2);
	diffEq(xt1, vt1, x, v, t);
		xim = x + xt1 * {dt}f;
		vim = v + vt1 * {dt}f + sqrtf({dt}f * {gam}f * {d0}f * 2.0f) * n1;
		t = ct + i * {dt}f;
	diffEq(xt2, vt2, xim, vim, t);
		x += 0.5f * {dt}f * (xt1 + xt2);
		v += 0.5f * {dt}f * (vt1 + vt2) + sqrtf(2.0f * {dt}f * {gam}f * {d0}f) *...


I am working on the file "CUDA/iCSE_PR_SDE.ipynb".
The first task is:
Wynikiem działania programu jest $N$ liczb określających końcowe położenie cząstki. Możemy zwizualizować je wykorzystując np. hostogram:
Can you write Python code for it?


I am working on the file "CUDA/iCSE_PR_SDE.ipynb".
The first task is:
lub:
Can you write Python code for it?

print('{zmienna} a nawiasy: {{}}'.format( **{"zmienna": 123} ))


I am working on the file "CUDA/iCSE_PR_map2d.ipynb".
The first task is:
Krok czwarty
Algorytm ten nie jest korzystny, gdyż rozmiar bloku determinuje rozmiar siatki na, której próbkujemy funkcje.
Optymalnie było by wykonywać operacje w blokach o zadanym rozmiarze, niezależnie od ilości próbek danego obszaru.
Poniższy przykład wykorzystuje dwuwymiarową strukturę zarówno bloku jak i gridu. Dzielimy wątki tak by w obrębie jednego bloku były wewnatrz kwadratu o bokach 4x4.
Can you write Python code for it?

import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    __global__ void sin2da(float *z)
    {
      int ix = threadIdx.x + blockIdx.x * blockDim.x;
      int iy = threadIdx.y + blockIdx.y * blockDim.y;

      int gid = ix + iy * blockDim.x * gridDim.x;
      float x = -4.0f+6.0f*float(ix)/(blockDim.x*gridDim.x);
      float y = -3.0f+6.0f*float(iy)/(blockDim.y*gridDim.y);

      z[gid] = sinf(powf(x,2.0f)+powf(y,2.0f));
    }
    """)

block_size = 4
Nx = 32*block_size
Ny = 32*block_size
x = np.linspace(-4,2,Nx).astype(np.float32)
y = np.linspace(-3,3,Ny).astype(np.float32)
XX,YY = np.meshgrid(x,y)
z = np.zeros(Nx*Ny).astype(np.float32)

func = mod.get_function("sin2da")
func(cuda.Out(z),\
     block=(block_size,block_size,1),\
     grid=(Nx//block_size,Ny//block_size,1)  )

plt.contourf(XX,YY,z.reshape(Ny,Nx) )


I am working on the file "CUDA/iCSE_PR_map2d.ipynb".
The first task is:
Krok drugi
Nie będziemy teraz przesyłać wartości $x$ do jądra, ale obliczymy je w locie, ze wzoru:
$$ x = x_0 + i \frac{\Delta x}{N}$$
Can you write Python code for it?

import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    __global__ void sin1da(float *y)
    {
      int idx = threadIdx.x + blockDim.x*blockIdx.x;
      float x = -3.0f+6.0f*float(idx)/blockDim.x;
      y[idx] = sinf(powf(x,2.0f));
    }
    """)

Nx = 128
x = np.linspace(-3,3,Nx).astype(np.float32)
y = np.empty_like(x)
func = mod.get_function("sin1da")
func(cuda.Out(y),block=(Nx,1,1),grid=(1,1,1))
plt.plot(x,y,'r')


I am working on the file "CUDA/iCSE_PR_map2d.ipynb".
The first task is:
Krok trzeci
Wykonamy probkowanie funkcji dwóch zmiennych, korzystając z wywołania jądra, które zawiera $N_x$ wątków w bloku i $N_y$ bloków.
Proszę zwrócić szczególną uwagę na linie:
int idx = threadIdx.x;
int idy = blockIdx.x;

zawierające wykorzystanie odpowiednich indeksów na CUDA, oraz sposób obliczania globalnego indeksu tablicy wartości, która jest w formacie "row-major"!
int gid = idx + blockDim.x*idy;
Can you write Python code for it?

import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    __global__ void sin2d(float *z)
    {
      int idx = threadIdx.x;
      int idy = blockIdx.x;

      int gid = idx + blockDim.x*idy;

      float x = -4.0f+6.0f*float(idx)/blockDim.x;
      float y = -3.0f+6.0f*float(idy)/gridDim.x;

      z[gid] = sinf(powf(x,2.0f)+powf(y,2.0f));
    }
    """)

Nx = 128
Ny = 64
x = np.linspace(-4,2,Nx).astype(np.float32)
y = np.linspace(-3,3,Ny).astype(np.float32)
XX,YY = np.meshgrid(x,y)
z = np.zeros(Nx*Ny).astype(np.float32)

func = mod.get_function("sin2d")
func(cuda.Out(z),block=(Nx,1,1),grid=(Ny,1,1))


I am working on the file "CX-Bot-Translate_Translation.ipynb".
The first task is:
END
Can you write Python code for it?

print('Translation Notebook: RAN successfully to desired point')


I am working on the file "CX-Bot-Translate_Translation.ipynb".
The first task is:
Translate CX Functions
Can you write Python code for it?

##################################################
def get_source_texts_for_translate(component, df):
    if component == SheetsName.Training_Phrases:
        source_lang_col = 2
    elif component == SheetsName.Pages:
        source_lang_col = 6
    else: #Entities, Flows & Route_Groups
        source_lang_col = 4

    source_lang = df.iloc[1,source_lang_col] #for translate
    print(f'translate_cx({component.name}): get_source_texts_for_translate({component.name},df) - Reading {len(df)} rows from Sheets')
    row = 2
    input_texts = []
    while row < len(df):
        if component == SheetsName.Training_Phrases or (component == SheetsName.Entities and df.iloc[row,2] == '1'):
            input_texts.append(df.iloc[row, source_lang_col])
        elif component == SheetsName.Entities and df.iloc[row,2] != '1':
            input_texts.append( set_no_translate_markings(df.iloc[row, source_lang_col]) )
        else: #Flows, Pages & Route_Groups
            input_texts.append( eval_no_translate_markings(df.iloc[row, source_lang_col]) )
        row += 1

    return input_texts

################################################################################
def translate_and_write_to_sheets(component, df, input_texts, target_lang, is_one_lang_only):
    if component == SheetsName.Training_Phrases:
        source_lang_col = 2
        target_lang_starting_col = 3
        mime_type = 'text/html'
    elif component == SheetsName.Pages:
        source_lang_col = 6
        target_lang_starting_col = 7
        mime_type = 'text/html'
    else: #Entities, Flows & Route_Groups
        source_lang_col = 4
        target_lang_starting_col = 5
        mime_type = 'text/html'

    source_lang = df.iloc[1,source_lang_col] #for translate

    if target_lang != None:
        col = df.columns.get_loc(target_lang)
    else:
        col = target_lang_starting_col

    if is_one_lang_only:
        target_lang = df.iloc[1,col] #for translate
        write_result(component.name, col+1, 4, [['translating...']])
    ...


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
Global Variables & Functions
Can you write Python code for it?

Google_Sheets_ID = Google_Sheets_URL[Google_Sheets_URL.find('spreadsheets/d/')+len('spreadsheets/d/'):]
Google_Sheets_ID = Google_Sheets_ID[:Google_Sheets_ID.find('/')]
print(f'Google_Sheets_URL: [{Google_Sheets_URL}]')
print(f'Google_Sheets_ID:  [{Google_Sheets_ID}]')

CX_Agent_Link = CX_Agent_URL[CX_Agent_URL.find('projects'):]
print(f'CX_Agent_URL:  [{CX_Agent_URL}]')
print(f'CX_Agent_Link: [{CX_Agent_Link}]')

CX_Project_ID  = get_cx_id(CX_Agent_Link,'projects','locations')
CX_Location_ID = get_cx_id(CX_Agent_Link,'locations','agents')
print(f'CX_Project_ID:  [{CX_Project_ID}]')
print(f'CX_Location_ID: [{CX_Location_ID}]')

GSheets_Creds  = None
GSheets_Scopes = ['https://www.googleapis.com/auth/spreadsheets']

CX_Creds = None
CX_Client_Options = None

CX_Agent_Default_Lang = None
CX_Agent_Supported_Langs = []
Translate_Source_Lang = None

__DEBUG = False
__INFO  = True

def get_cx_credentials():
    global CX_Creds
    global CX_Agent_JSON
    if len(CX_Agent_JSON) == 0:
        return None
    elif CX_Creds is None:
        CX_Creds = service_account.Credentials.from_service_account_file(CX_Agent_JSON)
    return CX_Creds

def get_cx_client_options():
    global CX_Client_Options
    if CX_Client_Options is None:
        if CX_Location_ID != 'global':
            # Reference: https://cloud.google.com/dialogflow/cx/docs/concept/region#api
            api_endpoint= f'{CX_Location_ID}-dialogflow.googleapis.com:443'
        else:
            api_endpoint= 'dialogflow.googleapis.com:443'
        CX_Client_Options = {'api_endpoint': api_endpoint}
    return CX_Client_Options

def get_agent():
    agents_client = cx_services.agents.AgentsClient(credentials=get_cx_credentials(), client_options=get_cx_client_options())

    agent_request = cx_types.GetAgentRequest()
    agent_request.name = CX_Agent_Link
    response = agents_client.get_agent(agent_request)

    return response

def get_cx_default_lang():
    global CX_Agent_Default_Lang
    if CX_Agent_Default_Lang == None:
        agent =...


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
Run the [Agent] Notebook and Update Sheets
Can you write Python code for it?

%run CX-Bot-Translate_Agent.ipynb

try:
    write_intents_to_sheets()
    write_entities_to_sheets()
    write_flows_to_sheets()
    write_pages_to_sheets()
    write_route_groups_to_sheets()
except Exception as e:
    print(f'Exception:\n{e}')


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
Run the [Translation] Notebook and Translate Agent in Sheets
Can you write Python code for it?

%run CX-Bot-Translate_Translation.ipynb

## Feel free to comment out CX components that does not need translation
try:
    translate_cx(SheetsName.Training_Phrases)
    translate_cx(SheetsName.Entities)
    translate_cx(SheetsName.Flows)
    translate_cx(SheetsName.Pages)
    translate_cx(SheetsName.Route_Groups)
except Exception as e:
    print(f'Exception:\n{e}')


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
Update Translations in Sheets to Agent
Run the cells below when you are satisfied with translations in Sheets and you are ready to apply it to Dialogflow CX
Can you write Python code for it?

## Feel free to comment out CX components that does not require updates from Sheets back to CX
try:
    update_all_intents()
#     update_intents_by_lang('Chinese - Simplified')
    update_all_entities()
#     update_entities_by_lang('Tamil')
    update_all_flows()
#     update_flows_by_lang('Tamil')
    update_all_pages()
#     update_pages_by_lang('Chinese - Simplified')
    update_all_route_groups()
#     update_route_groups_by_lang('Tamil')
except Exception as e:
    print(f'Exception:\n{e}')


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
Validation
Can you write Python code for it?

##############################################
### Validation - Check if we can access Sheets
##############################################

if len(GSheets_JSON) != 0:
    GSheets_Creds = service_account.Credentials.from_service_account_file(GSheets_JSON)
else:
    if os.path.exists('token.json'):
        GSheets_Creds = Credentials.from_authorized_user_file('token.json', GSheets_Scopes)
    # If there are no (valid) credentials available, let the user log in.
    if not GSheets_Creds or not GSheets_Creds.valid:
        if GSheets_Creds and GSheets_Creds.expired and GSheets_Creds.refresh_token:
            GSheets_Creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file('credentials.json', GSheets_Scopes)
            GSheets_Creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.json', 'w') as token:
            token.write(GSheets_Creds.to_json())

try:
    service = build('sheets', 'v4', credentials=GSheets_Creds)
    sheets  = service.spreadsheets().get(spreadsheetId=Google_Sheets_ID).execute()
    print(f'Validating Access to Sheets: \nSUCCESS in accessing Google Sheets at ID:[{Google_Sheets_ID}] and URL:[{Google_Sheets_URL}].\n')
except Exception as e:
    raise Exception(f'Validating Access to Sheets: \nFAILED to access Google Sheets at ID:[{Google_Sheets_ID}] and URL:[{Google_Sheets_URL}]. See below:\n{e}\n')

################################################
### Validation - Check if we can access CX Agent
################################################
try:
    get_agent()
    print(f'Validating Access to CX Agent: \nSUCCESS in accessing CX Agent at LINK:[{CX_Agent_Link}] and URL:[{CX_Agent_URL}].\n')
except Exception as e:
    raise Exception(f'Validating Access to CX Agent: \nFAILED to access CX Agent at LINK:[{CX_Agent_Link}] and URL:[{CX_Agent_URL}]. See below:\n{e}\n')


I am working on the file "CX-Bot-Translate__MAIN.ipynb".
The first task is:
ipynb env:
Can you write Python code for it?

!python3 -V
#!python3 -m pip list | wc -l
#!python3 -m pip list | grep google

from IPython.core.display import display, HTML
display(HTML("<style>.container { width:80% !important; }</style>"))


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
1. Introduction
In this notebook we will analyze the behavior of logistic regression and support vector machines on the dataset in file Dataset2D.mat. We first load the dataset.
Can you write Python code for it?

# Load dataset
matvar = scipy.io.loadmat('Dataset2D.mat')
Xtrain = matvar['xTrain']
Xtest = matvar['xTest']
Xval = matvar['xVal']

# We must use astype(int) to convert the original target values (which are unsigned integers) to int.
Ytrain = matvar['yTrain'].astype(int)
Ytest = matvar['yTest'].astype(int)
Yval = matvar['yVal'].astype(int)


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
1.1 Data Preparation.
Normalize the dataset. Remind that the same transformation must be applied to training, validation and test data. Store train, validation and test input data in variables Xtrain, Xval and Xtest
Can you write Python code for it?

# <SOL>
# Data normalization
def normalize(X, mx=None, sx=None):

    # Compute means and standard deviations
    if mx is None:
        mx = np.mean(X, axis=0)
    if sx is None:
        sx = np.std(X, axis=0)

    # Normalize
    X0 = (X-mx)/sx

    return X0, mx, sx

# Normalize data
Xtrain, mx, sx = normalize(Xtrain)
Xval, mx, sx = normalize(Xval, mx, sx)
Xtest, mx, sx = normalize(Xtest, mx, sx)

n_tr = Xtrain.shape[0]
n_val = Xval.shape[0]
n_tst = Xtest.shape[0]

print('The number of training samples is ' + str(n_tr))
print('The number of validation samples is ' + str(n_val))
print('The number of test samples is ' + str(n_tst))
print('The data dimension is ' + str(Xtrain.shape[1]))
# </SOL>

# Check normalization
print(np.mean(Xtrain, axis=0))
print(np.mean(Xval, axis=0))
print(np.mean(Xtest, axis=0))
print(np.std(Xtrain, axis=0))
print(np.std(Xval, axis=0))
print(np.std(Xtest, axis=0))


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
2. Linear Classification with Logistic Regression.
First we will analyze the behavior of logistic regression for this dataset.
2.1. MAP estimator.
Implement a function to compute the MAP estimate of the parameters of a linear logistic regression model with Gaussian prior and a given value of the inverse regularization parameter $C$. The method should return the estimated parameter and the negative log-likelihood, $\text{NLL}({\bf w})$. The sintaxis must be
    w, NLL = logregFitR(Z_tr, Y_tr, rho, C, n_it)
where

Z_tr is the input training data matrix (one instance per row)
Y_tr contains the labels of corresponding to each row in the data matrix
rho is the learning step
C is the inverse regularizer
n_it is the number of iterations
Can you write Python code for it?

# <SOL>
# Define the logistic function
def logistic(x):
   p = 1.0 / (1 + np.exp(-x))
   return p

# MAP trainer.
def logregFitR(Z_tr, Y_tr, rho, C, n_it):

    # Data dimension
    n_dim = Z_tr.shape[1]

    # Initialize variables
    nll_tr = np.zeros(n_it)
    pe_tr = np.zeros(n_it)
    w = np.random.randn(n_dim,1)

    # Running the gradient descent algorithm
    for n in range(n_it):

        # Compute posterior probabilities for weight w
        p1_tr = logistic(np.dot(Z_tr, w))
        p0_tr = logistic(-np.dot(Z_tr, w))

        # Compute negative log-likelihood
        nll_tr[n] = - np.dot(Y_tr.T, np.log(p1_tr)) - np.dot((1-Y_tr).T, np.log(p0_tr))

        # Update weights
        w = (1-2*rho/C)*w + rho*np.dot(Z_tr.T, Y_tr - p1_tr)

    return w, nll_tr

# Compute predictions for a given model
def logregPredict(Z, w):

    # Compute posterior probability of class 1 for weights w.
    p = logistic(np.dot(Z, w))

    # Classify
    D = [int(np.round(pn)) for pn in p]

    return p, D

#</SOL>


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
2.2 Log-likelihood
Compute the MAP estimate for a polynomial regression with degree 5, for $C$ ranging from -0.01 to 100. Sample $C$ uniformly in a log scale, an plot using plt.semilogx.
Plot the final value of $\text{NLL}$ as a function of $C$. Can you explain the qualitative behavior of $\text{NLL}$ as $C$ grows?
The plot may show some oscillation because of the random noise introduced by random initializations of the learning algoritm. In order to smooth the results, you can initialize the random seed right before calling the logregFitR method, using
np.random.seed(3)
Can you write Python code for it?

# <SOL>
# Set parameters
nC = 50

logC = np.linspace(-3.5, 2, num=nC)
C_all = np.exp(logC)
n_it = 2000
rho = 0.001

# Compute Z_tr
poly = PolynomialFeatures(degree=5)
Z_tr = poly.fit_transform(Xtrain)

# Normalize columns (this is useful to make algorithms more stable).)
Zn, mz, sz = normalize(Z_tr[:,1:])
Z_tr = np.concatenate((np.ones((n_tr, 1)), Zn), axis=1)

# Compute Z_val
Z_val = poly.fit_transform(Xval)
Zn, mz, sz = normalize(Z_val[:,1:], mz, sz)
Z_val = np.concatenate((np.ones((n_val,1)), Zn), axis=1)

dim = Z_tr.shape[1]
L = np.zeros((nC, 1))
w_all = np.zeros((nC, dim))

# Train models
for k, C in enumerate(C_all):

    np.random.seed(3)
    w, L_all = logregFitR(Z_tr, Ytrain, rho, C, n_it)

    L[k] = L_all[-1]
    w_all[k] = w.T

plt.figure()
plt.semilogx(C_all, L, '.')
plt.xlabel('C')
plt.ylabel('Negative log-likelihood')

print("As C grows, the regularization effect dissapears, and the fit method minimizes NLL.")
# </SOL>

# This is a plot for the last value of C used in the code above.

if Xtrain.shape[1]==2:

    # Create a regtangular grid.
    x_min, x_max = Xtrain[:, 0].min(), Xtrain[:, 0].max()
    y_min, y_max = Xtrain[:, 1].min(), Xtrain[:, 1].max()
    dx = x_max - x_min
    dy = y_max - y_min
    h = dy /400
    xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                         np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
    X_grid = np.array([xx.ravel(), yy.ravel()]).T

    # Compute Z_grid
    Z_grid = poly.fit_transform(X_grid)
    n_grid = Z_grid.shape[0]
    Zn, mz, sz = normalize(Z_grid[:,1:], mz, sz)
    Z_grid = np.concatenate((np.ones((n_grid,1)), Zn), axis=1)

    # Compute the classifier output for all samples in the grid.
    pp, dd = logregPredict(Z_grid, w)
    pp = pp.reshape(xx.shape)

    # Paint output maps
    plt.figure()
    pylab.rcParams['figure.figsize'] = 8, 4  # Set figure size
    for i in [1, 2]:
        ax = plt.subplot(1,2,i)
        ax.set_xlabel('$x_0$')
        ax.set_ylabel('$x_1$')
        ax.axis('equal')
       ...


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
2.3. Training and test errors.
Plot the training and validation error rates as a function of $C$. Compute the value of $C$ minimizing the validation error rate.
Can you write Python code for it?

# <SOL>
# Train models
pe_tr = np.zeros((nC, 1))
pe_val = np.zeros((nC, 1))

for k, C in enumerate(C_all):

    p_tr, D_tr = logregPredict(Z_tr, w_all[k])
    p_val, D_val = logregPredict(Z_val, w_all[k])

    # Compute error rates
    E_tr = D_tr!=Ytrain.T
    E_val = D_val!=Yval.T

    # Error rates
    pe_tr[k] = np.mean(E_tr)
    pe_val[k] = np.mean(E_val)

plt.figure()
plt.semilogx(C_all, pe_tr, '.-', label='Train')
plt.semilogx(C_all, pe_val, '.-', label='Validation')
plt.legend(loc='best')

print("The optimal value of C is {0}".format(C_all[np.argmin(pe_val)]))

# </SOL>


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
3. Non-linear classification with Support Vector Machines
In this section we will train an SVM with Gaussian kernels. In this case, we will select parameter $C$ of the SVM by cross-validation.
3.1. Dataset preparation.
Join the training and validation datasets in a single input matrix X_tr2 and a single label vector Y_tr2
Can you write Python code for it?

# <SOL>
X_tr2 = np.concatenate((Xtrain, Xval), axis = 0)
Y_tr2 = np.concatenate((Ytrain, Yval), axis = 0)
# </SOL>


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
3.3. Hyperparameter optimization.
Compute the value of $C$ minimizing the validation error rate.
Can you write Python code for it?

# <SOL>
C_opt = C_all[np.argmin(pe_val)]
print("The optimal value of C in the explored range is {0}".format(C_opt))
# </SOL>


I am working on the file "C_lab1 Clasificacion/Pract_classification_professor.ipynb".
The first task is:
Visualize the input variables from the training set in a 2-dimensional plot.
Can you write Python code for it?

# Data visualization. This works for dimension 2 only.
if Xtrain.shape[1]==2:
    plt.scatter(Xtrain[:, 0], Xtrain[:, 1], c=Ytrain.flatten(), s=50, cmap='copper')
    plt.xlabel("$x_0$", fontsize=14)
    plt.ylabel("$x_1$", fontsize=14)
plt.show()


I am working on the file "C_lab1 Clasificacion/Pract_classification_student.ipynb".
The first task is:
2.3. Training and test errors.
Plot the training and validation error rates as a function of $C$. Compute the value of $C$ minimizing the validation error rate.
Can you write Python code for it?


I am working on the file "C_lab1 Clasificacion/Pract_classification_student.ipynb".
The first task is:
3. Non-linear classification with Support Vector Machines
In this section we will train an SVM with Gaussian kernels. In this case, we will select parameter $C$ of the SVM by cross-validation.
3.1. Dataset preparation.
Join the training and validation datasets in a single input matrix X_tr2 and a single label vector Y_tr2
Can you write Python code for it?


I am working on the file "C_lab1 Clasificacion/Pract_classification_student.ipynb".
The first task is:
3.3. Hyperparameter optimization.
Compute the value of $C$ minimizing the validation error rate.
Can you write Python code for it?


I am working on the file "C_lab1 Clasificacion/Pract_classification_student.ipynb".
The first task is:
3.3. Influence of $C$.
Repeate exercise 3.2 for $\gamma=5$ and different values of $C$, ranging from $10^{-3}$ to $10^{4}$, obtained by uniform sampling in a logarithmic scale. Plot the average number of errors as function of $C$.
Note that fitting the SVM may take some time, specially for the largest values of $C$.
Can you write Python code for it?


I am working on the file "C_lab1 Clasificacion/Pract_classification_student.ipynb".
The first task is:
Visualize the input variables from the training set in a 2-dimensional plot.
Can you write Python code for it?


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
1.2. Data partitioning
To begin, we will perform a random partition of the available data into training and test sets of 60,000 and 10,000 digits, respectively.
The notebook also considers a binary problem consisting of the recognition of digits 7 and 9. We have selected this pair of digits as it is one of the most confusing, but if you wish, you can use any other pair of digits.
Exercise 1.2: Create the binary classification problem 7 vs 9. Save the corresponding data matrices with names X_tr_bin, y_tr_bin, X_tst_bin, y_tst_bin. Make sure that the target vectors contain just 0s (for class 7) and 1s (class 9).
Can you write Python code for it?

from sklearn.model_selection import train_test_split

train_samples = 60000
test_samples = 10000

X_tr, X_tst, y_tr, y_tst = train_test_split(
    Xlocal, ylocal, train_size=train_samples, test_size=test_samples,
    random_state=0
)

print('Shape of input training data (multiclass):', X_tr.shape)
print('Shape of target training vector (multiclass):', y_tr.shape)
print('Shape of input test data (multiclass):', X_tst.shape)
print('Shape of target test vector (multiclass):', y_tst.shape)

# Exercise: create the binary classification problem 7 vs 9
# <SOL>
digit1 = 7
digit2 = 9

# Train data
X_tr_bin = X_tr[(y_tr==digit1)+(y_tr==digit2),]
y_tr_bin = y_tr[(y_tr==digit1)+(y_tr==digit2)]
y_tr_bin[y_tr_bin==digit1] = 0
y_tr_bin[y_tr_bin==digit2] = 1

# Repeat for test
X_tst_bin = X_tst[(y_tst==digit1)+(y_tst==digit2),]
y_tst_bin = y_tst[(y_tst==digit1)+(y_tst==digit2)]
y_tst_bin[y_tst_bin==digit1] = 0
y_tst_bin[y_tst_bin==digit2] = 1
# </SOL>

print('\nShape of input training data (binary):', X_tr_bin.shape)
print('Shape of target training vector (binary):', y_tr_bin.shape)
print('Shape of input test data (binary):', X_tst_bin.shape)
print('Shape of target test vector (binary):', y_tst_bin.shape)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
1.3. Data normalization
Next we will normalize the data for the multiclass and binary problems. When working with images, it is frequent to normalize the input data, corresponding to the gray intensity values of the different pixels, so that they take values in the range $[-0.5, 0.5]$.
Exercise 1.3: Implement the normalization of the input data. Make sure to normalize data independently for the multiclass and binary classification cases. Make also sure not to refit the scaler object when transforming the test partitions.
Can you write Python code for it?

# <SOL>
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(-.5, .5))
X_tr = scaler.fit_transform(X_tr)
X_tst = scaler.transform(X_tst)

X_tr_bin = scaler.fit_transform(X_tr_bin)
X_tst_bin = scaler.transform(X_tst_bin)
# </SOL>


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
2.1.2. Linear Classification with Logistic Regression
First we will analyze the behavior of logistic regression for this dataset using just the two first dimensions of PCA as the input variables.
Exercise 2.2: Use sklearn GridSearchCV and LogisticRegression methods to calculate the average classification error for different values of the inverse regularization parameter $C$. Explore $C$ using a logarithmic scale, e.g.,
$$C = [100, 10, 1, 0.1, 0.01, 0.001, 0.0001]$$
Use a 5-fold strategy to obtain the best value of parameter $C$.
Exercise 2.3: Calculate the average classification error rate (CE) and negative log-likelihood (NLL) of the best classifier.
For the next two cells to work, you need to use the following variable names:

C_param: Array with the explored values of $C$
CE_param: Array with the validation error rates for the corresponding values of $C$
clf: Best classifier selected with 5 fold strategy. It needs to implement a clf.predict_proba method
CE_LR2D: Average Classification Error Rate of previous classifier calculated over the test data
NLL_LR2D: Negative log-lilelihood of previous classifier calculated over the test data
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression

# <SOL>
C_param = [100, 10, 1, 0.1, 0.01, 0.001, 0.0001]
parameters = {'C': C_param}

clf = GridSearchCV(LogisticRegression(max_iter=250, tol=1e-3), parameters, cv=5)
clf.fit(X_tr_2D, y_tr_bin)

CE_param = 100 - 100 * clf.cv_results_['mean_test_score']
CE_LR2D  = 100 - 100*np.mean(y_tst_bin==clf.predict(X_tst_2D))
y_pred = clf.predict_log_proba(X_tst_2D)
NLL_LR2D = - np.sum(y_tst_bin.T @ y_pred[:,1]) - np.sum((1-y_tst_bin).T @ y_pred[:,0])
# </SOL>

plt.semilogx(C_param, CE_param)
plt.xlabel('C'), plt.ylabel('(%)'), plt.title('Classification Error Rate (Linear Logistic Regression)')
plt.show()

print('CE of Linear Logistic Regression Classifier (%):', CE_LR2D)
print('NLL of Linear Logistic Regression Classifier:', NLL_LR2D)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
2.2. Classification with all input features
Exercise 2.6: Analyze the performance in the binary classification problem of the following classifiers:

Logistic regression (validate $C$)
K-nearest neighbors (validate $K$)
Multi-layer perceptron (validate size of hidden layers)

For comparison purposes you can use both the average classifiation error rate and the negative log-likelihood.
In this section you need to use all 784 available features.
2.2.1. Logistic Regression
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression

# <SOL>
C_param = [100, 10, 1, 0.1, 0.01]
parameters = {'C': C_param}

clf = GridSearchCV(LogisticRegression(max_iter=2500, tol=1e-3), parameters, cv=5)
clf.fit(X_tr_bin, y_tr_bin)

CE_param = 100 - 100 * clf.cv_results_['mean_test_score']
CE_LR  = 100 - 100*np.mean(y_tst_bin==clf.predict(X_tst_bin))
y_pred = clf.predict_log_proba(X_tst_bin)
NLL_LR = - np.sum(y_tst_bin.T @ y_pred[:,1]) - np.sum((1-y_tst_bin).T @ y_pred[:,0])
# </SOL>

plt.semilogx(C_param, CE_param)
plt.xlabel('C'), plt.ylabel('(%)'), plt.title('Classification Error Rate (Linear Logistic Regression)')
plt.show()

print('CE of Linear Logistic Regression Classifier (%):', CE_LR)
print('NLL of Linear Logistic Regression Classifier:', NLL_LR)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
2.2.3. Multi-Layer Perceptron
Be aware that, in this case, the number of test samples together with the dimension of the input data implies long processing times for classification (close to 1 min per execution in Google Colab). You might consider to subsample the test partition for the validation process.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.neural_network import MLPClassifier

# <SOL>
parameters = {'activation':('relu', 'tanh'), 'max_iter':[2000, 4000],
              'hidden_layer_sizes': ((400,100), (400,200), (200,100))}
clf = GridSearchCV(MLPClassifier(), parameters, cv=5, verbose=2)
clf.fit(X_tr_bin, y_tr_bin)

CE_MLP  = 100 - 100*np.mean(y_tst_bin==clf.predict(X_tst_bin))
y_pred = clf.predict_proba(X_tst_bin)
y_pred = np.log(y_pred + 1e-6)
NLL_MLP = - np.sum(y_tst_bin.T @ y_pred[:,1]) - np.sum((1-y_tst_bin).T @ y_pred[:,0])
# </SOL>

print('CE of MLP Classifier (%):', CE_MLP)
print('NLL of MLP Classifier:', NLL_MLP)
print('Selected parameters:', clf.best_params_)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
3.2. Nearest Neighbor Method
In this section, you will analyze the performance of the nearest neighbor (1-NN) algorithm. The complexity of this algorithm grows with the size of the training set, so it is proposed to analyze the behavior of the algorithm for a variable size of the training set.
Exercise 3.2: Use the 1-NN method with a varying training set size. Obtain for each size:

The average classifiation error rate calculated on the test set
The fit time for the 1-NN method
The time taken to classify the complete test partition (10000 samples)
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
import time

train_size = [250, 500, 1000, 2000, 5000, 10000, 25000, 60000]
fit_time = []
test_time = []
CE = []

for ntrain in train_size:
    print('Classifying with', ntrain, 'samples')
    # Write your code here
    # <SOL>
    start = time.time()
    knn = KNeighborsClassifier()
    knn.fit(X_tr[:ntrain,:], y_tr[:ntrain])
    etime = time.time() - start
    fit_time.append(etime)
    start = time.time()
    CE.append(np.mean(knn.predict(X_tst)!=y_tst))
    etime = time.time() - start
    test_time.append(etime)
    # </SOL>

plt.figure(figsize=(15,3.5))
plt.subplot(1, 3, 1), plt.plot(train_size, 100*np.array(CE)), plt.xlabel('Training Set Size'), plt.ylabel('(%)'), plt.title('CE')
plt.subplot(1, 3, 2), plt.plot(train_size, fit_time), plt.xlabel('Training Set Size'), plt.ylabel('Seconds'), plt.title('Fit Time')
plt.subplot(1, 3, 3), plt.plot(train_size, test_time), plt.xlabel('Training Set Size'), plt.ylabel('Seconds'), plt.title('Test Time')
plt.show()
print('Average Classification Error for the 1-NN approach', 100*np.min(CE))


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
5.1. Using torch nn.Module and nn.Parameter
PyTorch nn module provides many attributes and methods to make simple the implementation and training of Neural Networks.
nn.Module and nn.Parameter allow to implement a concise network configuration, and simplify the calculation of the gradients

nn.Module is a PyTorch class that will be used to encapsulate and design a specific neural network, thus, it is central to the implementation of deep neural nets using PyTorch

nn.Parameter allow the definition of trainable network parameters. In this way, we will simplify the implementation of the training loop.

All parameters defined with nn.Parameter will have requires_grad = True

Below you can see a PyTorch fragment for the definition of a single layer perceptron (SLP) network. You can see that at least two methods need to be defined: the initialization of the network (including parameter definition and initialization), and a forward method that implements how the network produces its output for a given input pattern. Other auxiliary functions may be defined as well.
However, you can see that there is no need to implement a backward method for gradient calculation.
````
from torch import nn
class my_multiclass_net(nn.Module):
    def init(self, nin, nout):
        """This method initializes the network parameters
        Parameters nin and nout stand for the number of input parameters (features in X)
        and output parameters (number of classes)"""
        super().init()
        self.W = nn.Parameter(.1 * torch.randn(nin, nout))
        self.b = nn.Parameter(torch.zeros(nout))
def forward(self, x):
    return softmax(x @ self.W + self.b)

def softmax(t):
    """Compute softmax values for each sets of scores in t"""
    return t.exp() / t.exp().sum(-1).unsqueeze(-1)

````
You can see that by using nn.Parameter and nn.Module you can easily implement any function of your choice. However, we need to be careful about matrix dimensions and some particularities which are required to correctly operate PyTorch...
Can you write Python code for it?

from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """Note that now, we do not even need to initialize network parameters ourselves"""
        super().__init__()
        self.lin = nn.Linear(nin, nout)

    def forward(self, x):
        return self.lin(x)

loss_func = nn.CrossEntropyLoss()


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
5.2. Network Optimization
We cover in this subsection two different aspects about network training using PyTorch:

Using torch.optim allows an easier and more interpretable encoding of neural network training, and opens the door to more sophisticated training algorithms

Using minibatches can speed up network convergence. The idea of minibatches is that, at each epoch, gradients are evaluated over just a subset of the training input data. Training of the network will normally require more epochs but, as each epoch requires the evaluation of the network output for a smaller subset of training samples, the overall training time is normally reduced significantly.

torch.optim provides two convenient methods for neural network training:
   * opt.step() updates all network parameters using current gradients
   * opt.zero_grad() resets all network parameters
Can you write Python code for it?

from torch.utils.data import TensorDataset, DataLoader

train_ds = TensorDataset(X_tr_torch, y_tr_torch)
train_dl = DataLoader(train_ds, batch_size=64)

from torch import optim

my_net = my_multiclass_net(X_tr_torch.size()[1], y_tr_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 150

loss_tr = np.zeros(epochs)
loss_val = np.zeros(epochs)
CE_tr = np.zeros(epochs)
CE_val = np.zeros(epochs)

start = time.time()
for epoch in range(epochs):

    print(f'Current epoch: {epoch+1}    \r', end="")

    # In each epoch we iterate over all minibatches
    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_tr_torch)
        loss_tr[epoch] = loss_func(pred, y_tr_torch.argmax(axis=-1)).item()
        CE_tr[epoch] = CE(y_tr_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        CE_val[epoch] = CE(y_val_torch, pred_val).item()

print('Neural Network training completed in', time.time()-start, 'seconds')

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_tr, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(100*CE_tr, 'b'), plt.plot(100*CE_val, 'r'), plt.legend(['train', 'val']), plt.title('Classification Error (%)')
plt.show()


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
<font color='teal'> 3. Multi Class Classification </font>
In this section we will train classification schemes that allow us to discriminate between the 10 digits that make up the complete dataset. In this case, you must use the parameters provided, and you will be asked to study the execution times required by the different methods. To do this, you can use the time library as follows:
```
import time
start = time.time()
Some code should go here
etime = time.time() - start
````
3.1. Principal Component Analysis (PCA)
In a preliminary way, and as you did in section 2.2.1, we can analyze the first two PCA components of the available data. You can see that in this case there is a very important overlap between the digits of all the classes on the first two components of PCA.
In this section, we will apply the different classification strategies using all the available features.
Exercise 3.1: Analyze the variance of the successive projections that the PCA method would obtain, and reflect on whether a smaller-dimensional representation of the input data could be used.
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA(2)  # project from 784 to 2 dimensions
projected = pca.fit_transform(X_tr)
plt.scatter(projected[:, 0], projected[:, 1],
            c=y_tr, edgecolor='none', alpha=0.5,
            cmap=plt.cm.get_cmap('rainbow', 10))
plt.xlabel('component 1')
plt.ylabel('component 2')
plt.colorbar();

# Solution to Exercise 3.1

# <SOL>
pca = PCA(100)  # project from 784 to 100 dimensions
projected = pca.fit_transform(X_tr)
plt.plot(pca.explained_variance_)
plt.xlabel('Projection')
plt.ylabel('Explained variance')
plt.show()
# </SOL>


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
<font color='teal'> 5. Feed Forward Networks using PyTorch </font>
In this section we are going to illustrate how we can implement a multilayer perceptron-type neural network using the features of PyTorch. The network thus implemented will be equivalent to the one you would train using the scikit-learn MLP function, but thanks to the use of PyTorch it will be able to be executed parallelizing many of the calculations in GPU, which will allow a much faster training.
A first possibility would be the direct implementation of the neural network, through an implementation based on PyTorch tensors of the evaluation functions of the neural network and of the derivatives of the cost function with respect to the different parameters (back-propagation). However, PyTorch module nn provides different levels of abstraction that considerably simplify the implementation of the network, in addition to making the theoretical calculation of derivatives unnecessary.
Before proceding, we need to import training and test data as PyTorch tensors. The fragment below can be used for that purpose. Note that if you are using a GPU, all tensors should be moved to the GPU.
Exercise 5.1: Complete the code below to create PyTorch tensors for the different MNIST variables. The tensors for the input data have already been created for you. For encoding class membership, you need to create y_tr and y_val using One-Hot-Encoding. You can easily do that using sklearn method LabelBinarizer.
Can you write Python code for it?

from sklearn.preprocessing import LabelBinarizer

#Convert to Torch tensors. Float type is required
X_tr_torch = torch.from_numpy(X_tr).float()
X_val_torch = torch.from_numpy(X_tst).float()
# <SOL>
#Note we are using One Hot Encoding for the labels
lb = LabelBinarizer()
y_tr_torch = torch.from_numpy(lb.fit_transform(y_tr))
y_val_torch = torch.from_numpy(lb.transform(y_tst))
# </SOL>


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
 Part 2. Implementing Deep Networks with PyTorch
 4. Pytorch Tutorial

Pytorch is a Python library that provides different levels of abstraction for implementing deep neural networks

The main features of PyTorch are:

Definition of numpy-like n-dimensional tensors. They can be stored in / moved to GPU for parallel execution of operations
Automatic calculation of gradients, making backward gradient calculation transparent to the user
Definition of common loss functions, NN layers of different types, optimization methods, data loaders, etc, simplifying NN implementation and training
Provides different levels of abstraction, thus a good balance between flexibility and simplicity

This notebook provides just a basic review of the main concepts necessary to train NNs with PyTorch taking materials from:

Learning PyTorch with Examples, by Justin Johnson
What is torch.nn really?, by Jeremy Howard
Pytorch Tutorial for Deep Learning Lovers, by Kaggle user kanncaa1

4.1. PyTorch Installation

PyTorch can be installed with or without GPU support

If you have an Anaconda installation, you can install from the command line, using the instructions of the project website

PyTorch is also preinstalled in Google Collab with free GPU access

Follow RunTime -> Change runtime type, and select GPU for HW acceleration

Please, refer to Pytorch getting started tutorial for a quick introduction regarding tensor definition, GPU vs CPU storage of tensors, operations, and bridge to Numpy

4.2. Torch tensors (very) general overview
Esentially, tensors are objects provided by PyTorch for numerical representation. They are generic n-dimensional arrays such as the ones used by Numpy. Apart from the library...
Can you write Python code for it?

import torch

x = torch.rand((100,200))
X_tr_tensor = torch.from_numpy(X_tr)

print(x.type())
print(X_tr_tensor.size())


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
A relevant parameter for the training of logistic regression is the optimization method used to update the parameter vector $\bf w$. You can try other methods different from the lbfgs method which is the default selection. You can find information about these methods online, for instance in this stackoverflow entry: Logistic regression python solvers' definitions
2.1.3. Polynomial Logistic Regression
A strategy for the implementation of classifiers that provide non-linear boundaries consists of transforming the input variables. To this end, in this section we will use polynomial transformations, exploring different values of the degree of the built-in terms to try to optimize performance.
The proposed processing scheme for this section consists of the following steps:

Polynomial expansion of the input variables using the PolynomialFeatures method of scikit-learn
Scaling of all variables to zero mean and unit variance (StandardScaler method)
Logistic regression

The free parameters to be adjusted are, therefore, the degree of the polynomial transformation and the parameter $ C $ of the logistic regression.
Exercise 2.4: Validate parameters degree and C of the proposed classification scheme using a 5-fold validation. Allow for polynomial transformations of degree up to 6. A very convenient way to do this is to define a processing pipeline in scikit-learn, and use it together with GridSearchCV.
Note that both parameters should be validated together, i.e., all possible combinations need to be evaluated.
For the next two cells to work, you need to use the following variable names:

clf: Best classifier selected with 5 fold strategy. It needs to implement a clf.predict_proba method
CE_poly: Average Classification Error Rate of previous classifier calculated over the test data
NLL_poly: Negative log-lilelihood of previous classifier calculated over the test data
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import StandardScaler

# <SOL>
parameters = {'lr__C': [10000, 1000, 100, 10, 1, 0.1, 0.01, 0.001, 0.0001],
              'poly__degree': [1, 2, 3, 4, 5, 6]}

pipe = Pipeline([('poly', PolynomialFeatures()),
                 ('scaler', StandardScaler()),
                 ('lr', LogisticRegression(max_iter=500, tol=1e-3))])

clf = GridSearchCV(pipe, parameters, cv=5)
clf.fit(X_tr_2D, y_tr_bin)

CE_poly  = 100 - 100*np.mean(y_tst_bin==clf.predict(X_tst_2D))
y_pred = clf.predict_log_proba(X_tst_2D)
NLL_poly = - np.sum(y_tst_bin.T @ y_pred[:,1]) - np.sum((1-y_tst_bin).T @ y_pred[:,0])
# </SOL>

print('CE of Polynomial Logistic Regression Classifier (%):', CE_poly)
print('NLL of Polynomial Logistic Regression Classifier:', NLL_poly)
print('Selected parameters:', clf.best_params_)

plot_proba_map(X_tr_2D, y_tr_bin, clf)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Adding underscore performs operations "in place", e.g., x.add_(y)
Parallelization of Operations using GPUs

If a GPU is available, tensors can be moved to and from the GPU device

Operations on tensors stored in a GPU will be carried out using GPU resources and will typically be highly parallelized
Can you write Python code for it?

if torch.cuda.is_available():
    device = torch.device('cuda')
    x = x.to(device)
    y = x.add(x)
    y = y.to('cpu')
else:
    print('No GPU card is available')


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Comparing this figures to those for the conventional gradient descent method, we can extract a number of conclusions:

Convergence is radically faster when using SGD with minibatches. Note that just after the first epoch the error (both in terms of loss function and CE) is already smaller than that achieved with conventional gradient descent

In this case, the error in the validation set starts increasing slightly after a number of epochs. I.e., even for a linear classifier overfitting may occur given the high dimensionality of the input data

Note that the final classification error is much larger than that observed in Section 3; however keep in mind that the network we have just implemented is constrained to linear classification.

Exercise 5.2: Implement network training with other optimization methods. You can refer to the official documentation and select a couple of methods. You can also try to implement adaptive learning rates using torch.optim.lr_scheduler
5.3. Multi Layer networks using nn.Sequential
As we have seen, PyTorch simplifies considerably the implementation of neural network training, since we do not need to implement derivatives ourselves.
We can also make a simpler implementation of multilayer networks using nn.Sequential function. It returns directly a network with the requested topology, including parameters and forward evaluation method
For instance, the cell below defines a Network with two hidden layers with 200 and 100 units for the resolution of the MNIST multiclass problem. Relu activation is used at the output of the neurons of the hidden layers.
Can you write Python code for it?

my_MLP_net = nn.Sequential(
    nn.Linear(X_tr_torch.size()[1], 200),
    nn.ReLU(),
    nn.Linear(200,100),
    nn.ReLU(),
    nn.Linear(100,y_tr_torch.size()[1])
)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Each row of X contains a different digit. You can display the original images by realigning the dimensions of each row of X, as shown in the following example:
Can you write Python code for it?

pos = 1900

plt.imshow(X[pos,].reshape(28, 28)), plt.axis('off'), plt.show()
print('The label of element %s is "%s"' % (pos, y[pos]))


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Exercise 1.1: Save data variables X and y so that you can use them in the future without donwloading the dataset again. Reload data from file to check the correctness of your solution
Can you write Python code for it?

# You may use numpy.savez
# Reload variables with names Xlocal and ylocal

# <SOL>
np.savez('MNIST.npz', X=X, y=y)
data = np.load('MNIST.npz')
Xlocal = data['X']
ylocal = data['y']
# </SOL>

# Check that the reloaded matrices shapes are the same as before
print('Size of Input Data Matrix:', Xlocal.shape)
print('Size of Label Vector:', ylocal.shape)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Exercise 3.3: Calculate the confusion matrix of the 1-NN classifier when using 1000 samples for the training set, and answer the following questions:
   - Which two digits are most frequently confused?
   - Which is the digit that gets misclassified most often?
Can you write Python code for it?

# <SOL>
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

ntrain = 1000
knn = KNeighborsClassifier()
knn.fit(X_tr[:ntrain,:], y_tr[:ntrain])

cm = confusion_matrix(y_tst, knn.predict(X_tst), labels=knn.classes_)
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=knn.classes_)
clg = disp.plot()
clg.figure_.set_size_inches(7,7)
plt.show()
# </SOL>


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Exercise 4.1:

Initialize a tensor x as X_tr[0,315:325]
Compute output vector y applying a function of your choice to x
Compute scalar value z as the sum of all elements in y squared
Check that x.grad calculation is correct using the backward method
Try to run your cell multiple times to see if the calculation is still correct. If not, implement the necessary modifications so that you can run the cell multiple times, but the gradient does not change from run to run

Note: The backward method can only be run on scalar variables
Can you write Python code for it?

if torch.cuda.is_available():
    x = torch.from_numpy(X_tr[0,315:325]).to(device)
else:
    x = torch.from_numpy(X_tr[0,315:325])

# <SOL>
x.requires_grad = True
y = x.add(torch.exp(x))
z = (y**2).sum()
z.backward()
print(x.grad)
print(2*y*(1 + torch.exp(x)))

x.requires_grad = False
x.grad.zero_()
print('Automatic gradient calculation is deactivated, and gradients set to zero')
# </SOL>


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Exercise 5.3: Train the MLP network we have just defined on the MNIST dataset using the following settings:
   * Loss function: nn.CrossEntropyLoss()
   * Optimization algorithm: optim.Adam() with learning rate 1e-4
   * Minibatch size: 256
   * Number of epochs: 100
Calculate the time required to train the network
Can you write Python code for it?

# <SOL>
from torch import nn
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
import time

train_ds = TensorDataset(X_tr_torch, y_tr_torch)
train_dl = DataLoader(train_ds, batch_size=256)

loss_func = nn.CrossEntropyLoss()
opt = optim.Adam(my_MLP_net.parameters(), lr=1e-4)

epochs = 100

loss_tr = np.zeros(epochs)
loss_val = np.zeros(epochs)
CE_tr = np.zeros(epochs)
CE_val = np.zeros(epochs)

start = time.time()
for epoch in range(epochs):

    print(f'Current epoch: {epoch+1}    \r', end="")

    #In each epoch we iterate over all minibatches
    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_MLP_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the **end of each epoch**, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_MLP_net(X_tr_torch)
        loss_tr[epoch] = loss_func(pred, y_tr_torch.argmax(axis=-1)).item()
        CE_tr[epoch] = CE(y_tr_torch, pred).item()
        pred_val = my_MLP_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        CE_val[epoch] = CE(y_val_torch, pred_val).item()

print('Neural Network training completed in', time.time()-start, 'seconds')

# </SOL>

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_tr, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(100*CE_tr, 'b'), plt.plot(100*CE_val, 'r'), plt.legend(['train', 'val']), plt.title('Classification Error (%)')
plt.show()


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Exercise 5.4: Modify your code so that network training is done using GPUs. Compare the training time when using CPU and GPU implementations.
Can you write Python code for it?

# Network, training and validation data should be moved to GPU.
# You can do that with .to(device) or .cuda() functions

my_MLP_cuda = nn.Sequential(
    nn.Linear(X_tr_torch.size()[1], 200),
    nn.ReLU(),
    nn.Linear(200,100),
    nn.ReLU(),
    nn.Linear(100,y_tr_torch.size()[1])
).cuda()

X_tr_cuda = X_tr_torch.to(device)
X_val_cuda = X_val_torch.to(device)
y_tr_cuda = y_tr_torch.to(device)
y_val_cuda = y_val_torch.to(device)

# Adapt training code
# <SOL>
train_ds = TensorDataset(X_tr_cuda, y_tr_cuda)
train_dl = DataLoader(train_ds, batch_size=256)

loss_func = nn.CrossEntropyLoss()
opt = optim.Adam(my_MLP_cuda.parameters(), lr=1e-4)

epochs = 100

loss_tr = np.zeros(epochs)
loss_val = np.zeros(epochs)
CE_tr = np.zeros(epochs)
CE_val = np.zeros(epochs)

start = time.time()
for epoch in range(epochs):

    print(f'Current epoch: {epoch+1}    \r', end="")

    #In each epoch we iterate over all minibatches
    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_MLP_cuda(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the **end of each epoch**, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_MLP_cuda(X_tr_cuda)
        loss_tr[epoch] = loss_func(pred, y_tr_cuda.argmax(axis=-1)).item()
        CE_tr[epoch] = CE(y_tr_cuda, pred).item()
        pred_val = my_MLP_cuda(X_val_cuda)
        loss_val[epoch] = loss_func(pred_val, y_val_cuda.argmax(axis=-1)).item()
        CE_val[epoch] = CE(y_val_cuda, pred_val).item()

print('Neural Network training completed in', time.time()-start, 'seconds')

# </SOL>

plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_tr, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(100*CE_tr,...


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
In this notebook we will explore different strategies for solving a  classification problem consisting of determining the handwritten digit corresponding to a $ 28 \times 28 $ pixel image (MNIST dataset).

We will start by tackling a binary classification problem using different classification algorithms whose implementations are available in scikit-learn, including the multilayer perceptron as an example of a multilayer type neural network (multilayer perceptron, MLP).

Next, we will consider multiclass classification, and calculate the performance of the previous algorithms in this more challenging problem.

The last part of the notebook contains an introduction to PyTorch, one of the most widely used libraries for neural network training. We will review the basic concepts of Pytorch and the different modules that allow simplifying the implementation and optimization of neural networks, including the design of convolutional neural networks (CNNs) that represent a more powerful alternative than MLPs in image classification problems.

For faster executions you must run PyTorch code on GPU. For this, it is sufficient to use Gooble Colab by properly configuring the runtime environment.
 Part 1. Scikit-learn Methods
 1. Data Preparation
1.1. Data load
MNIST is a handwritten digit classification problem that includes 60,000 training patterns and 10,000 test patterns, with representations obtained from the digitization of the corresponding grayscale images and with $28\times 28$ pixel resolution.
The database can be downloaded from the OpenML repository using tools available from Scikit-learn. It is recommended that after a first download you make a local copy of the data to speed up future executions.
Can you write Python code for it?

from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

# Load data from https://www.openml.org/d/554
X, y = fetch_openml("mnist_784", version=1, return_X_y=True, as_frame=False)
# Format data in y as integers
y = y.astype(np.int)

print('Size of Input Data Matrix:', X.shape)
print('Size of Label Vector:', y.shape)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Note: If you are using Google Colab and the previous cell indicates that you do not have access to a GPU, you may change your runtime type. However, note that doing so will restart your runtime, so that you will have to run again the initial cells of the notebook to load the data.
4.3. Automatic Gradient Calculation

PyTorch tensors have a property requires_grad. When true, PyTorch automatic gradient calculation will be activated for that variable

In order to compute these derivatives numerically, PyTorch keeps track of all operations carried out on these variables, organizing them in a forward computation graph.

When executing the backward() method, derivatives will be calculated

However, this should only be activated when necessary, to save computation
Can you write Python code for it?

x.requires_grad = True
y = (3 * torch.log(x)).sum()
y.backward()
print(x.grad[:2,:2])
print(3/x[:2,:2])

x.requires_grad = False
x.grad.zero_()
print('Automatic gradient calculation is deactivated, and gradients set to zero')


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
Operations and slicing with tensors
Operations and slicing involving tensors use a syntax similar to that of numpy
Can you write Python code for it?

print('Size of tensor x:', x.size())
print('Tranpose of vector has size', x.t().size()) #Transpose and compute size
print('Extracting upper left matrix of size 3 x 3:', x[:3,:3])
print((x @ x.t()).size())
xpx = x.add(x)
xpx2 = torch.add(x,x)
print((xpx!=xpx2).sum())   #Since all are equal, count of different terms is zero


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_professor.ipynb".
The first task is:
The next cell visualizes the classification border and the probabilistic map of the selected classifier (clf)
Can you write Python code for it?

def plot_proba_map(X, y, clf):

    # param X: Input Data Matrix (Size (N,2))
    # param y: Targets (Size (N,))
    # param clf: a classifier implementing the predict_proba method

    if X.shape[1] != 2:
        print('Can only plot 2D probability maps')
    else:
        # Create a regtangular grid.
        x_min, x_max = X[:, 0].min(), X[:, 0].max()
        y_min, y_max = X[:, 1].min(), X[:, 1].max()
        dx = x_max - x_min
        dy = y_max - y_min
        h = dy /400
        xx, yy = np.meshgrid(np.arange(x_min - 0.1 * dx, x_max + 0.1 * dx, h),
                              np.arange(y_min - 0.1 * dx, y_max + 0.1 * dy, h))
        X_grid = np.array([xx.ravel(), yy.ravel()]).T

        # Compute the classifier output for all samples in the grid.
        pp = clf.predict_proba(X_grid)[:,1]
        pp = pp.reshape(xx.shape)

        plt.figure(figsize=(10,5))
        plt.subplot(1, 2, 1)
        CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
        plt.contour(CS, levels=[0.5], colors='m', linewidths=(3,))
        plt.subplot(1, 2, 2)
        CS = plt.contourf(xx, yy, pp, cmap=plt.cm.copper)
        plt.scatter(X[:, 0], X[:, 1], c=y, s=4, cmap='summer')
        plt.contour(CS, levels=[0.5], colors='m', linewidths=(3,))
        plt.show()

plot_proba_map(X_tr_2D, y_tr_bin, clf)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_student.ipynb".
The first task is:
1.2. Data partitioning
To begin, we will perform a random partition of the available data into training and test sets of 60,000 and 10,000 digits, respectively.
The notebook also considers a binary problem consisting of the recognition of digits 7 and 9. We have selected this pair of digits as it is one of the most confusing, but if you wish, you can use any other pair of digits.
Exercise 1.2: Create the binary classification problem 7 vs 9. Save the corresponding data matrices with names X_tr_bin, y_tr_bin, X_tst_bin, y_tst_bin. Make sure that the target vectors contain just 0s (for class 7) and 1s (class 9).
Can you write Python code for it?

from sklearn.model_selection import train_test_split

train_samples = 60000
test_samples = 10000

X_tr, X_tst, y_tr, y_tst = train_test_split(
    Xlocal, ylocal, train_size=train_samples, test_size=test_samples,
    random_state=0
)

print('Shape of input training data (multiclass):', X_tr.shape)
print('Shape of target training vector (multiclass):', y_tr.shape)
print('Shape of input test data (multiclass):', X_tst.shape)
print('Shape of target test vector (multiclass):', y_tst.shape)

# Exercise: create the binary classification problem 7 vs 9
# <SOL>
# </SOL>

print('\nShape of input training data (binary):', X_tr_bin.shape)
print('Shape of target training vector (binary):', y_tr_bin.shape)
print('Shape of input test data (binary):', X_tst_bin.shape)
print('Shape of target test vector (binary):', y_tst_bin.shape)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_student.ipynb".
The first task is:
1.3. Data normalization
Next we will normalize the data for the multiclass and binary problems. When working with images, it is frequent to normalize the input data, corresponding to the gray intensity values of the different pixels, so that they take values in the range $[-0.5, 0.5]$.
Exercise 1.3: Implement the normalization of the input data. Make sure to normalize data independently for the multiclass and binary classification cases. Make also sure not to refit the scaler object when transforming the test partitions.
Can you write Python code for it?


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_student.ipynb".
The first task is:
2.1.4. Multi-Layer Perceptron
Rather than using a predefined type of transformation, such as polynomial, we can recur to neural networks with non-linear activation functions to learn the most appropriate representation of the input data to solve the classification problem.
Exercise 2.5: In this section you will use the MLPClassifier method provided by scikit-learn for the implementation of feed-forward networks. You will only need to adjust the following parameters:

activation: The activation function for the units of the intermediate layer. You can test relu and tanh.
max_iter: Number of iterations of the optimization method. Try different values and make sure that the algorithm has completely converged.
hidden_layer_sizes: Use just one hidden layer with 20 units
alpha: The L2 regularization parameters.

Save your results using the following variable names:

clf: Best classifier selected with 5 fold strategy. It needs to implement a clf.predict_proba method
CE_MLP: Average Classification Error Rate of previous classifier calculated over the test data
NLL_MLP: Negative log-lilelihood of previous classifier calculated over the test data
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.neural_network import MLPClassifier

# <SOL>
# </SOL>

print('CE of MLP Classifier (%):', CE_MLP)
print('NLL of MLP Classifier:', NLL_MLP)
print('Selected parameters:', clf.best_params_)

plot_proba_map(X_tr_2D, y_tr_bin, clf)


I am working on the file "C_lab2_NNs/Hand_Digit_with_NN_student.ipynb".
The first task is:
2.2. Classification with all input features
Exercise 2.6: Analyze the performance in the binary classification problem of the following classifiers:

Logistic regression (validate $C$)
K-nearest neighbors (validate $K$)
Multi-layer perceptron (validate size of hidden layers)

For comparison purposes you can use both the average classifiation error rate and the negative log-likelihood.
In this section you need to use all 784 available features.
2.2.1. Logistic Regression
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression

# <SOL>
# </SOL>

plt.semilogx(C_param, CE_param)
plt.xlabel('C'), plt.ylabel('(%)'), plt.title('Classification Error Rate (Linear Logistic Regression)')
plt.show()

print('CE of Linear Logistic Regression Classifier (%):', CE_LR)
print('NLL of Linear Logistic Regression Classifier:', NLL_LR)