I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
Convenient plot functions
Can you write Python code for it?

def newfig(title='?', xlabel='?', ylabel='?', xlim=None, ylim=None, xscale=None, yscale=None, figsize=(10, 8),
            fontsize=16):
    sizes = ['xx-small', 'x-small', 'small', 'medium', 'large', 'x-large', 'xx-large']

    assert isinstance(fontsize, int) or fontsize in sizes, \
        "fontsize not int and not in [{}]".format(', '.join(sizes))

    _, ax = plt.subplots(figsize=figsize)
    ax.set_title(title)
    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    if xlim: ax.set_xlim(xlim)
    if ylim: ax.set_ylim(ylim)
    if xscale: ax.set_xscale(xscale)
    if yscale: ax.set_yscale(yscale)
    ax.grid()

    for item in ([ax.title, ax.xaxis.label, ax.yaxis.label] +
             ax.get_xticklabels() + ax.get_yticklabels()):
        item.set_fontsize(fontsize)
    return ax

def newfigs(layout, titles=['?'], xlabels=['?'], ylabels=['?'], xscales=None, yscales=None,
            xlim=None, ylim=None, sharex=None, sharey=None, fontsize=16, figsize=(10, 8)):
    sizes = ['xx-small', 'x-small', 'small', 'medium', 'large', 'x-large', 'xx-large']

    assert isinstance(fontsize, int) or fontsize in sizes, \
        "fontsize not int and not in [{}]".format(', '.join(sizes))

    fig, axs = plt.subplots(*layout, sharex=sharex, sharey=sharey)
    fig.set_size_inches(figsize)

    assert isinstance(layout, tuple) and len(layout) == 2, 'layout must be a 2-tuple (nrows, ncols) not {}'.format(str(layout))
    n_axes = np.prod(layout)

    if xscales is None: xscales = [None for _ in range(n_axes)]
    if yscales is None: yscales = [None for _ in range(n_axes)]

    for items_name, items in zip(['titles', 'xlabels', 'ylabels', 'xscales', 'yscales'], [titles, xlabels, ylabels, xscales, yscales]):
        assert len(items) == np.prod(layout), 'len({}) == {} != len(layout) == {}'.format(items_name, len(items), np.prod(layout))

    for ax, title, xlabel, ylabel, xscale, yscale in zip(axs.ravel(), titles, xlabels, ylabels, xscales, yscales):
    ...


I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
The flow will be permanently outward, even in the steady state situation (t = 3650 d). This is due to the strong groundwater flow from the right to the left.
Q8: Find the point where the inflow is just zero. (This may be done analytically)
There is no point at which the river infiltrates. For this to happen either the base flow should be less, or the extraction should be must higher.
Q9: River stage varies by 2 m with T = 1 year = 365 days
Can you write Python code for it?

Tcycle = 365
omega = 2 * np.pi / Tcycle
a = np.sqrt(omega * Sy / (2 * kD))

# The answer for the point at which the amplitude is reduced from 2 to 0.1 m follows from: $\frac{0.1}{2} = e^{-a x}$
L = np.log(2 / 0.1) / a
print("omega = {:.4g} rad/d".format(omega))
print("a     = {:.4g} /m".format(a))
print("L     = {:.4g} m".format(L))


I am working on the file "Assignment/VScode/AssJan2022.ipynb".
The first task is:
print("The delay is {:.3g} d.".format(L * a / omega))
Q10: plot the envelopes around the actual situation. Just use steady-state drawdown for convenience
Can you write Python code for it?

kD, Sy, q, Q, L, A = 900, 0.1, -1.0, 1200, 1500, 2.0

Xline = np.linspace(-3000, 3000, 301)

s = -q / kD * Xline

for k in wells:
    well = wells[k]
    r = np.abs(Xline - well['x'])
    r[r < well['r0']] = well['r0']

    s += -well['sign'] * Q / (2 * np.pi * kD) * np.log(r)

env = A * np.exp(-a * np.abs(Xline))

ax = newfig("Head with envelopes due to flutuating river", "x [m]", "head [m]")
ax.plot(Xline, s, label="head average.")
ax.plot(Xline, s + env, label="Top envelope")
ax.plot(Xline, s - env, label='Bottom envelope')

ax.legend(loc='lower right')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
1.2.  Trainieren des Klassifizierers
Als nächstest wird der Klassifizierer mit dem Trainings-Datensatz trainiert.
Can you write Python code for it?

nbc = NaiveBayesClassifier()
nbc.train(train_set)


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
1.3.  Testen des Klassifizierers
Zu guter Letzt wird der Klassifizierer mit dem Development-Datensatz und folgenden Methoden überprüft:
* accuracy
* recall
* fpRate
* precision
* f1Score
Can you write Python code for it?

print('Accuracy: ', nbc.accuracy(dev_set))
print('Recall: ', nbc.recall(dev_set))
print('FP-Rate: ', nbc.fpRate(dev_set))
print('Precision: ', nbc.precision(dev_set))
print('F1-Score: ', nbc.f1Score(dev_set))


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
2..  Klassifizieren von unbekannten Daten
2.1. Vorberitung der Datensätze
Als erstes werden der Trainings- und Test-Datensatz vorbereitet. Der Trainings-Datensatz wird diesmal komplett (ohne zusäzlichen Development-Datensatz) zum trainieren des Klassifizieres benutzt. Der Test-Datensatz beinhaltet die Daten welche fremd sind und klassifiziert werden sollen.
Can you write Python code for it?

train_dict = csv.DictReader(open('train.csv', 'r'))
train_set = ([features(p, True) for p in train_dict])

test_dict = csv.DictReader(open('test.csv', 'r'))


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Cabin-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Cabin']).size()), "NotSurvived": Series(not_survived.groupby(['Cabin']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Der Klassifizierer in Kombination mit der Feature-Extraktions-Methode features

Trainieren und Testen

1.1. Vorberitung der Datensätze
Als erstes werden der Trainings- und Development-Datensatz vorbereitet. Der Trainings-Datensatz soll dabei vom Development-Datensatz verschiedene Daten beinhalten, um so auch einschätzen zu können, ob der Klassifizierer gut funktioniert. Größe des Development-Datensatz soll bei min. 100 Instanzen liegen.
Can you write Python code for it?

train_dict = csv.DictReader(open('train.csv', 'r'))
train_set = ([features(p, True) for p in train_dict])
size = int(len(train_set) * 0.12)
train_set, dev_set = train_set[size:], train_set[:size]


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Der Klassifizierer in Kombination mit der Feature-Extraktions-Methode randomFeatures

Trainieren und Klassifizieren mittels Ensemble-Klassifizierung

Da es nicht sehr effektiv wäre die Feature-Extraktions-Methode randomFeatures nur für einmaliges Klassifizieren zu benutzen und wahrscheinlich in vielen Fällen zu schlechteren Ergebnissen führen würde, als die Methode features, wird hier eine Art Ensemble-Klassifizierer implementiert. Dabei werden mehrere Klassifizierer verwendet, wobei jeder einzelne eine Instanz auf Grund von unterschiedlichen Features einer Klassse zuordnet. Sofern der Klassifizierer einen höheren Accuracy-Wert als 0.7 hat (eigentlich würden 0.5 reichen, aber es wurde sich hier für einen höheren Wert entscheiden) wird sein Ergebnis, also die Klasse, in das endgültige Ergebnis mit einberechnet. Am Ende wird dann zusammengezählt, wie oft eine bestimmte Klasse ausgewählt wurde und die, die am häufigsten auftaucht wird dann als Endergebnis genommen.
Can you write Python code for it?

selected_classes_for_instances_from_all_classifiers = []

for i in range(100):

    random_number = randint(3,10)
    random_numbers = []
    for j in range(random_number):
        tmp_random_number = randint(0,11)
        random_numbers.append(tmp_random_number)

    train_dict = csv.DictReader(open('train.csv', 'r'))
    train_set = ([randomFeatures(p, random_numbers, True) for p in train_dict])
    size = int(len(train_set) * 0.12)
    train_set, dev_set = train_set[size:], train_set[:size]

    nbc = NaiveBayesClassifier()
    nbc.train(train_set)
    accuracy = nbc.accuracy(dev_set)

    print('With accuracy: ', accuracy)
    print('With features:', DataFrame(train_set).keys())

    if accuracy > 0.7:

        train_dict = csv.DictReader(open('train.csv', 'r'))
        train_set = ([randomFeatures(p, random_numbers, True) for p in train_dict])

        nbc = NaiveBayesClassifier()
        nbc.train(train_set)

        test_dict = csv.DictReader(open('test.csv', 'r'))

        for instance in test_dict:
            instance_features = randomFeatures(instance, random_numbers, False)
            selected_classes_for_instances_from_all_classifiers.append({'PassengerId': instance['PassengerId'], 'Class': nbc.classify(instance_features)})

if len(selected_classes_for_instances_from_all_classifiers) > 0:
    grouped_and_counted_classes_for_instances = DataFrame(selected_classes_for_instances_from_all_classifiers).groupby(['PassengerId', 'Class']).size()


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Embarked-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Embarked']).size()), "NotSurvived": Series(not_survived.groupby(['Embarked']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Fare-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Fare']).size()), "NotSurvived": Series(not_survived.groupby(['Fare']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Hilfs-Funktionen zur Feature-Extraktion
Die folgenden Funktionen dienen den beiden Funktionen features und randomFeatures als Hilfe, um die Features besser extrahieren zu können.
getTitle
Die Funktion getTitle bekommt einen String (name) übergeben und prüft, ob in dem String ein Titel vorkommt und gibt diesen Titel dann zurück. Mögliche Titel sind dabei 'Miss', 'Mrs', 'Master' und 'Mr'. Sollte keiner der Titel vorkommen wird 'none' zurückgegeben, sollte der String leer sein wird 'unknown' zurückgegeben.
Can you write Python code for it?

def getTitle(name):
    if name == '':
        return 'unknown'
    elif 'Miss' in name:
        return 'Miss'
    elif 'Mrs' in name:
        return 'Mrs'
    elif 'Master' in name:
        return 'Master'
    elif 'Mr' in name:
        return 'Mr'
    else:
        return 'none'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Klassifizierung
Im Folgenden wird ein Naive-Bayes-Klassifizierer implementiert, der sich mittels eines Trainings-Datensatzes trainieren lässt und anschließend in der Lage ist, nicht bekannte Daten zu klassifizieren. Als Algorithmus wird hier das Bayes' Theorem verwendet.
Bayes' Theorem
Naive-Bayes-Klassifizierer werden gerne im Bereich des Machine-Learning benutz und gehoren dort zu der Familie der eher einfachen wahrscheinlichkeitstheoretisch Klassifizierern. Der Klassifizierer baut dabei auch das Bayes' Theorem auf. Das Bayes' Theorem lässt sich wie folgt als Formel darstellen:
$$p(c|x) = \frac{p(x|c) * p(c)}{p(x)}$$
Wobei sich p(x|c) wie folgt berechnet:
$$p(x|c) = \prod_{i=1}^{n} p(x_i|c)$$
Zu beachten ist hierbei, dass davon ausgegenagen wird, dass die einzelenen Features (in der Formel x_i) von einander unabhägig sind.
Naive-Bayes-Klassifizierer
Im folgenden ist eine Klasse implementiert, welche einen Naive-Bayes-Klassifizierer representiert. Dabei wurde die zuvor erwähnte Fromel verwendet, wobei noch ein Smoothing hinzugefügt wurde und mit dem Logarithmus gerechnet wurde, um zu kleine Werte und somit ein Floating-Point-Underflow zu vermeiden. Da man zudem die Teilung duch p(x) weglassen kann, da diese bei allen Berechnungen gleichermaßen passiert, ergibt sich die folgende Formel:
$$p(c|x) = p(x|c) * p(c)$$
Wobei sich p(x|c) wie folgt berechnet:
$$p(x|c) = \sum_{i=1}^{n} log(p(x_i|c))$$
und wobei sich p(x_i|c) wie folgt berechnet:
$$p(x_i|c) = \frac{p(x_i \bigwedge c) + 1}{p(c) + m}$$
Diese Formel ist so auch in dem Klassifizierer implementiert. Folgende Methoden sind in der Klasse NaiveBayesClassifier enthalten:
1. init: Macht lediglich eine Konsolen-Ausgabe, um zu zeigen, dass ein Instanz erzeugt wurde
2. train: Trainiert den Klassifizierer. Dabei wird zum einen der übergebene Trainigs-Datensatz abgespeichert und verschiedene Werte vorberechnet. Vorberechente Werte sind: ein Array, besthend aus allen verschiedenen Klassen, die Anzahl der Instanzen mit einer bestimmten Klasse, die Wahrscheinlichkeiten,...
Can you write Python code for it?

class NaiveBayesClassifier:

    def __init__(self):
            print('New Naive Bayes Classifier initalized...')

    def train(self, train_set):
        self.train_set = DataFrame(train_set)

        self.classes = np.unique(self.train_set['Class'].values.ravel())

        self.all_instances_with_class = {}
        self.probability_of_classes = {}
        for class_ in self.classes:
            self.all_instances_with_class[class_] = self.train_set.groupby(['Class']).size().loc[class_]
            self.probability_of_classes[class_] = self.all_instances_with_class[class_] / self.train_set.shape[0]

        self.number_of_different_forms_of_features = {}
        for feature in self.train_set.keys():
            self.number_of_different_forms_of_features[feature] = self.train_set.groupby([feature]).size().count()

    def classify(self, instance):
        probabilities_of_classes_provided_by_features = {}

        for class_ in self.classes:
            # Frage: Sollte nicht auch von self.probability_of_classes[class_] der Logarithmus genommen werden?
            probability_of_class_provided_by_features = log(self.probability_of_classes[class_])
            probability_of_features_provided_by_class = 0

            for feature in instance:
                if feature != 'Class' :
                    try:
                        all_instances_with_class_and_feature = self.train_set.groupby(['Class', feature]).size().loc[class_].loc[instance[feature]]
                    except KeyError:
                        all_instances_with_class_and_feature = 0

                    probability_of_features_provided_by_class += log((all_instances_with_class_and_feature + 1) / (self.all_instances_with_class[class_] + self.number_of_different_forms_of_features[feature]))

            probability_of_class_provided_by_features += probability_of_features_provided_by_class
           ...


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Name
Title-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Title']).size()), "NotSurvived": Series(not_survived.groupby(['Title']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Parch-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Parch']).size()), "NotSurvived": Series(not_survived.groupby(['Parch']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Pclass-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Pclass']).size()), "NotSurvived": Series(not_survived.groupby(['Pclass']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Plots zur Visualisierung der extrahierten Features
Die folgeneden Plots sind dazu da, um sich ein Bild darüber zu machen, wie entscheidend ein bestimmtes Feature ist, wenn es darum geht, ob ein Passagier überlebt hat oder nicht. Bei der Bearbeitung der Aufgabe, wurd bei der Feature-Extraktion verschiedenes ausprobiert und anschließend durch die Plots geprüft.
Die Plots sind dabei so aufgebaut, dass an der x-Achse alle Ausprägungen eines Features zu finden sind. Für jede Ausprägung eines Features gibt es dann jeweils zwei Balken: Der blaue Balken für die Pasagiere, die die Auspägung hatten und nicht überlebt haben und der orange Balken für die Pasagiere, die die Auspägung hatten und überlebt haben. Die y-Achse gibt an, wie viele Pasagiere mit einer bestimmten Ausprägung jeweils überlebt oder nicht überlebt haben.
Vorbereitung
der beiden Datensätze survived und not_survived, welche einemal alle überlebenden Passagiere und einmal alle nicht überlebenden Passagiere beinhalten.
Can you write Python code for it?

train_dict = csv.DictReader(open('train.csv', 'r'))
train_df = DataFrame([features(p, True) for p in train_dict])

survived = train_df.query('Class==1')
not_survived =  train_df.query('Class==0')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Sex-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['Sex']).size()), "NotSurvived": Series(not_survived.groupby(['Sex']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
SibSp-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['SibSp']).size()), "NotSurvived": Series(not_survived.groupby(['SibSp']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
Ticket
SingleTicketNumber-Balken-Diagramm
Can you write Python code for it?

#x = DataFrame({"Survived": Series(survived.groupby(['SingleTicketNumber']).size()), "NotSurvived": Series(not_survived.groupby(['SingleTicketNumber']).size())})
#ax = x.plot(kind='bar')
#patches, labels = ax.get_legend_handles_labels()
#ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
TicketPrefix-Balken-Diagramm
Can you write Python code for it?

x = DataFrame({"Survived": Series(survived.groupby(['TicketPrefix']).size()), "NotSurvived": Series(not_survived.groupby(['TicketPrefix']).size())})
ax = x.plot(kind='bar')
patches, labels = ax.get_legend_handles_labels()
ax.legend(patches, labels, loc='best')


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
getAge
Die Funktion getAge bekommt einen String (age) übergeben und prüft, ob der String leer ist. Ist dies der Fall wird -1 zurückgegeben, ansonsten wird der String erst zu einer Float-Zahl gecastet und anschließend zu einem Integer, um so eine sinnvolle Rundung vorzunehmen.
Can you write Python code for it?

def getAge(age):
    if age == '':
        return -1
    else:
        return int(float(age))


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
getAgeClass
Die Funktion getAgeClass bekommt eine Integer-Zahl (age) übergeben und ordnet diese Zahl einer vordefinierten Klasse zu. Die Klassen sind dabei wie folgt eingeteilt:
* 'unknown': age == -1
* 'child': age >= 0 and age <= 15
* 'young person': age > 15 and age <= 23
* 'young adult': age > 23 and age <= 36
* 'adult': age > 36 and age <= 54
* 'old adult': age > 54 and age <= 63
* 'old person': age > 63
* alles was keiner dieser Klassen zugeordnet werden kann ist 'undefined'
Um die Klassen sinnvoll festlegen zu können, wurden als erstes alle Alters-Daten in der Rohform und nach dem Casting zum Interger graphisch visualisiert.
Can you write Python code for it?

def getAgeClass(age):
    if age == -1:
        return 'unknown'
    elif age >= 0 and age <= 15:
        return 'child'
    elif age > 15 and age <= 23:
        return 'young person'
    elif age > 23 and age <= 36:
        return 'young adult'
    elif age > 36 and age <= 54:
        return 'adult'
    elif age > 54 and age <= 63:
        return 'old adult'
    elif age > 63:
        return 'old person'
    else:
        return 'undefined'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
getCabinPrefix
Die Funktion getCabinPrefix bekommt einen String (cabin) übergeben und parst aus diesem das Prefix heraus. Das Perfix definiert sich dabei als der Teil, der vor der eigentlichen Kabinien-Nummer steht. Sollte der String leer sein, wird 'unknown' zurückgegeben. Ansonsten werden alle Leerzeichen und alle Zahlen, die am Ende stehen, heraus geparst. Der resultierende String wird zurückgegeben, außer er ist leer, dann wird 'none' zurückgegeben.
Can you write Python code for it?

def getCabinPrefix(cabin):
    if cabin == '':
        return 'unknown'
    else:
        cabin = re.sub(r"\s*", "", cabin)
        cabin = re.sub(r"[0-9]*$", "", cabin)
        if cabin != '':
            return cabin
        else:
            return 'none'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
getFareClass
Die Funktion getFareClass bekommt eine Integer-Zahl (fare) übergeben und ordnet diese Zahl einer vordefinierten Klasse zu. Die Klassen sind dabei wie folgt eingeteilt:
* 'unknown': age == -1
* 'cheap': fare >= 0 and fare <= 10
* 'economical': fare > 10 and fare <= 26
* 'normal': fare > 26 and fare <= 50
* 'expensive': fare > 50 and fare <= 79
* 'first class': fare > 79
* alles was keiner dieser Klassen zugeordnet werden kann ist 'undefined'
Um die Klassen sinnvoll festlegen zu können, wurden als erstes alle Fare-Daten in der Rohform und nach dem Casting zum Interger graphisch visualisiert.
Can you write Python code for it?

def getFareClass(fare):
    if fare == -1:
        return 'unknown'
    elif fare >= 0 and fare <= 10:
        return 'cheap'
    elif fare > 10 and fare <= 26:
        return 'economical'
    elif fare > 26 and fare <= 50:
        return 'normal'
    elif fare > 50 and fare <= 79:
        return 'expensive'
    elif fare > 79:
        return 'first class'
    else:
        return 'undefined'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
getTicketPrefix
Die Funktion getTicketPrefix bekommt einen String (ticket) übergeben und parst aus diesem das Prefix heraus. Das Perfix definiert sich dabei als der Teil, der vor der eigentlichen Ticket-Nummer steht. Sollte der String leer sein oder nur aus der Zeichenkette 'LINE' bestehen, wird beim ersten Fall 'unknown' und beim zweiten Fall 'LINE' zurückgegeben. Ansonsten werden Sonderzeichen wie Punkte, Leerzeichen und Back-Slashes aus dem String heraus geparst. Der resultierende String wird zurückgegeben, außer er ist leer, dann wird 'none' zurückgegeben.
Can you write Python code for it?

def getTicketPrefix(ticket):
    if ticket == '':
        return 'unknown'
    else:
        if ticket == 'LINE':
            return ticket
        else:
            ticket = ticket.upper()
            ticket = re.sub(r"[0-9]*$", "", ticket)
            ticket = re.sub(r"\.", "", ticket)
            ticket = re.sub(r"\s*", "", ticket)
            ticket = re.sub(r"\/*", "", ticket)
            if ticket != '':
                return ticket
            else:
                return 'none'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
initTicketNumberDf
Die Funktion initTicketNumberDf erzeugt ein Dataframe mit allen bekannten Ticket-Nummern (ohne Prefix, das wird zuvor raus geparst) und gibt dieses zurück.
Can you write Python code for it?

def initTicketNumberDf():
    arr = []
    with open('train.csv') as csvfile:
        csv_dict = csv.DictReader(csvfile, delimiter=',', quotechar='"')
        ([arr.append(row['Ticket']) for row in csv_dict])

    regex_alp = re.compile(r"[a-zA-Z]*", re.IGNORECASE)
    regex_pun = re.compile(r"\.", re.IGNORECASE)
    regex_sla = re.compile(r"\/", re.IGNORECASE)
    regex_num = re.compile(r"^[0-9]* ", re.IGNORECASE)
    regex_spa = re.compile(r"\s*", re.IGNORECASE)

    new_arr = []
    for row in arr:
        tmp_row = regex_alp.sub("", row)
        tmp_row = regex_pun.sub("", tmp_row)
        tmp_row = regex_sla.sub("", tmp_row)
        tmp_row = regex_num.sub("", tmp_row)
        tmp_row = regex_spa.sub("", tmp_row)
        if tmp_row != '':
            new_arr.append(tmp_row)

    df = pd.DataFrame(new_arr)
    df = df.apply(pd.value_counts)
    return df


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
isSingleTicketNumber
Die Funktion isSingleTicketNumber bekommt ein String (ticket) übergeben, ist dieser String leer, wird 'unknown' zurück gegeben, ansonsten wird die Ticket-Nummer aus dem Ticket geparst. Anschließend wird in dem ebenfalls übergebenen DataFrame ticket_number_df geguckt, ob die Nummer bereits existiert oder nicht. Jenachdem wird True oder False zurückgegeben. Sollte nach dem Parsen nur noch ein leerer String übrig sein, so wird 'none' zurückgegeben
Can you write Python code for it?

def isSingleTicketNumber(ticket, ticket_number_df):
    if ticket == '':
        return 'unknown'
    else:
        ticket = re.sub(r"[a-zA-Z]*", "", ticket)
        ticket = re.sub(r"\.", "", ticket)
        ticket = re.sub(r"\/", "", ticket)
        ticket = re.sub(r"^[0-9]* ", "", ticket)
        ticket = re.sub(r"\s*", "", ticket)
        if ticket != '':
            return (int((ticket_number_df.loc[ticket, :])[0]) == 1)
        else:
            return 'none'


I am working on the file "Assignment01/Assignment01.ipynb".
The first task is:
randomFeatures
Die Funktion randomFeatures extrahiert bestimmte, zufällige Eigenschaften einer Instanze, die wichtig sind um zu bestimmen, welcher Klasse die Instanz zugehört.
Dafür bekommt sie als Übergabeparamter eine Instanz (instance), von welcher die Eigenschaften extrahiert werden sollen, eine Reihe von zufälligen Zahlen (random_numbers), welche darüber entscheiden, welche Eigenschaften der Instanz ausgewählt werden und einen booloschen Wert (train), welcher besagt, ob es sich um eine Trainings-Instanz oder nicht handelt und somit die Klasse (0 oder 1, bzw. überlebt oder nicht überlebt) der Instanz mit extrahiert wird oder nicht.
Folgende Features können dabei extrahiert werden:
* Pclass: Ticket-Klassse
* Title: Titel
* Rev: Kommt 'Rev.' im Namen vor
* Jr: Kommt 'Jr' im Namen vor
* Sex: Geschlecht
* Age: Alter
* SibSp: Anzahl von Geschwistern / Ehepartnern auf der Titanic
* Parch: Anzahl von Kindern / Eltern auf der Titanic
* SingleTicketNumber: Ist die Ticket-Nummer nur an eine Person vergeben worden
* TicketPrefix: Ticket-Prefix
* Fare: Ticket-Preis
* Cabin: Kabine
* Embarked: Hafen der Anbordnahme
Das Feature SingleTicketNumber wurde im laufe der Bearbeitung der Aufgabe raus genommen, da dieses Feature nicht richtig extrahiert werden kann, da nicht alle vergebenen Ticket-Nummern bekannt sind und man somit nicht sicher sagen kann, ob eine Ticket-Nummer nur an eine Person vergeben wurde oder nicht. Dennoch wird es hier der Vollständigkeit halber mit aufgelistet.
Can you write Python code for it?

def randomFeatures(instance, random_numbers, train=False):
    features = {}
    if train == True:
        features['Class'] = int(instance['Survived'])
    for random_number in random_numbers:
        if random_number == 0:
            features['Pclass'] = instance['Pclass']
        elif random_number == 1:
            features['Title'] = getTitle(instance['Name'])
        elif random_number == 2:
            features['Rev'] = 'Rev' in instance['Name']
        elif random_number == 3:
            features['Jr'] = 'Jr' in instance['Name']
        elif random_number == 4:
            features['Sex'] = instance['Sex']
        elif random_number == 5:
            features['Age'] = getAgeClass(getAge(instance['Age']))
        elif random_number == 6:
            features['SibSp'] = instance['SibSp']
        elif random_number == 7:
            features['Parch'] = instance['Parch']
        #elif random_number == 8:
            #features['SingleTicketNumber'] = isSingleTicketNumber(passenger['Ticket'], ticket_number_df)
        elif random_number == 8:
            features['TicketPrefix'] = getTicketPrefix(instance['Ticket'])
        elif random_number == 9:
            features['Fare'] = getFareClass(getFare(instance['Fare']))
        elif random_number == 10:
            features['Cabin'] = getCabinPrefix(instance['Cabin'])
        elif random_number == 11:
            features['Embarked'] = instance['Embarked']
    return features


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Berechnung aller vorkommenden Wörter
Um alle vorkommenden Wörter zu berechnen, werden alle Texte der Dokumente durchlaufen und mittels des nltk-Wörter-Tokinisierers tokinisiert. Jedes resultierende Wort wird anschließend analysiert und sofern es kein Zeichen oder ein Stopwort ist, dem term_set hinzugefügt.
Can you write Python code for it?

for page in pages:
    for term in nltk.word_tokenize(page['text']):
        if term not in exclude and term not in stopwords:
            term_set.add(porter.stem(term).lower())


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Berechnung des Term-Frequency-Indexes
Um den Term-Frequency-Indexes aufzustellen, wird jedes Dokument durchlaufen, wobei zunächst für jedes Wort aus dem term_set ein Wert von 0 initialisiert wird. Anschließend wird jedes Wort innherhalb des Dokumentes analysiert und, sofern es sich um kein Zeichen oder Stopwort handelt, der passende Zähler um eins erhöht.
Can you write Python code for it?

for page in pages:
    tf_dict[page['id']] = {}
    for term in term_set:
        tf_dict[page['id']][term] = 0
    for term in nltk.word_tokenize(page['text']):
        if term not in exclude and term not in stopwords:
            tf_dict[page['id']][porter.stem(term).lower()] += 1


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Die Funktion set_ranks() setzt, wenn rank+1 nicht None ist, für alle Seiten den älteren PageRank-Wert rank auf den aktuellen PageRank-Wert rank+1.
Can you write Python code for it?

def set_ranks():
    for page in pages:
        if  page['rank+1'] is not None:
            page['rank'] = page['rank+1']


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Die Funktion termination_condition() gibt einen boolischen Wert zurück, der sagt, ob die Abbruchbedingung erfüllt ist oder nicht. Im Falle, dass es keine zwei Vergleichswerte gibt, also rank+1 gleich None ist, wird False zurückgegeben, ansonsten wird der Rückgabewert mit folgender Formel berechnet:
$$ \sum_{p_i,i \in 1,...,N} abs(r_{k+1}(p_i) - r_k(p_i)) <= \delta $$
Can you write Python code for it?

def termination_condition():
    delta = 0
    for page in pages:
        if page['rank+1'] is None:
            return False
        else:
            delta += abs(page['rank+1'] - page['rank'])
    return  delta <= δ


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Funktionen
PageRank-Funktion
Die Funktion calculate_page_rank() berechnet den PageRank für die übergebene Seite page_i. Folgende Formel wurde dabei implementiert:
$$ r_{k+1}(p_i) = d * \bigg ( \displaystyle\sum_{p_j \in B_{pi}} \frac{r_k(p_j)}{|p_j|} + \displaystyle\sum_{p_j,|p_j|=0} \frac{r_k(p_j)}{N} \bigg ) + \frac{t}{N} $$
Can you write Python code for it?

def calculate_page_rank(page_i):
    sum_result = 0
    for page_j in pages:
        if page_i['id'] in page_j['back_links']:
            sum_result += page_j['rank'] / len(page_j['back_links'])
        if len(page_j['back_links']) == 0:
            sum_result += page_j['rank'] / n
    return d * sum_result + t / n


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Gewichteter TF-IDF-Wert
Die Funktion get_weighted_tf_idf() brechnet mittels der folgenden Formel den gewichteten TF-IDF-Wert für den übergebenen Term im Bezug auf das Dokument mit der übergebenen Dokumenten-Id.
$$ \text{tf-idf}{t,d} = tf{t,d} \times idf_t $$
Can you write Python code for it?

def get_weighted_tf_idf(doc_id, term):
    return get_weighted_tf(doc_id, term) * get_weighted_idf(term)


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Im folgenden Code-Block wird jeder Such-Term durchgegangen und der jeweilige Wert für jede Seite berechnet. Der Wert setzt sich aus dem TF-IDF-Wert und PageRank zusammen. Anschließend wird das Ergebnis in der Datei pageranke_search.txt gespeichert.
Can you write Python code for it?

with open('pageranke_search.txt', 'w') as f:
    for search_term in search_terms:
        f.write('Suchwort: %s\n\n' % ', '.join(search_term))
        result = search(search_term, page_rank=True)
        for key in result:
            f.write("%s: %s\n" % (key, result[key]))
        f.write('\n\n')


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Initalisierung der Stopwörter
Im folgenden Code-Block werden die Stopwörter aus der Datei stop_words.txt ausgelesen und in der Variablen stopwords gespeichert.
Can you write Python code for it?

with open('stop_words.txt') as line:
    stopwords += re.sub('[^a-zA-Z0-9,]', '', line.read()).split(',')


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Konstanten
Die folgenden Konstenten sind vorgegeben bzw. müssen zuvor berechnet werden:
* n = Anzahl der betrachteten Seiten
* t = Teleportationsrate
* d = Dämpfungsfaktor
* δ = Abbruchswert
Can you write Python code for it?


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Search
Die Funktion search() brechnet für jede Seite einen Wert, der aussagt, wie gut die jeweilge Seite zu den gesuchten Termen passt. Dabei berechnet sich der Wert ohne Einbeziehung des PageRanks wie folgt:
$$ \text{Score}(q, d) = \sum_{t \in q} \text{tf-idf}_{t,d}$$
Mit Einbeziehung des PageRanks berechnet sich der Wert wie folgt:
$$ \text{Score}(q, d) = \sum_{t \in q} \text{tf-idf}_{t,d} \times PageRank_d $$
Can you write Python code for it?

def search(terms, page_rank=False):
    result = {}
    for page in pages:
        result[page['id']] = 0
        for term in terms:
            result[page['id']] += get_weighted_tf_idf(page['id'], porter.stem(term).lower())
            if page_rank:
                result[page['id']] *= page['rank+1']
    return result


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Speichern des Term-Frequency-Indexes
Das Resultat wird in der Datei index.txt gespeichert. Dabei stellt jede Zeile ein Wort und jede Spalte ein Dokument dar.
Can you write Python code for it?

tf_df = pd.DataFrame(tf_dict)
tf_df.to_csv('index.txt', header=True, index=True, sep=';')


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
Suche mit TF-IDF und anschließende Speicherung
Gesucht wird nach folgenden Termen:
* token
* index
* classification
* classification, token
Can you write Python code for it?

search_terms = [['token'],['index'],['classification'],['classification', 'token']]


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
a) Implement a crawler
Der Crawler wurde mit Hilfe der Bibliothek 'scrapy' geschrieben. Die dazu implementierte Anwendung ist im Verzeichnis ~/sitespider zu finden und der eigentliche Crawler in der Datei ~/sitespider/sitespider/spiders/site_spider.py. Der Code dazu sieht wie folgt aus:
```python
-- coding: utf-8 --
import scrapy
import re
import json
class SiteSpiderSpider(scrapy.Spider):
    name = 'site_spider'
start_urls = [
    'http://people.f4.htw-berlin.de/~zhangg/pages/teaching/pages/d01.html',
    'http://people.f4.htw-berlin.de/~zhangg/pages/teaching/pages/d06.html',
    'http://people.f4.htw-berlin.de/~zhangg/pages/teaching/pages/d08.html'
]

def parse(self, response):
    id = response.url.split('/')[-1].split('.')[0]

    with open("sites/%s.json" % id, 'w') as f:
        data = {'id': id,
                'text': "".join(line for line in response.css('body::text').extract()),
                'url': response.url,
                'back_links': [link.split('.')[0] for link in response.css('a::attr(href)').extract()]
                }

        json.dump(data, f)

    for next_page in response.css('a::attr(href)').extract():
        yield response.follow(next_page, callback=self.parse)

```
Die Klasse SiteSpiderSpider erbt von der Klasse scrapy.Spider. So kann die geschriebene Klasse später als Crawler benutzen werden.
Das Attribut name identifizieren den Crwaler und wird dazu benutzt, den Crawler später über die Konsole aufzurufen. Das Attribut start_urls legt fest, welche URLs vom Crawler initial angefragt werden sollen. Die Methode parse() nimmt dann anschließend die Rückgabe jeder Anfrage entgegen und bearbeitet diese. Da die Methode parse() die default Callback-Methode ist, muss diese nicht explizit angegeben werden.
Die parse()-Methode erzeugt einen String, der die Anfrage eindeutig beschreibt. Dieser Wert wird anschließend als Dateiname für die Datei verwendet, in der die Daten der Anfrage gespeichert werden. Gewähltes Fromat ist hierbei JSON. In der Datei selber wird die Id der Anfrage, der...
Can you write Python code for it?

directory = 'sitespider/sites'
files = [x[2] for x in os.walk(directory)][0]
pages = []


I am working on the file "Assignment02/Assignment02.ipynb".
The first task is:
e) Implement a function search to search for documents containing given words
Folgend wird ein TF-Index für alle Wörter innerhalb der Dokumente (Seiten) berechnet.
Variablen
Folgende Variablen werden verwendet:
* df_dict: Dictionary, das für jedes Wort die Anzahl festhält, in wie vielen Dokumenten das Wort vorkommt und die Dokumenten, in denen es vorkommt.
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
A very powerful feature of NumPy and Python are List Comprehensions. These can replace many for loops and are much more efficient to run. Here we square every element in the vector a from above:
Can you write Python code for it?

a_squared = [i**2 for i in a]

a_squared


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
As you can see you can use Pandas to quickly manipulate and access tabular data. Here we access the first 10 rows of the Size_Uniformity column:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Columns can also be accessed using the name of the column as an index:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Combined with functions, more complex curves can be plotted. Let's plot the sigmoid curve:
Can you write Python code for it?

import math
def sigmoid(x):
    a = []
    for item in x:
        a.append(1/(1+math.exp(-item)))
    return a

x = np.arange(-10., 10., 0.1)
sig = sigmoid(x)

plt.plot(x,sig)


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Entire books have been written about NumPy. Let's move on to Matplotlib.
Matplotlib
First we will load Matplotlib, a 2D plotting library, and run %matplotlib inline. This magic function (functions beginning with the % symbol are called magic functions) will display any plots inline in the notebook.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Get some information about the vector:
Can you write Python code for it?

print("The vector a has " + str(a.ndim) + " dimension(s) and has the shape " + str(a.shape) + ".")


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
However, it is convention in machine learning to use a 0-based index to represent classes. Let's replace the 2s with 0s and the 4s with 1s:
Can you write Python code for it?

df = df.replace({"Class": {2: 0, 4: 1}})
df.Class.value_counts()


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Let's load the well known Wisconsin Breast Cancer dataset:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Load the Iris dataset (a flower data set often used for demonstration purposes):
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Make a vector with 6 elements:
Can you write Python code for it?

a = np.array([1,2,3,4,5,6])

# Print the contents of a
a


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Now we will try to fit the k-nearest neighbours classifier to the training data:
Can you write Python code for it?

knn = KNeighborsClassifier() # Initialise the classifier.
knn.fit(X_train, y_train)    # Fit the classifier.


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Or calculate the standard deviation for a certain column:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Or plot three lines at once by supplying three pairs of x y values:
Can you write Python code for it?

p = np.arange(1,10) # Get a range of numbers form 1 to 10
plt.plot(p, p, p, p**2, p, p**3)


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Or we can slice the first column only. Using the , symbol we can ask for specfic rows and columns. The first integer always specifies the rows, which is followed by , and the second integer specifies the columns. The colon character : is shorthand for all rows or all columns. Here we select all rows of matrix m using : and select the first column, using 0:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Pandas
Let's now import Pandas. This library provides R-style dataframe table functionality. Like NumPy, it is convention to import the Pandas library as pd for brevity.
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Pandas also provides useful plotting tools. To look for correlations in data, a scatter matrix is often useful.
Here we will plot only three columns of the data, and only the first 100 rows of the data, as a scatter plot with so many columns can take some time to render and can result in a very large plot.
Can you write Python code for it?

from pandas.tools.plotting import scatter_matrix

# Manually select three of the table's columns by passing an array of column names:
df_subset = df[['Clump_Thickness','Size_Uniformity', 'Shape_Uniformity']]

# The semi colon at the end of this line is to suppress informational output (we only want to see the plot)
scatter_matrix(df_subset.head(100), alpha=0.2, figsize=(6,6), diagonal='kde');


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
SciKit-Learn
Last but not least, we shall import some modules from SciKit-Learn. SciKit-Learn is the main machine learning library for Python. It is a large library and is not normally loaded entirely; in general you load only the modules you need from the main library. Here we will load the datasets module and the k-nearest neighbours module (KNeighborsClassifier):
Can you write Python code for it?

from sklearn import datasets
from sklearn.neighbors import KNeighborsClassifier


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
So you see only 2s and 4s are contained in this column. See a breakdown of the counts using the value_counts() function:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
The classifier has now been trained on the training data (X_train). We can now check how well it predicts newly seen data (using our test set, X_test):
Can you write Python code for it?

y_pred = knn.predict(X_test)

# The classifier's predicted labels are now contained in y_pred:
y_pred


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
The column Bare_Nucleoli appears as type object as it contains some missing data, which appear as ? in the dataset. Later in the course we will learn how to handle missing data.
You can also use Pandas to perform quick statistical analyses. Here we calculate the standard deviation for each column:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
The k-Nearest Neighbour algorithm is possibly the simplest classifier. Given a new observation, take the label of the sample closest to it in the n-dimensional feature space.
First we must randomise the data, but we must ensure we randomise the labels as well in sync. We can use a NumPy feature to create indices that then correlate to both the targets and the data:
Can you write Python code for it?

np.random.seed(376483)
random_indices = np.random.permutation(len(y))

random_indices


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Then, the data must be split into a test set and a training set (again we are using naming conventions here for the training and test data X_train and X_test and their labels y_train and y_test):
Can you write Python code for it?

X_train = X[random_indices[:-10]]
X_test  = X[random_indices[-10:]]

y_train = y[random_indices[:-10]]
y_test  = y[random_indices[-10:]]

print("Number of training samples: %d. Number of test samples: %d." % (len(X_train), len(X_test)) )


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Using NumPy we can select rows and columns of data very easily (known as array slicing).
For example, we can print the first row of the matrix, m:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
Using the plt.plot() function, you can plot many types of data and Matplotlib will try to figure out what you want to do with the data:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
We can now element-wise compare our predicted results, in y_pred, with the true labels stored in y_test:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
We can select a specific element using the matrix's column and row index, for example we want to select the item in the second column's second row:
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
You can examine the data types (Pandas dataframes can contain multiple types):
Can you write Python code for it?


I am working on the file "Assignment1-Complete.ipynb".
The first task is:
You can view a summary of the data using the describe() function:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
A very powerful feature of NumPy and Python are List Comprehensions. These can replace many for loops and are much more efficient to run. Here we square every element in the vector a from above:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
As you can see you can use Pandas to quickly manipulate and access tabular data. Here we access the first 10 rows of the Size_Uniformity column:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Combined with functions, more complex curves can be plotted. Let's plot the sigmoid curve:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Convention states that matrices are represented using uppercase letters, often the letter X, and label vectors are represented using lower case letters, often y:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Entire books have been written about NumPy. Let's move on to Matplotlib.
Matplotlib
First we will load Matplotlib, a 2D plotting library, and run %matplotlib inline. This magic function (functions beginning with the % symbol are called magic functions) will display any plots inline in the notebook.
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Get some information about the matrix:
Can you write Python code for it?

print("The matrix m has " + str(m.ndim) + " dimension(s) and has the shape " + str(m.shape) + ".")


I am working on the file "Assignment1.ipynb".
The first task is:
Get some information about the vector:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
However, it is convention in machine learning to use a 0-based index to represent classes. Let's replace the 2s with 0s and the 4s with 1s:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Let's rename the column names:
Can you write Python code for it?

df.columns = ["ID","Clump_Thickness","Size_Uniformity","Shape_Uniformity","Marginal_Adhesion","Epithelial_Size","Bare_Nucleoli","Bland_Chromatin","Normal_Nucleoli","Mitoses","Class"]

# Print the new header names:
df.columns


I am working on the file "Assignment1.ipynb".
The first task is:
Make a vector with 6 elements:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Notice how the Class column (the last column in the table) consists of 2s and 4s. In this case 2 stands for malignant and 4 stands for benign. You can check this quickly using:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Now we will try to fit the k-nearest neighbours classifier to the training data:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Or calculate the standard deviation for a certain column:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Or plot three lines at once by supplying three pairs of x y values:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Pandas
Let's now import Pandas. This library provides R-style dataframe table functionality. Like NumPy, it is convention to import the Pandas library as pd for brevity.
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Perhaps now we would like to drop the ID column:
Can you write Python code for it?

df = df.drop(["ID"], axis=1)
df.describe()


I am working on the file "Assignment1.ipynb".
The first task is:
SciKit-Learn
Last but not least, we shall import some modules from SciKit-Learn. SciKit-Learn is the main machine learning library for Python. It is a large library and is not normally loaded entirely; in general you load only the modules you need from the main library. Here we will load the datasets module and the k-nearest neighbours module (KNeighborsClassifier):
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
So you see only 2s and 4s are contained in this column. See a breakdown of the counts using the value_counts() function:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
The k-Nearest Neighbour algorithm is possibly the simplest classifier. Given a new observation, take the label of the sample closest to it in the n-dimensional feature space.
First we must randomise the data, but we must ensure we randomise the labels as well in sync. We can use a NumPy feature to create indices that then correlate to both the targets and the data:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
The plt.plot() function will take an arbritrary number of x y argument pairs:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Then, the data must be split into a test set and a training set (again we are using naming conventions here for the training and test data X_train and X_test and their labels y_train and y_test):
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
Using the plt.plot() function, you can plot many types of data and Matplotlib will try to figure out what you want to do with the data:
Can you write Python code for it?


I am working on the file "Assignment1.ipynb".
The first task is:
We can now element-wise compare our predicted results, in y_pred, with the true labels stored in y_test:
Can you write Python code for it?


I am working on the file "Assignment1/Assignment1.ipynb".
The first task is:
Set configurables:
Can you write Python code for it?

_DATA_FILEPATH = 'datagovdatasetsviewmetrics.csv'
_ROTATION_DEGREES = 90
_BOTTOM_MARGIN = 0.35
_COLOR_THEME = 'coolwarm'
_LABEL_X = 'Organizations'
_LABEL_Y = 'Views'
_TITLE = 'Organizations with Most Views'
_ORGANIZATION_COUNT = 10
_MAX_LABEL_LENGTH = 20


I am working on the file "Assignment1/Assignment1.ipynb".
The first task is:
Use seaborn (one of the nicest chart-libraries for Python) to make a bar-chart. Notice that pandas is just a layer around numpy.
Can you write Python code for it?

def plot_chart(distilled):
    # Split the series into separate vectors of labels and values.

    labels_raw = []
    values_raw = []
    for (name, views) in distilled:
        labels_raw.append(name)
        values_raw.append(views)

    labels = numpy.array(labels_raw)
    values = numpy.array(values_raw)

    # Create one plot.

    seaborn.set(style="white", context="talk")

    (f, ax) = matplotlib.pyplot.subplots(1)

    b = seaborn.barplot(
        labels,
        values,
        ci=None,
        palette=_COLOR_THEME,
        hline=0,
        ax=ax,
        x_order=labels)

    # Set labels.

    ax.set_title(_TITLE)
    ax.set_xlabel(_LABEL_X)
    ax.set_ylabel(_LABEL_Y)

    # Rotate the x-labels (otherwise they'll overlap). Seaborn also doesn't do
    # very well with diagonal labels so we'll go vertical.
    b.set_xticklabels(labels, rotation=_ROTATION_DEGREES)

    # Add some margin to the bottom so the labels aren't cut-off.
    matplotlib.pyplot.subplots_adjust(bottom=_BOTTOM_MARGIN)


I am working on the file "Assignment1/Assignment1.ipynb".
The first task is:
Use the tabulate library to render a nice table. This is one of my most favorite Python tools:
Can you write Python code for it?

def print_table(distilled):
    headings = ['Organization', 'Views']
    print(tabulate.tabulate(distilled, headers=headings))


I am working on the file "Assignment1/Assignment1.ipynb".
The first task is:
We use pandas to read, group, and sort our data:
Can you write Python code for it?

def read_data():
    d = pandas.read_csv(_DATA_FILEPATH)

    return d

def process_data(d):
    # Group by organization.

    def sum_views(df):
        return sum(df['Views per Month'])

    g = d.groupby('Organization Name').apply(sum_views)

    # Sort by views (descendingly).

    g.sort(ascending=False)

    # Grab the first N to plot.

    items = g.iteritems()
    s = itertools.islice(items, 0, _ORGANIZATION_COUNT)

    s = list(s)

    # Sort them in ascending order, this time, so that the larger ones are on
    # the right (in red) in the chart. This has a side-effect of flattening the
    # generator while we're at it.
    s = sorted(s, key=lambda (n, v): v)

    # Truncate the names (otherwise they're unwieldy).

    distilled = []
    for (name, views) in s:
        if len(name) > (_MAX_LABEL_LENGTH - 3):
            name = name[:17] + '...'

        distilled.append((name, views))

    return distilled


I am working on the file "Assignment1/KristinaMilkovich-EarthquakeStats.ipynb".
The first task is:

2. Query Data
The two examples below allow you to look at specific parts of your data.
1. Get places with the highest magnitude earthquakes
2. Find all the earthquakes in California
Can you write Python code for it?

earthquakes_ds = earthquakes_ds.set_index(earthquakes_ds["place"])
earthquakes_ds[["mag"]].sort_values(by = "mag", ascending=False).head(10)

#Query that returns all the earthquakes in California.
earthquakes_ds[earthquakes_ds['place'].str.contains("California")]


I am working on the file "Assignment1/KristinaMilkovich-EarthquakeStats.ipynb".
The first task is:

4. Example: Find the places with the most Earthquakes

Parse data in the places column to determine closest major city to every earthquake.
Get a unique list of cities and determine how many earthquakes were in each city.
See Pie Chart below - each city's slice is proportional to how many earthquakes occurred during the dataset time frame.

The winner is Road Town, British Virgin Islands.
Runner up is Mooreland, Oklahoma.
Can you write Python code for it?

earthquakes_ds = earthquakes_ds.set_index(earthquakes_ds["place"])
s = pd.Series(earthquakes_ds['place'])
section =s.str.split("of")
series_name = section.str.get(1)
set_series = set(series_name)
set_series = list(set_series)
series_name

from collections import *
series_count = Counter(series_name)
Counter(series_name)

# group keys and count, likely an better method in Panda lib
import matplotlib.pyplot as plt, numpy as np
names = []
counts = []

for item in series_count.keys():
    names.append(item)
    counts.append(series_count[item])

# earthquake pie - Percentage Earthquakes between 10/8/16 to 10/15/16
plt.axis('equal')
plt.pie(
    counts,
    labels= names,
    colors=['blue', 'green', 'red', 'turquoise', 'magenta','yellow', 'purple'],
    autopct="%1.2f%%",
    radius=4);


I am working on the file "Assignment1/KristinaMilkovich-EarthquakeStats.ipynb".
The first task is:

3. Visual Representations of Queries
Here, queries are plotted into different graphs to allow better understanding and visualization of data. The library matplotlib is used extensively.
1. Below is a bar graph of earthquake magitude by depth of epicenter.
2. Further down is a line graph of longitude versus latitude of all earthquakes (Note: certain regions on the graph contain clusters)
Can you write Python code for it?

%matplotlib inline

# shows relationship between magnitude of an earthquake and depth of the epicenter
earthquakes_ds = earthquakes_ds.set_index(earthquakes_ds['depth']).sort_values(by = "depth", ascending=True)
mag = earthquakes_ds['mag']
depthplot = mag.plot(kind='bar', figsize=(22,5), title="Magnitude by Depth, Earthquakes in North America Oct 8, 2016 to Oct 15, 2016")
depthplot.set_ylabel("Magnitude using Log Scale")
depthplot.set_xlabel("Depth (meters)");

# maps longitude versus latitude to display earthquake clusters by location
earthquakes_ds = earthquakes_ds.set_index(earthquakes_ds['longitude']).sort_values(by = "longitude", ascending=True)
time = earthquakes_ds[["latitude"]].sort_values(by = "latitude", ascending=True)
ax = time.plot(figsize=(20,8), marker='o', linestyle='-', title="Earthquakes, Longitude versus Latitude in Current Dataset")
ax.set_xlabel("Longitude")
ax.set_ylabel("Latitude");


I am working on the file "Assignment1/features.ipynb".
The first task is:
Extract Features
For each image we will compute a Histogram of Oriented
Gradients (HOG) as well as a color histogram using the hue channel in HSV
color space. We form our final feature vector for each image by concatenating
the HOG and color histogram feature vectors.
Roughly speaking, HOG should capture the texture of the image while ignoring
color information, and the color histogram represents the color of the input
image while ignoring texture. As a result, we expect that using both together
ought to work better than using either alone. Verifying this assumption would
be a good thing to try for the bonus section.
The hog_feature and color_histogram_hsv functions both operate on a single
image and return a feature vector for that image. The extract_features
function takes a set of images and a list of feature functions and evaluates
each feature function on each image, storing the results in a matrix where
each column is the concatenation of all feature vectors for a single image.
Can you write Python code for it?

from cs231n.features import *

num_color_bins = 10 # Number of bins in the color histogram
feature_fns = [hog_feature, lambda img: color_histogram_hsv(img, nbin=num_color_bins)]
X_train_feats = extract_features(X_train, feature_fns, verbose=True)
X_val_feats = extract_features(X_val, feature_fns)
X_test_feats = extract_features(X_test, feature_fns)

# Preprocessing: Subtract the mean feature
mean_feat = np.mean(X_train_feats, axis=0, keepdims=True)
X_train_feats -= mean_feat
X_val_feats -= mean_feat
X_test_feats -= mean_feat

# Preprocessing: Divide by standard deviation. This ensures that each feature
# has roughly the same scale.
std_feat = np.std(X_train_feats, axis=0, keepdims=True)
X_train_feats /= std_feat
X_val_feats /= std_feat
X_test_feats /= std_feat

# Preprocessing: Add a bias dimension
X_train_feats = np.hstack([X_train_feats, np.ones((X_train_feats.shape[0], 1))])
X_val_feats = np.hstack([X_val_feats, np.ones((X_val_feats.shape[0], 1))])
X_test_feats = np.hstack([X_test_feats, np.ones((X_test_feats.shape[0], 1))])


I am working on the file "Assignment1/features.ipynb".
The first task is:
Load data
Similar to previous exercises, we will load CIFAR-10 data from disk.
Can you write Python code for it?

from cs231n.features import color_histogram_hsv, hog_feature

def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):
  # Load the raw CIFAR-10 data
  cifar10_dir = 'cs231n/datasets/cifar-10-batches-py'
  X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)

  # Subsample the data
  mask = range(num_training, num_training + num_validation)
  X_val = X_train[mask]
  y_val = y_train[mask]
  mask = range(num_training)
  X_train = X_train[mask]
  y_train = y_train[mask]
  mask = range(num_test)
  X_test = X_test[mask]
  y_test = y_test[mask]

  return X_train, y_train, X_val, y_val, X_test, y_test

X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()


I am working on the file "Assignment1/knn.ipynb".
The first task is:
We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:

First we must compute the distances between all test examples and all train examples.
Given these distances, for each test example we find the k nearest examples and have them vote for the label

Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.
First, open cs231n/classifiers/k_nearest_neighbor.py and implement the function compute_distances_two_loops that uses a (very inefficient) double loop over all pairs of (test, train) examples and computes the distance matrix one element at a time.
Can you write Python code for it?

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print dists.shape

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()


I am working on the file "Assignment1/knn.ipynb".
The first task is:
You should expect to see a slightly better performance than with k = 1.
Can you write Python code for it?

# Now lets speed up distance matrix computation by using partial vectorization
# with one loop. Implement the function compute_distances_one_loop and run the
# code below:
dists_one = classifier.compute_distances_one_loop(X_test)

# To ensure that our vectorized implementation is correct, we make sure that it
# agrees with the naive implementation. There are many ways to decide whether
# two matrices are similar; one of the simplest is the Frobenius norm. In case
# you haven't seen it before, the Frobenius norm of two matrices is the square
# root of the squared sum of differences of all elements; in other words, reshape
# the matrices into vectors and compute the Euclidean distance between them.
difference = np.linalg.norm(dists - dists_one, ord='fro')
print 'Difference was: %f' % (difference, )
if difference < 0.001:
    print 'Good! The distance matrices are the same'
else:
    print 'Uh-oh! The distance matrices are different'

# Now implement the fully vectorized version inside compute_distances_no_loops
# and run the code
dists_two = classifier.compute_distances_no_loops(X_test)

# check that the distance matrix agrees with the one we computed before:
difference = np.linalg.norm(dists - dists_two, ord='fro')
print 'Difference was: %f' % (difference, )
if difference < 0.001:
    print 'Good! The distance matrices are the same'
else:
    print 'Uh-oh! The distance matrices are different'

# Let's compare how fast the implementations are
def time_function(f, *args):
    """
    Call a function f with args and return the time (in seconds) that it took to execute.
    """
    import time
    tic = time.time()
    f(*args)
    toc = time.time()
    return toc - tic

two_loop_time = time_function(classifier.compute_distances_two_loops, X_test)
print 'Two loop version took %f seconds' % two_loop_time

one_loop_time = time_function(classifier.compute_distances_one_loop, X_test)
print 'One loop version took %f seconds' % one_loop_time

no_loop_time = time_function(classifier.compute_distances_no_loops,...


I am working on the file "Assignment1/knn.ipynb".
The first task is:
You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:
Can you write Python code for it?

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print 'Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy)


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Backward pass
Implement the rest of the function. This will compute the gradient of the loss with respect to the variables W1, b1, W2, and b2. Now that you (hopefully!) have a correctly implemented forward pass, you can debug your backward pass using a numeric gradient check:
Can you write Python code for it?

from cs231n.gradient_check import eval_numerical_gradient

# Use numeric gradient checking to check your implementation of the backward pass.
# If your implementation is correct, the difference between the numeric and
# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.

loss, grads = net.loss(X, y, reg=0.1)

# these should all be less than 1e-8 or so
for param_name in grads:
  f = lambda W: net.loss(X, y, reg=0.1)[0]
  param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False)
  print '%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name]))


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Debug the training
With the default parameters we provided above, you should get a validation accuracy of about 0.29 on the validation set. This isn't very good.
One strategy for getting insight into what's wrong is to plot the loss function and the accuracies on the training and validation sets during optimization.
Another strategy is to visualize the weights that were learned in the first layer of the network. In most neural networks trained on visual data, the first layer weights typically show some visible structure when visualized.
Can you write Python code for it?

# Plot the loss function and train / validation accuracies
plt.subplot(2, 1, 1)
plt.plot(stats['loss_history'])
plt.title('Loss history')
plt.xlabel('Iteration')
plt.ylabel('Loss')

plt.subplot(2, 1, 2)
plt.plot(stats['train_acc_history'], label='train')
plt.plot(stats['val_acc_history'], label='val')
plt.title('Classification accuracy history')
plt.xlabel('Epoch')
plt.ylabel('Clasification accuracy')
plt.show()

from cs231n.vis_utils import visualize_grid

# Visualize the weights of the network

def show_net_weights(net):
  W1 = net.params['W1']
  W1 = W1.reshape(32, 32, 3, -1).transpose(3, 0, 1, 2)
  plt.imshow(visualize_grid(W1, padding=3).astype('uint8'))
  plt.gca().axis('off')
  plt.show()

show_net_weights(net)


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Forward pass: compute loss
In the same function, implement the second part that computes the data and regularizaion loss.
Can you write Python code for it?

loss, _ = net.loss(X, y, reg=0.1)
correct_loss = 1.30378789133

# should be very small, we get < 1e-12
print 'Difference between your loss and correct loss:'
print np.sum(np.abs(loss - correct_loss))


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Run on the test set
When you are done experimenting, you should evaluate your final trained network on the test set; you should get above 48%.
We will give you extra bonus point for every 1% of accuracy above 52%.
Can you write Python code for it?

test_acc = (best_net.predict(X_test) == y_test).mean()
print 'Test accuracy: ', test_acc


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Train a network
To train our network we will use SGD with momentum. In addition, we will adjust the learning rate with an exponential learning rate schedule as optimization proceeds; after each epoch, we will reduce the learning rate by multiplying it by a decay rate.
Can you write Python code for it?

input_size = 32 * 32 * 3
hidden_size = 50
num_classes = 10
net = TwoLayerNet(input_size, hidden_size, num_classes)

# Train the network
stats = net.train(X_train, y_train, X_val, y_val,
            num_iters=1000, batch_size=200,
            learning_rate=1e-4, learning_rate_decay=0.95,
            reg=0.5, verbose=True)

# Predict on the validation set
val_acc = (net.predict(X_val) == y_val).mean()
print 'Validation accuracy: ', val_acc


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Train the network
To train the network we will use stochastic gradient descent (SGD), similar to the SVM and Softmax classifiers. Look at the function TwoLayerNet.train and fill in the missing sections to implement the training procedure. This should be very similar to the training procedure you used for the SVM and Softmax classifiers. You will also have to implement TwoLayerNet.predict, as the training process periodically performs prediction to keep track of accuracy over time while the network trains.
Once you have implemented the method, run the code below to train a two-layer network on toy data. You should achieve a training loss less than 0.2.
Can you write Python code for it?

net = init_toy_model()
stats = net.train(X, y, X, y,
            learning_rate=1e-1, reg=1e-5,
            num_iters=100, verbose=False)

print 'Final training loss: ', stats['loss_history'][-1]

# plot the loss history
plt.plot(stats['loss_history'])
plt.xlabel('iteration')
plt.ylabel('training loss')
plt.title('Training Loss history')
plt.show()


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
Tune your hyperparameters
What's wrong?. Looking at the visualizations above, we see that the loss is decreasing more or less linearly, which seems to suggest that the learning rate may be too low. Moreover, there is no gap between the training and validation accuracy, suggesting that the model we used has low capacity, and that we should increase its size. On the other hand, with a very large model we would expect to see more overfitting, which would manifest itself as a very large gap between the training and validation accuracy.
Tuning. Tuning the hyperparameters and developing intuition for how they affect the final performance is a large part of using Neural Networks, so we want you to get a lot of practice. Below, you should experiment with different values of the various hyperparameters, including hidden layer size, learning rate, numer of training epochs, and regularization strength. You might also consider tuning the learning rate decay, but you should be able to get good performance using the default value.
Approximate results. You should be aim to achieve a classification accuracy of greater than 48% on the validation set. Our best network gets over 52% on the validation set.
Experiment: You goal in this exercise is to get as good of a result on CIFAR-10 as you can, with a fully-connected Neural Network. For every 1% above 52% on the Test set we will award you with one extra bonus point. Feel free implement your own techniques (e.g. PCA to reduce dimensionality, or adding dropout, or adding features to the solver, etc.).
Can you write Python code for it?

best_net = None # store the best model into this

learning = [1e-5, 1e-3]
regularization = [0, 1]
decay = [0.9, 1]

results = {}
best_val = -1

for num_hidden in np.arange(50, 300, 50):
    for _ in np.arange(0, 50):
            i = np.random.uniform(low=learning[0], high=learning[1])
            j = np.random.uniform(low=regularization[0], high=regularization[1])
            k = np.random.uniform(low=decay[0], high=decay[1])

            # Train the network
            net = TwoLayerNet(input_size, num_hidden, num_classes)
            stats = net.train(X_train, y_train, X_val, y_val,
                              num_iters=500, batch_size=200,
                              learning_rate=i, learning_rate_decay=k,
                              reg=j, verbose=False)

            # Predict on the validation set
            val_acc = (net.predict(X_val) == y_val).mean()

            results[(num_hidden, i, j, k)] = val_acc
            if val_acc > best_val:
                best_val = val_acc

# Print the obtained accuracies
for nh, lr, reg, dec in sorted(results):
    print 'Hidden: %d, learning rate: %f, regularisation: %f, decay: %f -> %f' % ( \
    nh, lr, reg, dec, results[nh, lr, reg, dec])

# Find the best learning rate and regularization strength
best_hidden = 25
best_lr = 0.000958
best_reg = 0.952745
best_decay = 0.935156

best_val = -1
for nh, lr, reg, dec in sorted(results):
    if results[(nh, lr, reg, dec)] > best_val:
        best_val = results[(nh, lr, reg, dec)]
        best_hidden = nh
        best_lr = lr
        best_reg = reg
        best_decay = dec

# Train the best_svm with more iterations
best_net = TwoLayerNet(input_size, best_hidden, num_classes)
stats = best_net.train(X_train, y_train, X_val, y_val,
                  num_iters=2000, batch_size=200,
                  learning_rate=best_lr, learning_rate_decay=best_decay,
                  reg=best_reg, verbose=True)

# Predict on the validation set
val_acc = (net.predict(X_val) == y_val).mean()

print 'Best validation...


I am working on the file "Assignment1/two_layer_net.ipynb".
The first task is:
We will use the class TwoLayerNet in the file cs231n/classifiers/neural_net.py to represent instances of our network. The network parameters are stored in the instance variable self.params where keys are string parameter names and values are numpy arrays. Below, we initialize toy data and a toy model that we will use to develop your implementation.
Can you write Python code for it?

# Create a small net and some toy data to check your implementations.
# Note that we set the random seed for repeatable experiments.

input_size = 4
hidden_size = 10
num_classes = 3
num_inputs = 5

def init_toy_model():
  np.random.seed(0)
  return TwoLayerNet(input_size, hidden_size, num_classes, std=1e-1)

def init_toy_data():
  np.random.seed(1)
  X = 10 * np.random.randn(num_inputs, input_size)
  y = np.array([0, 1, 2, 2, 1])
  return X, y

net = init_toy_model()
X, y = init_toy_data()


I am working on the file "Assignment2.ipynb".
The first task is:
Classification
Task: Use Support Vector Machines and a One Vs. All (OVA) approach to replicate the results from the Ramaswamy et al. paper.
SciKit Learn provides an SVM package for Support Vector Machines (see http://scikit-learn.org/stable/modules/svm.html).
Visualise your results appropriately using plots and tables to describe classification results on the test set.
Can you write Python code for it?

# Your classification code. Use as many cells as required, use Markdown cells to document where necessary.


I am working on the file "Assignment2.ipynb".
The first task is:
Data Exploration
Now that the data have been loaded in X_train, X_test, y_train, and y_test, we can take a look a closer look at our data. Note: It is convention to use large X for data matrices, and small y for target vectors.
As can be seen, in our training set we have $p=16,063$ genes/features and $N=144$ tumours/samples:
Can you write Python code for it?


I am working on the file "Assignment2.ipynb".
The first task is:
Let's see how the classes are distributed. First let's look at the number of unique values, which should equal 14, as we know we have 14 different cancer types:
Can you write Python code for it?


I am working on the file "Assignment2.ipynb".
The first task is:
Summary
Now that we have read the data in a form which we can easily use, we move on to the deliverables that must be completed for Assignment 2.
Deliverables for Assignment 2
Clustering
Task: Perform hierarchical clustering mimicking the approaches used by Ramaswamy et al. in their paper cited above. Plot a dendogram of your results (SciPy provides dendogram plotting functions, see http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.cluster.hierarchy.dendrogram.html for example) - or visualise your clustering in any other way you deem reasonable.
Both SciKit Learn and SciPy offer hierarchical clustering algorithms, see http://docs.scipy.org/doc/scipy/reference/cluster.hierarchy.html and http://scikit-learn.org/stable/modules/clustering.html.
Notice that not all clustering techniques are useful for all purposes. In the case of this assignment, we know the number of clusters we are searching for - this is a requirement for certain clustering algorithms. Other algorithms may require parameters you might not immediately have available to you.
Can you write Python code for it?

# Your clustering code. Use as many cells as required, use Markdown cells to document where necessary.


I am working on the file "Assignment2.ipynb".
The first task is:
Using the cancer_names_longform list we declared above, we can print tumour frequencies nicely:
Can you write Python code for it?

for freq in stats.itemfreq(y_train):
    print "%s samples appear %s times (shortform: %s)." % (cancer_names_longform[int(freq[0])-1],
                                                           freq[1],
                                                           cancer_names_shortform[int(freq[0])-1])


I am working on the file "Assignment2.ipynb".
The first task is:
We can see how the cancer types are distrubuted using the itemfreq function of the SciPy stats package:
Can you write Python code for it?


I am working on the file "Assignment2.ipynb".
The first task is:
You can take a quick look at some statistics values for each gene using the useful describe function (we use transpose to perform the analysis on a gene-by-gene basis). For example you may want to look at mean expression levels for each gene to see if they are over-expressed or under-expressed:
Can you write Python code for it?

# Note: The transpose() function here does not permanently transpose the data stored in X_train.
X_train.transpose().describe()


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Batch Normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For the sigmoid function, it turns out that you can derive a very simple formula for the backward pass by simplifying gradients on paper.
Surprisingly, it turns out that you can also derive a simple expression for the batch normalization backward pass if you work out derivatives on paper and simplify. After doing so, implement the simplified batch normalization backward pass in the function batchnorm_backward_alt and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.
NOTE: You can still complete the rest of the assignment if you don't figure this part out, so don't worry too much if you can't get it.
Can you write Python code for it?

N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print 'dx difference: ', rel_error(dx1, dx2)
print 'dgamma difference: ', rel_error(dgamma1, dgamma2)
print 'dbeta difference: ', rel_error(dbeta1, dbeta2)
print 'speedup: %.2fx' % ((t2 - t1) / (t3 - t2))


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Batch Normalization: backward
Now implement the backward pass for batch normalization in the function batchnorm_backward.
To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.
Once you have finished, run the following to numerically check your backward pass.
Can you write Python code for it?

# Gradient check batchnorm backward pass

N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, gamma, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, beta, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma, dout)
db_num = eval_numerical_gradient_array(fb, beta, dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
print 'dx error: ', rel_error(dx_num, dx)
print 'dgamma error: ', rel_error(da_num, dgamma)
print 'dbeta error: ', rel_error(db_num, dbeta)


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.
The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.
Can you write Python code for it?

# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers = {}
solvers = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print 'Running weight scale %d / %d' % (i + 1, len(weight_scales))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=True)
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=False)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers[weight_scale] = solver

# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers[ws].train_acc_history))

  best_val_accs.append(max(solvers[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val...


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Batch normalization: Forward
In the file cs231n/layers.py, implement the batch normalization forward pass in the function batchnorm_forward. Once you have done so, run the following to test your implementation.
Can you write Python code for it?

# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print 'Before batch normalization:'
print '  means: ', a.mean(axis=0)
print '  stds: ', a.std(axis=0)

# Means should be close to zero and stds close to one
print 'After batch normalization (gamma=1, beta=0)'
a_norm, _ = batchnorm_forward(a, np.ones(D3), np.zeros(D3), {'mode': 'train'})
print '  mean: ', a_norm.mean(axis=0)
print '  std: ', a_norm.std(axis=0)

# Now means should be close to beta and stds close to gamma
gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print 'After batch normalization (nontrivial gamma, beta)'
print '  means: ', a_norm.mean(axis=0)
print '  stds: ', a_norm.std(axis=0)

# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)
for t in xrange(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)
bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print 'After batch normalization (test-time):'
print '  means: ', a_norm.mean(axis=0)
print '  stds: ', a_norm.std(axis=0)


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.
Can you write Python code for it?

# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=True)
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, use_batchnorm=False)

bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=200)
bn_solver.train()

solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=200)
solver.train()


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your FullyConnectedNet in the file cs2312n/classifiers/fc_net.py. Modify your implementation to add batch normalization.
Concretely, when the flag use_batchnorm is True in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.
HINT: You might find it useful to define an additional helper layer similar to those in the file cs231n/layer_utils.py. If you decide to do so, do it in the file cs231n/classifiers/fc_net.py.
Can you write Python code for it?

N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

for reg in [0, 3.14]:
  print 'Running check with reg = ', reg
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            use_batchnorm=True)

  loss, grads = model.loss(X, y)
  print 'Initial loss: ', loss

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print '%s relative error: %.2e' % (name, rel_error(grad_num, grads[name]))
  if reg == 0: print


I am working on the file "Assignment2/BatchNormalization.ipynb".
The first task is:
Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.
Can you write Python code for it?

plt.subplot(3, 1, 1)
plt.title('Training loss')
plt.xlabel('Iteration')

plt.subplot(3, 1, 2)
plt.title('Training accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 3)
plt.title('Validation accuracy')
plt.xlabel('Epoch')

plt.subplot(3, 1, 1)
plt.plot(solver.loss_history, 'o', label='baseline')
plt.plot(bn_solver.loss_history, 'o', label='batchnorm')

plt.subplot(3, 1, 2)
plt.plot(solver.train_acc_history, '-o', label='baseline')
plt.plot(bn_solver.train_acc_history, '-o', label='batchnorm')

plt.subplot(3, 1, 3)
plt.plot(solver.val_acc_history, '-o', label='baseline')
plt.plot(bn_solver.val_acc_history, '-o', label='batchnorm')

for i in [1, 2, 3]:
  plt.subplot(3, 1, i)
  plt.legend(loc='upper center', ncol=4)
plt.gcf().set_size_inches(15, 15)
plt.show()


I am working on the file "Assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Convolution: Naive backward pass
Implement the backward pass for the convolution operation in the function conv_backward_naive in the file cs231n/layers.py. Again, you don't need to worry too much about computational efficiency.
When you are done, run the following to check your backward pass with a numeric gradient check.
Can you write Python code for it?

x = np.random.randn(4, 3, 5, 5)
w = np.random.randn(2, 3, 3, 3)
b = np.random.randn(2,)
dout = np.random.randn(4, 2, 5, 5)
conv_param = {'stride': 1, 'pad': 1}

dx_num = eval_numerical_gradient_array(lambda x: conv_forward_naive(x, w, b, conv_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_forward_naive(x, w, b, conv_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_forward_naive(x, w, b, conv_param)[0], b, dout)

out, cache = conv_forward_naive(x, w, b, conv_param)
dx, dw, db = conv_backward_naive(dout, cache)

# Your errors should be around 1e-9'
print 'Testing conv_backward_naive function'
print 'dx error: ', rel_error(dx, dx_num)
print 'dw error: ', rel_error(dw, dw_num)
print 'db error: ', rel_error(db, db_num)


I am working on the file "Assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Convolution: Naive forward pass
The core of a convolutional network is the convolution operation. In the file cs231n/layers.py, implement the forward pass for the convolution layer in the function conv_forward_naive.
You don't have to worry too much about efficiency at this point; just write the code in whatever way you find most clear.
You can test your implementation by running the following:
Can you write Python code for it?


I am working on the file "Assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Convolutional "sandwich" layers
Previously we introduced the concept of "sandwich" layers that combine multiple operations into commonly used patterns. In the file cs231n/layer_utils.py you will find sandwich layers that implement a few commonly used patterns for convolutional networks.
Can you write Python code for it?

from cs231n.layer_utils import conv_relu_pool_forward, conv_relu_pool_backward

x = np.random.randn(2, 3, 16, 16)
w = np.random.randn(3, 3, 3, 3)
b = np.random.randn(3,)
dout = np.random.randn(2, 3, 8, 8)
conv_param = {'stride': 1, 'pad': 1}
pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}

out, cache = conv_relu_pool_forward(x, w, b, conv_param, pool_param)
dx, dw, db = conv_relu_pool_backward(dout, cache)

dx_num = eval_numerical_gradient_array(lambda x: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_relu_pool_forward(x, w, b, conv_param, pool_param)[0], b, dout)

print 'Testing conv_relu_pool'
print 'dx error: ', rel_error(dx_num, dx)
print 'dw error: ', rel_error(dw_num, dw)
print 'db error: ', rel_error(db_num, db)

from cs231n.layer_utils import conv_relu_forward, conv_relu_backward

x = np.random.randn(2, 3, 8, 8)
w = np.random.randn(3, 3, 3, 3)
b = np.random.randn(3,)
dout = np.random.randn(2, 3, 8, 8)
conv_param = {'stride': 1, 'pad': 1}

out, cache = conv_relu_forward(x, w, b, conv_param)
dx, dw, db = conv_relu_backward(dout, cache)

dx_num = eval_numerical_gradient_array(lambda x: conv_relu_forward(x, w, b, conv_param)[0], x, dout)
dw_num = eval_numerical_gradient_array(lambda w: conv_relu_forward(x, w, b, conv_param)[0], w, dout)
db_num = eval_numerical_gradient_array(lambda b: conv_relu_forward(x, w, b, conv_param)[0], b, dout)

print 'Testing conv_relu:'
print 'dx error: ', rel_error(dx_num, dx)
print 'dw error: ', rel_error(dw_num, dw)
print 'db error: ', rel_error(db_num, db)


I am working on the file "Assignment2/ConvolutionalNetworks.ipynb".
The first task is:
Experiment!
Experiment and try to get the best performance that you can on CIFAR-10 using a ConvNet. Here are some ideas to get you started:
Things you should try:

Filter size: Above we used 7x7; this makes pretty pictures but smaller filters may be more efficient
Number of filters: Above we used 32 filters. Do more or fewer do better?
Batch normalization: Try adding spatial batch normalization after convolution layers and vanilla batch normalization aafter affine layers. Do your networks train faster?
Network architecture: The network above has two layers of trainable parameters. Can you do better with a deeper network? You can implement alternative architectures in the file cs231n/classifiers/convnet.py. Some good architectures to try include:
[conv-relu-pool]xN - conv - relu - [affine]xM - [softmax or SVM]
[conv-relu-pool]XN - [affine]XM - [softmax or SVM]
[conv-relu-conv-relu-pool]xN - [affine]xM - [softmax or SVM]

Tips for training
For each network architecture that you try, you should tune the learning rate and regularization strength. When doing this there are a couple important things to keep in mind:

If the parameters are working well, you should see improvement within a few hundred iterations
Remember the course-to-fine approach for hyperparameter tuning: start by testing a large range of hyperparameters for just a few training iterations to find the combinations of parameters that are working at all.
Once you have found some sets of parameters that seem to work, search more finely around these parameters. You may need to train for more epochs.

Going above and beyond
If you are feeling adventurous there are many other features you can implement to try and improve your performance. You are not required to implement any of these; however they would be good things to try for extra credit.

Alternative update steps: For the assignment we implemented SGD+momentum, RMSprop, and Adam; you could try alternatives like AdaGrad or AdaDelta.
Alternative activation functions such as leaky ReLU, parametric...
Can you write Python code for it?

model = ChaudharyNet(hidden_dim=4096, reg=0.5)

solver = Solver(model, data,
                num_epochs=50, batch_size=500,
                update_rule='adadelta',
                optim_config={
                  'rho': 0.95,
                  'epsilon': 1e-8,
                  'learning_rate': 0.001
                },
                verbose=True, print_every=5)
solver.train()

plt.subplot(2, 1, 1)
plt.plot(solver.loss_history, '-')
plt.xlabel('iteration')
plt.ylabel('loss')

plt.subplot(2, 1, 2)
plt.plot(solver.train_acc_history, '-')
plt.plot(solver.val_acc_history, '-')
plt.legend(['train', 'val'], loc='upper left')
plt.xlabel('epoch')
plt.ylabel('accuracy')
plt.show()

model