I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Challenges of Streaming Data:
Building an ANTARES-like Pipeline for Data Management and Discovery
Version 0.1

By AA Miller 2017 Apr 10
Edited by Gautham Narayan, 2017 Apr 26
As we just saw in Gautham's lecture - LSST will produce an unprecedented volume of time-domain information for the astronomical sky. $>37$ trillion individual photometric measurements will be recorded. While the vast, vast majority of these measurements will simply confirm the status quo, some will represent rarities that have never been seen before (e.g., LSST may be the first telescope to discover the electromagnetic counterpart to a LIGO graviational wave event), which the community will need to know about in ~real time.
Storing, filtering, and serving this data is going to be a huge <del>nightmare</del> challenge. ANTARES, as detailed by Gautham, is one proposed solution to this challenge. In this exercise you will build a miniature version of ANTARES, which will require the application of several of the lessons from earlier this week. Many of the difficult, and essential, steps necessary for ANTARES will be skipped here as they are too time consuming or beyond the scope of what we have previously covered. We will point out these challenges are we come across them.
Can you write Python code for it?

import numpy as np
import scipy.stats as spstat
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib notebook


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Finally, we close by commiting the changes we made to the database.
Note that strictly speaking this is not needed, however, were we to update any values in the database then we would need to commit those changes.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Let's unpack everything that happened in these two commands. First - if the table rawPhot already exists, we drop it to start over from scratch. (this is useful here, but should not be adopted as general practice)
Second - we create the new table rawPhot, which has 6 columns: id - a running index for every row in the table, objId - an ID to identify which source the row belongs to, t - the time of observation in MJD, pb - the passband of the observation, flux the observation flux, and dflux the uncertainty on the flux measurement. In addition to naming the columns, we also must declare their type. We have declared id as the primary key, which means this value will automatically be assigned and incremented for all data inserted into the database. We have also declared pb as a variable character of length 1, which is more useful and restrictive than simply declaring pb as text, which allows any freeform string.
Now we need to insert the raw flux measurements into the database. To do so, we will use the ANTARESlc class that we created earlier. As an initial example, we will insert the first 3 observations from the source FAKE010.
Can you write Python code for it?

filename = "training_set_for_LSST_DSFP/FAKE001.dat"
lc = ANTARESlc(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[0])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[1])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[2])))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Now that we have a cursor object, we can populate the database. As an example we will start by creating a table to hold all the raw photometry (though ultimately we will not use this table for analysis).
Note - there are many cursor methods capable of interacting with the database. The most common, execute, takes a single SQL command as its argument and executes that command. Other useful methods include executemany, which is useful for inserting data into the database, and executescript, which take an SQL script as its argument and executes the script.
In many cases, as below, it will be useful to use triple quotes in order to improve the legibility of your code.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if is already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Now that we have loaded the raw observations, we need to create a new table to store summary statistics for each object. This table will include everything we've added to the ANTARESlc class.
Can you write Python code for it?

cur.execute("""drop table if exists lcFeats""") # drop the table if it already exists
cur.execute("""create table lcFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES rawPhot(objId)
                )
""")


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
One thing that we brushed over previously is that the brightness measurements have units of flux, rather than the traditional use of magnitudes. The reason for this is that LSST will measure flux variations via image differencing, which will for some sources in some filters result in a measurement of negative flux. (You may have already noticed this in 1a.) Statistically there is nothing wrong with such a measurement, but it is impossible to convert a negative flux into a magnitude. Thus we will use flux measurements throughout this exercise. [Aside - if you are bored during the next break, I'd be happy to rant about why we should have ditched the magnitude system years ago.]
Using flux measurements will allow us to make unbiased measurements of the statistical distributions of the variations of the sources we care about.
Problem 1d
What is FAKE010 the source that is plotted above?
Hint 1 - if you have no idea that's fine, move on.
Hint 2 - ask Szymon or Tomas...
Solution 1d
FAKE010 is a transient, as can be seen by the rapid rise followed by a gradual decline in the light curve. In this particular case, we can further guess that FAKE010 is a Type Ia supernova due to the secondary maxima in the $i'$ and $z'$ light curves. These secondary peaks are not present in any other known type of transient.
Problem 1e
To get a better sense of the data, plot the multiband light curves for sources FAKE060 and FAKE073.
Can you write Python code for it?

lc59 = ANTARESlc("training_set_for_LSST_DSFP/FAKE060.dat")
lc59.plot_multicolor_lc()

lc60 = ANTARESlc("training_set_for_LSST_DSFP/FAKE073.dat")
lc60.plot_multicolor_lc()


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 1c
Confirm the corrections made in 1b by plotting the multiband light curve for the source FAKE010.
Can you write Python code for it?

lc = ANTARESlc('training_set_for_LSST_DSFP/FAKE010.dat')

lc.plot_multicolor_lc()


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 2b
Confirm your solution to 2a by measuring the mean colors of source FAKE010. Does your measurement make sense given the plot you made in 1c?
Can you write Python code for it?

lc = ANTARESlc('training_set_for_LSST_DSFP/FAKE010.dat')

lc.filter_flux()
lc.weighted_mean_flux()
lc.mean_colors()

print("The g'-r', r'-i', and 'i-z' colors are: {:.3f}, {:.3f}, and {:.3f}, respectively.". format(lc.gMinusR, lc.rMinusI, lc.iMinusZ))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 3b
Load all of the raw photometric observations into the rawPhot table in the database.
Hint - you can use glob to select all of the files being loaded.
Hint 2 - you have already loaded the data from FAKE001 into the table.
Can you write Python code for it?

import glob

filenames = glob.glob("training_set_for_LSST_DSFP/FAKE*.dat")

for filename in filenames[1:]:
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples

    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 3e
Confirm that the data loaded correctly by counting the number of sources with gAmp > 2.
How many sources have gMinusR = -999?
Hint - you should find 9 and 2, respectively.
Can you write Python code for it?

cur.execute("""select count(*) from lcFeats where gAmp > 2""")

nAmp2 = cur.fetchone()[0]

cur.execute("""select count(*) from lcFeats where gMinusR = -999""")
nNoColor = cur.fetchone()[0]

print("There are {:d} sources with gAmp > 2".format(nAmp2))
print("There are {:d} sources with no measured i' - z' color".format(nNoColor))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 4) Build a Classification Model
One of the primary goals for ANTARES is to separate the Wheat from the Chaff, in other words, given that ~10 million alerts will be issued by LSST on a nightly basis, what is the single (or 10, or 100) most interesting alert.
Here we will build on the skills developed during the DSFP Session 2 to construct a machine-learning model to classify new light curves.
Fortunately - the data that has already been loaded to miniANTARES.db is a suitable training set for the classifier (we simply haven't provided you with labels just yet). Execute the cell below to add a new table to the database which includes the appropriate labels.
Can you write Python code for it?

cur.execute("""drop table if exists lcLabels""") # drop the table if it already exists
cur.execute("""create table lcLabels(
               objId int,
               label int,
               foreign key(objId) references rawPhot(objId)
               )""")

labels = np.zeros(100)
labels[20:60] = 1
labels[60:] = 2

data = np.append(np.arange(1,101)[np.newaxis].T, labels[np.newaxis].T, axis = 1)
tup_data = [tuple(x) for x in data]

cur.executemany("""insert into lcLabels(objId, label) values (?,?)""", tup_data)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 4b
Train a SVM model (SVC in scikit-learn) using a radial basis function (RBF) kernel with penalty parameter, $C = 1$, and kernel coefficient, $\gamma = 0.1$.
Evaluate the accuracy of the model via $k = 5$ fold cross validation.
Hint - you may find the cross_val_score module helpful.
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.cross_validation import cross_val_score

cv_scores = cross_val_score(SVC(C = 1.0, gamma = 0.1, kernel = 'rbf'), X, y, cv = 5)

print("The SVM model produces a CV accuracy of {:.4f}".format(np.mean(cv_scores)))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 4d
Does the model you selected perform better than the SVM model? Perform a $k = 5$ fold cross validation to determine which model provides superior accuracy.
Can you write Python code for it?

cv_scores = cross_val_score(RandomForestClassifier(), X, y, cv = 5)

print("The RF model produces a CV accuracy of {:.4f}".format(np.mean(cv_scores)))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 5b
Calculate features for the new observations and insert those features into a database table.
Hint - again, you may want to create a new table for this, up to you. I'm using the testFeats table.
Can you write Python code for it?

for filename in new_obs_filenames:
    lc = ANTARESlc(filename)

    # simple HACK to get rid of data with too few observations (fails because std is nan with just one observation)
    if len(lc.DFlc) <= 14:
        continue

    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    lc.filter_flux()
    lc.weighted_mean_flux()
    lc.normalized_flux_std()
    lc.normalized_amplitude()
    lc.normalized_MAD()
    lc.normalized_beyond_1std()
    lc.skew()
    lc.mean_colors()

    feats = (objId, lc.gStd, lc.rStd, lc.iStd, lc.zStd,
            lc.gAmp,  lc.rAmp,  lc.iAmp,  lc.zAmp,
            lc.gMAD, lc.rMAD, lc.iMAD, lc.zMAD,
            lc.gBeyond, lc.rBeyond, lc.iBeyond, lc.zBeyond,
            lc.gSkew, lc.rSkew, lc.iSkew, lc.zSkew,
            lc.gMinusR, lc.rMinusI, lc.iMinusZ)

    cur.execute("""insert into testFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 5c
Train the model that you adopted in 4e on the training set, and produce predictions for the newly observed sources.
What is the class distribution for the newly detected sources?
Hint - the training set was constructed to have a nearly uniform class distribution, that may not be the case for the actual observed distribution of sources.
Can you write Python code for it?

svm_clf = SVC(C=1.0, gamma = 0.1, kernel = 'rbf').fit(X, y)

cur.execute("""select gStd, rStd, iStd, zStd,
                      gAmp,  rAmp,  iAmp,  zAmp,
                      gMAD, rMAD, iMAD, zMAD,
                      gBeyond, rBeyond, iBeyond, zBeyond,
                      gSkew, rSkew, iSkew, zSkew,
                      gMinusR, rMinusI, iMinusZ
               from testFeats
               order by objId asc""")
X_new = np.array(cur.fetchall())

y_preds = svm_clf.predict(X_new)

print("""There are {:d}, {:d}, and {:d} sources
         in classes 1, 2, 3, respectively""".format(*list(np.bincount(y_preds)))) # be careful using bincount


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
Problem 6b
Plot the light curves of the 2 most anomalous sources.
Can you identify why these sources have been selected as outliers?
Can you write Python code for it?

lc485 = ANTARESlc("test_set_for_LSST_DSFP/FAKE00485.dat")
lc485.plot_multicolor_lc()

lc2030 = ANTARESlc("test_set_for_LSST_DSFP/FAKE02030.dat")
lc2030.plot_multicolor_lc()


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
The above procedure should look familiar to above, with one exception: the addition of the foreign key in the lcFeats table. The inclusion of the foreign key ensures a connected relationship between rawPhot and lcFeats. In brief, a row cannot be inserted into lcFeats unless a corresponding row, i.e. objId, exists in rawPhot. Additionally, rows in rawPhot cannot be deleted if there are dependent rows in lcFeats.
Problem 3d
Calculate features for every source in rawPhot and insert those features into the lcFeats table.
Can you write Python code for it?

for filename in filenames:
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    lc.filter_flux()
    lc.weighted_mean_flux()
    lc.normalized_flux_std()
    lc.normalized_amplitude()
    lc.normalized_MAD()
    lc.normalized_beyond_1std()
    lc.skew()
    lc.mean_colors()

    feats = (objId, lc.gStd, lc.rStd, lc.iStd, lc.zStd,
            lc.gAmp,  lc.rAmp,  lc.iAmp,  lc.zAmp,
            lc.gMAD, lc.rMAD, lc.iMAD, lc.zMAD,
            lc.gBeyond, lc.rBeyond, lc.iBeyond, lc.zBeyond,
            lc.gSkew, lc.rSkew, lc.iSkew, lc.zSkew,
            lc.gMinusR, lc.rMinusI, lc.iMinusZ)

    cur.execute("""insert into lcFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntaresSolutions_serial.ipynb".
The first task is:
We now have a database connection object, conn. To interact with the database (create tables, load data, write queries) we need a cursor object.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
And again, map the function onto the filenames, and load the results into the table.
Can you write Python code for it?

# and then lets map the function onto all the data
result = lview.map(calculate_features, filenames)

# and get the result
all_feats = result.get()

# and now load it all into our table
for feats in all_feats:
    cur.execute("""insert into lcFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
And now that you have results, load them in the database as before.
Can you write Python code for it?

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
As we have many light curve files (in principle as many as 37 billion...), we will define a light curve class to ease our handling of the data.
 Problem 1b
Fix the lc class definition below.
Hint - the only purpose of this problem is to make sure you actually read each line of code below, it is not intended to be difficult.
Can you write Python code for it?

class ANTARESlc():
    '''Light curve object for NOAO formatted data'''

    def __init__(self, filename):
        DFlc = pd.read_csv(filename, delim_whitespace=True, comment = '#')
        self.DFlc = DFlc
        self.filename = filename

    def plot_multicolor_lc(self):
        '''Plot the 4 band light curve'''
        fig, ax = plt.subplots()
        g = ax.plot_the_errorbars_please(self.DFlc['t'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'g'],
                     fmt = 'o', color = '#78A5A3', label = r"$g'$")
        r = ax.plot_the_errorbars_please(self.DFlc['t'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'r'],
                     fmt = 'o', color = '#CE5A57', label = r"$r'$")
        i = ax.plot_the_errorbars_please(self.DFlc['t'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'i'],
                     fmt = 'o', color = '#E1B16A', label = r"$i'$")
        z = ax.plot_the_errorbars_please(self.DFlc['t'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'z'],
                     fmt = 'o', color = '#444C5C', label = r"$z'$")
        ax.legend(fancybox = True)
        ax.set_xlabel(r"$\mathrm{MJD}$")
        ax.set_ylabel(r"$\mathrm{flux}$")


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Eeek. That's way slower than the regular feature computation!
It actually turns out that this is OK for us, because the implementation of Gaussian Process Regression in sklearn is pretty slow, and there are various better packages and approximations we can make to speed things up dramatically.
For the purposes of this exercise though, we'll stop here, since it's no fun to run these GPs on 5000 lightcurves (and our implementation isn't really robust either).
Problem 6) Anomaly Detection
As we learned earlier - one of the primary goals of ANTARES is to reduce the stream of 10 million alerts on any given night to the single (or 10, or 100) most interesting objects. One possible definition of "interesting" is rarity - in which case it would be useful to add some form of anomaly detection to the pipeline. scikit-learn has several different algorithms that can be used for anomaly detection. Here we will employ isolation forest which has many parallels to random forests, which we have previously learned about.
In brief, isolation forest builds an ensemble of decision trees where the splitting parameter in each node of the tree is selected randomly. In each tree the number of branches necessary to isolate each source is measured - outlier sources will, on average, require fewer splittings to be isolated than sources in high-density regions of the feature space. Averaging the number of branchings over many trees results in a relative ranking of the anomalousness (yes, I just made up a word) of each source.
Problem 6a
Using IsolationForest in sklearn.ensemble - determine the 10 most isolated sources in the data set.
Hint - for IsolationForest you will want to use the decision_function() method rather than predict_proba(), which is what we have previously used with sklearn.ensemble models to get relative rankings from the model.
Can you write Python code for it?

from sklearn.ensemble import IsolationForest

isoF_clf = # complete
isoF_clf.fit( # complete
anomaly_score = isoF_clf.decision_function(X_new)

print("The 10 most anomalous sources are: {}".format( # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Finally, we close by commiting the changes we made to the database.
Note that strictly speaking this is not needed, however, were we to update any values in the database then we would need to commit those changes.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
For now - don't worry about what the labels mean (though if you inspect the light curves you may be able to figure this out...)
Problem 4a
Query the database to select features and labels for the light curves in your training set. Store the results of these queries in numpy arrays, X and y, respectively, which are suitable for the various scikit-learn machine learning algorithms.
Hint - recall that databases do not store ordered results.
Hint 2 - recall that scikit-learn expects y to be a 1d array. You will likely need to convert a 2d array to 1d.
Can you write Python code for it?

cur.execute( # complete
y = # complete

cur.execute(# complete

X = # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Let's unpack everything that happened in these two commands. First - if the table rawPhot already exists, we drop it to start over from scratch. (this is useful here, but should not be adopted as general practice)
Second - we create the new table rawPhot, which has 6 columns: id - a running index for every row in the table, objId - an ID to identify which source the row belongs to, t - the time of observation in MJD, pb - the passband of the observation, flux the observation flux, and dflux the uncertainty on the flux measurement. In addition to naming the columns, we also must declare their type. We have declared id as the primary key, which means this value will automatically be assigned and incremented for all data inserted into the database. We have also declared pb as a variable character of length 1, which is more useful and restrictive than simply declaring pb as text, which allows any freeform string.
Now we need to insert the raw flux measurements into the database. To do so, we will use the ANTARESlc class that we created earlier. As an initial example, we will insert the first 3 observations from the source FAKE010.
Can you write Python code for it?

filename = "training_set_for_LSST_DSFP/FAKE001.dat"
lc = ANTARESlc(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[0])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[1])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(lc.DFlc.ix[2])))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Lets look at one fit to see what's happening:
Can you write Python code for it?

ftest = filenames[0:20]
gtime, gflux, gdflux, gnewtime, gnewflux = calculate_gpfit(ftest[3])

fig, ax = plt.subplots()
ax.errorbar(gtime, gflux, gdflux, fmt = 'o', color = '#78A5A3')
ax.plot(gnewtime, gnewflux, 'k-')
ax.set_xlabel(r"$\mathrm{MJD}$")
ax.set_ylabel(r"$\mathrm{flux}$")


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Load all of the raw photometric observations into the rawPhot table in the database.
Normally, you'd load these files one by one in serial, looping over each file. This is what happens in miniAntares_serial.ipynb. Instead, we're going to use our cluster to parallelize this operation.
Problem 3b
First, build a list of filenames:
Hint - you can use glob to select all of the files being loaded.
Hint 2 - you have already loaded the data from FAKE001 into the table.
Can you write Python code for it?

# build your list of filenames here
import glob
filenames = # complete

# execute this
@require(ANTARESlc, 'pandas as pd')
def load_data(filename):
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples
    return data


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Now let's see how long that took
Can you write Python code for it?

%load_ext snakeviz
%snakeviz dummy_func()


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Now that we have a cursor object, we can populate the database. As an example we will start by creating a table to hold all the raw photometry (though ultimately we will not use this table for analysis).
Note - there are many cursor methods capable of interacting with the database. The most common, execute, takes a single SQL command as its argument and executes that command. Other useful methods include executemany, which is useful for inserting data into the database, and executescript, which take an SQL script as its argument and executes the script.
In many cases, as below, it will be useful to use triple quotes in order to improve the legibility of your code.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if is already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Now, figure out how to use the DirectView access to the cluster to map this function on to all the filenames, and get the results. Examine a few of them.
Can you write Python code for it?

result = lview.map(load_data, filenames[1:])
all_data = result.get()


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
One thing that we brushed over previously is that the brightness measurements have units of flux, rather than the traditional use of magnitudes. The reason for this is that LSST will measure flux variations via image differencing, which will for some sources in some filters result in a measurement of negative flux. (You may have already noticed this in 1a.) Statistically there is nothing wrong with such a measurement, but it is impossible to convert a negative flux into a magnitude. Thus we will use flux measurements throughout this exercise.
[Aside - if you are bored during the next break, I'd be happy to rant about why we should have ditched the magnitude system years ago.]
Using flux measurements will allow us to make unbiased measurements of the statistical distributions of the variations of the sources we care about.
Problem 1d
What is FAKE010 the source that is plotted above?
Hint 1 - "That's no moon!".
Hint 2 - ask Szymon or Tomas...
Solution 1d
Write your solution to 1d here
Problem 1e
To get a better sense of the data, plot the multiband light curves for sources FAKE060 and FAKE073.
Can you write Python code for it?

lc60 # complete
lc60 # complete

lc73 # complete
lc73 # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
One thing you've probably noticed is that there are observations that have very large uncertainties, and our features (which are really just moments of the flux distribution) aren't really all that robust to outliers.
Let's try a more complex description of the shape of the light curves with a Gaussian process regression.
We'll need the sklearn.gaussian_process module
Can you write Python code for it?

import sklearn.gaussian_process as gp

# we'll create a function that just fits the g band lightcurve
def calculate_gpfit(filename):
    """
    Do a quick gaussian process regression of just g-band lightcurve to get a smooth representation
    This should be more robust to outliers
    """
    lc = ANTARESlc(filename)
    t = lc.gtime.values
    y = lc.gFlux.values
    dy = lc.gFluxUnc.values
    t = t.reshape(-1, 1)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    # a Gaussian Process is defined by a correlation function that relates each point to each other
    # I've chosen a simple common kernel - the Matern kernel.
    # It has a nice property in that it is 'stationary' - the covariance between points
    # only depends on their separation

    base_kernel = gp.kernels.Matern(length_scale=10.,nu=2.5, length_scale_bounds=(1.,20.))
    gkernel = 0.5*(np.median(dy)**2.)*base_kernel

    # the second thing you need to define a Gaussian Process is a mean function, which in our case is
    # implictly zero (this is fine for transients - they spend most of eternity at the background!)
    gband = gp.GaussianProcessRegressor(kernel=gkernel, alpha=dy**2.).fit(t, y)

    # now lets use the Gaussian process to predict the lightcurve on an evenly sampled grid
    # this is appropriate for Wavelets, or generally robust feature extraction

    newtime = np.linspace(t.min(), t.max(), 200, endpoint=True)
    newtime = newtime.reshape(-1, 1)
    gnew = gband.predict(newtime, return_cov=False)
    return (t, y , dy, newtime, gnew)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 1c
Confirm the corrections made in 1b by plotting the multiband light curve for the source FAKE010.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 2) Data Preparation
While we could create a database table that includes every single photometric measurement made by LSST, this ~37 trillion row db would be enormous without providing a lot of added value beyond the raw flux measurements [while this table is necessary, alternative tables may provide more useful]. Furthermore, extracting individual light curves from such a database will be slow. Instead, we are going to develop summary statistics for every source which will make it easier to select individual sources and develop classifiers to identify objects of interest.
Below we will redefine the ANTARESlc class to include additional methods so we can (eventually) store summary statistics in a database table. In the interest of time, we limit the summary statistics to a relatively small list all of which have been shown to be useful for classification (see Richards et al. 2011 for further details). The statistics that we include (for now) are:

Std -- the standard deviation of the flux measurements
Amp -- the amplitude of flux deviations
MAD -- the median absolute deviation of the flux measurements
beyond1std -- the fraction of flux measurements beyond 1 standard deviation
the mean $g' - r'$, $r' - i'$, and $i' - z'$ color

Problem 2a
Read through the expandanded definition of the ANTARESlc class. Feel free to improve the doc strings.
food for thought - if a source is observed in different filters but the observations are not simultaneous (or quasi-simultaneous), what is the meaning of a "mean color"?
Solution to food for thought - in this case we simply want to take the mean flux in each filter and create a statistic that is $-2.5 \log \frac{\langle f_X \rangle}{\langle f_{Y} \rangle}$, where ${\langle f_{Y} \rangle}$ is the mean flux in band $Y$, while $\langle f_X \rangle$ is the mean flux in band $X$, which can be $g', r', i', z'$. Note that our use of image-difference flux measurements, which can be negative, means you'll need to add some form a case excpetion if $\langle f_X \rangle$ or...
Can you write Python code for it?

class ANTARESlc():
    '''Light curve object for NOAO formatted data'''

    def __init__(self, filename):
        '''Read in light curve data'''
        DFlc = pd.read_csv(filename, delim_whitespace=True, comment = '#')
        self.DFlc = DFlc
        self.filename = filename
        self.filter_flux()

    def plot_multicolor_lc(self):
        '''Plot the 4 band light curve'''
        fig, ax = plt.subplots()
        g = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'g'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'g'],
                     fmt = 'o', color = '#78A5A3', label = r"$g'$")
        r = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'r'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'r'],
                     fmt = 'o', color = '#CE5A57', label = r"$r'$")
        i = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'i'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'i'],
                     fmt = 'o', color = '#E1B16A', label = r"$i'$")
        z = ax.errorbar(self.DFlc['t'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['flux'].ix[self.DFlc['pb'] == 'z'],
                     self.DFlc['dflux'].ix[self.DFlc['pb'] == 'z'],
                     fmt = 'o', color = '#444C5C', label = r"$z'$")
        ax.legend(fancybox = True)
        ax.set_xlabel(r"$\mathrm{MJD}$")
        ax.set_ylabel(r"$\mathrm{flux}$")

    def filter_flux(self):
        '''Store individual passband fluxes as object attributes'''

        self.gtime = self.DFlc['t'].ix[self.DFlc['pb'] == 'g']
        self.gFlux = self.DFlc['flux'].ix[self.DFlc['pb'] == 'g']
        self.gFluxUnc = self.DFlc['dflux'].ix[self.DFlc['pb'] == 'g']

        self.rtime = self.DFlc['t'].ix[self.DFlc['pb'] == 'r']
        self.rFlux =...


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 2b
Confirm your solution to 2a by measuring the mean colors of source FAKE010. Does your measurement make sense given the plot you made in 1c?
Can you write Python code for it?

lc = # complete

lc. # complete

print("The g'-r', r'-i', and 'i-z' colors are:\
    {:.3f}, {:.3f}, and {:.3f}, respectively.". format( # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 3) Store the sources in a database
Building (and managing) a database from scratch is a challenging task. For (very) small projects one solution to this problem is to use SQLite, which is a self-contained, publicly available SQL engine. One of the primary advantages of SQLite is that no server setup is required, unlike other popular tools such as postgres and MySQL. In fact, SQLite is already integrated with python so everything we want to do (create database, add tables, load data, write queries, etc.) can be done within Python.
Without diving too deep into the details, here are situations where SQLite has advantages and disadvantages according to their own documentation:
Advantages

Situations where expert human support is not needed
For basic data analysis (SQLite is easy to install and manage for new projects)
Education and training

Disadvantages

Client/Server applications (SQLite does not behave well if multiple systems need to access db at the same time)
Very large data sets (SQLite stores entire db in a single disk file, other solutions can store data across multiple files/volumes)
High concurrency (Only 1 writer allowed at a time for SQLite)

From the (limited) lists above, you can see that while SQLite is perfect for our application right now, if you were building an actual ANTARES-like system a more sophisticated database solution would be required.
Problem 3a
Import sqlite3 into the notebook.
Hint - if this doesn't work, you may need to conda install sqlite3 or pip install sqlite3.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 3e
Confirm that the data loaded correctly by counting the number of sources with gAmp > 2.
How many sources have gMinusR = -999?
Hint - you should find 9 and 2, respectively.
Can you write Python code for it?

cur.execute( # complete

nAmp2 = # complete

cur.execute( # complete
nNoColor = # complete

print("There are {:d} sources with gAmp > 2".format( # complete
print("There are {:d} sources with no measured i' - z' color".format( # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 4b
Train a SVM model (SVC in scikit-learn) using a radial basis function (RBF) kernel with penalty parameter, $C = 1$, and kernel coefficient, $\gamma = 0.1$.
Evaluate the accuracy of the model via $k = 5$ fold cross validation.
Hint - you may find the model_selection module helpful.
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

cv_scores = # complete

print("The SVM model produces a CV accuracy of {:.4f}".format( # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 4d
Does the model you selected perform better than the SVM model? Perform a $k = 5$ fold cross validation to determine which model provides superior accuracy.
Can you write Python code for it?

cv_scores = cross_val_score( # complete

print("The RF model produces a CV accuracy of {:.4f}".format( # complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 4e
Which model are you going to use in your miniANTARES? Justify your answer.
Write solution to 4e here
Problem 5) Class Predictions for New Sources
Now that we have developed a basic infrastructure for dealing with streaming data, we may reap the rewards of our efforts. We will use our ANTARES-like software to classify newly observed sources.
Problem 5a
Load the light curves and features for the new observations (found in full_testset_for_LSST_DSP) into new tables, called testPhot and testFeats, respectively, in the database.
Can you write Python code for it?

cur.execute("""drop table if exists testPhot""") # drop the table if is already exists
cur.execute( # complete
cur.execute("""drop table if exists testFeats""") # drop the table if it already exists
cur.execute( # complete

new_obs_filenames = glob.glob("test_set_for_LSST_DSFP/FAKE*.dat")

result = lview.map(load_data, new_obs_filenames)
all_data = result.get()

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 5b
Calculate features for the new observations and insert those features into a database table.
Can you write Python code for it?

result = lview.map( # complete
all_feats = result.get()

for feats in all_feats:
    cur.execute("""insert into testFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Problem 5c
Train the model that you adopted in 4e on the training set, and produce predictions for the newly observed sources.
What is the class distribution for the newly detected sources?
Hint - the training set was constructed to have a nearly uniform class distribution, that may not be the case for the actual observed distribution of sources.
Can you write Python code for it?

svm_clf = SVC(C=1.0, gamma = 0.1, kernel = 'rbf').fit( # complete

cur.execute( # complete
X_new = # complete

y_preds = # complete

print("""There are {:d}, {:d}, and {:d} sources
         in classes 1, 2, 3, respectively""".format(*list(np.bincount(y_preds)))) # be careful using bincount


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
Putting what you've learnt together:
Building an mini-ANTARES (ish) Pipeline for Data Management and Discovery
========
Version 0.2

By AA Miller 2017 Apr 10
Edited G. Narayan 2017 Apr 26
As we just saw LSST will produce an unprecedented volume of time-domain information for the astronomical sky. $>37$ trillion individual photometric measurements will be recorded. While the vast, vast majority of these measurements will simply confirm the status quo, some will represent rarities that have never been seen before (e.g., LSST may be the first telescope to discover the electromagnetic counterpart to a LIGO graviational wave event), which the community will need to know about in ~real time.
Storing, filtering, and serving this data is going to be a huge <del>nightmare</del> challenge. ANTARES is one proposed solution to this challenge. In this exercise you will build a miniature version of ANTARES, which will require the application of several of the lessons from earlier this week.
Many of the difficult, and essential, steps necessary for ANTARES will be skipped here as they are too time consuming or beyond the scope of what we have previously covered. We will point out these challenges are we come across them.

0) There is a README.MD file along with this notebook.
Before running this notebook you will need to follow the instructions in the README to setup your notebook and download the data.

If everything worked OK up to this point, congratulations! You now have access to a (mini) parallel computing environment of your own.
Can you write Python code for it?

# first we need to construct a client that will interface with our cluster
from ipyparallel import Client, require
worker = Client()

# once we create a client, we can decide how to allocate tasks across the cluster
# we've got however many 'engines' you started in the cluster
# lets just use all of them

lview = worker[:]

# now if you want to import packages, you can import them across all the 'engines'
with lview.sync_imports():
    import numpy as np
    import scipy.stats as spstat
    import pandas as pd
    import os

# there's not much point in creating plots on the engines - we want to actually see them presumably
import matplotlib.pyplot as plt
%matplotlib notebook

# If everything has worked so far, you should see a list of worker IDs, and a corresponding list of process IDs.
# You can verify it with a `ps`
ar = lview.apply_async(os.getpid)
print("Engine ID:PID mapping: ", ar.get_dict())


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
The SVM model does a decent job of classifying the data. However - we are going to have 10 million alerts every night. Therefore, we need something that runs quickly. For most ML models the training step is slow, while predictions (relatively) are fast.
Problem 4c
Pick any other classification model from scikit-learn, and "profile" the time it takes to train that model vs. the time it takes to train an SVM model.
Is the model that you have selected faster than SVM?
Hint - you should import the model outside your timing loop as we only care about the training step in this case.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
rf_clf = # complete
svm_clf = # complete

%%timeit
# timing solution for RF model
# complete

%%timeit
# timing solution for SVM model
# complete


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
There are two things to highlight above: (1) we do not specify an id for the data as this is automatically generated, and (2) the data insertion happens via a tuple. In this case, we are taking advantage of the fact that a Python tuple is can be concatenated:
(objId,) + tuple(lc10.DFlc.ix[0]))

While the above example demonstrates the insertion of a single row to the database, it is far more efficient to bulk load the data. To do so we will delete, i.e. DROP, the rawPhot table and use some pandas manipulation to load the contents of an entire file at once via executemany.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")

# next 3 lines are already in name space; repeated for clarity
filename = "training_set_for_LSST_DSFP/FAKE001.dat"
lc = ANTARESlc(filename)
objId = int(filename.split('FAKE')[1].split(".dat")[0])

data = [(objId,) + tuple(x) for x in lc.DFlc.values] # array of tuples

cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
This procedure should look familiar to above, with one exception: the addition of the foreign key in the lcFeats table. The inclusion of the foreign key ensures a connected relationship between rawPhot and lcFeats. In brief, a row cannot be inserted into lcFeats unless a corresponding row, i.e. objId, exists in rawPhot. Additionally, rows in rawPhot cannot be deleted if there are dependent rows in lcFeats.
Problem 3d
Calculate features for every source in rawPhot and insert those features into the lcFeats table. You'll first need a function to calculate the features.
Can you write Python code for it?

# just like last time, we'll define a function to calculate the features for one lightcurve
# execute this

@require(ANTARESlc, 'pandas as pd', 'numpy as np', 'scipy.stats as spstat')
def calculate_features(filename):
    lc = ANTARESlc(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    lc.weighted_mean_flux()
    lc.normalized_flux_std()
    lc.normalized_amplitude()
    lc.normalized_MAD()
    lc.normalized_beyond_1std()
    lc.skew()
    lc.mean_colors()
    feats = (objId, lc.gStd, lc.rStd, lc.iStd, lc.zStd,
            lc.gAmp,  lc.rAmp,  lc.iAmp,  lc.zAmp,
            lc.gMAD, lc.rMAD, lc.iMAD, lc.zMAD,
            lc.gBeyond, lc.rBeyond, lc.iBeyond, lc.zBeyond,
            lc.gSkew, lc.rSkew, lc.iSkew, lc.zSkew,
            lc.gMinusR, lc.rMinusI, lc.iMinusZ)
    return feats


I am working on the file "Sessions/Session03/Day4/ANTARES/miniAntares_parallel.ipynb".
The first task is:
We now have a database connection object, conn. To interact with the database (create tables, load data, write queries) we need a cursor object.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/Parallel.ipynb".
The first task is:
It can be hard to guess which code is going to operate faster just by looking at it because the interactions between software and computers can be extremely complex.  The best way to optimize code is through using profilers to identify bottlenecks in your code and then attempt to address these problems through optimization.  Let's give it a whirl.
Problem 1) Searching lists by bisecting them
Problem 1a
Let's say you have a sorted list of random elements, and you want to see where a new number fits into them to remain ordered.  The simplest way to search through an ordered list to find the location for your desired element is to just step through it one-by-one, but this is not algorithmically ideal.  Please write out such a function, and comment on the complexity of this algorithm (seen below) in big O notation.  Then use timeit to test how fast it is.
Can you write Python code for it?

# Create sorted random array and random element of that array; this just sets up the problem.
def rand_arr(n_elements=100000):

    random_list = [random.random() for i in np.arange(n_elements)]
    random_list.sort()
    return random_list

random_list = rand_arr()
new_number = random.random()

%%timeit
# complete


I am working on the file "Sessions/Session03/Day4/Parallel.ipynb".
The first task is:
Problem 1b
We can cut corners all we want to optimize that code, but at the end of the day, it's still going to not be very good.  So let's try to make a better algorithm for searching the ordered array to find where to put our new number.  One such way is by continually bisecting the array in what is called a binary search, and checking if the special_number is greater than, or less than the value at the bisection, then shifting to the remaining values to continue the search.  Please code this up, and comment on its complexity in big O notation, and use timeit to get its speed.  Did you get the result you expected to get?  If not, why do you think that happened?  You may get different results if you change the input parameters.  Think about it.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/Parallel.ipynb".
The first task is:
Problem 1d
But it gets better!  We can actually use multiprocessing.Pool.map() to do the same thing, but this way it uses every processor available to it!  Load this up and see what you get, or if you run into any problems.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/Profiling.ipynb".
The first task is:
It can be hard to guess which code is going to operate faster just by looking at it because the interactions between software and computers can be extremely complex.  The best way to optimize code is through using profilers to identify bottlenecks in your code and then attempt to address these problems through optimization.  Let's give it a whirl.
Problem 1) Using timeit
We will begin our experience with profilers by using the time and timeit commands.  time can be run on any size of program, but it returns coarse level time information on how long something took to run overall.
There are a lot of small optimizations that can add up to a lot of time in real-world software.  Let's look at a few of the non-obvious ones.
Problem 1a
What is the best way to join a bunch of strings into a larger string?  There are several ways of doing this, but some are clearly superior to others.  Let's use timeit to test things out.
Below, in each of the cells after the string_list is defined, put a new code snippet using the following three methods for building a string:
--Use the builtin + operator to add strings together in an iterative way
--Use the join method, as in "".join(list).
--Iteratively add the strings from the list together using "%s %s" string composition.
Guess which method you think will be fastest?  Now test it out and see if you're right!
Can you write Python code for it?

string_list = ['the ', 'quick ', 'brown ', 'fox ', 'jumped ', 'over ', 'the ', 'lazy ', 'dog']

%%timeit
output = ""
# complete

%%timeit
# complete

%%timeit
output = ""
# complete


I am working on the file "Sessions/Session03/Day4/Profiling.ipynb".
The first task is:
Problem 2b
OK, now try running the cProfile module with it in order to produce some profiling statistics.  You can do this by running:
python -m cProfile -o sort.prof sort.py
This will produce an output profile file called sort.prof.  You can do a variety of things with sort.prof, but you'll need a few programs to do this.  First, install pyprof2html with: pip install pyprof2html.  Then, try:
pyprof2html sort.prof
This will produce a html directory, and you can just open up the enclosed index.html file to bring it to your browser.  You can see function by function, what is taking the most time!  You can click on column headers to change which sorting occurs.
Problem 2c
But there are graphical ways of representing these data effectively.  Download snakeviz, another means of viewing your profile data.  You can do this with pip install snakeviz.  And then open up the same file with snakeviz:
snakeviz sort.prof
This should bring up another graphical interface for analyzing the profile data.  Switch to icicle mode, and explore the information a bit.  Try to figure out where the "hot" sections of the code are.  Namely, what is the most expensive function that is running in terms of time?
Problem 2d
OK, so if that's the most expensive, we better speed it up.  We can investigate line-by-line how slow/fast things are, but we need another package for that called line_profiler.  Go ahead and install this with pip install line_profiler.
Go back to the source code file, and add a @profile line directly above the slow function.  line_profiler automatically installed a file called kernprof to your $PYTHONPATH, which is used with the following format at the command line:
kernprof.py -v -l your_script your_script_args
Start up kernprof and we'll look at the slow function in our sort program!  See if you can find where the slowdown is, based on the amount of time spent on a particular line.  Can you fix this line to not be so inefficient?
Hint: Remember the ways we discussed for optimizing code: In-place operations, string...
Can you write Python code for it?

from matplotlib import pyplot as plt
import numpy as np
import random

class Galaxy():
    """
    Galaxy class for simply representing a galaxy.
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, age=0):
        self.total_mass = total_mass
        self.cold_gas_mass = cold_gas_mass
        self.stellar_mass = stellar_mass
        self.age = age
        self.SFR = 0
        self.color = 'red'

    def __repr__(self):
        return "Galaxy (m_total = %.1g; m_cold = %.1g; m_stars = %.1g; age = %.1g; SFR = %0.2f)" % \
                (self.total_mass, self.cold_gas_mass, self.stellar_mass, self.age, self.SFR)

class EvolvingGalaxy(Galaxy):
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """
    def current_state(self):
        """
        Return a tuple of the galaxy's total_mass, cold_gas_mass, stellar_mass, age, and SFR
        """
        return (self.total_mass, self.cold_gas_mass, self.stellar_mass, self.age, self.SFR)

    def calculate_star_formation_rate(self):
        """
        Calculate the star formation rate by taking a random number between 0 and 1
        normalized by the galaxy total mass / 1e12;

        Also updates the galaxy's color to blue if SFR > 0.01, otherwise color = red
        """
        self.SFR = random.random() * (self.total_mass / 1e12)
        if self.SFR > 0.01:
            self.color = 'blue'
        else:
            self.color = 'red'

    def accrete_gas_from_IGM(self, time):
        """
        Allow the galaxy to accrete cold gas from the IGM at a variable rate normalized to
        the galaxy's mass
        """
        cold_gas_accreted = random.random() * 0.1 * time * (self.total_mass / 1e12)
        self.cold_gas_mass += cold_gas_accreted
        self.total_mass += cold_gas_accreted

    def form_stars(self, time):
        """
        Form stars according to the current star formation rate and time available
        If unable cold gas, then shut off star...


I am working on the file "Sessions/Session03/Day4/Profiling.ipynb".
The first task is:
Whoa!  We were able to see a >100x efficiency increase by just switching these methods slightly!  Numpy arrays are awesome, but I'm sort of surprised that the lambda function won compared to native numpy.
Problem 2) Deeper profiling with cProfile and line_profiler
Problem 2a
OK, so what about larger program?  Here is a sorting algorithm that I wrote, which may possess some inefficiencies. But it is hard to know which bugs are causing the biggest problems (some actually aren't that big of a deal in the long term). Let's see if we can speed it up.  First, take this code and copy it into a file called sort.py.  Read through the code to make sure you understand it.  Then, run it with the time command, and write down the total time it took to run.
Can you write Python code for it?

# Sort version1

import random

def create_random_list(n_elements):
    """
    Create a list made up of random elements in random order
    """
    random_list = []
    for i in range(n_elements):
        random_list.append(random.random())
    return random_list

def find_minimum_index(random_list):
    """
    Find the index of the minimum value in the list
    """
    # current minimum
    min_value = 1
    i = 0

    # Find minimum in list
    for element in random_list:
        if element < min_value:
            min_value = element

    # Find value that matches minimum
    for element in random_list:
        if element == min_value:
            return i
        i += 1

def sort_list(random_list):
    """
    Sort a list into ascending order
    """
    output_list = []
    for _ in range(len(random_list)):
        i = find_minimum_index(random_list)
        minimum = random_list[i]
        output_list.append(minimum)
        del random_list[i]
    return output_list

if __name__ == '__main__':
    l = create_random_list(10000)
    o = sort_list(l)


I am working on the file "Sessions/Session03/Day4/Profiling_solns.ipynb".
The first task is:
It can be hard to guess which code is going to operate faster just by looking at it because the interactions between software and computers can be extremely complex.  The best way to optimize code is through using profilers to identify bottlenecks in your code and then attempt to address these problems through optimization.  Let's give it a whirl.
Problem 1) Using timeit
We will begin our experience with profilers by using the time and timeit commands.  time can be run on any size of program, but it returns coarse level time information on how long something took to run overall.
There are a lot of small optimizations that can add up to a lot of time in real-world software.  Let's look at a few of the non-obvious ones.
Problem 1a
What is the best way to join a bunch of strings into a larger string?  There are several ways of doing this, but some are clearly superior to others.  Let's use timeit to test things out.
Below, in each of the cells after the string_list is defined, put a new code snippet using the following three methods for building a string:
--Use the builtin + operator to add strings together in an iterative way
--Use the join method, as in "".join(list).
--Iteratively add the strings from the list together using "%s %s" string composition.
Guess which method you think will be fastest?  Now test it out and see if you're right!
Can you write Python code for it?

string_list = ['the ', 'quick ', 'brown ', 'fox ', 'jumped ', 'over ', 'the ', 'lazy ', 'dog']

%%timeit
output = ""
for string in string_list:
    output+=string

%%timeit
"".join(string_list)

%%timeit
output = ""
for word in string_list:
    output = "%s %s" % (output, word)


I am working on the file "Sessions/Session03/Day4/Profiling_solns.ipynb".
The first task is:
Problem 2b
OK, now try running the cProfile module with it in order to produce some profiling statistics.  You can do this by running:
python -m cProfile -o sort.prof sort.py
This will produce an output profile file called sort.prof.  You can do a variety of things with sort.prof, but you'll need a few programs to do this.  First, install pyprof2html with: pip install pyprof2html.  Then, try:
pyprof2html sort.prof
This will produce a html directory, and you can just open up the enclosed index.html file to bring it to your browser.  You can see function by function, what is taking the most time!  You can click on column headers to change which sorting occurs.
Problem 2c
But there are graphical ways of representing these data effectively.  Download snakeviz, another means of viewing your profile data.  You can do this with pip install snakeviz.  And then open up the same file with snakeviz:
snakeviz sort.prof
This should bring up another graphical interface for analyzing the profile data.  Switch to icicle mode, and explore the information a bit.  Try to figure out where the "hot" sections of the code are.  Namely, what is the most expensive function that is running in terms of time?
Problem 2d
OK, so if that's the most expensive, we better speed it up.  We can investigate line-by-line how slow/fast things are, but we need another package for that called line_profiler.  Go ahead and install this with pip install line_profiler.
Go back to the source code file, and add a @profile line directly above the slow function.  line_profiler automatically installed a file called kernprof to your $PYTHONPATH, which is used with the following format at the command line:
kernprof.py -v -l your_script your_script_args
Start up kernprof and we'll look at the slow function in our sort program!  See if you can find where the slowdown is, based on the amount of time spent on a particular line.  Can you fix this line to not be so inefficient?
Hint: Remember the ways we discussed for optimizing code: In-place operations, string...
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day4/Profiling_solns.ipynb".
The first task is:
Whoa!  We were able to see a >100x efficiency increase by just switching these methods slightly!  Numpy arrays are awesome, but I'm sort of surprised that the lambda function won compared to native numpy.
Problem 2) Deeper profiling with cProfile and line_profiler
Problem 2a
OK, so what about larger program?  Here is a sorting algorithm that I wrote, which may possess some inefficiencies. But it is hard to know which bugs are causing the biggest problems (some actually aren't that big of a deal in the long term). Let's see if we can speed it up.  First, take this code and copy it into a file called sort.py.  Read through the code to make sure you understand it.  Then, run it with the time command, and write down the total time it took to run.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 3e
Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?

normalized_cm = cm.astype('float')/cm.sum(axis = 1)[:,np.newaxis]

normalized_cm


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values. [In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.] In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, but the documentation for essentially everything within scikit-learn is excellent, so read the docs.
To further develop our intuition, we will now explore the Iris dataset a little further.
Problem 1a What is the pythonic type of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
I have used my own domain knowledge to specifically choose features that may be useful when clustering galaxies. If you know a bit about SDSS and can think of other features that may be useful feel free to add them to the query.
One nice feature of astropy tables is that they can readily be turned into pandas DataFrames, which can in turn easily be turned into a sklearn X array with NumPy. For example:
X = np.array(SDSSgals.to_pandas())

And you are ready to go.
Challenge Problem Using the SDSS dataset above, identify interesting clusters within the data [this is intentionally very open ended, if you uncover anything especially exciting you'll have a chance to share it with the group]. Feel free to use the algorithms discussed above, or any other packages available via sklearn. Can you make sense of the clusters in the context of galaxy evolution?
Hint - don't fret if you know nothing about galaxy evolution (neither do I!). Just take a critical look at the clusters that are identified
Can you write Python code for it?

Xgal = np.array(SDSSgals.to_pandas())

galScaler = StandardScaler().fit(Xgal)

dbs = DBSCAN(eps = .25, min_samples=55)

dbs.fit(galScaler.transform(Xgal))

cluster_members = dbs.labels_ != -1
outliers = dbs.labels_ == -1

plt.figure(figsize = (10,8))
plt.scatter(Xgal[:,0][outliers], Xgal[:,3][outliers],
            c = "k",
            s = 4, alpha = 0.1)
plt.scatter(Xgal[:,0][cluster_members], Xgal[:,3][cluster_members],
            c = dbs.labels_[cluster_members],
            alpha = 0.4, edgecolor = "None", cmap = "viridis")

plt.xlim(-1,5)
plt.ylim(-0,3.5)


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
I was unable to obtain 3 clusters with DBSCAN. While these results are, on the surface, worse than what we got with $k$-means, my suspicion is that the 4 features do not adequately separate the 3 classes. [See - a nayseyer can always make that argument.] This is not a problem for DBSCAN as an algorithm, but rather, evidence that no single algorithm works well in all cases.
Challenge Problem) Cluster SDSS Galaxy Data
The following query will select 10k likely galaxies from the SDSS database and return the results of that query into an astropy.Table object. (For now, if you are not familiar with the SDSS DB schema, don't worry about this query, just know that it returns a bunch of photometric features.)
Can you write Python code for it?

from astroquery.sdss import SDSS  # enables direct queries to the SDSS database

GALquery = """SELECT TOP 10000
             p.dered_u - p.dered_g as ug, p.dered_g - p.dered_r as gr,
             p.dered_g - p.dered_i as gi, p.dered_g - p.dered_z as gz,
             p.petroRad_i, p.petroR50_i, p.deVAB_i
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND p.type = 3
               """
SDSSgals = SDSS.query_sql(GALquery)
SDSSgals


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Note - I was unable to get the galaxies to clusster using DBSCAN.
Problem 3) Supervised Machine Learning
Supervised machine learning, on the other hand, aims to predict a target class or produce a regression result based on the location of labelled sources (i.e. the training set) in the multidimensional feature space. The "supervised" comes from the fact that we are specifying the allowed outputs from the model. As there are labels available for the training set, it is possible to estimate the accuracy of the model (though there are generally important caveats about generalization, which we will explore in further detail later).
We will begin with a simple, but nevertheless, elegant algorithm for classification and regression: $k$-nearest-neighbors ($k$NN). In brief, the classification or regression output is determined by examining the $k$ nearest neighbors in the training set, where $k$ is a user defined number. Typically, though not always, distances between sources are Euclidean, and the final classification is assigned to whichever class has a plurality within the $k$ nearest neighbors (in the case of regression, the average of the $k$ neighbors is the output from the model). We will experiment with the steps necessary to optimize $k$, and other tuning parameters, in the detailed break-out problem.
In scikit-learn the KNeighborsClassifer algorithm is implemented as part of the sklearn.neighbors module.
Problem 3a
Fit two different $k$NN models to the iris data, one with 3 neighbors and one with 10 neighbors. Plot the resulting class predictions in the sepal length-sepal width plane (same plot as above). How do the results compare to the true classifications? Is there any reason to be suspect of this procedure?
Hint - after you have constructed the model, it is possible to obtain model predictions using the .predict() method, which requires a feature array, including the same features and order as the training set, as input.
Hint that isn't essential, but is worth thinking about - should the features be...
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

KNNclf = KNeighborsClassifier(n_neighbors = 3).fit(iris.data, iris.target)
preds = KNNclf.predict(iris.data)
plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1],
            c = preds, cmap = "viridis", s = 30, edgecolor = "None")

KNNclf = KNeighborsClassifier(n_neighbors = 10).fit(iris.data, iris.target)
preds = KNNclf.predict(iris.data)
plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1],
            c = preds, cmap = "viridis", s = 30, edgecolor = "None")


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Petal length has the largest range and standard deviation, thus, it will have the most "weight" when determining the $k$ clusters.
The truth is that the iris data set is fairly small and straightfoward. Nevertheless, we will now examine the clustering results after re-scaling the features. [Some algorithms, cough Support Vector Machines cough, are notoriously sensitive to the feature scaling, so it is important to know about this step.] Imagine you are classifying stellar light curves: the data set will include contact binaries with periods of $\sim 0.1 \; \mathrm{d}$ and Mira variables with periods of $\gg 100 \; \mathrm{d}$. Without re-scaling, this feature that covers 4 orders of magnitude may dominate all others in the final model projections.
The two most common forms of re-scaling are to rescale to a guassian with mean $= 0$ and variance $= 1$, or to rescale the min and max of the feature to $[0, 1]$. The best normalization is problem dependent. The sklearn.preprocessing module makes it easy to re-scale the feature set. It is essential that the same scaling used for the training set be used for all other data run through the model. The testing, validation, and field observations cannot be re-scaled independently. This would result in meaningless final classifications/predictions.
Problem 2d Re-scale the features to normal distributions, and perform $k$-means clustering on the iris data. How do the results compare to those obtained earlier?
Hint - you may find 'StandardScaler()' within the sklearn.preprocessing module useful.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit(iris.data)

Kcluster = KMeans(n_clusters = 3)
Kcluster.fit(scaler.transform(iris.data))

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Problem 1d What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
Problem 2) Unsupervised Machine Learning
Unsupervised machine learning, sometimes referred to as clustering or data mining, aims to group or classify sources in the multidimensional feature space. The "unsupervised" comes from the fact that there are no target labels provided to the algorithm, so the machine is asked to cluster the data "on its own." The lack of labels means there is no (simple) method for validating the accuracy of the solution provided by the machine (though sometimes simple examination can show the results are terrible).
For this reason [note - this is my (AAM) opinion and there many be many others who disagree], unsupervised methods are not particularly useful for astronomy. Supposing one did find some useful clustering structure, an adversarial researcher could always claim that the current feature space does not accurately capture the physics of the system and as such the clustering result is not interesting or, worse, erroneous. The one potentially powerful exception to this broad statement is outlier detection, which can be a branch of both unsupervised and supervised learning. Finding weirdo objects is an astronomical pastime, and there are unsupervised methods that may help in that regard in the LSST era.
To begin today we will examine one of the most famous, and simple, clustering algorithms: $k$-means. $k$-means clustering looks to identify $k$ convex clusters, where $k$ is a user defined number. And here-in lies the rub: if we truly knew the number of clusters in advance, we likely wouldn't need to perform any clustering in the first place. This is the major downside to $k$-means. Operationally, pseudocode for the algorithm can be summarized as the following:
initiate search by identifying k points (i.e. the cluster centers)
loop
    assign each point in the data set to the closest cluster center
    calculate new cluster centers based on mean position of all points within cluster
    if diff(new center - old center) < threshold:
        stop (i.e. clusters are...
Can you write Python code for it?

from sklearn.cluster import KMeans

Kcluster = KMeans(n_clusters = 2)
Kcluster.fit(iris.data)

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')

Kcluster = KMeans(n_clusters = 3)
Kcluster.fit(iris.data)

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
These results are almost identical to the training classifications. However, we have cheated! In this case we are evaluating the accuracy of the model (98% in this case) using the same data that defines the model. Thus, what we have really evaluated here is the training error. The relevant parameter, however, is the generalization error: how accurate are the model predictions on new data?
Without going into too much detail, we will test this using cross validation (CV). In brief, CV provides predictions on the training set using a subset of the data to generate a model that predicts the class of the remaining sources. Using cross_val_predict, we can get a better sense of the model accuracy. Predictions from cross_val_predict are produced in the following manner:
from sklearn.cross_validation import cross_val_predict
CVpreds = cross_val_predict(sklearn.model(), X, y)

where sklearn.model() is the desired model, X is the feature array, and y is the label array.
Problem 3b
Produce cross-validation predictions for the iris dataset and a $k$NN with 5 neighbors. Plot the resulting classifications, as above, and estimate the accuracy of the model as applied to new data. How does this accuracy compare to a $k$NN with 50 neighbors?
Can you write Python code for it?

from sklearn.cross_validation import cross_val_predict

CVpreds = cross_val_predict(KNeighborsClassifier(n_neighbors=5), iris.data, iris.target)
plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1],
            c = preds, cmap = "viridis", s = 30, edgecolor = "None")
print("The accuracy of the kNN = 5 model is ~{:.4}".format( sum(CVpreds == iris.target)/len(CVpreds) ))

CVpreds50 = cross_val_predict(KNeighborsClassifier(n_neighbors=50), iris.data, iris.target)

print("The accuracy of the kNN = 50 model is ~{:.4}".format( sum(CVpreds50 == iris.target)/len(CVpreds50) ))


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
We just found that the classifier does a much better job classifying setosa and versicolor than it does for virginica. The main reason for this is some viginica flowers lie far outside the main virginica locus, and within predominantly versicolor "neighborhoods". In addition to knowing the accuracy for the individual classes, it is also useful to know class predictions for the misclassified sources, or in other words where there is "confusion" for the classifier. The best way to summarize this information is with a confusion matrix. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion matrix. This can be accomplished with the following:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_prep)

Problem 3d
Calculate the confusion matrix for the iris training set and the $k$NN = 50 model.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(iris.target, CVpreds50)
print(cm)


I am working on the file "Sessions/Session04/Day0/TooBriefMLSolutions.ipynb".
The first task is:
You likely haven't encountered a scikit-learn Bunch before. It's functionality is essentially the same as a dictionary.
Problem 1b What are the keys of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 3e
Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values. [In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.] In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, but the documentation for essentially everything within scikit-learn is excellent, so read the docs.
To further develop our intuition, we will now explore the Iris dataset a little further.
Problem 1a What is the pythonic type of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
I was unable to obtain 3 clusters with DBSCAN. While these results are, on the surface, worse than what we got with $k$-means, my suspicion is that the 4 features do not adequately separate the 3 classes. [See - a nayseyer can always make that argument.] This is not a problem for DBSCAN as an algorithm, but rather, evidence that no single algorithm works well in all cases.
Challenge Problem) Cluster SDSS Galaxy Data
The following query will select 10k likely galaxies from the SDSS database and return the results of that query into an astropy.Table object. (For now, if you are not familiar with the SDSS DB schema, don't worry about this query, just know that it returns a bunch of photometric features.)
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Note - I was unable to get the galaxies to clusster using DBSCAN.
Problem 3) Supervised Machine Learning
Supervised machine learning, on the other hand, aims to predict a target class or produce a regression result based on the location of labelled sources (i.e. the training set) in the multidimensional feature space. The "supervised" comes from the fact that we are specifying the allowed outputs from the model. As there are labels available for the training set, it is possible to estimate the accuracy of the model (though there are generally important caveats about generalization, which we will explore in further detail later).
We will begin with a simple, but nevertheless, elegant algorithm for classification and regression: $k$-nearest-neighbors ($k$NN). In brief, the classification or regression output is determined by examining the $k$ nearest neighbors in the training set, where $k$ is a user defined number. Typically, though not always, distances between sources are Euclidean, and the final classification is assigned to whichever class has a plurality within the $k$ nearest neighbors (in the case of regression, the average of the $k$ neighbors is the output from the model). We will experiment with the steps necessary to optimize $k$, and other tuning parameters, in the detailed break-out problem.
In scikit-learn the KNeighborsClassifer algorithm is implemented as part of the sklearn.neighbors module.
Problem 3a
Fit two different $k$NN models to the iris data, one with 3 neighbors and one with 10 neighbors. Plot the resulting class predictions in the sepal length-sepal width plane (same plot as above). How do the results compare to the true classifications? Is there any reason to be suspect of this procedure?
Hint - after you have constructed the model, it is possible to obtain model predictions using the .predict() method, which requires a feature array, including the same features and order as the training set, as input.
Hint that isn't essential, but is worth thinking about - should the features be...
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

KNNclf = KNeighborsClassifier( # complete
preds = KNNclf.predict( # complete
plt.figure()
plt.scatter( # complete

# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Petal length has the largest range and standard deviation, thus, it will have the most "weight" when determining the $k$ clusters.
The truth is that the iris data set is fairly small and straightfoward. Nevertheless, we will now examine the clustering results after re-scaling the features. [Some algorithms, cough Support Vector Machines cough, are notoriously sensitive to the feature scaling, so it is important to know about this step.] Imagine you are classifying stellar light curves: the data set will include contact binaries with periods of $\sim 0.1 \; \mathrm{d}$ and Mira variables with periods of $\gg 100 \; \mathrm{d}$. Without re-scaling, this feature that covers 4 orders of magnitude may dominate all others in the final model projections.
The two most common forms of re-scaling are to rescale to a guassian with mean $= 0$ and variance $= 1$, or to rescale the min and max of the feature to $[0, 1]$. The best normalization is problem dependent. The sklearn.preprocessing module makes it easy to re-scale the feature set. It is essential that the same scaling used for the training set be used for all other data run through the model. The testing, validation, and field observations cannot be re-scaled independently. This would result in meaningless final classifications/predictions.
Problem 2d Re-scale the features to normal distributions, and perform $k$-means clustering on the iris data. How do the results compare to those obtained earlier?
Hint - you may find 'StandardScaler()' within the sklearn.preprocessing module useful.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit( # complete

# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Problem 1d What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
Problem 2) Unsupervised Machine Learning
Unsupervised machine learning, sometimes referred to as clustering or data mining, aims to group or classify sources in the multidimensional feature space. The "unsupervised" comes from the fact that there are no target labels provided to the algorithm, so the machine is asked to cluster the data "on its own." The lack of labels means there is no (simple) method for validating the accuracy of the solution provided by the machine (though sometimes simple examination can show the results are terrible).
For this reason [note - this is my (AAM) opinion and there many be many others who disagree], unsupervised methods are not particularly useful for astronomy. Supposing one did find some useful clustering structure, an adversarial researcher could always claim that the current feature space does not accurately capture the physics of the system and as such the clustering result is not interesting or, worse, erroneous. The one potentially powerful exception to this broad statement is outlier detection, which can be a branch of both unsupervised and supervised learning. Finding weirdo objects is an astronomical pastime, and there are unsupervised methods that may help in that regard in the LSST era.
To begin today we will examine one of the most famous, and simple, clustering algorithms: $k$-means. $k$-means clustering looks to identify $k$ convex clusters, where $k$ is a user defined number. And here-in lies the rub: if we truly knew the number of clusters in advance, we likely wouldn't need to perform any clustering in the first place. This is the major downside to $k$-means. Operationally, pseudocode for the algorithm can be summarized as the following:
initiate search by identifying k points (i.e. the cluster centers)
loop
    assign each point in the data set to the closest cluster center
    calculate new cluster centers based on mean position of all points within cluster
    if diff(new center - old center) < threshold:
        stop (i.e. clusters are...
Can you write Python code for it?

from sklearn.cluster import KMeans

Kcluster = KMeans( # complete
Kcluster.fit( # complete

plt.figure()
plt.scatter( # complete

# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
The normalization makes it easier to compare the classes, since each class has a different number of sources. Now we can procede with a visual representation of the confusion matrix. This is best done using imshow() within pyplot. You will also need to plot a colorbar, and labeling the axes will also be helpful.
Problem 3f
Plot the confusion matrix. Be sure to label each of the axeses.
Hint - you might find the sklearn confusion matrix tutorial helpful for making a nice plot.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
These results are almost identical to the training classifications. However, we have cheated! In this case we are evaluating the accuracy of the model (98% in this case) using the same data that defines the model. Thus, what we have really evaluated here is the training error. The relevant parameter, however, is the generalization error: how accurate are the model predictions on new data?
Without going into too much detail, we will test this using cross validation (CV). In brief, CV provides predictions on the training set using a subset of the data to generate a model that predicts the class of the remaining sources. Using cross_val_predict, we can get a better sense of the model accuracy. Predictions from cross_val_predict are produced in the following manner:
from sklearn.cross_validation import cross_val_predict
CVpreds = cross_val_predict(sklearn.model(), X, y)

where sklearn.model() is the desired model, X is the feature array, and y is the label array.
Problem 3b
Produce cross-validation predictions for the iris dataset and a $k$NN with 5 neighbors. Plot the resulting classifications, as above, and estimate the accuracy of the model as applied to new data. How does this accuracy compare to a $k$NN with 50 neighbors?
Can you write Python code for it?

from sklearn.cross_validation import cross_val_predict

CVpreds = cross_val_predict( # complete

plt.scatter( # complete
print("The accuracy of the kNN = 5 model is ~{:.4}".format( # complete

# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
We just found that the classifier does a much better job classifying setosa and versicolor than it does for virginica. The main reason for this is some viginica flowers lie far outside the main virginica locus, and within predominantly versicolor "neighborhoods". In addition to knowing the accuracy for the individual classes, it is also useful to know class predictions for the misclassified sources, or in other words where there is "confusion" for the classifier. The best way to summarize this information is with a confusion matrix. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion matrix. This can be accomplished with the following:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_prep)

Problem 3d
Calculate the confusion matrix for the iris training set and the $k$NN = 50 model.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
cm = confusion_matrix( # complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
With 3 clusters the algorithm does a good job of separating the three classes. However, without the a priori knowledge that there are 3 different types of iris, the 2 cluster solution would appear to be superior.
Problem 2b How do the results change if the 3 cluster model is called with n_init = 1 and init = 'random' options? Use rs for the random state [this allows me to cheat in service of making a point].
*Note - the respective defaults for these two parameters are 10 and k-means++, respectively. Read the docs to see why these choices are, likely, better than those in 2b.
Can you write Python code for it?

rs = 14
Kcluster1 = KMeans( # complete
# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefMachLearn.ipynb".
The first task is:
You likely haven't encountered a scikit-learn Bunch before. It's functionality is essentially the same as a dictionary.
Problem 1b What are the keys of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Already with this simple plot we see a problem - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. Intuitively this seems incorrect, so lets examine how the histogram changes if we change the number of bins or the bin centers.
Problem 1b
Using the same data make 2 new histograms: (i) one with 5 bins (bins = 5), and (ii) one with the bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
But! Histograms and scatter plots can be problematic as we have discussed many times before.
Problem 3d
Re-create the plot above but set kind='kde' to produce density estimates of the distributions.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?

g = sns.PairGrid(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
                 hue = "species", diag_sharey=False)
g.map_lower(sns.kdeplot)
g.map_upper(plt.scatter, edgecolor='white')
g.map_diag(sns.kdeplot, lw=3)


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
For data sets where we have classification labels, we can even color the various points using the hue option, and produce KDEs along the diagonal with diag_type = 'kde'.
Can you write Python code for it?

sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')