I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
With 3 clusters the algorithm does a good job of separating the three classes. However, without the a priori knowledge that there are 3 different types of iris, the 2 cluster solution would appear to be superior.
Problem 2b How do the results change if the 3 cluster model is called with n_init = 1 and init = 'random' options? Use rs for the random state [this allows me to cheat in service of making a point].
*Note - the respective defaults for these two parameters are 10 and k-means++, respectively. Read the docs to see why these choices are, likely, better than those in 2b.
Can you write Python code for it?

rs = 14
Kcluster1 = KMeans(n_clusters = 3, n_init = 1, init = 'random', random_state = rs)
Kcluster1.fit(iris.data)

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster1.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Finally, as a baseline for the exercises that follow, we will now make a simple 2D plot showing the separation of the 3 classes in the iris dataset. This plot will serve as the reference for examining the quality of the clustering algorithms.
Problem 1e Make a scatter plot showing sepal length vs. sepal width for the iris data set. Color the points according to their respective classes.
Hint - determine which columns of data correspond to sepal length and sepal width.
Can you write Python code for it?

plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 3e Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?

normalized_cm = cm.astype('float')/cm.sum(axis = 1)[:,np.newaxis]

normalized_cm


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values. [In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.] In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, but the documentation for essentially everything within scikit-learn is excellent, so read the docs.
To further develop our intuition, we will now explore the Iris dataset a little further.
Problem 1a What is the pythonic type of iris?
You likely haven't encountered a scikit-learn Bunch before. It's functionality is essentially the same as a dictionary.
Problem 1b What are the keys of iris?
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?

print(np.shape( # complete
print( # complete


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
I have used my own domain knowledge to specificly choose features that may be useful when clustering galaxies. If you know a bit about SDSS and can think of other features that may be useful feel free to add them to the query.
One nice feature of astropy tables is that they can readily be turned into pandas DataFrames, which can in turn easily be turned into a sklearn X array with NumPy. For example:
X = np.array(SDSSgals.to_pandas())

And you are ready to go.
Challenge Problem Using the SDSS dataset above, identify interesting clusters within the data [this is intentionally very open ended, if you uncover anything especially exciting you'll have a chance to share it with the group]. Feel free to use the algorithms discussed above, or any other packages available via sklearn. Can you make sense of the clusters in the context of galaxy evolution?
Hint - don't fret if you know nothing about galaxy evolution (neither do I!). Just take a critical look at the clusters that are identified
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
I was unable to obtain 3 clusters with DBSCAN. While these results are, on the surface, worse than what we got with $k$-means, my suspicion is that the 4 features do not adequately separate the 3 classes. [See - a nayseyer can always make that argument.] This is not a problem for DBSCAN as an algorithm, but rather, evidence that no single algorithm works well in all cases.
Challenge Problem) Cluster SDSS Galaxy Data
The following query will select 10k likely galaxies from the SDSS database and return the results of that query into an astropy.Table object. (For now, if you are not familiar with the SDSS DB schema, don't worry about this query, just know that it returns a bunch of photometric features.)
Can you write Python code for it?

from astroquery.sdss import SDSS  # enables direct queries to the SDSS database

GALquery = """SELECT TOP 10000
             p.dered_u - p.dered_g as ug, p.dered_g - p.dered_r as gr,
             p.dered_g - p.dered_i as gi, p.dered_g - p.dered_z as gz,
             p.petroRad_i, p.petroR50_i, p.deVAB_i
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND p.type = 3
               """
SDSSgals = SDSS.query_sql(GALquery)
SDSSgals


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Introduction to Machine Learning:
Examples of Unsupervised and Supervised Machine-Learning Algorithms
Version 0.1
Broadly speaking, machine-learning methods constitute a diverse collection of data-driven algorithms designed to classify/characterize/analyze sources in multi-dimensional spaces. The topics and studies that fall under the umbrella of machine learning is growing, and there is no good catch-all definition. The number (and variation) of algorithms is vast, and beyond the scope of these exercises. While we will discuss a few specific algorithms today, more importantly, we will explore the scope of the two general methods: unsupervised learning and supervised learning and introduce the powerful (and dangerous?) Python package scikit-learn.

By AA Miller (Jet Propulsion Laboratory, California Institute of Technology.)
(c) 2016 California Institute of Technology. Government sponsorship acknowledged.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Petal length has the largest range and standard deviation, thus, it will have the most "weight" when determining the $k$ clusters.
The truth is that the iris data set is fairly small and straightfoward. Nevertheless, we will now examine the clustering results after re-scaling the features. [Some algorithms, cough Support Vector Machines cough, are notoriously sensitive to the feature scaling, so it is important to know about this step.] Imagine you are classifying stellar light curves: the data set will include contact binaries with periods of $\sim 0.1 \; \mathrm{d}$ and Mira variables with periods of $\gg 100 \; \mathrm{d}$. Without re-scaling, this feature that covers 4 orders of magnitude may dominate all others in the final model projections.
The two most common forms of re-scaling are to rescale to a guassian with mean $= 0$ and variance $= 1$, or to rescale the min and max of the feature to $[0, 1]$. The best normalization is problem dependent. The sklearn.preprocessing module makes it easy to re-scale the feature set. It is essential that the same scaling used for the training set be used for all other data run through the model. The testing, validation, and field observations cannot be re-scaled independently. This would result in meaningless final classifications/predictions.
Problem 2d Re-scale the features to normal distributions, and perform $k$-means clustering on the iris data. How do the results compare to those obtained earlier?
Hint - you may find 'StandardScaler()' within the sklearn.preprocessing module useful.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit( # complete

Kcluster = KMeans( # complete

plt.figure()
plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Problem 1d What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
Problem 2) Unsupervised Machine Learning
Unsupervised machine learning, sometimes referred to as clustering or data mining, aims to group or classify sources in the multidimensional feature space. The "unsupervised" comes from the fact that there are no target labels provided to the algorithm, so the machine is asked to cluster the data "on its own." The lack of labels means there is no (simple) method for validating the accuracy of the solution provided by the machine (though sometimes simple examination can show the results are terrible).
For this reason [note - this is my (AAM) opinion and there many be many others who disagree], unsupervised methods are not particularly useful for astronomy. Supposing one did find some useful clustering structure, an adversarial researcher could always claim that the current feature space does not accurately capture the physics of the system and as such the clustering result is not interesting or, worse, erroneous. The one potentially powerful exception to this broad statement is outlier detection, which can be a branch of both unsupervised and supervised learning. Finding weirdo objects is an astronomical pastime, and there are unsupervised methods that may help in that regard in the LSST era.
To begin today we will examine one of the most famous, and simple, clustering algorithms: $k$-means. $k$-means clustering looks to identify $k$ convex clusters, where $k$ is a user defined number. And here-in lies the rub: if we truly knew the number of clusters in advance, we likely wouldn't need to perform any clustering in the first place. This is the major downside to $k$-means. Operationally, pseudocode for the algorithm can be summarized as the following:
initiate search by identifying k points (i.e. the cluster centers)
loop
    assign each point in the data set to the closest cluster center
    calculate new cluster centers based on mean position of all cluster points
    if diff(new center - old center) < threshold:
        stop (i.e. clusters are defined)

The...
Can you write Python code for it?

from sklearn.cluster import KMeans

Kcluster = KMeans( # complete
# complete

plt.figure()
plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')

Kcluster = KMeans( # complete
# complete

plt.figure()
plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
The normalization makes it easier to compare the classes, since each class has a different number of sources. Now we can procede with a visual representation of the confusion matrix. This is best done using imshow() within pyplot. You will also need to plot a colorbar, and labeling the axes will also be helpful.
Problem C3 Plot the confusion matrix. Be sure to label each of the axeses.
Hint - you might find the sklearn confusion matrix tutorial helpful for making a nice plot.
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
These results are almost identical to those obtained without scaling. This is due to the simplicity of the iris data set.
How do I test the accuracy of my clusters?
Essentially - you don't. There are some methods that are available, but they essentially compare clusters to labeled samples, and if the samples are labeled it is likely that supervised learning is more useful anyway. If you are curious, scikit-learn does provide some built-in functions for analyzing clustering, but again, it is difficult to evaluate the validity of any newly discovered clusters.
What if I don't know how many clusters are present in the data?
An excellent question, as you will almost never know this a priori. Many algorithms, like $k$-means, do require the number of clusters to be specified, but some other methods do not. As an example DBSCAN. In brief, DBSCAN requires two parameters: minPts, the minimum number of points necessary for a cluster, and $\epsilon$, a distance measure. Clusters are grown by identifying core points, objects that have at least minPts located within a distance $\epsilon$. Reachable points are those within a distance $\epsilon$ of at least one core point but less than minPts core points. Identically, these points define the outskirts of the clusters. Finally, there are also outliers which are points that are $> \epsilon$ away from any core points. Thus, DBSCAN naturally identifies clusters, does not assume clusters are convex, and even provides a notion of outliers. The downsides to the algorithm are that the results are highly dependent on the two tuning parameters, and that clusters of highly different densities can be difficult to recover (because $\epsilon$ and minPts is specified for all clusters.
In scitkit-learn the
DBSCAN algorithm is part of the sklearn.cluster module. $\epsilon$ and minPts are set by eps and min_samples, respectively.
Problem 2e Cluster the iris data using DBSCAN. Play around with the tuning parameters to see how they affect the final clustering results. How does the use of...
Can you write Python code for it?

from sklearn.cluster import DBSCAN

dbs = DBSCAN( # complete
dbs.fit( # complete

dbs_outliers = # complete

plt.figure()
plt.scatter( # complete
plt.scatter( # complete

plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachineLearning.ipynb".
The first task is:
While it is useful to understand the overall accuracy of the model, it is even more useful to understand the nature of the misclassifications that occur.
Problem 3c Calculate the accuracy for each class in the iris set, as determined via CV for the $k$NN = 50 model.
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
A standard method for evaluating classification models is the confusion matrix, which summarizes both the accuracy for each individual class, as well the most-likely misclassifications for each class. [For the 2 class problem, such as ours, these quantities are essentially one and the same, but for multi-class problems this is highly helpful.] In examining the matrix, "confusion" for the classifier is summarized. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion_matrix within the sklearn.metrics module.
Problem 4b Calculate (and examine) the confusion matrix for the training set from Part 1.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(val_y, val_predsNorm)
print(cm)


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Challenge 1 Calculate the accuracy with which the model classifies QSOs based on the 10k QSOs selected with the above command. How does that accuracy compare to that estimated by the validation set?
Can you write Python code for it?

qso_X = np.array(QSOts[feats].to_pandas())
qso_y = np.ones(len(QSOts))    # we are defining QSOs as stars for this exercise
qso_Xnorm = qso_X[:,0][:,np.newaxis] - qso_X[:,1:]

qso_preds = RFmod.predict(qso_Xnorm)

print("The RF model correctly classifies ~{:.4f} of the QSOs".format(accuracy_score(qso_y, qso_preds)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
From the above results, we can see that 1 tree is likely too few, while the results seem to be stabilizing with $N_\mathrm{tree} \gtrsim 10$. So - how does one actually determine the optimal set of tuning parameters? Brute force. At least in our case we will use brute force, as the data set and number of tuning parameters is small (there are alternative approaches that are less time consuming when this isn't the case). We will perform a grid search over the three tuning parameters, measuring the CV accuracy at each point within the 3D grid, and picking the point with the highest accuracy. Two general rules of thumb: (i) if the highest accuracy occurs at the edge of the grid, it is best to refit a new grid centered on that point, and (ii) the results should be stable in the vicinity of the grid maximum. If this is not the case, then it is likely that the model has been overfit.
Once again, scikit-learn has made our lives significantly easier by providing a convenience function GridSearchCV, which will enable a single line call to optimize the model. One thing to keep in mind, which likely won't be a problem here, depending on how GridSearchCV is called it is possible to run into memory issues.
Problem 3c Perform a 3-fold CV grid search to optimize the RF star-galaxy model. Remember the rules of thumb. What are the optimal tuning parameters for the model?
Hint - think about the computational runtime based on the number of points in the grid. Do not start with a very dense or large grid.
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV

grid_results = GridSearchCV(RandomForestClassifier(),
                            {'n_estimators': [30, 100, 300], 'max_features': [1, 3, 7], 'min_samples_leaf': [1,10]},
                            cv = 5)
grid_results.fit(train_Xnorm, train_y)

print("The optimal parameters are:")
for key, item in grid_results.best_params_.items(): # warning - slightly different meanings in Py2 & Py3
    print("{}: {}".format(key, item))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Not bad for a model with no optimization!
For reference, the SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag} - \mathtt{cmodelMag} > 0.145,$$
sources that satisfy this criteria are considered galaxies.
Problem 2b Determine the accuracy of the SDSS photometric model on the 10k spectroscopic sources obtained above. This accuracy will represent the "benchmark" upon which we hope to improve with the machine learning model.
Can you write Python code for it?

SDSSpred = SDSSts['psfMag_r'] - SDSSts['cModelMag_r'] > 0.145
SDSSphot_class = np.empty(len(SDSSpred), dtype = '|S6')
SDSSphot_class[SDSSpred] = 'GALAXY'
SDSSphot_class[SDSSpred == False] = 'STAR'

print("The SDSS phot model produces an accuracy ~{:.4f}".format(accuracy_score(SDSSts['class'], SDSSphot_class)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Now, let's try to develop some intuition for the effects of changing the tuning parameters.
Problem 3b Determine the 5-fold CV accuracy for models with $N_\mathrm{tree}$ = 1, 10, 100.
Can you write Python code for it?

CVscores1 = cross_val_score(RandomForestClassifier(n_estimators = 1), train_Xnorm, train_y, cv = 5)

CVscores10 = cross_val_score(RandomForestClassifier(n_estimators = 10), train_Xnorm, train_y, cv = 5)

CVscores100 = cross_val_score(RandomForestClassifier(n_estimators = 100), train_Xnorm, train_y, cv = 5)

print("The CV accuracy for 1, 10, 100 trees is {:.4f}, {:.4f}, {:.4f}".format(np.mean(CVscores1), np.mean(CVscores10), np.mean(CVscores100)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
One advantage of the SDSS model is that the magnitude measurements have been normalized. The SDSS spectroscopic training set is biased towards stars at bright the end, and towards galaxies at the faint end. (This can be quickly confirmed by plotting histograms, or better yet KDEs, of each.) While the universe is biased in this way, it is not clear that the SDSS targeting function properly captured this bias. Thus, we will attempt to engineer the features to remove this bias (similar to SDSS).
Problem 2c Normalize the feature vector relative to psfMag_r, and refit the RF model to see if the accuracy improves when predicting the class of sources in the validation set.
Hint - be sure you apply the eaxct same normalization to both the training and validation set
Can you write Python code for it?

train_Xnorm = train_X[:,0][:,np.newaxis] - train_X[:,1:]
val_Xnorm = val_X[:,0][:,np.newaxis] - val_X[:,1:]

RFmod = RandomForestClassifier(n_estimators = 50)
RFmod.fit(train_Xnorm, train_y)
val_predsNorm = RFmod.predict(val_Xnorm)

print("The normalized features produce a model with accuracy ~{:.4f}".format(accuracy_score(val_y, val_predsNorm)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Problem 2) Feature Engineering
It has been said that all machine learning is an exercise in feature engineering. In other words, all machine-learning are worthless without the correct set of features. Here we will quickly examine the set of features selected above to test how well they separate stars and galaxies. We will use the Random Forest (RF) algorithm Breiman 2001 as implemented by sklearn. RandomForestClassifier is part of the sklearn.ensemble module.
Problem 2a Train a RF model on your training set, and determine the accuracy of the model using the validation set. Set n_estimators, the number of trees in the forest, to 25 in the model.
Hint - you may find sklearn.metrics.accuracy_score useful for this exercise.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

RFmod = RandomForestClassifier(n_estimators = 50)
RFmod.fit(train_X, train_y)
val_preds = RFmod.predict(val_X)

print("The raw features produce a model with accuracy ~{:.4f}".format(accuracy_score(val_y, val_preds)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Problem 4) Evaluating the Accuracy of the Model
Now that the model has been fully optimized, we can estimate its accuracy with the validation set. This estimate provides insight for the model accuracy expected for photometric data where we do not have spectroscopy, and thus cannot validate the predictions.
Problem 4a Train a RF model using the optimized tuning parameters from 3c, and estimate the classification accuracy of the model using the validation set.
Can you write Python code for it?

RFmod = RandomForestClassifier(n_estimators=300, max_features=3, min_samples_leaf=10, n_jobs=-1).fit(train_Xnorm, train_y)

val_predsNorm = RFmod.predict(val_Xnorm)

print("The accuracy of the optimized model is ~{:.4f}".format(accuracy_score(val_y, val_predsNorm)))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
That's it! You now have the basic outline necessary to create a full-blown, publication-ready machine-learning model.
Now, before you run off importing sklearn left and right and re-writing all your code to use machine-learning, remember: you should always, always, always, always (how many times can I copy paste the word "always"?), always be extremely concerned about the bias present in your training set. (Even after the paper is accepted, the community loves it, and you've discovered some new great thing - you should be worried about the training set.) In my (AAM) personal experience, the vast majority of my time as a machine-learning practicioner is spent chasing down unusual things in the training set. So, proceed with caution.
If you have time - try the challenge problem below, which should reinforce my words of caution.
Challenge Problem) Taking the Plunge
Applying the model to field data
QSOs are unresolved sources that look like stars in optical imaging data. We will now download photometric measurements for 10k QSOs from SDSS and see how accurate the RF model performs for these sources.
Can you write Python code for it?

QSOquery = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class = 'QSO'
             ORDER BY s.specobjid ASC
               """
QSOts = SDSS.query_sql(QSOquery)


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
The ROC curve should come very close to (0, 1), which is the sign of a good model (and with ~97% accuracy we already knew we have a pretty good model). We will now explore the results of thresholding. Suppose you are searching for supernovae in a time domain survey, and it is essential you examine transients in as many galaxies as possible. While searching, there will be many false positives in the form of stars, thus, you need a model that rejects stars (high TPR) with a very low FPR (does not misclassify galaxies).
Problem 4e Determine the classification threshold and TPR corresponding to a FPR = 0.01.
Can you write Python code for it?

fpr01_idx = (np.abs(fpr-0.01)).argmin()

tpr01 = tpr[fpr01_idx]
thresh01 = thresh[fpr01_idx]

print("At FPR = 0.01, the TPR = {:.3f} corresponding to decision threshold = {:.3f}".format(tpr01, thresh01))


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
To reiterate a point from above: data-driven models are only as good as the training set. Now that we have a potential training set, it is essential to inspect the data for any peculiarities.
Problem 1a Visualize the training set (an intentially open-ended question, think back to yesterday's exercises) to inspect the features and whether or not they may be useful for a machine-learning model.
Hint astropy Tables can be converted to pandas DataFrames with the .to_pandas() operator, which may be helpful.
Can you write Python code for it?

import seaborn as sns
sns.pairplot(SDSSts.to_pandas(), hue = 'class', diag_kind = 'kde')


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Uber Challenge Problem) The 3-class problem
Uber Challenge Create a 3 class RF model that separates stars, galaxies, and QSOs in SDSS imaging data. Limit the size of your training set to $< 25000$ sources, but that's it as far as suggestions and hints are concerned. Can you achieve a classification accuracy around 97%?
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
Visual representations are often the most useful means for interpreting the confusion matrix (this is especially true for multiclass problems).
Problem 4c Plot the normalized confusion matrix for the validation set.
Hint - you might find the sklearn confusion matrix tutorial helpful.
Can you write Python code for it?

cmNorm = cm.astype(float)/cm.sum(axis = 1)[:,np.newaxis]

plt.imshow(cmNorm, interpolation = 'Nearest', cmap = 'Blues', vmin = 0, vmax = 1)
plt.colorbar()
tick_names = ["GALAXY", "STAR"]
tick_marks = np.arange(len(tick_names))
plt.xticks(tick_marks, tick_names, rotation=45)
plt.yticks(tick_marks, tick_names)
plt.ylabel('True label')
plt.xlabel('Predicted label')


I am working on the file "Sessions/Session01/Day4/SGRandForestSolutions.ipynb".
The first task is:
While it is possible to look up each of the names of the $r$-band magnitudes in the SDSS PhotoObjAll schema, the schema list is long, and thus there is a chance of missing one column. Better to identify the desired columns using the database itself:
SELECT COLUMN_NAME
FROM INFORMATION_SCHEMA.Columns
WHERE table_name = 'PhotoObjAll' AND
COLUMN_NAME LIKE '%Mag/_r' ESCAPE '/'

which returns the following list of columns: psfMag_r, fiberMag_r, fiber2Mag_r, petroMag_r, deVMag_r, expMag_r, modelMag_r, cModelMag_r.
We now select these magnitude measurements for 10000 stars and galaxies from SDSS. Additionally, we join these results with the SpecObjAll table to obtain their spectroscopic classifications, which will serve as labels for the machine-learning model.
Note - the SDSS database contains duplicate observations, flagged observations, and non-detections, which we condition the query to exclude (as explained further below). We also exclude quasars, as the precise photometric classification of these objects is ambiguous: low-$z$ AGN have resolvable host galaxies, while high-$z$ QSOs are point-sources. Query conditions:

p.mode = 1 select only the primary photometric detection of a source
s.sciencePrimary = 1 select only the primary spectroscopic detection of a source (together with above, prevents duplicates)
p.clean = 1 the SDSS clean flag excludes flagged observations and sources with non-detections
s.class != 'QSO' removes potentially ambiguous QSOs from the training set
Can you write Python code for it?

TSquery = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             ORDER BY p.objid ASC
               """
SDSSts = SDSS.query_sql(TSquery)
SDSSts


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
A standard method for evaluating classification models is the confusion matrix, which summarizes both the accuracy for each individual class, as well the most-likely misclassifications for each class. [For the 2 class problem, such as ours, these quantities are essentially one and the same, but for multi-class problems this is highly helpful.] In examining the matrix, "confusion" for the classifier is summarized. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion_matrix within the sklearn.metrics module.
Problem 4b Calculate (and examine) the confusion matrix for the training set from Part 1.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

cm = confusion_matrix( # complete
print(cm)


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Challenge 1 Calculate the accuracy with which the model classifies QSOs based on the 10k QSOs selected with the above command. How does that accuracy compare to that estimated by the validation set?
Can you write Python code for it?

# complete

print("The RF model correctly classifies ~{:.4f} of the QSOs".format(accuracy_score( # complete


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Challenge 2 Can you think of any reasons why the performance would be so much worse for the QSOs than it is for the stars? Can you obtain a ~.97 accuracy when classifying QSOs?
Can you write Python code for it?

# As discussed above, low-z AGN have resolved host galaxies which will confuse the classifier,
# this can be resolved by only selecting high-z QSOs (z > 1.5)

QSOquery = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class = 'QSO'
             AND s.z > 1.5
             ORDER BY s.specobjid ASC
               """
QSOts = SDSS.query_sql(QSOquery)

qso_X = # complete
qso_y = # complete
qso_Xnorm = # complete

qso_preds = RFmod.predict( # complete

print("The RF model correctly classifies ~{:.4f} of the QSOs".format(accuracy_score( # complete


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
From the above results, we can see that 1 tree is likely too few, while the results seem to be stabilizing with $N_\mathrm{tree} \gtrsim 10$. So - how does one actually determine the optimal set of tuning parameters? Brute force. At least in our case we will use brute force, as the data set and number of tuning parameters is small (there are alternative approaches that are less time consuming when this isn't the case). We will perform a grid search over the three tuning parameters, measuring the CV accuracy at each point within the 3D grid, and picking the point with the highest accuracy. Two general rules of thumb: (i) if the highest accuracy occurs at the edge of the grid, it is best to refit a new grid centered on that point, and (ii) the results should be stable in the vicinity of the grid maximum. If this is not the case, then it is likely that the model has been overfit.
Once again, scikit-learn has made our lives significantly easier by providing a convenience function GridSearchCV, which will enable a single line call to optimize the model. One thing to keep in mind, which likely won't be a problem here, depending on how GridSearchCV is called it is possible to run into memory issues.
Problem 3c Perform a 3-fold CV grid search to optimize the RF star-galaxy model. Remember the rules of thumb. What are the optimal tuning parameters for the model?
Hint - think about the computational runtime based on the number of points in the grid. Do not start with a very dense or large grid.
Can you write Python code for it?

from sklearn.grid_search import GridSearchCV

grid_results = GridSearchCV(  # complete
grid_results.fit(  # complete

print("The optimal parameters are:")
for key, item in grid_results.best_params_.items(): # warning - slightly different meanings in Py2 & Py3
    print("{}: {}".format(key, item))


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Not bad for a model with no optimization!
For reference, the SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag} - \mathtt{cmodelMag} > 0.145,$$
sources that satisfy this criteria are considered galaxies.
Problem 2b Determine the accuracy of the SDSS photometric model on the 10k spectroscopic sources obtained above. This accuracy will represent the "benchmark" upon which we hope to improve with the machine learning model.
Can you write Python code for it?

 # complete

print("The SDSS phot model produces an accuracy ~{:.4f}".format(accuracy_score(SDSSts['class'], SDSSphot_class)))


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Note - if your plot looks a little funny, that is likely because you imported seaborne earlier. Remember, import seaborne also changes defaults, but you can change the default appearance of seaborne.
For this problem, where the accuracy is very high and there are only two classes, the confusion matrix is not necessarily that insightful. Another common machine-learning evalutation tool is the Receiver Operating Characteristic (ROC) curve, which is espeically useful for threshold tuning.
One of the useful properties of RF models, which has not been mentioned yet, is that RF provides probabilistic predictions. In short, a source that is classified as a star in 95% of trees is much more likely to be a star than a source classified as such in only 53% of trees. [It is better to think of these probabilities as relative rankings based on the training set rather than true probabilities. Biases in the training set, which will almost certainly always exist in astronomy, result in "probabilities" that are not properly calibrated.] These "probabilities" are highly useful because they allow us to tune the purity and completeness of the final model, and they can be access for some (not all) sklearn models using the .predict_proba() function.
The model output is tuned via the ROC curve (or the somewhat related precision-recall curve), which plots the  the true positive rate (TPR) against the false positive rate (FPR):
$$ \mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}},$$
$$ \mathrm{FPR} = \frac{\mathrm{FP}}{\mathrm{FP} + \mathrm{TN}},$$
where $\mathrm{TP}$ is the number of true positives, $\mathrm{FN}$ is the number of false negatives, $\mathrm{FP}$ is the number of false positives, and $\mathrm{TN}$ is the number of true negatives. The ideal model has high TPR and low FPR. The ROC curve comes from varying the classification decision threshold from 1 to 0.
Once again, scikit-learn comes to the rescue by making it easy to calculate ROC curves with roc_curve in the sklearn.metrics module.
Problem 4d Calculate...
Can you write Python code for it?

from sklearn.metrics import roc_curve

train_yInt = # complete
# complete
val_yInt = # complete
# complete

RFmod = RandomForestClassifier(# complete
# complete
val_predsProba = # complete

fpr, tpr, thresh = roc_curve( # complete

plt.plot( # complete
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Now, let's try to develop some intuition for the effects of changing the tuning parameters.
Problem 3b Determine the 5-fold CV accuracy for models with $N_\mathrm{tree}$ = 1, 10, 100.
Can you write Python code for it?

CVscores1 = cross_val_score(  # complete

CVscores10 = cross_val_score(  # complete

CVscores100 = cross_val_score(  # complete

print("The CV accuracy for 1, 10, 100 trees is {:.4f}, {:.4f}, {:.4f}".format(  # complete


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
One advantage of the SDSS model is that the magnitude measurements have been normalized. The SDSS spectroscopic training set is biased towards stars at bright the end, and towards galaxies at the faint end. (This can be quickly confirmed by plotting histograms, or better yet KDEs, of each.) While the universe is biased in this way, it is not clear that the SDSS targeting function properly captured this bias. Thus, we will attempt to engineer the features to remove this bias (similar to SDSS).
Problem 2c Normalize the feature vector relative to psfMag_r, and refit the RF model to see if the accuracy improves when predicting the class of sources in the validation set.
Hint - be sure you apply the eaxct same normalization to both the training and validation set
Can you write Python code for it?

train_Xnorm =  # complete
val_Xnorm =  # complete

RFmod =  # complete
RFmod.fit(  # complete
val_predsNorm = RFmod.predict(  # complete

print("The normalized features produce a model with accuracy ~{:.4f}".format(accuracy_score(val_y, val_predsNorm)))


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Problem 1) Obtain and Examine Training Data
As a reminder, for supervised-learning problems we use a training set, sources with known labels, i.e. they have been confirmed as normal stars, QSOs, or galaxies, to build a model to classify new observations where we do not know the source label.
The training set for this exercise will be pulled from SDSS. For features, we will start with each Magnitude measured in the $r$-band. This results in a total of 8 features (more than the Iris data set, but significantly fewer than everything in SDSS).
The first step when pursuring a machine learning problem is to examine the potential training set. A machine-learning model is only as good as its training set. This point cannot be emphasized enough. Machine-learning models are data-driven, they do not capture any physical theory, and thus it is essential that the training set satisfy several criteria. Two of the most important criteria for a good training set are:

the training set should be unbiased [this is actually really hard to achieve in astronomy since most surveys are magnitude limited]
the training set should be representative of the (unobserved or field) population of sources [a training set with no stars will yield a model incapable of discovering point sources]

As a first step (and this is always a good idea), we are going to examine the training set to see if anything suspicious is going on. We will use astroquery to directly access the SDSS database, and store the results in an astropy Table.
Note The SDSS API for astroquery is not standard for the package, which leads to a warning. This is not, however, a problem for our purposes.
Can you write Python code for it?

# execute this cell
from astroquery.sdss import SDSS  # enables direct queries to the SDSS database


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Problem 2) Feature Engineering
It has been said that all machine learning is an exercise in feature engineering. In other words, all machine-learning are worthless without the correct set of features. Here we will quickly examine the set of features selected above to test how well they separate stars and galaxies. We will use the Random Forest (RF) algorithm Breiman 2001 as implemented by sklearn. RandomForestClassifier is part of the sklearn.ensemble module.
Problem 2a Train a RF model on your training set, and determine the accuracy of the model using the validation set. Set n_estimators, the number of trees in the forest, to 25 in the model.
Hint - you may find sklearn.metrics.accuracy_score useful for this exercise.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

RFmod =  # complete

val_preds =  # complete

print("The raw features produce a model with accuracy ~{:.4f}".format(accuracy_score(val_y, val_preds)))


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Problem 2d How does the accuracy of each of these 3 models compare?
If normalizing the features did not improve the accuracy of the model, can you think of any reasons why?
Hint - think about the name of the algorithm.
Type response to 2d here
While there are additional feature engineering steps one could take:

remove correlated features
add contextual features (e.g., galactic latitude? distance to nearest neighbor?)
create new features (ellipticity measurements?)

we will stop at this stage. Generally speaking, RF is immune to correlated features (this is not true of all machine-learning algorithms), contextual features, especially galactic latitude, can bias the model due to the manner in which the training data were obtained.
Finally, we exploit one of the most novel aspects of RF - the ability to measure the relative importance of each feature. This is accomplished by randomly shuffling the values of a particular feature, and examining the decrease in performance of the model's overall accuracy. The relative feature importances can be accessed using the .feature_importances_ attribute associated with the RandomForestClassifer() class. The higher the value, the more important feature.
Problem 2e Calculate the relative importance of each feature. Which feature is most important? Can you make sense of the feature ordering?
Hint - do not dwell too long on the final ordering of the features.
Can you write Python code for it?

 # complete

print("The relative importance of the features is: \n{:s}".format(  # complete


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Problem 4) Evaluating the Accuracy of the Model
Now that the model has been fully optimized, we can estimate its accuracy with the validation set. This estimate provides insight for the model accuracy expected for photometric data where we do not have spectroscopy, and thus cannot validate the predictions.
Problem 4a Train a RF model using the optimized tuning parameters from 3c, and estimate the classification accuracy of the model using the validation set.
Can you write Python code for it?

RFmod = RandomForestClassifier( # complete

val_predsNorm = RFmod.predict(  # complete

print("The accuracy of the optimized model is ~{:.4f}".format(accuracy_score(val_y, val_predsNorm)))


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
Supervised Machine Learning Break Out:
Separating Stars and Galaxies from SDSS
Version 0.1
Many (nearly all?) of the science applications for LSST data will rely on the accurate separation of stars and galaxies in the LSST imaging data. As an example, imagine measuring galaxy clustering without knowing which sources are galaxies and which are stars. During this exercise, we will utilize supervised machine-learning methods to separate extended (galaxies) and point sources (stars, QSOs) in imaging data. These methods are highly flexible, and as a result can classify sources at higher fidelity than methods that simply make cuts in a low-dimensional space.

By AA Miller (c) 2016 Jul 11
Can you write Python code for it?

import numpy as np
from astropy.table import Table
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
That's it! You now have the basic outline necessary to create a full-blown, publication-ready machine-learning model.
Now, before you run off importing sklearn left and right and re-writing all your code to use machine-learning, remember: you should always, always, always, always (how many times can I copy paste the word "always"?), always be extremely concerned about the bias present in your training set. (Even after the paper is accepted, the community loves it, and you've discovered some new great thing - you should be worried about the training set.) In my (AAM) personal experience, the vast majority of my time as a machine-learning practicioner is spent chasing down unusual things in the training set. So, proceed with caution.
If you have time - try the challenge problem below, which should reinforce my words of caution.
Challenge Problem) Taking the Plunge
Applying the model to field data
QSOs are unresolved sources that look like stars in optical imaging data. We will now download photometric measurements for 10k QSOs from SDSS and see how accurate the RF model performs for these sources.
Can you write Python code for it?

QSOquery = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class = 'QSO'
             ORDER BY s.specobjid ASC
               """
QSOts = SDSS.query_sql(QSOquery)


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
To reiterate a point from above: data-driven models are only as good as the training set. Now that we have a potential training set, it is essential to inspect the data for any peculiarities.
Problem 1a Visualize the training set (an intentially open-ended question, think back to yesterday's exercises) to inspect the features and whether or not they may be useful for a machine-learning model.
Hint astropy Tables can be converted to pandas DataFrames with the .to_pandas() operator, which may be helpful.
Problem 1b Can the acquired training set be used to separate stars and galaxies? [Write your reasoning below]
Type response to 1b here
To test the efficacy of the machine-learning model we need to separate the spectroscopic sample into an independent training and validation set. There is no set number for the precise fraction of the data to include in the validation set, and typical choices vary between $\sim{0.2}-0.4$. For this problem we will adopt 0.3.
sklearn.cross_validation has a handy function train_test_split, which will simplify this process.
Problem 1c Split the 10k spectroscopic sources 70-30 into training and validation sets. Save the results in arrays called: train_X, train_y, val_X, val_y, respectively. Use rs for the random_state when selecting a random portion of the spectroscopic set.
Hint - recall that sklearn utilizes X, a 2D np.array(), and y as the features and labels arrays, respecitively.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
rs = 23  # we are in Chicago after all

# complete

X = np.array( # complete
y = np.array( # complete

train_X, val_X, train_y, val_y = train_test_split( X, y,  # complete


I am working on the file "Sessions/Session01/Day4/StarGalaxyRandomForest.ipynb".
The first task is:
While it is possible to look up each of the names of the $r$-band magnitudes in the SDSS PhotoObjAll schema, the schema list is long, and thus there is a chance of missing one column. Better to identify the desired columns using the database itself:
SELECT COLUMN_NAME
FROM INFORMATION_SCHEMA.Columns
WHERE table_name = 'PhotoObjAll' AND
COLUMN_NAME LIKE '%Mag/_r' ESCAPE '/'

which returns the following list of columns: psfMag_r, fiberMag_r, fiber2Mag_r, petroMag_r, deVMag_r, expMag_r, modelMag_r, cModelMag_r.
We now select these magnitude measurements for 10000 stars and galaxies from SDSS. Additionally, we join these results with the SpecObjAll table to obtain their spectroscopic classifications, which will serve as labels for the machine-learning model.
Note - the SDSS database contains duplicate observations, flagged observations, and non-detections, which we condition the query to exclude (as explained further below). We also exclude quasars, as the precise photometric classification of these objects is ambiguous: low-$z$ AGN have resolvable host galaxies, while high-$z$ QSOs are point-sources. Query conditions:

p.mode = 1 select only the primary photometric detection of a source
s.sciencePrimary = 1 select only the primary spectroscopic detection of a source (together with above, prevents duplicates)
p.clean = 1 the SDSS clean flag excludes flagged observations and sources with non-detections
s.class != 'QSO' removes potentially ambiguous QSOs from the training set
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
EX. 4. GAUSSIAN PROCESSES
Ex. 4.1. SIMULATION
Three Gaussian Process covariance kernels are implemented in the following cell. The parameter h for each means the decay rate of the correlation between observations. For the quasi-periodic kernel, this is multiplied by a periodic term (added in the exponent): the correlation is periodically strengthens and weakens.
Can you write Python code for it?

from sklearn.gaussian_process import GaussianProcess

## Define the three kernel functions we'll use:
def exponential(x1, x2, h):
    return np.exp(-0.5 * np.abs(x1 - x2) / h)
def squared_exponential(x1, x2, h):
    return np.exp(-0.5 * (x1 - x2) ** 2 / h ** 2)
def quasi_periodic(x1, x2, h, l, nu):
    return np.exp(-0.5 * (x1 - x2) ** 2 / h ** 2 - 0.5 * np.sin(2*np.pi*nu*(x1 - x2)) ** 2 / l)

x = np.linspace(52610, 54625, 115)
h = 50.0
l = 0.1
nu = 0.1

mu = np.zeros(len(x))

Cse = squared_exponential(x, x[:, None], h)
Ce = exponential(x, x[:, None], h)
Cq = quasi_periodic(x, x[:, None], h = h, l = l, nu = nu)

qper_rdvar = np.random.multivariate_normal(mu, Cq, 3)
ex_rdvar = np.random.multivariate_normal(mu, Ce, 3)
sqex_rdvar = np.random.multivariate_normal(mu, Cse, 3)

plt.rcParams["figure.figsize"] = [5., 10.]
fig = plt.figure(1)
plt.subplots_adjust(left=0.07, bottom=0.05, right=0.95, top=0.95, wspace=None, hspace=0.35)

fig.add_subplot(311)
for ii in [0,1,2]:
    plt.plot(x, ex_rdvar[ii], 'k', alpha=1)
plt.title('Exponential')
plt.xlabel('x')
plt.ylabel('GP draw')
plt.margins(0.01, 0.05)

fig.add_subplot(312)
for ii in [0,1,2]:
    plt.plot(x, sqex_rdvar[ii], 'k', alpha=1)
plt.title('Squared exponential')
plt.xlabel('x')
plt.ylabel('GP draw')
plt.margins(0.01, 0.05)
#plt.xlim(-0.1,4.)

fig.add_subplot(313)
for ii in [0,1,2]:
    plt.plot(x, qper_rdvar[ii], 'k', alpha=1)
plt.title('Quasi-periodic')
plt.xlabel('x')
plt.ylabel('GP draw')
plt.margins(0.01, 0.05)

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
Ex. 3. TIME-RESOLVED ANALYSIS
Ex. 3.1 A PERIOD-CHANGING OGLE CEPHEID
Do a time-resolved analysis of the former period-changing Cepheid in an interval around its average frequency, and represent it as a color map (time & frequency) (in frequency: between 0.1 and 0.11 c/d, using a window of 2 years).
Can you write Python code for it?

print(cep2['time'].max())
print(cep2['time'].min())
centers_tmp = np.linspace(470, 4950, num = 449)

fgrid_short_cep2 = np.linspace(0.1, 0.11, 4000)
trsvd_cep2 = np.zeros([len(fgrid_short_cep2), len(centers_tmp)])
for ii in np.arange(0,len(centers_tmp)):
    df = cep2[(cep2['time'] < centers_tmp[ii] + 365) & (cep2['time'] > centers_tmp[ii] - 365)]
    trsvd_cep2[:,ii] = lomb_scargle(t = df['time'], y = df['mag'], dy = df['mag.error'], omega = 2*np.pi*fgrid_short_cep2, generalized = True)

# Find the candidate frequencies in each window (where the periodogram takes its maximum value):
bestfr_vs_time = np.zeros(len(centers_tmp))
for jj in np.arange(len(centers_tmp)):
    bestfr_vs_time[jj] = fgrid_short_cep2[trsvd_cep2[:,jj] == trsvd_cep2[:,jj].max()]

# Plot the spectrogram, and add the constant catalog frequency and the best local frequencies from the sliding windows:
plt.rcParams["figure.figsize"] = [8., 8.]
fig = plt.figure()

plt.imshow(trsvd_cep2, origin = 'lower', aspect = 'auto',
           extent = [centers_tmp[0], centers_tmp[-1], fgrid_short_cep2[0], fgrid_short_cep2[-1]])
plt.title('Cepheid 3')
plt.ylabel('Frequency [1/d]')
plt.xlabel('Time [d]')
plt.margins(0.01, 0.05)
plt.axhline(1/p_cep2, xmin = 0, xmax = 1, color='w')
plt.plot(cep2['time'], [0.1]*len(cep2['time']), '|', color='k')
plt.plot(centers_tmp, bestfr_vs_time, 'k')

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
Ex. 4.2. CANDIDATE QUASAR FROM LINEAR
The Ornstein-Uhlenbeck process can be applied to fit quasar data. We will use the fact that the OU process is a Gaussian process, and apply Gaussian process modelling for its light curve, both with the squared exponential and the exponential kernel.
Can you write Python code for it?

qso = pd.read_csv("./data/q2803474.csv")

qso_gp_sqex1 = GaussianProcess(corr='squared_exponential', theta0=0.5, thetaL = 0.001, thetaU = 100)
qso_gp_ex1 = GaussianProcess(corr='absolute_exponential', theta0=0.5, thetaL = 0.001, thetaU = 100)

qso_gp_sqex1.fit(qso['time'].reshape(-1,1), qso['mag'].reshape(-1,1))


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
For the Cepheid, everything looks all right. For the quasar, the found frequency depended on the definition of the minimal frequency searched: here, the lowest frequency of the grid is the grid resolution, whereas with the pipeline that produced the results on the quasar, it was 1/timespan of observations (that is, oversampling factor times the grid resolution). The two eclipsing binaries exhibit the limitation of the generalized Lomb-Scargle method, when applied to light curves that are nonsinusoidal. The eclipsing binaries typically have two minima in one cycle, and thus, the GLS finds the double frequency (the half-period), which produces one single minimum in one cycle.
Fold the light curves of the two eclipsing binaries to take a look. In the file, there is a column 'phase' giving the phases according to the catalog period, so we have to compute only those according to our period estimate.
Can you write Python code for it?

# Compute the period corresponding to the maximum of the periodogram:
p_est_ecl1 = 1 / fgrid_ecl1[pgram_ecl1 == pgram_ecl1.max()]
p_est_ecl2 = 1 / fgrid_ecl2[pgram_ecl2 == pgram_ecl2.max()]
# Check that it is indeed roughly the half of the catalog period:
print(p_est_ecl1 *2)
print(p_ecl1)

# Compute  the phases and add them as column to the dataframes:
ecl1['phase2'] = (ecl1['time'] % p_est_ecl1) / p_est_ecl1
ecl2['phase2'] = (ecl2['time'] % p_est_ecl2) / p_est_ecl2

# plot:
plt.rcParams["figure.figsize"] = [12., 8.]
fig = plt.figure()
plt.subplots_adjust(left=0.05, bottom=0.1, right=0.95, top=0.95, wspace=None, hspace=0.35)

ax = fig.add_subplot(221)
plt.gca().invert_yaxis()
ax.errorbar(ecl1['phase2'], ecl1['mag'], ecl1['mag.error'], fmt='.k', ecolor='gray')
plt.title("ecl1, GLS period")
plt.xlabel('Phase')
plt.ylabel('Magnitude')

ax = fig.add_subplot(222)
plt.gca().invert_yaxis()
ax.errorbar(ecl1['phase'], ecl1['mag'], ecl1['mag.error'], fmt='.k', ecolor='gray')
plt.title("ecl1, catalog period")
plt.xlabel('Phase')
plt.ylabel('Magnitude')

ax = fig.add_subplot(223)
plt.gca().invert_yaxis()
ax.errorbar(ecl2['phase2'], ecl2['mag'], ecl2['mag.error'], fmt='.k', ecolor='gray')
plt.title("ecl2, GLS period")
plt.xlabel('Phase')
plt.ylabel('Magnitude')

ax = fig.add_subplot(224)
plt.gca().invert_yaxis()
ax.errorbar(ecl2['phase'], ecl2['mag'], ecl2['mag.error'], fmt='.k', ecolor='gray')
plt.title("ecl2, catalog period")
plt.xlabel('Phase')
plt.ylabel('Magnitude')

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
Note the data does not contain any noise as yet: these periodograms are the pure consequence of the funny time sampling.
Ex. 2. PERIOD SEARCH ON REAL DATA
2.1 PERIODOGRAM OF DIFFERENT VARIABLES
The time series of a Cepheid and two eclipsing binaries from OGLE-III, a quasar from LINEAR, and an X-ray binary are given in the csv files. Compute the periodograms for each. Compare and analyse the results, by plotting the folded light curves.
Note %> pip install astroML_addons may be required to ensure that the LS code below is not super slow.
Can you write Python code for it?

from astroML.time_series import lomb_scargle

cep1 = pd.read_csv("./data/OGLE-LMC-CEP-0727.csv")
ecl1 = pd.read_csv("./data/OGLE-SMC-ECL-0322.csv")
ecl2 = pd.read_csv("./data/OGLE-SMC-ECL-0124.csv")
qso = pd.read_csv("./data/q2803474.csv")
## Periods:
p_cep1 = 14.4891397
p_ecl1 = 71.6113338
p_ecl2 = 0.3270255
p_qso = 10**(-0.00138752348664116)

## The resolution computed from the timespan of the observations:
def fgrid_fun(times, ofac, fmax):
    r_tmp = 1 / (times.max() - times.min()) / ofac
    n_tmp = np.floor(fmax/r_tmp)
    return np.linspace(r_tmp, fmax, n_tmp)

# The frequency grids for each object
fgrid_cep1 = fgrid_fun(cep1['time'], ofac = 10, fmax = 3)
fgrid_ecl1 = fgrid_fun(ecl1['time'], ofac = 10, fmax = 3)
fgrid_ecl2 = fgrid_fun(ecl2['time'], ofac = 10, fmax = 10)
fgrid_qso = fgrid_fun(qso['time'], ofac = 10, fmax = 6)

pgram_cep1 = lomb_scargle(t = cep1['time'], y = cep1['mag'], dy = cep1['mag.error'], omega = 2*np.pi*fgrid_cep1, generalized = True)
pgram_ecl1 = lomb_scargle(t = ecl1['time'], y = ecl1['mag'], dy = ecl1['mag.error'], omega = 2*np.pi*fgrid_ecl1, generalized = True)
pgram_ecl2 = lomb_scargle(t = ecl2['time'], y = ecl2['mag'], dy = ecl2['mag.error'], omega = 2*np.pi*fgrid_ecl2, generalized = True)
pgram_qso = lomb_scargle(t = qso['time'], y = qso['mag'], dy = qso['mag.error'], omega = 2*np.pi*fgrid_qso, generalized = True)

plt.rcParams["figure.figsize"] = [16., 8.]
fig = plt.figure(1)
plt.subplots_adjust(left=0.05, bottom=0.05, right=0.95, top=0.95, wspace=None, hspace=0.35)

fig.add_subplot(221)
plt.plot(fgrid_cep1, pgram_cep1, 'k', alpha=1)
plt.title('Cepheid')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Periodogram')
plt.margins(0.01, 0.05)
plt.axvline(1/p_cep1, ymin = 0, ymax = 0.1, color='c', lw = 2)

fig.add_subplot(222)
plt.plot(fgrid_ecl1, pgram_ecl1, 'k', alpha=1)
plt.title('Detached(?) binary')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Periodogram')
plt.margins(0.01, 0.05)
plt.axvline(1/p_ecl1, ymin = 0, ymax = 0.1, color='c', lw =...


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
Plot the spectrogram (leaving out the high values of the red noise at low frequencies):
Can you write Python code for it?

plt.rcParams["figure.figsize"] = [8., 8.]
fig = plt.figure()
plt.imshow(psd_specgram[20:,:], origin = 'lower', aspect = 'auto',
           extent = [centers_specgram[0], centers_specgram[-1], freqs_specgram[20], freqs_specgram[-1]])
plt.title('Raw PSD')
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [s]')
plt.margins(0.01, 0.05)

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
TAKE-HOME MESSAGE:
1. In an evenly-sampled case, the time intervals between the observations determine fmax, the maximum detectable frequency in the periodogram.
2. The span of observations, together with the number of observations, determine the frequency resolution df/fmax of the resulting periodogram (this is equivalent to the separation between the Fourier frequencies).
Ex. 1.2 MORE COMPLEX REGULARITIES (DAILY RYTHM)
Take time sampling (c), and drop every halfday (keep the first 2 obs of every four). Compute the spectral window again. Take a look also what happens at the Fourier frequencies. Compute also the periodograms of the signals.
Can you write Python code for it?

drop_half_ind = (np.arange(360) % 4 < 2)
half_df = df3[drop_half_ind]

half_spw3 = deemingPSD_fun(times = half_df['time'], obs = half_df['cst'], freqs = freqs3)
half_psd31 = deemingPSD_fun(times = half_df['time'], obs = half_df['sin1'], freqs = freqs3)
half_psd32 = deemingPSD_fun(times = half_df['time'], obs = half_df['sin2'], freqs = freqs3)

fig = plt.figure(1)
plt.subplots_adjust(left=0.05, bottom=0.05, right=0.95, top=0.95, wspace=None, hspace=0.35)

ax1 = fig.add_subplot(311)
plt.plot(freqs3, half_spw3, 'k', alpha=1)
plt.title('90 days, sampling 4/d with 50% missing data \n Spectral window')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Spectral window')
plt.margins(0.01, 0.05)
#plt.vlines(F_freqs3, ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(0.95,1.05)

ax2 = fig.add_subplot(312)
plt.plot(freqs3, half_psd32, 'k', alpha=1)
plt.title('Sine at Fourier frequency')
plt.xlabel('Frequency [1/d]')
plt.ylabel('PSD')
plt.margins(0.01, 0.05)
#plt.vlines(F_freqs2, ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(71./90.-.05, 71./90.+.05)
#plt.ylim(0.,0.3)
plt.axvline(71./90., ymin = 0, ymax = 1, color='c', linestyle='dashed')

ax3 = fig.add_subplot(313)
plt.plot(freqs3, half_psd31, 'k', alpha=1)
plt.title('Sine, F = 0.2284')
plt.xlabel('Frequency [1/d]')
plt.ylabel('PSD')
plt.margins(0.01, 0.05)
#plt.vlines(F_freqs3, ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(truefreq-.05, truefreq+.05)
#plt.ylim(0.,0.3)
plt.axvline(truefreq, ymin = 0, ymax = 1, color='c', linestyle='dashed')

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
TAKE-HOME MESSAGE:
Any (quasi-)periodicity imposed on the time sampling will show up as additional peaks in the spectral window.
1.3 UNEVEN SAMPLING
Now randomly drop half of the dataframe rows, and add a random element to each time (uniformly sampled from the interval (-0.1,0.1). See what happens with the spectral window and the periodograms.
Can you write Python code for it?

rnd_df = half_df.sample(frac = 0.5).sort_values(axis = 0, by = 'time')
rnd_df['time'] = rnd_df['time'] + np.random.uniform(-0.1,0.1, size = 90)

rnd_spw2 = deemingPSD_fun(times = rnd_df['time'], obs = rnd_df['cst'], freqs = freqs2)
rnd_psd21 = deemingPSD_fun(times = rnd_df['time'], obs = rnd_df['sin1'], freqs = freqs2)
rnd_psd22 = deemingPSD_fun(times = rnd_df['time'], obs = rnd_df['sin2'], freqs = freqs2)


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
The smoothing in frequency domain achieved that the periodograms are now consistent estimators of the spectral density of the true signal in the background. The smoothing in time-domain averaged (with Gaussian weights, so putting more weight on the central time) the close-by periodograms, emphasising those frequencies that are more persistently present in the spectrum. The QPO is now clearly emerging from the background.
However, the time-domain smoothing implies a loss of temporal resolution. Take a look at the QPO using wavelets now. The dataset is big, so choose (based on the previous exercise) a frequency window and a time window such that you can expect to find the QPO. Since we don't know the appropriate scale Q, it is necessary to try a few values.
Can you write Python code for it?

Q = 2000
freq_wavelet = np.arange(800,900, 0.25)
qpo_waveletpsd = wavelet_PSD(t = times[0:200000], h = qpo['flux'][0:200000],
           f0 = freq_wavelet, Q = Q)

plt.rcParams["figure.figsize"] = [8., 8.]
fig = plt.figure(1)

plt.imshow(qpo_waveletpsd, origin = 'lower', aspect = 'auto',
           extent = [times[0], times[200000], freq_wavelet[0], freq_wavelet[-1]])
plt.title('Wavelet PSD')
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [s]')
plt.margins(0.01, 0.05)

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
The time-resolved sliding window periodogram shows well the period change of the Cepheid.
3.2 HIGH-FREQUENCY QUASI-PERIODIC OSCILLATION OF AN X-RAY BINARY
High-frequency QPOs, broad humps concentrated around 1 kHz in the periodogram of the X-ray binary, are exciting and not well understood phenomena of X-ray binaries. Low-frequency QPOs may be related to Lense-Thirring precession of the inner parts of the accretion disk around the compact component, which produces strong X-ray radiation. For the high-frequency QPOs, no such commonly accepted explanations exist. Their characteristic frequency, the center of the hump, tends to move around, and the QPO may sometimes even disappear for some time. This is best to follow by some time-resolved methods.
The data set consists of photon counts, binned in 2^(-11) s long time intervals. The record was very long (almost an hour), so we have an equally-spaced time series of almost 7 million counts. The binning was selected in a way that the sampling is dense enough to find such a high frequency in the spectrum.
First go: we use sliding window spectrograms from matplotlib.mlab.
Can you write Python code for it?

from scipy import fftpack
from matplotlib.mlab import specgram
from astroML.fourier import wavelet_PSD
from scipy import ndimage

qpo = pd.read_csv("./data/qpo.csv")

# The values are photon counts in 2**(-11) s long intervals.
dt = 2**(-11)
# The times of the flux values thus:
times = np.arange(0, np.shape(qpo)[0]*dt, dt, dtype = float)
#qpo_slices = np.linspace(qpo['t'].min(), qpo['t'].max(), 400)
# We wish to compute the PSD in 8 s long windows. The time limits for selection:
qpo_slices = np.arange(0, 426*8, 8, dtype = float)
# The centers of the slices (for plotting):
center_times = np.arange(0, 425*8, 8, dtype = float)+4
# Spacing of the Fourier frequencies in a 8 s window:
df = 1. / 8
# the Fourier frequencies:
ffreqs = df * np.arange(16384)

psd_specgram, freqs_specgram, centers_specgram = specgram(qpo['flux'],
            NFFT = 16384, Fs=2**11, noverlap=16384/2)


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
Visualize first the three spectral windows. What are the differences between them? After inspecting it on the whole test frequency range, enlarge it
(remove the comments from the last two lines).
Can you write Python code for it?

plt.rcParams["figure.figsize"] = [12.0, 9.0]
fig = plt.figure(1)
plt.subplots_adjust(left=0.05, bottom=0.05, right=0.95, top=0.95, wspace=None, hspace=0.35)
ax1 = fig.add_subplot(311)
plt.plot(freqs1, spw1, 'k', alpha=1)
plt.title('90 days, sampling 1/d')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Sp.win.')
plt.margins(.01, .05)
#plt.vlines(F_freqs1[F_freqs1 < 0.1], ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(0., .1)

ax2 = fig.add_subplot(312)
plt.plot(freqs2, spw2, 'k', alpha=1)
plt.title('90 days, sampling 4/d')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Sp.win.')
plt.margins(.01, .05)
#plt.vlines(F_freqs2[F_freqs2 < 0.1], ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(0., .1)

ax3 = fig.add_subplot(313)
plt.plot(freqs3, spw3, 'k', alpha=1)
plt.title('360 days, sampling 1/d')
plt.xlabel('Frequency [1/d]')
plt.ylabel('Sp.win.')
plt.margins(.01, .05)
#plt.vlines(F_freqs3[F_freqs3 < 0.1], ymin = 0, ymax = 1, color='r', linestyle='solid')
#plt.xlim(0., .1)

plt.show()


I am working on the file "Sessions/Session01/Day5/TimeSeries.ipynb".
The first task is:
When omitting the low-frequency terms around f = 0 in the periodograms, the plot of the spectrogram would show just predominant noise.
We know that the periodogram is an inconsistent estimator of the spectral density (that's because its values are distributed exponentially, and for the exponential distribution, its variance is equal to the square of its mean). With a moving average or a kernel smoother, we can make it a consistent one. Also, our time slices might have been too short, and therefore, the periodograms have too high variance. Therefore we do a 2D smoothing.
Can you write Python code for it?

psd_specgram_smoo = ndimage.filters.gaussian_filter(psd_specgram, [8,8], mode='constant', cval = 0)

plt.rcParams["figure.figsize"] = [16., 8.]
fig = plt.figure()
plt.subplots_adjust(left=0.05, bottom=0.05, right=0.95, top=0.95, wspace=None, hspace=0.35)

fig.add_subplot(121)
plt.imshow(psd_specgram[20:,:], origin = 'lower', aspect = 'auto',
           extent = [centers_specgram[0], centers_specgram[-1], freqs_specgram[20], freqs_specgram[-1]])
plt.title('Raw PSD')
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [s]')
plt.margins(0.01, 0.05)

fig.add_subplot(122)
plt.imshow(psd_specgram_smoo[40:,:], origin = 'lower', aspect = 'auto',
           extent = [centers_specgram[0], centers_specgram[-1], freqs_specgram[40], freqs_specgram[-1]])
plt.title('Smoothed PSD')
plt.ylabel('Frequency [Hz]')
plt.xlabel('Time [s]')
plt.margins(0.01, 0.05)

plt.show()


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Challenge Problem) ROC Curves
Earlier today we learned about receiver operating characteristic (ROC) curves as a means of measuring model performance. In brief, ROC curves plot the true positive rate (TPR) as a function of the false positive rate (FPR). Typically, the model that gets closest to TPR = 1 and FPR = 0 is considered best.
Measuring TPR as a function of FPR requires classification probabilities, which are not typically available for $kNN$ models, but this is possible using sklearn (in brief, the probabilities are just the relative fraction of each class within the $k$ neighbors).
Challenge Problem
Plot the ROC curve for each class in the iris data set using a $k = 50$ $kNN$ model and 10-fold cross validation predictions. Be sure to clearly label each of the curves.
Hint 1 - ROC curves only work for 2 class problems. You need to create three 1 vs. all models in this case.
Hint 2 - in cross_val_predict you'll want to set method = 'predict_proba' in order to return class probabilities.
Hint 3 - (sklearn to the rescue again!) sklearn.metrics.roc_curve quickly calculates the FPR and TPR when given class labels and prediction probabilities.
Can you write Python code for it?

from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score

sns.set_style("darkgrid")
fig, ax = plt.subplots()
# loop over each iris type for 1 vs all classification
for iris_type in ['setosa', 'versicolor', 'virginica']:
    y_iris_type = np.zeros(len(iris.target), dtype = int)
    y_iris_type[np.where(iris.target == int(np.where(iris.target_names == iris_type)[0]))] = 1
    y_preds = cross_val_predict(KNeighborsClassifier(n_neighbors=50), iris.data, y_iris_type ,
                                 method = 'predict_proba', cv = 10)
    fpr, tpr, thresholds = roc_curve(y_iris_type, y_preds[:,1])
    auc = roc_auc_score(y_iris_type, y_preds[:,1])
    ax.plot(fpr, tpr, label = r'$\mathrm{{{:s}}}, AUC = {{{:.3f}}}$'.format(iris_type, auc))

ax.legend(loc = 4, fontsize = 18)
ax.set_xlim(-0.03, 0.3)
ax.set_ylim(0.4,1.03)
ax.set_xlabel(r'$\mathrm{False \; Positive \; Rate}$', fontsize = 18)
ax.set_ylabel(r'$\mathrm{True \; Positive \; Rate}$', fontsize = 18)


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Problem 1) Visualize the Iris Data Set
Before building the model, we visualize the iris data using Seaborn. As previously covered, seaborn can be really handy when visualizing $2 < N \lesssim 10$ -dimension data sets.
As a reminder, the Iris data set measures 4 different features of 3 different types of Iris flowers. There are 150 different flowers in the data set.
Note - for those familiar with pandas seaborn is designed to integrate easily and directly with pandas DataFrame objects. In the example below the Iris data are loaded into a DataFrame. iPython notebooks also display the DataFrame data in a nice readable format.
Can you write Python code for it?

import seaborn as sns
iris = sns.load_dataset("iris")
iris


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Problem 1a
Can you identify anything interesting about the Iris data set in Table format?
Solution 1a
The petal length and petal width for setosa and virginica clearly separate flowers belonging to those two classes.
The iris data set is (probably) best visualized using a seaborn pair plot, which shows the pair-wise distributions of all the features included in the data set.
Recall - KDEs are (typically) better than histograms, and color, when used properly conveys a lot of information.
Thus, we set diag_kind = 'kde' and color the data by class, using hue = 'species' in the seaborn pair plot
Can you write Python code for it?

sns.set_style("darkgrid")
sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Problem 1c
Make a scatter plot of sepal length vs. sepal width for the iris data set. Color the points by their respective iris type (i.e. labels).
Can you write Python code for it?

plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')

sns.set_style("whitegrid")
print(iris.feature_names)  # shows that sepal length is first feature and sepal width is second feature

plt.scatter(iris.data[:,0], iris.data[:,1], c = iris.target, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Re-Introduction to Machine Learning:
Classifying the Iris Dataset with K-Nearest Neighbors
Version 0.1

By AA Miller (Northwestern University, Adler Planetarium)
During the first session of the LSSTC DSFP we had an opportunity to work with unsupervised algorithms while clustering flowers in the famous iris data set. Here we will explore the use of the K-nearest neighbors algorithm to actually classify the flowers in the iris data set, while also re-familiarizing ourselves with scikit-learn.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
The confusion matrix reveals that most of the virginica misclassifications are predicted to be versicolor. The iris data set features 50 members of each class, but problems with class imbalance are more difficult to visualize in this way. Thus, sometimes it's helpful to normalize each value relative to the true number of sources.
Better still, we can visualize the confusion matrix in a readily digestible fashion. First - let's normalize the confusion matrix.
Problem 2e
Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?

normalized_cm = cm.astype('float')/cm.sum(axis = 1)[:,np.newaxis]

normalized_cm


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
These results are almost identical to the training classifications. However, we have cheated!
We are evaluating the accuracy of the model (98% in this case) using the same data that defines the model. Thus, what we have really evaluated here is the training error.
The true test of a good model is the generalization error: how accurate are the model predictions on new data?
We can approximate the generalization error (under the assumption that new observations are similar to the training set - an often poor assumption in astronomy...) via cross validation (CV).
In brief, CV provides predictions for training set objects that are withheld from the model construction in order to avoid "double-dipping." The most common forms of CV iterate over all sources in the training set.
Using cross_val_predict, we can obtain CV predictions for each source in the training set.
from sklearn.model_selection import cross_val_predict
CVpreds = cross_val_predict(sklearn.model(), X, y)

where sklearn.model() is the desired model, X is the feature array, and y is the label array.
Hint - if you are running an old (< 0.18) version of scikit-learn you may need to run conda update.
Problem 2b
Produce cross-validation predictions for the iris dataset and a $k$NN model with 5 neighbors.
Plot the resulting classifications, as above, and estimate the accuracy of the model as applied to new data.
How does this accuracy compare to a $k$NN model with 50 neighbors?
Can you write Python code for it?

from sklearn.model_selection import cross_val_predict

CVpreds = cross_val_predict(# complete
plt.figure()
plt.scatter( # complete
print("The accuracy of the kNN = 5 model is ~{:.4}".format( # complete

CVpreds50 = cross_val_predict( # complete

print("The accuracy of the kNN = 50 model is ~{:.4}".format( # complete

from sklearn.model_selection import cross_val_predict

CVpreds = cross_val_predict(KNeighborsClassifier(n_neighbors=5), iris.data, iris.target)
plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1],
            c = preds, cmap = "viridis", s = 30, edgecolor = "None")
print("The accuracy of the kNN = 5 model is ~{:.4}".format( sum(CVpreds == iris.target)/len(CVpreds) ))

CVpreds50 = cross_val_predict(KNeighborsClassifier(n_neighbors=50), iris.data, iris.target)

print("The accuracy of the kNN = 50 model is ~{:.4}".format( sum(CVpreds50 == iris.target)/len(CVpreds50) ))


I am working on the file "Sessions/Session02/Day1/ReIntroToMachLearnSolutions.ipynb".
The first task is:
Wow! The 5-neighbor model only misclassifies 2 of the flowers via CV. The 50-neighbor model misclassifies 17 flowers. While this overall accuracy is still relatively high, it would be useful to understand which flowers are being misclassified.
Problem 2c
Calculate the accuracy for each class in the iris set, as determined via CV for the $k$ = 50 model.
Which class is most accurate? Does this meet your expectations?
Can you write Python code for it?

# complete

for iris_type in range(3):
    iris_acc = sum( (CVpreds50 == iris_type) & (iris.target == iris_type)) / sum(iris.target == iris_type)

    print("The accuracy for the {:s} class is ~{:.4f}".format(iris.target_names[iris_type], iris_acc))


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Challenge Problem) ROC Curves
Earlier today we learned about receiver operating characteristic (ROC) curves as a means of measuring model performance. In brief, ROC curves plot the true positive rate (TPR) as a function of the false positive rate (FPR). Typically, the model that gets closest to TPR = 1 and FPR = 0 is considered best.
Measuring TPR as a function of FPR requires classification probabilities, which are not typically available for $kNN$ models, but this is possible using sklearn (in brief, the probabilities are just the relative fraction of each class within the $k$ neighbors).
Challenge Problem
Plot the ROC curve for each class in the iris data set using a $k = 50$ $kNN$ model and 10-fold cross validation predictions. Be sure to clearly label each of the curves.
Hint 1 - ROC curves only work for 2 class problems. You need to create three 1 vs. all models in this case.
Hint 2 - in cross_val_predict you'll want to set method = 'predict_proba' in order to return class probabilities.
Hint 3 - (sklearn to the rescue again!) sklearn.metrics.roc_curve quickly calculates the FPR and TPR when given class labels and prediction probabilities.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Normalization makes it easier to compare the classes when there is class imbalance.
We can visualize the confusion matrix using imshow() within pyplot. A colorbar and axes labels will be needed.
Problem 2f
Plot the confusion matrix. Be sure to label each axis.
Hint - you might find the sklearn confusion matrix tutorial helpful for making a nice plot.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Problem 1a
Can you identify anything interesting about the Iris data set in Table format?
Solution 1a
Type answer here
The iris data set is (probably) best visualized using a seaborn pair plot, which shows the pair-wise distributions of all the features included in the data set.
Recall - KDEs are (typically) better than histograms, and color, when used properly conveys a lot of information.
Thus, we set diag_kind = 'kde' and color the data by class, using hue = 'species' in the seaborn pair plot
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Problem 2) Supervised Machine Learning
Supervised machine learning aims to predict a target class or produce a regression result based on the location of labelled sources (i.e. the training set) in the multidimensional feature space. The "supervised" comes from the fact that we are specifying the allowed outputs from the model. Using the training set labels, we can estimate the accuracy of each model we generate. (though there are generally important caveats about generalization, which we will explore in further detail later).
We will begin with a simple, but nevertheless, elegant algorithm for classification and regression: $k$-nearest-neighbors ($k$NN). In brief, the classification or regression output is determined by examining the $k$ nearest neighbors in the training set, where $k$ is a user defined number. Typically, though not always, calculated distances are Euclidean, and the final classification is assigned to whichever class has a plurality within the $k$ nearest neighbors (in the case of regression, the average of the $k$ neighbors is the output from the model).
Note - you should be worried about how to select the number $k$. We will re-visit this in further detail on Friday.
In scikit-learn the KNeighborsClassifer algorithm is implemented as part of the sklearn.neighbors module.
    from sklearn.neighbors import KNeighborsClassifier
    KNNclf = KNeighborsClassifier()

See the docs to learn about the default options for KNeighborsClassifer.
Problem 2a
Fit two different $k$NN models to the iris data, one with 3 neighbors and one with 10 neighbors. Plot the resulting class predictions in the sepal length-sepal width plane (same plot as above).
How do the results compare to the true classifications?
Is there any reason to be suspect of this procedure?
Hint - recall that sklearn models are fit using the .fit(X,y) method, where X is the training data, or feature array, with shape [n_samples, n_features] and y is the targets, or label array, with shape [n_samples].
Hint 2 - after you have...
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

KNNclf = KNeighborsClassifier( # complete
preds = # complete
plt.figure()
plt.scatter( # complete

KNNclf = KNeighborsClassifier( # complete
preds = # complete
plt.figure()
plt.scatter( # complete


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Re-Introduction to Machine Learning:
Classifying the Iris Dataset with K-Nearest Neighbors
Version 0.1

By AA Miller (Northwestern University, Adler Planetarium)
During the first session of the LSSTC DSFP we had an opportunity to work with unsupervised algorithms while clustering flowers in the famous iris data set. Here we will explore the use of the K-nearest neighbors algorithm to actually classify the flowers in the iris data set, while also re-familiarizing ourselves with scikit-learn.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
The confusion matrix reveals that most of the virginica misclassifications are predicted to be versicolor. The iris data set features 50 members of each class, but problems with class imbalance are more difficult to visualize in this way. Thus, sometimes it's helpful to normalize each value relative to the true number of sources.
Better still, we can visualize the confusion matrix in a readily digestible fashion. First - let's normalize the confusion matrix.
Problem 2e
Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/ReIntroToMachineLearning.ipynb".
The first task is:
Wow! The 5-neighbor model only misclassifies 2 of the flowers via CV. The 50-neighbor model misclassifies 17 flowers. While this overall accuracy is still relatively high, it would be useful to understand which flowers are being misclassified.
Problem 2c
Calculate the accuracy for each class in the iris set, as determined via CV for the $k$ = 50 model.
Which class is most accurate? Does this meet your expectations?
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
DBSCAN
DBSCAN is a highly popular clustering algorithm that does not require the number of clusters to be specified. Instead, it defines clusters according to two parameters: minPts, the minimum number of points necessary for a cluster, and $\epsilon$, a distance measure. scikit-learn has an implementation of the algorithm as part of the cluster module: DBSCAN. $\epsilon$ and minPts are set by the implementation parameters eps and min_samples, respectively.
Problem: Cluster the SDSS color data with DBSCAN and see what effect changing the fine tuning parameters has on the results. How does DBSCAN compare to $k$-means?
Note: DBSCAN labels outliers as -1 and so plt.scatter() will plot all these points as the same color.
Can you write Python code for it?

from sklearn.cluster import DBSCAN

dbs = DBSCAN( # complete
dbs.fit( # complete

# Class numbers
labels = dbs.labels_
unique_labels = np.unique(labels)
count = [(labels == lab).sum() for lab in unique_labels]

plt.figure()
for i in range(len(unique_labels)):
    plt.plot( # complete

plt.xlabel('u - g')
plt.ylabel('g - r')


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
Different implementations
scikit-learn also has an alternate implementation of the $k$-means algorithm, MiniBatchKMeans which is faster but at the expense of accuracy.
Problem: See what the performance differences between the two implementations are and also whether different results are obtained.
Hint - The iPython %timeit command is very good for performance evaluations.
Can you write Python code for it?

from sklearn.cluster import MiniBatchKMeans

%timeit MiniBatchKMeans( # complete

%timeit KMeans( # complete

# complete

plt.figure()
plt.scatter(# complete
plt.xlabel('u - g')
plt.ylabel('g - r')


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
Dimensional reduction
There are a number of unsupervised learning algorithms that can produce a low-dimensional representation of a high-dimensional data set that preserves topological information, i.e., clustering. The state-of-the-art is the t-distributed stochastic neighborhood embedding (t-SNE).
Problem: Fit the data set with a t-SNE (this may take some time as the algorithm is computationally intensive)
Note: The TSNE class within the sklearn.manifold modules uses a slightly different syntax from the other scikit-learn implementations. The call is .fit_transform(X).
Can you write Python code for it?

from sklearn.manifold import TSNE

tsne = TSNE( # complete
# complete

plt.figure()
plt.scatter( # complete


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
How many clusters?
One of the issues with the $k$-means algorithm is that the number of clusters to use, $k$, is a parameter. There are a number of ways of determining this for a given data set. One of them is to calculate the mean silhouette for (a subset of) the data for a variety of $k$ values and pick the value closest to 1.
Problem: Calculate the mean silhouette value for $k$ in the range 2 to 10 and select the most appropriate one.
Hint - The silhouette_score() class within the metrics module may prove useful.
Note: This will take some time to run
Can you write Python code for it?

from sklearn import metrics

# Select a random subset of the data
N_small = 100
dfSample = df[ # complete

ss = []
nn = [2,3,5,10]

for n in # complete
    Kcluster = Kmeans( # complete

    ss.append(metrics.silhouette_score( # complete

plt.figure()
plt.scatter(# complete
plt.xlabel('k')
plt.ylabel('Mean silhouette')


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
Kernel density estimation
Kernel density estimation fits a kernel to each data point and then estimates the pdf from the sum of all of these.
Problem: Fit the data with a standard KDE. How des it compare against histograms?
Hint - The KernelDensity class in the sklearn.neighbors module may be useful.
Can you write Python code for it?

from sklearn.neighbors import KernelDensity

kde = KernelDensity( # complete

# complete

plt.figure()
plt.scatter( # complete


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
Revision: scikit-learn
You're going to use the Python package scikit-learn to attack a number of problems which are best handled with unsupervised machine learning algorithms (models). In scikit-learn, different models are accessed via the various modules with user-defined tuning parameters. Data (features) for the models are stored in a 2D numpy array, X, with each row representing an individual object and each column a specific feature. Unsupervised models are invoked by calling .fit(X). Once a model has fit a data set, it can be used to generate observations for a new data set Xnew by calling .predict(Xnew). Any other method calls are described in the documentation.
$k$-means clustering
You were introduced to the $k$-means clustering algorithm in the first DSFP session in August 2016 (and in the lecture after lunch today) and its application to the gold standard Iris dataset. We're going to revisit a bit of that material first but moving straight away to an astronomical data set from SDSS: SDSS_colors.csv.
Problem: So let's start with loading the data set and plotting the axes against each other to get a feel for what the data looks like (you should always do this as a matter of course when getting a new (or not so new) data set).
Can you write Python code for it?

import pandas as pd

df = pd.read_csv("SDSS_colors.csv")
# complete


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
So far we've just been looking at one dimension but now let's consider a two-dimensional case: the density of galaxies in terms of their radius (petroRad_r_kpc) and magnitude (absPetroMag_r).
Problem: Repeat finding the optimal bandwidth via cross-validation and plot the resulting pdf.
Note: The method score_samples gives results as $\log$.
Can you write Python code for it?

data = # complete

# use grid search cross-validation to optimize the bandwidth
params = {'bandwidth': # complete
grid = GridSearchCV( # complete
# complete

x = np.arange(.0, 40, 1)
y = np.arange(-24, -15, 0.2)
X, Y = np.meshgrid(x, y)
XY = np.array([X.flatten(), Y.flatten()]).transpose()
pdf = # complete

# Use imshow to make a color density plot
plt.imshow(# complete, interpolation='none', cmap=pl.cm.jet, origin='lower',clip_on=True))


I am working on the file "Sessions/Session02/Day1/Unsupervised machine learning.ipynb".
The first task is:
Unsupervised machine learning problems
By Matthew J. Graham (California Institute of Technology)
Acknowledgements to A. A. Miller and G. Cabrera.
(c) 2017, California Institute of Technology
Version 0.1
Given a data set of point sources, a typical machine learning problem is how many different kinds of objects can we distinguish in it? This is best answered with unsupervised algorithms, which aim to identify hidden structures in data sets. Of course, interpretation of any such structures, i.e., whether they actually indicate physically meaningful correlations or relationships, is left to the domain expert.
Can you write Python code for it?

import numpy as np
import matplot.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Challenge Problem: Interpreting Results
Earlier today, we talked about interpreting machine learning models. Let's see how you would go about this in practice.

Repeat your classification with a logistic regression model.
Is the logistic regression model easier or harder to interpret? Why?
Assume you're interested in which features are the most relevant to your classification (because they might have some bearing on the underlying physics). Would you do your classification on the original features or the PCA transformation? Why?
Change the subset of parameters used in the logistic regression models. Look at the weights. Do they change? How? Does that affect your interpretability?
Can you write Python code for it?

from sklearn.linear_model import LogisticRegressionCV

lr =


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Choosing The Right Scoring Function
As a standard, the algorithms in scikit-learn use accuracy to score results. The accuracy is basically the raw fraction of correctly classified samples in your validation or test set.
Question: Is this scoring function always the best method to use? Why (not)? Can you think of alternatives to use?
Let's make a heavily biased data set:
Can you write Python code for it?

# all stars
star_ind = np.argwhere(y == b"STAR").T[0]
# all galaxies
galaxy_ind = np.argwhere(y == b"GALAXY").T[0]

np.random.seed(100)
# new array with much fewer stars
star_ind_new = np.random.choice(star_ind, replace=False, size=int(len(star_ind)/80.0))

X_new = np.vstack((X[galaxy_ind], X[star_ind_new]))
y_new = np.hstack((y[galaxy_ind], y[star_ind_new]))