I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Comparing Algorithms
So far, we've just picked PCA because it's common. But what if there's a better algorithm for dimensionality reduction out there for our problem? Or what if you'd want to compare random forests to other classifiers?
In this case, your best option is to split off a separate validation set, perform cross-validation for each algorithm separately, and then compare the results using hold-out cross validation and your validation set (Note: Do not use your test set for this! Your test set is only used for your final error estimate!)
Doing CV across algorithms is difficult, since the KFoldCV object needs to know which parameters belong to which algorithms, which is difficult to do.
Exercise 7: Pick an algorithm from the manifold learning library in scikit-learn, cross-validate a random forest for both, and compare the performance of both.
Important: Do not choose t-SNE. The reason is that t-SNE does not generalize to new samples! This means while it's useful for data visualization, you cannot train a t-SNE transformation (in the scikit-learn implementation) on one part of your data and apply it to another!
Can you write Python code for it?

# First, let's redo the train-test split to split the training data
# into training and hold-out validation set

# make a list of name-estimator tuples
estimators =

# instantiate the pipeline
pipe =

# make a dictionary of parameters
params =

# perform the grid search
grid_search =

# complete the print functions
print("Best score: ")
print("Best parameter set: " )
print("Validation score for model with PCA: ")

# Now repeat the same procedure with the second algorithm you've picked.


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Even More Challenging Challenge Problem: Implementing Your Own Estimator
Sometimes, you might want to use algorithms, for example for feature engineering, that are not implemented in scikit-learn. But perhaps these transformations still have free parameters to estimate. What to do?
scikit-learn classes inherit from certain base classes that make it easy to implement your own objects. Below is an example I wrote for a machine learning model on time series, where I wanted to re-bin the time series in different ways and and optimize the rebinning factor with respect to the classification afterwards.
Can you write Python code for it?

from sklearn.base import BaseEstimator, TransformerMixin

class RebinTimeseries(BaseEstimator, TransformerMixin):

    def __init__(self, n=4, method="average"):

        """
        Initialize hyperparameters

        :param n: number of samples to bin
        :param method: "average" or "sum" the samples within a bin?
        :return:
        """

        self.n = n ## save number of bins to average together
        self.method = method

        return

    def fit(self,X):
        """
        I don't really need a fit method!
        """

        ## set number of light curves (L) and
        ## number of samples per light curve (k)
        return self

    def transform(self, X):
        self.L, self.K = X.shape

        ## set the number of binned samples per light curve
        K_binned = int(self.K/self.n)

        ## if the number of samples in the original light curve
        ## is not divisible by n, then chop off the last few samples of
        ## the light curve to make it divisible
        #print("X shape: " + str(X.shape))

        if K_binned*self.n < self.K:
            X = X[:,:self.n*K_binned]

        ## the array for the new, binned light curves
        X_binned = np.zeros((self.L, K_binned))

        if self.method in ["average", "mean"]:
            method = np.mean
        elif self.method == "sum":
            method = np.sum
        else:
            raise Exception("Method not recognized!")

        #print("X shape: " + str(X.shape))
        #print("L: " + str(self.L))
        for i in xrange(self.L):
            t_reshape = X[i,:].reshape((K_binned, self.n))
            X_binned[i,:] = method(t_reshape, axis=1)

        return X_binned

    def predict(self, X):
        pass

    def score(self, X):
        pass

    def fit_transform(self, X, y=None):

        self.fit(X)
        X_binned = self.transform(X)

        return X_binned


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Exercise 10: Run a logistic regression classifier on this data, for a very low regularization (0.0001) and a
very large regularization (10000) parameter. Print the accuracy and a confusion matrix of the results for each run. How many mis-classified samples are in each? Where do the mis-classifications end up? If you were to run a cross validation on this, could you be sure to get a good model? Why (not)?
As part of this exercise, you should plot a confusion matrix. A confusion matrix takes the true labels and the predicted labels, and then plots a grid for all samples where true labels and predicted labels match versus do not match. You can use the scikit-learn function confusion_matrix to create one. pyplot.matshow is useful for plotting it, but just printing it on the screen works pretty well, too (at least for the two classes considered here).
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score

X_train2, X_test2, y_train2, y_test2 = train_test_split(X_new, y_new,
                                                        test_size = 0.3,
                                                        random_state = 20)

C_all =

for C in C_all:

    lr =

    # ... insert code here ...

    # make predictions for the validation set
    y_pred =

    # print accuracy score for this regularization:

    # make and print a confusion matrix
    cm =


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Exercise 11: Take a look at the metrics implemented for model evaluation in scikit-learn, in particular the different versions of the F1 score. Is there a metric that may be more suited to the task above? Which one?
Hint: Our imbalanced class, the one we're interested in, is the STAR class. Make sure you set the keyword pos_label in the f1_score function correctly.
Can you write Python code for it?

for C in C_all:

    lr =

    # ... insert code here ...

    # predict the validdation set
    y_pred = lr.predict(X_test2)

    # print both accuracy and F1 score for comparison:

    # create and plot a confusion matrix:
    cm =


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Exercise 1: Visualize this data set. What representation is most appropriate, do you think?
Exercise 2: Let's now do some machine learning. In this exercise, you are going to use a random forest classifier to classify this data set. Here are the steps you'll need to perform:
* Split the column with the classes (stars and galaxies) from the rest of the data
* Cast the features and the classes to numpy arrays
* Split the data into a test set and a training set. The training set will be used to train the classifier; the test set we'll reserve for the very end to test the final performance of the model (more on this on Friday). You can use the scikit-learn function test_train_split for this task
* Define a RandomForest object from the sklearn.ensemble module. Note that the RandomForest class has three parameters:
    - n_estimators: The number of decision trees in the random forest
    - max_features: The maximum number of features to use for the decision trees
    - min_samples_leaf: The minimum number of samples that need to end up in a terminal leaf (this effectively limits the number of branchings each tree can make)
* We'll want to use cross-validation to decide between parameters. You can do this with the scikit-learn class GridSearchCV. This class takes a classifier as an input, along with a dictionary of the parameter values to search over.
In the earlier lecture, you learned about four different types of cross-validation:
* hold-out cross validation, where you take a single validation set to compare your algorithm's performance to
* k-fold cross validation, where you split your training set into k subsets, each of which holds out a different portion of the data
* leave-one-out cross validation, where you have N different subsets, each of which leaves just one sample as a validation set
* random subset cross validation, where you pick a random subset of your data points k times as your validation set.
Exercise 2a: Which of the four algorithms is most appropriate here? And why?
Answer: In this case, k-fold...
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# set the random state
rs = 23

# extract feature names, remove class

# cast astropy table to pandas and then to a numpy array, remove classes

# our classes are the outcomes to classify on

# let's do a split in training and test set:
# we'll leave the test set for later.

# instantiate the random forest classifier:

# do a grid search over the free random forest parameters:
pars =

grid_results =


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Exercise 2c: Take a look at the different validation scores for the different parameter combinations. Are they very different or are they similar?
It looks like the scores are very similar, and have very small variance between the different cross validation instances. It can be useful to do this kind of representation to see for example whether there is a large variance in the cross-validation results.
Cross-validating Multiple Model Components
In most machine learning applications, your machine learning algorithm might not be the only component having free parameters. You might not even be sure which machine learning algorithm to use!
For demonstration purposes, imagine you have many features, but many of them might be correlated. A standard dimensionality reduction technique to use is Principal Component Analysis.
Exercise 4: The number of features in our present data set is pretty small, but let's nevertheless attempt to reduce dimensionality with PCA. Run a PCA decomposition in 2 dimensions and plot the results. Colour-code stars versus galaxies. How well do they separate along the principal components?
Hint: Think about whether you can run PCA on training and test set separately, or whether you need to run it on both together before doing the train-test split?
Can you write Python code for it?

from sklearn.decomposition import PCA

# instantiate the PCA object
pca =

# fit and transform the samples:
X_pca =

# make a plot of the PCA components colour-coded by stars and galaxies
fig, ax = plt.subplots(1, 1, figsize=(12,8))


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
First, we're going to need some data. We'll work with the star-galaxy data from the first session. This uses the astroquery package and then queries the top 10000 observations from SDSS (see this exercise for more details):
Can you write Python code for it?

# execute this line:
from astroquery.sdss import SDSS

TSquery = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             ORDER BY p.objid ASC
               """
SDSSts = SDSS.query_sql(TSquery)
SDSSts


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Here are the important things about writing transformer objects for use in scikit-learn:
* The class must have the following methods:
    - fit: fit your training data
    - transform: transform your training data into the new representation
    - predict: predict new examples
    - score: score predictions
    - fit_transform is optional (I think)
* The __init__ method only sets up parameters. Don't put any relevant code in there (this is convention more than anything else, but it's a good one to follow!)
* The fit method is always called in a Pipeline object (either on its own or as part of fit_transform). It usually modifies the internal state of the object, so returning self (i.e. the object itself) is usually fine.
* For transformer objects, which don't need scoring and prediction methods, you can just return pass as above.
Exercise 8: Last time, you learned that the SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag} - \mathtt{cmodelMag} \gt 0.145,$$
sources that satisfy this criteria are considered galaxies.

Implement an object that takes $\mathtt{psfMag}$ and $\mathtt{cmodelMag}$ as inputs and has a free parameter p that sets the value above which a source is considered a galaxy.
Implement a transform methods that returns a single binary feature that is one if $$\mathtt{psfMag} - \mathtt{cmodelMag} \gt p$$ and zero otherwise.
Add this feature to your optimized set of features consisting of either the PCA or your alternative representation, and run a random forest classifier on both. Run a CV on all components involved.

Hint: $\mathtt{psfMag}$ and $\mathtt{cmodelMag}$ are the first and the last column in your feature vector, respectively.
Hint: You can use FeatureUnion to combine the outputs of two transformers in a single data set. (Note that using pipeline with all three will chain them, rather than compute the feature union, followed by a classifier). You can input your FeatureUnion object into Pipeline.
Can you write Python code for it?

class PSFMagThreshold(BaseEstimator, TransformerMixin):

    def __init__(self, p=1.45,):

    def fit(self,X):

    def transform(self, X):

    def predict(self, X):

    def score(self, X):

    def fit_transform(self, X, y=None):


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Now let's make a feature set that combines this feature with the PCA features:
Can you write Python code for it?

from sklearn.pipeline import FeatureUnion

transformers =

feat_union =

X_transformed =


I am working on the file "Sessions/Session02/Day2/ModelSelection_Exercise.ipynb".
The first task is:
Now we can build the pipeline:
Can you write Python code for it?

# combine the transformers
transformers =

# make the feature union
feat_union =

# combine estimators for the pipeline
estimators =

# define the pipeline object
pipe_c =

# make the parameter set
params =

# perform the grid search
grid_search_c =

# complete the print statements:
print("Best score: ")
print("Best parameter set: ")
print("Validation score: ")


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Answer 1: Whether the model is easier or harder to interpret depends on what type of interpretability is desired. If you are interested in how the features influence the classification, the logistic regression model is easier to interpret: because random forests is an ensemble method, it's very hard to understand in detail how a prediction comes about (since the individual decision trees may have very different structures). However, for very large feature spaces with complicated, engineered features, your linear model (the logistic regression model) loses interpretability in how the parameters affect the outcomes just as much.
Answer 2: The more feature engineering you do, the harder it will be to interpret the results. The PCA features are a linear transformation of your original eight features. But what do they mean in physical terms? Who knows?
Can you write Python code for it?

# let's leave out the first parameter and see whether the coefficients change:
lr.fit(X_train[:,1:], y_train)
lr.coef_


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Choosing The Right Scoring Function
As a standard, the algorithms in scikit-learn use accuracy to score results. The accuracy is basically the raw fraction of correctly classified samples in your validation or test set.
Question: Is this scoring function always the best method to use? Why (not)? Can you think of alternatives to use?
Let's make a heavily biased data set:
Can you write Python code for it?

# all stars
star_ind = np.argwhere(y == b"STAR").T[0]
# all galaxies
galaxy_ind = np.argwhere(y == b"GALAXY").T[0]

np.random.seed(100)
# new array with much fewer stars
star_ind_new = np.random.choice(star_ind, replace=False, size=int(len(star_ind)/80.0))

X_new = np.vstack((X[galaxy_ind], X[star_ind_new]))
y_new = np.hstack((y[galaxy_ind], y[star_ind_new]))


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Exercise 10: Run a logistic regression classifier on this data, for a very low regularization (0.0001) and a very large regularization (10000) parameter. Print the accuracy and a confusion matrix of the results for each run. How many mis-classified samples are in each? Where do the mis-classifications end up? If you were to run a cross validation on this, could you be sure to get a good model? Why (not)?
Hint: Our imbalanced class, the one we're interested in, is the
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score

X_train2, X_test2, y_train2, y_test2 = train_test_split(X_new, y_new,
                                                        test_size = 0.3,
                                                        random_state = 20)

C_all = [0.0001, 10000]

for C in C_all:

    lr = LogisticRegression(penalty='l2', C=C)

    lr.fit(X_train2, y_train2)
    y_pred = lr.predict(X_test2)

    print("The accuracy score for C = %i is %.4f"%(C, accuracy_score(y_test2, y_pred)))

    cm = confusion_matrix(y_test2, y_pred, labels=np.unique(y))
    print(cm)


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Exercise 11: Take a look at the metrics implemented for model evaluation in scikit-learn, in particular the different versions of the F1 score. Is there a metric that may be more suited to the task above? Which one?
Can you write Python code for it?

for C in C_all:

    lr = LogisticRegression(penalty='l2', C=C)

    lr.fit(X_train2, y_train2)
    y_pred = lr.predict(X_test2)

    print("The accuracy score for C = %i is %.4f"%(C, accuracy_score(y_test2, y_pred)))
    print("The F1 score for C = %.5f is %.4f"%(C, f1_score(y_test2, y_pred,
                                                         pos_label=b"STAR",
                                                         average="binary")))

    cm = confusion_matrix(y_test2, y_pred, labels=np.unique(y))
    print(cm)


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Exercise 1: Visualize this data set. What representation is most appropriate, do you think?
Exercise 2: Let's now do some machine learning. In this exercise, you are going to use a random forest classifier to classify this data set. Here are the steps you'll need to perform:
* Split the column with the classes (stars and galaxies) from the rest of the data
* Cast the features and the classes to numpy arrays
* Split the data into a test set and a training set. The training set will be used to train the classifier; the test set we'll reserve for the very end to test the final performance of the model (more on this on Friday). You can use the scikit-learn function test_train_split for this task
* Define a RandomForest object from the sklearn.ensemble module. Note that the RandomForest class has three parameters:
    - n_estimators: The number of decision trees in the random forest
    - max_features: The maximum number of features to use for the decision trees
    - min_samples_leaf: The minimum number of samples that need to end up in a terminal leaf (this effectively limits the number of branchings each tree can make)
* We'll want to use cross-validation to decide between parameters. You can do this with the scikit-learn class GridSearchCV. This class takes a classifier as an input, along with a dictionary of the parameter values to search over.
In the earlier lecture, you learned about four different types of cross-validation:
* hold-out cross validation, where you take a single validation set to compare your algorithm's performance to
* k-fold cross validation, where you split your training set into k subsets, each of which holds out a different portion of the data
* leave-one-out cross validation, where you have N different subsets, each of which leaves just one sample as a validation set
* random subset cross validation, where you pick a random subset of your data points k times as your validation set.
Exercise 2a: Which of the four algorithms is most appropriate here? And why?
Answer: In this case, k-fold...
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
from sklearn.grid_search import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# set the random state
rs = 23  # we are in Chicago after all

# extract feature names, remove class
feats = list(SDSSts.columns)
feats.remove('class')

# cast astropy table to pandas, remove classes
X = np.array(SDSSts[feats].to_pandas())

# our classes are the outcomes to classify on
y = np.array(SDSSts['class'])

# let's do a split in training and test set:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = rs)
# we'll leave the test set for later.

# instantiate the random forest classifier:
RFmod = RandomForestClassifier()

# do a grid search over the free random forest parameters:
pars = {"n_estimators": [10, 100, 300],
        "max_features": [1, 3, 7],
        "min_samples_leaf": [1,10]}

grid_results = GridSearchCV(RandomForestClassifier(),
                            pars,
                            cv = 5)

grid_results.fit(X_train, y_train)


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Exercise 2c: Take a look at the different validation scores for the different parameter combinations. Are they very different or are they similar?
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Exercise 5: Re-do the classification on the PCA components instead of the original features.
Can you write Python code for it?

# Train PCA on training data set
X_pca_train = pca.fit_transform(X_train)

# apply to test set
X_pca_test = pca.transform(X_test)

# we'll leave the test set for later.

# instantiate the random forest classifier:
RFmod = RandomForestClassifier()

# do a grid search over the free random forest parameters:
pars = {"n_estimators": [10, 100, 300],
        "max_features": [1, 2],
        "min_samples_leaf": [1,10]}

grid_results = GridSearchCV(RandomForestClassifier(),
                            pars,
                            cv = 5)

grid_results.fit(X_pca_train, y_train)

grid_results.best_score_


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
First, we're going to need some data. We'll work with the star-galaxy data from the first session. This uses the astroquery package and then queries the top 10000 observations from SDSS (see this exercise for more details):
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Here are the important things about writing transformer objects for use in scikit-learn:
* The class must have the following methods:
    - fit: fit your training data
    - transform: transform your training data into the new representation
    - predict: predict new examples
    - score: score predictions
    - fit_transform is optional (I think)
* The __init__ method only sets up parameters. Don't put any relevant code in there (this is convention more than anything else, but it's a good one to follow!)
* The fit method is always called in a Pipeline object (either on its own or as part of fit_transform). It usually modifies the internal state of the object, so returning self (i.e. the object itself) is usually fine.
* For transformer objects, which don't need scoring and prediction methods, you can just return pass as above.
Exercise 8: Last time, you learned that the SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag} - \mathtt{cmodelMag} \gt 0.145,$$
sources that satisfy this criteria are considered galaxies.

Implement an object that takes $\mathtt{psfMag}$ and $\mathtt{cmodelMag}$ as inputs and has a free parameter s that sets the value above which a source is considered a galaxy.
Implement a transform methods that returns a single binary feature that is one if $$\mathtt{psfMag} - \mathtt{cmodelMag} \gt p$$ and zero otherwise.
Add this feature to your optimized set of features consisting of either the PCA or your alternative representation, and run a random forest classifier on both. Run a CV on all components involved.

Hint: $\mathtt{psfMag}$ and $\mathtt{cmodelMag}$ are the first and the last column in your feature vector, respectively.
Hint: You can use FeatureUnion to combine the outputs of two transformers in a single data set. (Note that using pipeline with all three will chain them, rather than compute the feature union, followed by a classifier). You can input your FeatureUnion object into Pipeline.
Can you write Python code for it?

class PSFMagThreshold(BaseEstimator, TransformerMixin):

    def __init__(self, p=1.45,):

        """
        Initialize hyperparameters

        Parameters
        ----------
        p : float
            The threshold for the magnitude - model magnitude
        """

        self.p = p # store parameter in object

        return

    def fit(self,X):
        """
        I don't really need a fit method!
        """
        return self

    def transform(self, X):

        # extract relevant columns
        psfmag = X[:,0]
        c_psfmag = X[:,-1]

        # compute difference
        d_psfmag = psfmag - c_psfmag

        # make a 1D array of length N
        X_new = np.zeros(X.shape[0])

        X_new[d_psfmag > self.p] = 1.0

        # IMPORTANT: Your output vector must be a COLUMN vector
        # You can achieve this with the numpy function atleast_2D()
        # and the numpy function transpose()
        return np.atleast_2d(X_new).T

    def predict(self, X):
        pass

    def score(self, X):
        pass

    def fit_transform(self, X, y=None):

        self.fit(X)
        X_new = self.transform(X)

        return X_new

pt = PSFMagThreshold(p=1.45)
X_pt = pt.fit_transform(X)


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Now let's make a feature set that combines this feature with the PCA features:
Can you write Python code for it?

from sklearn.pipeline import FeatureUnion

transformers = [("pca", PCA(n_components=2)),
                ("pt", PSFMagThreshold(p=1.45))]

feat_union = FeatureUnion(transformers)

X_transformed = feat_union.fit_transform(X_train)


I am working on the file "Sessions/Session02/Day2/ModelSelection_ExerciseSolutions.ipynb".
The first task is:
Now we can build the pipeline:
Can you write Python code for it?

# combine the
transformers = [("pca", PCA()),
                ("pt", PSFMagThreshold(p=1.45))]

feat_union = FeatureUnion(transformers)

estimators = [("feats", feat_union),
        ("clf", RandomForestClassifier())]

pipe_c = Pipeline(estimators)

# make the parameter set
params = dict(feats__pca__n_components=[2, 4, 6, 8],
              feats__pt__p=[0.5, 0.9, 1.45, 2.0],
              clf__n_estimators=[10, 100, 300],
              clf__min_samples_leaf=[1,10])

# perform the grid search
grid_search_c = GridSearchCV(pipe_c, param_grid=params)
grid_search_c.fit(X_train_new, y_train_new)

# print validation score
print("Best score: " + str(grid_search_c.best_score_))
print("Best parameter set: " + str(grid_search_c.best_params_))
print("Validation score: " + str(grid_search_c.score(X_val, y_val)))


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Before starting, take a moment to review the results above and predict how they will change...
Exercise: Uncertain Heteroscedastic Errors
In this exercise, the errors reported for each point are uncertain: the true errors are different from the reported errors by an unknown constant scale factor.  This introduces a new parameter, the scale factor, that we will call s:
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Combine all the walkers into a single array of generated samples.  You could trim off some initial "burn-in" here, but I don't recommend it (see lecture notes):
Can you write Python code for it?

samples = sampler.chain[:, :, :].reshape((-1, ndim))

samples.shape


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Combine the priors and likelihood to calculate the log of the posterior $\log P(\theta|D)$:
Can you write Python code for it?

def log_posterior(theta):
    lp = log_prior(theta)
    if np.isfinite(lp):
        # Only calculate likelihood if necessary.
        lp += log_likelihood(theta)
    return lp


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Consecutive samples are certainly correlated, but the ensemble of all samples should have the correct distribution for inferences.  Make a corner plot to see this distribution.
Can you write Python code for it?

fig = corner.corner(samples, labels=['m', 'b'], truths=[truth['m'], truth['b']])


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Defined the likelihood $P(D|\theta)$ of our data $D$ given the parameters $\theta$ with a function that returns $\log P(D|\theta)$:
Can you write Python code for it?

def log_likelihood(theta):
    m, b = theta
    # Calculate the predicted value at each x.
    y_pred = m * x_data + b
    # Calculate the residuals at each x.
    dy = y_data - y_pred
    # Assume Gaussian errors on each data point.
    return -0.5 * np.sum(dy ** 2) / rms_noise ** 2


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Each sample represents a possible model.  Pick a few at random to superimpose on our data:
Can you write Python code for it?

shuffled = samples[gen.permutation(len(samples))]
shuffled.shape

plt.errorbar(x_data, y_data, yerr=rms_noise, fmt='.k', label='Toy MC')
plt.plot(x_plot, y_plot, 'r-', label='Truth')

for m, b in shuffled[:100]:
    y_sample = m * x_plot + b
    plt.plot(x_plot, y_sample, 'b', alpha=0.05)

plt.grid()
plt.legend();


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Exercise: Marginal Distributions and Confidence Limits
Create a function that uses the samples array to plot the distribution of predicted values for $y(x)^8$ given $x$ and prints the (central) 68% and 95% confidence limits on this prediction.  (Hint: np.percentile)
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Give each "walker" a unique starting point in parameter space.  The advice on this in the docs is:

The best technique seems to be to start in a small ball around the a priori preferred position. Don’t worry, the walkers quickly branch out and explore the rest of the space.

Pick something close but exactly the truth for our a priori preferred position.
Can you write Python code for it?

a_priori = np.array([-1.1, 5.5])
ball_size = 1e-4
initial = [a_priori + ball_size * gen.normal(size=ndim) for i in range(nwalkers)]


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Initialize a Markov chain sampler using the emcee package:
Can you write Python code for it?

ndim = 2  # the number of parameters
nwalkers = 100  # bigger is better
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior)


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Introduction
This first section is a walk through to introduce you to the terminology and define some useful functions. Read it carefully, evaluate each cell, then move to the following sections where you will be repeating this basic study with variations.
We will study one of the simplest models, the straight line (which still has many subtleties):
$$
f(x) = m x + b
$$
This exercise is based on a more complicated example in the emcee documentation.
Initialization
First, specify our true parameters.  A dictionary is overkill with only two parameters, but is useful when tracking a larger number.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Markov Chain Monte Carlo
Define our priors $P(\theta)$ on the two fit parameters $\theta$ with a function to calculate $logP(\theta)$. The function should return -np.inf for any parameter values that are forbidden.
Start with simple "top hat" priors on both parameters:
Can you write Python code for it?

def log_prior(theta):
    m, b = theta
    if -5 < m < 0.5 and 0 < b < 10:
        return 0 # log(1) = 0
    else:
        return -np.inf # some parameter is outside its allowed range.


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Next, generate a "toy Monte Carlo" sample of n_data = 50 observations using the true model:
Can you write Python code for it?

# Reproducible "randomness"
gen = np.random.RandomState(seed=123)

# Random x values, in increasing order.
n_data = 50
x_data = np.sort(gen.uniform(low=0, high=10, size=n_data))
# True y values without any noise.
y_true = truth['m'] * x_data + truth['b']

# Add Gaussian noise.
rms_noise = 0.5
y_error = np.full(n_data, rms_noise)
y_data = y_true + gen.normal(scale=y_error)


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Plot the chains from the first 3 walkers for each parameter. Notice how they all start from the same small ball around our a priori initial point.
Can you write Python code for it?

plt.plot(sampler.chain[0, :, 0])
plt.plot(sampler.chain[1, :, 0])
plt.plot(sampler.chain[2, :, 0]);

plt.plot(sampler.chain[0, :, 1])
plt.plot(sampler.chain[1, :, 1])
plt.plot(sampler.chain[2, :, 1]);


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Plot the generated sample with the model superimposed:
Can you write Python code for it?

plt.errorbar(x_data, y_data, yerr=y_error, fmt='.k', label='Toy MC')
plt.plot(x_plot, y_plot, 'r-', label='Truth')
plt.grid()
plt.legend();


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Plot the true model over the range [0, 10]:
Can you write Python code for it?

x_lo, x_hi = 0, 10
x_plot = np.linspace(x_lo, x_hi, 100)  # Really only need 2 points here
y_plot = truth['m'] * x_plot + truth['b']

plt.plot(x_plot, y_plot, 'r-', label='Truth')
plt.grid()
plt.legend();


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
Regenerate the data using this larger error, while keeping the reported error the same:
Can you write Python code for it?

y_data = y_true + gen.normal(scale=y_error * truth['s'])


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
The weighted linear least squares method also provides a covariance matrix for the best-fit parameters, but unfortunately sklearn does not calculate this for you.  However, we will soon have something even better using MCMC.
Add this fit to our plot:
Can you write Python code for it?

y_fit = fit['m'] * x_plot + fit['b']

plt.errorbar(x_data, y_data, yerr=rms_noise, fmt='.k', label='Toy MC')
plt.plot(x_plot, y_plot, 'r-', label='Truth')
plt.plot(x_plot, y_fit, 'b--', label='Fit')
plt.grid()
plt.legend();


I am working on the file "Sessions/Session02/Day3/MCMC-Exercises.ipynb".
The first task is:
scikit-learn provides a convenient class for (weighted) linear least squares:
Can you write Python code for it?

import sklearn.linear_model

linear_model = sklearn.linear_model.LinearRegression()
linear_model.fit(x_data.reshape(-1, 1), y_data, sample_weight=weights)

fit = dict(m=linear_model.coef_[0], b=linear_model.intercept_)
print fit


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 1: Fully-connected network for digit recognition
Handwritten digit recognition is one of the most famous neural network applications; even before the recent advances in "deep learning," neural networks were able to achieve excellent performance for this problem and have been used in real-world applications for many years.
The canonical digit recognition example uses the so-called MNIST dataset, which consists of 60,000 training and 10,000 test examples. Each consists of a 28x28 black and white image of a handwritten digit. First we'll load the MNIST dataset and visualize the first few training examples (this may take a minute or two the first time you download the data).
Can you write Python code for it?

from keras.datasets import mnist
from keras.utils import np_utils

nrow = 28; ncol = 28; nb_classes = 10  # MNIST data parameters
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32'); X_test = X_test.astype('float32')  # int -> float
X_train = X_train.reshape(-1, nrow * ncol); X_test = X_test.reshape(-1, nrow * ncol)  # flatten
X_train /= 255; X_test /= 255  # normalize pixels to between 0 and 1

# convert class vectors to binary class matrices (i.e., one-hot encoding)
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

fig, ax = plt.subplots(2, 5, figsize=(15, 8))
for i in range(10):
    plt.sca(ax.ravel()[i])
    plt.imshow(X_train[i].reshape(nrow, ncol))


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 1b: Multi-layer classifier
Write a function that takes parameters hidden_size and num_layers and returns a classifier like the one above but with multiple fully-connected layers. Compare the performance of your multi-layer classifier with that of the single-layer network above. Also, check the output of model.summary() and see how the number of parameters varies with hidden_size and num_layers.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 2: Convolutional network
In the above example, we treated our images as one-dimensional vectors and input them to a simple feed-forward network. This has the major disadvantage of ignoring the local structure of the image: each pixel is considered separately from its surroundings. Convolutional networks, on the other hand, apply a number of filters to small regions of each image; these filters are trained to recognize common shapes that appear in the image and are useful for distinguishing classes. Here we'll train a basic convolutional network to perform the same image classification task.
Can you write Python code for it?

# First we'll reshape the data back into two-dimensional form
X_train = X_train.reshape(X_train.shape[0], nrow, ncol, 1)
X_test = X_test.reshape(X_test.shape[0], nrow, ncol, 1)
input_shape = (nrow, ncol, 1)  # only 1 channel since the images are black and white


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 2a: Simple convolutional network
First, implement a simple convolutional network consisting of a single convolutional layer, a single dense ReLU layer, and a softmax output. The structure should look basically like the single-layer network from earlier, but with a couple of additions.
A couple of hints:
- The Conv2D layer takes parameters nb_filter for the number of filters, and nb_row and nb_col for the filter dimensions; try a small number (e.g. 12) of small (e.g. 3 x 3) filters to start.
- Since a convolutional layer expects 2D inputs and a fully-connected layer expects 1D inputs, you'll want to add a Flatten() layer inbetween; this is just like the flattening we performed the MNIST inputs themselves before passing them to our fully-connected network, except applied to the intermediate values of our network.
- This is a more computationally-intensive training procedure, so try reducing nb_epoch to something like 5.
Can you write Python code for it?

from keras.layers import Conv2D, Flatten

# Convolutional network here
pass


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 2b: Deep-ish convolutional network
Part of the reason the above network is so slow is because of the connection between the convolutional and fully-connected layers: the convolutional layer's weights consist of just a few small filters, but its output is large (roughly 28x28x{# filters}), so the following fully-connected layer needs roughly 28x28x{# filters}x{hidden layer size} parameters...in other words, a lot. For this reason, it's common to insert a pooling layer after a convolutional layer, which reduces the output size considerably. The most common type of pooling is max pooling, primarily because of its simplicity. For more about pooling see, e.g., the CS 231n lecture notes about convolutional architectures.
Extend the network above by first replacing the first dense layer with a max pooling layer; the MaxPooling2D layer takes a 2-dimensional tuple pool_size which controls the size of regions that are pooled together. Experiment with this parameter and see how the training time changes; then try adding additional convolutional+pooling layers to form a deeper network. Check the output of model.summary() to see how the number of parameters changes depending on whether you apply pooling after a convolutional layer.
Can you write Python code for it?

from keras.layers import MaxPooling2D

# Convolutional network w/ pooling here
pass


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
Part 4: CIFAR10 dataset (optional)
Another standard neural network test case is the CIFAR10 image dataset, which consists of 60,000 32x32 color images from 10 classes. This dataset is available in keras as keras.datasets.cifar10. Use one of the above network architectures and see how it performs on this (more difficult) problem.
Can you write Python code for it?

from keras.datasets import cifar10

# Repeat w/ cifar10 dataset
pass


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkExercise.ipynb".
The first task is:
The most popular type of recurrent cell is called a "Long Short-Term Memory" cell, or LSTM. Follow the same structure as above and implement a simple recurrent neural network classifier using the LSTM layer.

Note that a recurrent cell takes sequences as inputs, but can output either sequences (i.e. a new value each time it processes a value from the input), or individual values (i.e. only one value after the entire sequence is processed). In keras this is controlled using the return_sequences keyword argument.
Recurrent networks generally use different activations than convolutional networks; you can omit the Activation layers here and just use the LSTM cell's default, which is tanh.
Training LSTMs is computationally-intensive so for now keep your hidden layer sizes small and only use the first 250 training examples; this won't be enough to train an effective model but you'll at least see the steps needed to do so given more time/computing power.
Can you write Python code for it?

from keras.layers import LSTM

# LSTM classifier here
pass


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
Part 1b: Multi-layer classifier
Write a function that takes parameters hidden_size and num_layers and returns a classifier like the one above but with multiple fully-connected layers. Compare the performance of your multi-layer classifier with that of the single-layer network above. Also, check the output of model.summary() and see how the number of parameters varies with hidden_size and num_layers.
Can you write Python code for it?

def fully_connected(hidden_size, num_layers):
    model = Sequential()
    for i in range(num_layers):
        model.add(Dense(hidden_size, input_dim=nrow * ncol if i == 0 else None))
        model.add(Activation('relu'))
    model.add(Dense(nb_classes))
    model.add(Activation('softmax'))
    return model

model = fully_connected(64, 3)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    validation_data=(X_test, Y_test),
                    verbose=0, callbacks=[TQDMNotebookCallback()])
score, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Test score: {}; test accuracy: {}'.format(score, accuracy))
model.summary()


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
Part 2: Convolutional network
In the above example, we treated our images as one-dimensional vectors and input them to a simple feed-forward network. This has the major disadvantage of ignoring the local structure of the image: each pixel is considered separately from its surroundings. Convolutional networks, on the other hand, apply a number of filters to small regions of each image; these filters are trained to recognize common shapes that appear in the image and are useful for distinguishing classes. Here we'll train a basic convolutional network to perform the same image classification task.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
Part 2b: Deep-ish convolutional network
Part of the reason the above network is so slow is because of the connection between the convolutional and fully-connected layers: the convolutional layer's weights consist of just a few small filters, but its output is large (roughly 28x28x{# filters}), so the following fully-connected layer needs roughly 28x28x{# filters}x{hidden layer size} parameters...in other words, a lot. For this reason, it's common to insert a pooling layer after a convolutional layer, which reduces the output size considerably. The most common type of pooling is max pooling, primarily because of its simplicity. For more about pooling see, e.g., the CS 231n lecture notes about convolutional architectures.
Extend the network above by first replacing the first dense layer with a max pooling layer; the MaxPooling2D layer takes a 2-dimensional tuple pool_size which controls the size of regions that are pooled together. Experiment with this parameter and see how the training time changes; then try adding additional convolutional+pooling layers to form a deeper network. Check the output of model.summary() to see how the number of parameters changes depending on whether you apply pooling after a convolutional layer.
Can you write Python code for it?

from keras.layers import MaxPooling2D

model = Sequential()
model.add(Conv2D(32, 3, 3, input_shape=input_shape))
model.add(MaxPooling2D((3, 3)))
model.add(Conv2D(32, 3, 3))
model.add(MaxPooling2D((3, 3)))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

nb_epoch = 5
history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    validation_data=(X_test, Y_test),
                    verbose=0, callbacks=[TQDMNotebookCallback()])
score, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Test score: {}; test accuracy: {}'.format(score, accuracy))


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
Part 2c: Regularization
Because of the extremely high number of parameters contained in deep convolutional networks, the chances of overfitting your model to the test data are rather high. The term "regularization" is something of a catch-all for techniques that try to mitigate this overfitting behavior. Here we'll try adding a couple of approaches to our existing network: dropout and weight penalization.
Dropout is a technique that randomly sets some fraction neuron activations to zero during training; the idea is that the network will learn to cope with this by developing multiple different representations for the various patterns in the data, and thereby increase its robustness to unseen data. For validation/test data, dropout is then disabled to generate the best predictions possible.
Weight penalization is a standard idea from linear regression (there it's called ridge regression/LASSO/elastic net, among other things). For each set of weights, a penalty is added to the loss function that is proportional to the magnitude of the coefficients; the result is that the network prefers a combination of many smaller weights rather than some extremely large weights, which again (hopefully) should lead to a more robust model.
Extend your network by adding dropout layers between the convolutional layers, and/or by adding $\ell_2$ regularization to the weights in an existing layer. Does this help reduce the gap between the training and test error? Dropout in keras is simply another layer; regularization can be passed into an existing layer with the keyword argument W_regularizer=l2({some small value}).
Can you write Python code for it?

from keras.layers import Dropout
from keras.regularizers import l2

model = Sequential()
model.add(Conv2D(32, 3, 3, input_shape=input_shape))
model.add(MaxPooling2D((3, 3)))
model.add(Dropout(0.25))
model.add(Conv2D(32, 3, 3))
model.add(MaxPooling2D((3, 3)))
model.add(Dropout(0.25))
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

nb_epoch = 5
history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    validation_data=(X_test, Y_test),
                    verbose=0, callbacks=[TQDMNotebookCallback()])
score, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Test score: {}; test accuracy: {}'.format(score, accuracy))


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
The most popular type of recurrent cell is called a "Long Short-Term Memory" cell, or LSTM. Follow the same structure as above and implement a simple recurrent neural network classifier using the LSTM layer.

Note that a recurrent cell takes sequences as inputs, but can output either sequences (i.e. a new value each time it processes a value from the input), or individual values (i.e. only one value after the entire sequence is processed). In keras this is controlled using the return_sequences keyword argument.
Recurrent networks generally use different activations than convolutional networks; you can omit the Activation layers here and just use the LSTM cell's default, which is tanh.
Training LSTMs is computationally-intensive so for now keep your hidden layer sizes small and only use the first 250 training examples; this won't be enough to train an effective model but you'll at least see the steps needed to do so given more time/computing power.
Can you write Python code for it?

from keras.layers import LSTM

model = Sequential()
model.add(LSTM(16, input_shape=input_shape, return_sequences=True))
model.add(LSTM(16, return_sequences=False))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

nb_epoch = 5
history = model.fit(X_train[:250], Y_train[:250],
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    validation_data=(X_test, Y_test),
                    verbose=0, callbacks=[TQDMNotebookCallback()])
score, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Test score: {}; test accuracy: {}'.format(score, accuracy))


I am working on the file "Sessions/Session02/Day4/AdvancedNeuralNetworkSolution.ipynb".
The first task is:
We'll start with our old friend the single-layer perceptron that we implemented in the "Basic Neural Network Exercise." The perceptron consists of a single fully-connected (a.k.a. dense) layer with some activation function, plus an output that we pass to the softmax function. An example of a keras implementation of such a network is given below; you'll want to use this as the template for the rest of your models in this exercise.
The steps below should mostly be self-explanatory, but here's a quick breakdown:
- Sequential is a keras class that allows us to add new layers one at a time
- Dense is a fully-connected layer; the input_shape argument is only needed in the first layer
- Activation passes the output of the previous layer through a specified activation function
- compile prepares the underlying tensorflow or theano graph corresponding to the requested model
- rmsprop is a variant of gradient descent that converges more quickly; vanilla gradient descent is rarely used for training neural networks
- "Categorical crossentropy" is the same standard loss function we used in the previous example for evaluating predicted class probabilities
Can you write Python code for it?

# Define model architecture
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation

model = Sequential()
model.add(Dense(8, input_dim=nrow * ncol))
model.add(Activation('relu'))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.summary()

# Fit model to training data and check accuracy
from keras_tqdm import TQDMNotebookCallback

batch_size = 128
nb_epoch = 20

model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
history = model.fit(X_train, Y_train,
                    batch_size=batch_size, nb_epoch=nb_epoch,
                    validation_data=(X_test, Y_test),
                    verbose=0, callbacks=[TQDMNotebookCallback()])
score, accuracy = model.evaluate(X_test, Y_test, verbose=0)
print('Test score: {}; test accuracy: {}'.format(score, accuracy))


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkExercise.ipynb".
The first task is:
2c) (optional) Numerically check the gradient implementation in dloss
This is an extremely important step when coding derivatives by hand: check the gradients returned by dloss by perturbing a single weight value by $\epsilon$ and comparing the resulting change in loss to what you'd expect from the gradient.
* This is a bit of an abuse of notation since we're taking some derivatives with respect to vectors/matrices; this just means a vector/matrix of derivatives with respect to each element.
** For an excrutiating amount of detail about backprop, see e.g. Neural Networks and Deep Learning.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkExercise.ipynb".
The first task is:
Part 1: Forward pass
The "forward pass" step (i.e., computing the output for a given input) can be written as
$$
\begin{align}
\mathbf{h} &= f(W_1 \mathbf{x} + \mathbf{b_1}) \
\mathbf{z} &= W_2 \mathbf{h} + \mathbf{b_2} \
\mathbf{p} &= \operatorname{softmax}(\mathbf{z})
\end{align}
$$
where $f(\cdot)$ is an activation function, $W_1$ and $W_2$ are weight matrices, and $\mathbf{b}1$ and $\mathbf{b}_2$ are bias vectors. The softmax function is given by
$$
\operatorname{softmax}(\mathbf{z}) = \exp\left(\mathbf{z}\right) / \left(\sum{k=1}^K \exp\left( z_k\right)\right)
$$
and the ReLU activation is
$$
f(x) = \max(x, 0).
$$
1a) Implement predict function
Your function should loop over the input points (i.e., the rows of $X$) and compute the estimated class probabilities for each. The model parameters $W_1, W_2, b_1, b_2$ are treated as fixed.
1b) Examine predictions
The function visualize_predictions below can be used to inspect the decision boundaries for a given model, dataset, and class labels. Initialize a network and see what the classifications look like. Try varying the number of hidden units and see how the decision boundary shapes change.
Can you write Python code for it?

def visualize_predictions(model, X, y, step=0.02):
    x_min, x_max = X[:, 0].min(), X[:, 0].max()
    y_min, y_max = X[:, 1].min(), X[:, 1].max()
    xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=40)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.title("Accuracy: {}%".format(100 * np.mean(y == model.predict(X))))

# Test `predict` function for part 1b:
pass


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkExercise.ipynb".
The first task is:
Part 2: Training
Obviously generating classifications at random isn't the way to go. Now we'll use gradient descent to iteratively improve the weights of our network and hopefully improve our classification accuracy.
If $y$ is some true label and $\mathbf{p}$ are the estimated class probabilities from our network, then our loss function for a single training example can be written as
$$L(y, \mathbf{p}) = -\log p_y,$$
that is, the negative log of the predicted probability for the true class: 0 if our predicted probability was 1, and $+\infty$ if it was 0.
In order to perform gradient descent, we'll need the derivatives of the loss with respect to all of our model parameters. We can compute these derivatives via the backpropagation (effectively just the chain rule)*:
$$
\begin{aligned}
\frac{\partial L}{\partial z_k}   &= p_k - \text{{1 if $y=k$, 0 otherwise}} \
\frac{\partial L}{\partial W_2}          &= \mathbf{h} \left(\frac{\partial L}{\partial \mathbf{z}}\right)^T \
\frac{\partial L}{\partial \mathbf{b}_2} &= \frac{\partial L}{\partial \mathbf{z}}\
\frac{\partial L}{\partial \mathbf{h}}   &= W_2 \frac{\partial L}{\partial \mathbf{z}} \circ f'(\mathbf{h}) \
\frac{\partial L}{\partial W_1}          &= \mathbf{x} \left(\frac{\partial L}{\partial \mathbf{h}}\right)^T\
\frac{\partial L}{\partial \mathbf{b}_1} &= \frac{\partial L}{\partial \mathbf{h}}\
\end{aligned}
$$
Even for very deep or complicated networks, the gradient of the loss function can always be decomposed in this way, one connection at a time.
The actual quantity we wish to minimize is the sum of this loss value over all the samples in our data set. The gradient of this total loss is just the sum of the gradient for each sample.
2a) Implement fit function
Our gradient descent training consists of the following steps:
- Compute loss gradient with respect to model parameters using dloss. Note: for the sake of efficiency, this function returns the sum of the gradient of over all samples, so you should pass in the entire...
Can you write Python code for it?

# Initialize model, train on sample data, and visualize predictions for part 2b:
pass


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkExercise.ipynb".
The first task is:
Part 3: Compare with scikit-learn
The scikit-learn package does contain a minimal implementation of a neural network classifier in scikit-learn.neural_networks.MLPClassifier. This classifier is nowhere near as powerful or fully-featured as the neural networks provided by other packages such as tensorflow, theano, etc., but it still can represent complicated non-linear relationships within small datasets.
3a) Implement a simple neural network classifier using scikit-learn
What parameter values give a network comparable to the one we implemented above? Note that the results may not be equivalent since scikit-learn version could reach a different set of (locally) optimal parameters.
Hint: try using solver='lbfgs' (a more complicated optimization method) for better results.
3b) Experiment with different activations/network sizes
How do the decision boundaries change for different models?
3c) Experiment with different numbers of classes and noise levels
Go back to where we initialized the data and change the parameters K and sigma. Can a single- or multi-layer perceptron still represent more complicated patterns?
Can you write Python code for it?

# Train `scikit-learn` multi-layer perceptron
from sklearn.neural_network import MLPClassifier

?MLPClassifier


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkSolution.ipynb".
The first task is:
2c) (optional) Numerically check the gradient implementation in dloss
This is an extremely important step when coding derivatives by hand: check the gradients returned by dloss by perturbing a single weight value by $\epsilon$ and comparing the resulting change in loss to what you'd expect from the gradient.
* This is a bit of an abuse of notation since we're taking some derivatives with respect to vectors/matrices; this just means a vector/matrix of derivatives with respect to each element.
** For an excrutiating amount of detail about backprop, see e.g. Neural Networks and Deep Learning.
Can you write Python code for it?

dW1, dW2, db1, db2 = model.dloss(X, y)
h = 1e-6
i = 0; j = 0
l0 = model.loss(y, model.predict_proba(X))
model.W1[i, j] += h
lh = model.loss(y, model.predict_proba(X))
print("Derivative: {}; Numerical esitmate: {}".format(dW1[i, j], (lh - l0) / h))


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkSolution.ipynb".
The first task is:
Part 1: Forward pass
The "forward pass" step (i.e., computing the output for a given input) can be written as
$$
\begin{align}
\mathbf{h} &= f(W_1 \mathbf{x} + \mathbf{b_1}) \
\mathbf{z} &= W_2 \mathbf{h} + \mathbf{b_2} \
\mathbf{p} &= \operatorname{softmax}(\mathbf{z})
\end{align}
$$
where $f(\cdot)$ is an activation function, $W_1$ and $W_2$ are weight matrices, and $\mathbf{b}1$ and $\mathbf{b}_2$ are bias vectors. The softmax function is given by
$$
\operatorname{softmax}(\mathbf{z}) = \exp\left(\mathbf{z}\right) / \left(\sum{k=1}^K \exp\left( z_k\right)\right)
$$
and the ReLU activation is
$$
f(x) = \max(x, 0).
$$
1a) Implement predict function
Your function should loop over the input points (i.e., the rows of $X$) and compute the estimated class probabilities for each. The model parameters $W_1, W_2, b_1, b_2$ are treated as fixed.
1b) Examine predictions
The function visualize_predictions below can be used to inspect the decision boundaries for a given model, dataset, and class labels. Initialize a network and see what the classifications look like. Try varying the number of hidden units and see how the decision boundary shapes change.
Can you write Python code for it?

def visualize_predictions(model, X, y, step=0.02):
    x_min, x_max = X[:, 0].min(), X[:, 0].max()
    y_min, y_max = X[:, 1].min(), X[:, 1].max()
    xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=40)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.title("Accuracy: {}%".format(100 * np.mean(y == model.predict(X))))

model = SingleLayerReLU(100, K)
visualize_predictions(model, X, y)


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkSolution.ipynb".
The first task is:
Part 2: Training
Obviously generating classifications at random isn't the way to go. Now we'll use gradient descent to iteratively improve the weights of our network and hopefully improve our classification accuracy.
If $y$ is some true label and $\mathbf{p}$ are the estimated class probabilities from our network, then our loss function for a single training example can be written as
$$L(y, \mathbf{p}) = -\log p_y,$$
that is, the negative log of the predicted probability for the true class: 0 if our predicted probability was 1, and $+\infty$ if it was 0.
In order to perform gradient descent, we'll need the derivatives of the loss with respect to all of our model parameters. We can compute these derivatives via the backpropagation (effectively just the chain rule)*:
$$
\begin{aligned}
\frac{\partial L}{\partial z_k}   &= p_k - \text{{1 if $y=k$, 0 otherwise}} \
\frac{\partial L}{\partial W_2}          &= \mathbf{h} \left(\frac{\partial L}{\partial \mathbf{z}}\right)^T \
\frac{\partial L}{\partial \mathbf{b}_2} &= \frac{\partial L}{\partial \mathbf{z}}\
\frac{\partial L}{\partial \mathbf{h}}   &= W_2 \frac{\partial L}{\partial \mathbf{z}} \circ f'(\mathbf{h}) \
\frac{\partial L}{\partial W_1}          &= \mathbf{x} \left(\frac{\partial L}{\partial \mathbf{h}}\right)^T\
\frac{\partial L}{\partial \mathbf{b}_1} &= \frac{\partial L}{\partial \mathbf{h}}\
\end{aligned}
$$
Even for very deep or complicated networks, the gradient of the loss function can always be decomposed in this way, one connection at a time.
The actual quantity we wish to minimize is the sum of this loss value over all the samples in our data set. The gradient of this total loss is just the sum of the gradient for each sample.
2a) Implement fit function
Our gradient descent training consists of the following steps:
- Compute loss gradient with respect to model parameters using dloss. Note: for the sake of efficiency, this function returns the sum of the gradient of over all samples, so you should pass in the entire...
Can you write Python code for it?

model = SingleLayerReLU(100, K)
losses = model.fit(X, y, step_size=2e-3, n_iter=20000)
plt.plot(losses, '-')
plt.xlabel('Iteration')
plt.ylabel('Total loss');
plt.figure()
visualize_predictions(model, X, y)


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkSolution.ipynb".
The first task is:
Part 4: Visualize neuron activations (optional)
Interpreting neural network classifiers can be quite difficult, especially when the network contains many layers. One way of attempting to make sense of the model is by visualizing the activations of individual neurons: by plotting the activation of a neuron over the entire input space, we can see what patterns the neurons are learning to represent.
Try below to generate plots that show the activation of an individual neuron. scikit-learn doesn't make this immediately available, but we can make it work: try setting the last layer's weights (in coefs_[-1]) to zero for everything except a single unit, and setting that unit's weights to 1 (so that the activation just passes straight through). You can re-use most of the code from visualize_predictions to generate a contour map of activation strengths.
Can you write Python code for it?

from copy import deepcopy

X, y = spiral_data(100, 2, 0.1)
multi_layer_model = MLPClassifier((64, 8), alpha=0.5, activation='relu', solver='lbfgs')
multi_layer_model.fit(X, y)
visualize_predictions(multi_layer_model, X, y)

def visualize_activations(model, unit, X, y, step=0.02):
    """Visualize activations of ith neuron of last layer."""
    model = deepcopy(model)
    model.coefs_[-1][:unit] = 0  # zero out other units
    model.coefs_[-1][unit] = 1  # just want the activation function
    model.coefs_[-1][(unit + 1):] = 0  # zero out other units

    x_min, x_max = X[:, 0].min(), X[:, 0].max()
    y_min, y_max = X[:, 1].min(), X[:, 1].max()
    xx, yy = np.meshgrid(np.arange(x_min, x_max, step), np.arange(y_min, y_max, step))
    Z = model.predict_proba(np.c_[xx.ravel(), yy.ravel()])[:, 0]
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, alpha=0.8)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=40)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())

fig, ax = plt.subplots(4, 2, figsize=(14, 20))
for i in range(len(multi_layer_model.coefs_[-1])):
    plt.sca(ax.ravel()[i])
    visualize_activations(multi_layer_model, i, X, y)


I am working on the file "Sessions/Session02/Day4/BasicNeuralNetworkSolution.ipynb".
The first task is:
Setup
Our neural network will take an $\mathbf{x} = (x_1, x_2)$ vector as input and output a $K$-dimensional vector $\mathbf{p}=(p_1,\dots,p_K)$ of class probabilities. For simplicity we'll focus on a single choice of activation function, the ReLU function $f(x) = \max(x, 0)$.

We'll follow the scikit-learn model API and construct a simple model with fit and predict methods. The exercises below will step you through the necessary steps to implement a fully-functional neural network classifier.
Can you write Python code for it?

from sklearn.base import BaseEstimator, ClassifierMixin

class SingleLayerReLU(BaseEstimator, ClassifierMixin):
    """Skeleton code for single-layer multi-class neural network classifier w/ ReLU activation.
    NOTE: Whenever you change the code below, you need to re-run this cell AND re-initialize
    your model (`model = SingleLayerNet(...)`) in order to update your specific `model` object.
    """
    def __init__(self, hidden_size, num_classes, sigma_init=0.01):
        """Initialize weights with Gaussian noise scaled by `sigma_init` and
        biases with zeros.
        """
        self.hidden_size = hidden_size
        self.num_classes = num_classes
        self.W1 = sigma_init * np.random.randn(hidden_size, 2)
        self.W2 = sigma_init * np.random.randn(num_classes, hidden_size)
        self.b1 = np.zeros(hidden_size)
        self.b2 = np.zeros(num_classes)

    def loss(self, y, P):
        """Compute total softmax loss.
        Inputs:  y -> (N,) array of true (integer) labels
                 p -> (N, K) array of predicted probabilities
        Outputs: L -> total loss value
        """
        return -np.sum(np.log(P[range(len(P)), y]))

    def dloss(self, X, y):
        """Compute gradient of softmax loss with respect to network weights.
        Inputs:  X -> (N, 2) array of network inputs
                 y -> (N,) array of true labels
        Outputs: dW1 -> (hidden_size, 2) array of weight derivatives
                 dW2 -> (hidden_size, 2) array of weight derivatives
                 db1 -> (hidden_size,) array of bias derivatives
                 db2 -> (hidden_size,) array of bias derivatives
        """
        H = np.maximum(0, X @ self.W1.T + self.b1)  # ReLU activation
        Z = H @ self.W2.T + self.b2
        P = np.exp(Z) / np.sum(np.exp(Z), axis=1, keepdims=True)

        dZ = P
        dZ[range(len(X)), y] -= 1

        dW2 = (H.T @ dZ).T
        db2 = np.sum(dZ, axis=0)

        dH = dZ @ self.W2
        dH[H <= 0] = 0  # backprop ReLU  activation

      ...


I am working on the file "Sessions/Session02/Day5/ImageVizExercises.ipynb".
The first task is:
3. RGB colors of Herschel-SPIRE images
Reproject the 250 um image of W5 in ./data/0259p6031_1342192088_SpirePhoto_L20_PMP250_SPG14.0.fits.gz (extension 1) and the 350 micron image in ./data/0259p6031_1342192088_SpirePhoto_L20_PMP350_SPG14.0.fits.gz onto the 500 micron image in that directory, and try the same 3-color procedures.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
After quitting glue, try linking the data
Can you write Python code for it?

from glue.core.data_factories import load_data
from glue.core import DataCollection
from glue.core.link_helpers import LinkSame
from glue.app.qt.application import GlueApplication
import warnings
from astropy.utils.exceptions import AstropyWarning

#load 2 datasets from files
image = load_data('./data/w5.fits')
with warnings.catch_warnings():
    warnings.simplefilter('ignore', AstropyWarning)
    catalog = load_data('./data/w5_psc.vot')
dc = DataCollection([image, catalog])

# link positional information
dc.add_link(LinkSame(image.id['Right Ascension'], catalog.id['RAJ2000']))
dc.add_link(LinkSame(image.id['Declination'], catalog.id['DEJ2000']))

#start Glue
app = GlueApplication(dc)
app.start()

image.component_ids()

dc.links

dc.subset_groups

g1 = dc.subset_groups[0]

g1.label


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Astropy visualization functions for scaling and stretching
Can you write Python code for it?

# Scale to image minimum and maximum, stretch with log function
norm = ImageNormalize(image, interval=MinMaxInterval(),
                      stretch=LogStretch())


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Contours with matplotlib
Can you write Python code for it?

%matplotlib inline
from astropy.visualization import (ManualInterval, LogStretch,
                                   ImageNormalize)
norm = ImageNormalize(image, interval=ManualInterval(vmin=370.0, vmax=1000.0),
                      stretch=LogStretch())

%matplotlib inline
fig = plt.figure(figsize=(8,8))
#plt.imshow(image, norm=norm, origin="lower", cmap='Greys_r')
plt.contour(image, [550, 750, 950], origin='lower',
            colors=['grey', 'green', 'blue']);


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Firefly
Web-based visualizer
Basis for LSST Science User Interface
Remote server sits close to the data
Many operations performed by your browser
Developed for Spitzer Heritage Archive
Employed as IRSA Viewer
Can access user interface from IRSA
Example: Herschel High-Level Images http://irsa.ipac.caltech.edu/data/Herschel/HHLI/index.html
Search on W5
Overlay a WISE catalog
Easy way to start a Firefly server
Download Firefly Standalone from release page (https://github.com/Caltech-IPAC/firefly/releases
Only requirement is java 1.8
Start with java -jar firefly-exec.war
By default, will be available on localhost port 80: http://localhost:8080/firefly/
Python client for Firefly
Install with pip install firefly_client
This part requires a local Firefly server running
Can you write Python code for it?

from firefly_client import FireflyClient

fc = FireflyClient('localhost:8080', channel='mine')

fc.launch_browser()

fval = fc.upload_file('./data/w5.fits')

fc.show_fits(fval, plot_id='w5_wise')

tval = fc.upload_file('./data/w5_wise.tbl')

fc.show_table(tval, title='Bright WISE W4')


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Here we use only the numpy array of the data.
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
In the case the World Coordinate System is a simple tangent projection
Can you write Python code for it?

wcs

norm = ImageNormalize(hdu.data, interval=MinMaxInterval(),
                     stretch=LogStretch())


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Make sure to pick a good sequential colormap and avoid the default 'jet'!
Can you write Python code for it?

%matplotlib inline
fig = plt.figure(figsize=(8,8))
ax = plt.subplot(projection=wcs)
plt.imshow(hdu.data, norm=norm, cmap='viridis', origin="lower")
plt.grid(color='white', ls='solid')
plt.xlabel('Right Ascension')
plt.ylabel('Declination')


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
NOTE: if you don't have the webbrowser module, open the link that was printed in the cell above in a new window to get the viewer.
You can open as many of these viewers as you want--just keep a handle to it and use a different name for each unique one.
Keyboard/mouse bindings in the viewer window: http://ginga.readthedocs.io/en/latest/quickref.html
Can you write Python code for it?

# Load an image into the viewer
# (change the path to where you downloaded the sample images, or use your own)
v1.load('./data/w5.fits')

# Example of embedding a viewer
v1.embed(height=650)

# capture the current state of the screen
v1.show()


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
Note that astronomical images in FITS use origin="lower" which is not the matplotlib default
Can you write Python code for it?

%matplotlib inline
fig = plt.figure(figsize=(8,8))
plt.imshow(image, norm=norm, origin="lower", cmap='Greys_r');


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
See this contour demo from the matplotlib documentation for more to do with contours
Adding coordinates (wcsaxes)
Prior to this point, we have visualized numpy arrays
Can you write Python code for it?

from astropy.wcs import WCS

hdu = fits.open('data/w5.fits')[0]
wcs = WCS(hdu.header)


I am working on the file "Sessions/Session02/Day5/ImageVizSlides.ipynb".
The first task is:
The LSST stack can display images to Firefly.
Experimental Jupyter widgets have been developed.
Glue for linking data
Based on brushing-and-linking paradigm
Used for JWST
Images, scatter plots, histograms
To start from the notebook, execute the next line in its own cell
Can you write Python code for it?

%gui qt

from glue import qglue

qglue()


I am working on the file "Sessions/Session02/Day5/ImageVizSolutions.ipynb".
The first task is:
2. RGB-3-color images
Using astropy and reproject (installable with pip install reproject), follow these instructions in the Astropy documentation to make color RGB images. Compare the second one to Figure 1 of Lupton et al 2004.
Can you write Python code for it?

import numpy as np
from astropy.visualization import make_lupton_rgb
from astropy.io import fits
from reproject import reproject_interp

# Read in the three images downloaded from here:
g = fits.open('http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-g-001737-5-0039.fits.bz2')[0]
r = fits.open('http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-r-001737-5-0039.fits.bz2')[0]
i = fits.open('http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-i-001737-5-0039.fits.bz2')[0]

# remap r and i onto g
r_new, r_mask = reproject_interp(r, g.header)
i_new, i_mask = reproject_interp(i, g.header)

# zero out the unmapped values
i_new[np.logical_not(i_mask)] = 0
r_new[np.logical_not(r_mask)] = 0

# red=i, green=r, blue=g
# make a file with the default scaling
rgb_default = make_lupton_rgb(i_new, r_new, g.data, filename="ngc6976-default.jpeg")
# this scaling is very similar to the one used in Lupton et al. (2004)
rgb = make_lupton_rgb(i_new, r_new, g.data, Q=10, stretch=0.5, filename="ngc6976.jpeg")


I am working on the file "Sessions/Session02/Day5/ImageVizSolutions.ipynb".
The first task is:
3. RGB colors of Herschel-SPIRE images
Reproject the 250 um image of W5 in ./data/0259p6031_1342192088_SpirePhoto_L20_PMP250_SPG14.0.fits.gz (extension 1) and the 350 micron image in ./data/0259p6031_1342192088_SpirePhoto_L20_PMP350_SPG14.0.fits.gz onto the 500 micron image in that directory, and try the same 3-color procedures.
Can you write Python code for it?

w5_250 = fits.open('./data/0259p6031_1342192088_SpirePhoto_L20_PMP250_SPG14.0.fits.gz')[1]
w5_350 = fits.open('./data/0259p6031_1342192088_SpirePhoto_L20_PMP350_SPG14.0.fits.gz')[1]
w5_500 = fits.open('./data/0259p6031_1342192088_SpirePhoto_L20_PMP500_SPG14.0.fits.gz')[1]

im250, msk250 = reproject_interp(w5_250, w5_500.header)
im350, msk350 = reproject_interp(w5_350, w5_500.header)

# zero out the unmapped values
im250[np.logical_not(msk250)] = 0
im350[np.logical_not(msk350)] = 0

rgb_w5_default = make_lupton_rgb(im250, im350, w5_500.data, filename="w5-default.jpeg")


I am working on the file "Sessions/Session02/Day5/ImageVizSolutions.ipynb".
The first task is:
Solutions for image visualization
Some exercises make use of code in the notebook. Other exercises will require a GUI interaction. In those cases, one or more images of results are included.
1. matplotlib: Contours from another image, and secondary axes
Using matplotlib and astropy:

display the file .data/w5.fits as a bitmap with log stretch and min-max scaling
overlay the data in image extension 1 of ./data/0259p6031_1342192088_SpirePhoto_L20_PMP350_SPG14.0.fits.fits.gz as white contours with levels drawn at [0.7, 1.4, 3] image units (Jy/beam)
display coordinate axes and grid (green, alpha=1) in (RA, Dec)
overlay a coordinate grid (cyan, alpha=1) and axis labels in Galactic longitude and latitude
Can you write Python code for it?

import matplotlib.pyplot as plt
from astropy.io import fits
from astropy.wcs import WCS
from astropy.visualization import (MinMaxInterval,
                                  LogStretch,
                                  ImageNormalize)
%matplotlib inline

hdu = fits.open('./data/w5.fits')[0]
wcs = WCS(hdu.header)

hdu2 = fits.open('./data/0259p6031_1342192088_SpirePhoto_L20_PMP350_SPG14.0.fits.gz')[1]

norm = ImageNormalize(hdu.data, interval=MinMaxInterval(),
                      stretch=LogStretch())

fig = plt.figure(figsize=(8,8))
ax = plt.subplot(projection=wcs)

overlay = ax.get_coords_overlay('galactic')

plt.imshow(hdu.data, norm=norm, origin="lower", cmap='Greys_r')

ax.coords['ra'].set_ticks(color='green')
ax.coords['dec'].set_ticks(color='green')

ax.coords['ra'].set_axislabel('Right Ascension')
ax.coords['dec'].set_axislabel('Declination')

ax.coords.grid(color='green', linestyle='solid', alpha=1.0)

overlay['l'].set_ticks(color='cyan')
overlay['b'].set_ticks(color='cyan')

overlay['l'].set_axislabel('Galactic Longitude')
overlay['b'].set_axislabel('Galactic Latitude')

overlay.grid(color='cyan', linestyle='solid', alpha=1.0)

ax.contour(hdu2.data, transform=ax.get_transform(WCS(hdu2.header)),
           levels=[0.7,1.4,3], colors='white');


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
A Practical Guide to the Machine Learning Workflow:
Separating Stars and Galaxies from SDSS
Version 0.1

By AA Miller 2017 Jan 22
We will now follow the steps from the machine learning workflow lecture to develop an end-to-end machine learning model using actual astronomical data. As a reminder the workflow is as follows:

Data Preparation
Model Building
Model Evaluation
Model Optimization
Model Predictions

Some of these steps will be streamlined to allow us to fully build a model within the alloted time.
Science background: Many (nearly all?) of the science applications for LSST data will rely on the accurate separation of stars and galaxies in the LSST imaging data. As an example, imagine measuring galaxy clustering without knowing which sources are galaxies and which are stars.
During this exercise, we will utilize supervised machine-learning methods to separate extended (galaxies) and point sources (stars, QSOs) in imaging data. These methods are highly flexible, and as a result can classify sources at higher fidelity than methods that simply make cuts in a low-dimensional space.
Can you write Python code for it?

import numpy as np
from astropy.table import Table
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Challenge 3
Perform an actual test of the model using "field" sources. The SDSS photometric classifier is nearly perfect for sources brighter than $r = 21$ mag. Download a random sample of $r < 21$ mag photometric sources, and classify them using the optimized RF model. Adopting the photometric classifications as ground truth, what is the accuracy of the RF model?
Hint - you'll need to look up the parameter describing photometric classification in SDSS
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Problem 1) Obtain and Examine Training Data
As a reminder, for supervised-learning problems we use a training set, sources with known labels, i.e. they have been confirmed as normal stars, QSOs, or galaxies, to build a model to classify new observations where we do not know the source label.
The training set for this exercise uses Sloan Digital Sky Survey (SDSS) data. For features, we will start with each $r$-band magnitude measurement made by SDSS. This yields 8 features (twice that of the Iris data set, but significantly fewer than the 454 properties measured for each source in SDSS).
Step 1 in the ML workflow is data preparation - we must curate the training set. As a reminder:
A machine-learning model is only as good as its training set.
This point cannot be emphasized enough. Machine-learning models are data-driven, they do not capture any physical theory, and thus it is essential that the training set satisfy several criteria.
Two of the most important criteria for a good training set are:

the training set should be unbiased [this is actually really hard to achieve in astronomy since most surveys are magnitude limited]
the training set should be representative of the (unobserved or field) population of sources [a training set with no stars will yield a model incapable of discovering point sources]

So, step 1 (this is a must), we are going to examine the training set to see if anything suspicious is going on. We will use astroquery to directly access the SDSS database, and store the results in an astropy Table.
Note The SDSS API for astroquery is not standard for the package, which leads to a warning. This is not, however, a problem for our purposes.
Can you write Python code for it?

from astroquery.sdss import SDSS  # enables direct queries to the SDSS database


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Problem 2) An Aside on the Importance of Feature Engineering
It has been said that all machine learning is an exercise in feature engineering.
Feature engineering - the process of creating new features, combining features, removing features, collecting new data to supplement existing features, etc. is essential in the machine learning workflow. As part of the data preparation stage, it is useful to apply domain knowledge to engineer features prior to model construction. [Though it is important to know that feature engineering may be needed at any point in the ML workflow if the model does not provide desired results.]
Due to a peculiarity of our SDSS training set, we need to briefly craft a separate problem to demonstrate the importance of feature engineering.
For this aside, we will train the model on bright ($r' < 18.5$ mag) sources and test the model on faint ($r' > 19.5$ mag) sources. As you might guess the model will not perform well. Following some clever feature engineering, we will be able to improve this.
aside-to-the-aside
This exact situation happens in astronomy all the time, and it is known as sample selection bias. In brief, any time a larger aperture telescope is built, or instrumentation is greatly improved, a large swath of sources that were previously undetectable can now be observed. These fainter sources, however, may contain entirely different populations than their brighter counterparts, and thus any models trained on the bright sources will be biased when making predictions on the faint sources.
We train and test the model with 10000 sources using an identical query to the one employed above, with the added condition restricting the training set to bright sources and the test set to faint sources.
Can you write Python code for it?

bright_query = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             AND p.cModelMag_r < 18.5
             ORDER BY p.objid ASC
               """
bright_set = SDSS.query_sql(bright_query)
bright_set

faint_query = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             AND p.cModelMag_r > 19.5
             ORDER BY p.objid ASC
               """
faint_set = SDSS.query_sql(faint_query)
faint_set


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Solution 1b
Write your answer here
Finally, to finish off our preparation of the data - we need to create an independent test that will be used to evalute the accuracy/generalization properies of the model after everything has been tuned. Often, independent test sets are generated by witholding a fraction of the training set. No hard and fast rules apply for the fraction to be withheld, though typical choices vary between $\sim{0.2}-0.5$. For this problem we will adopt 0.3.
sklearn.model_selection has a handy function train_test_split, which will simplify this process.
Problem 1c Split the 10k spectroscopic sources 70-30 into training and test sets. Save the results in arrays called: train_X, train_y, test_X, test_y, respectively. Use rs for the random_state in train_test_split.
Hint - recall that sklearn utilizes X, a 2D np.array(), and y as the features and labels arrays, respecitively.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
rs = 2  # we are in second biggest metropolitan area in the US

# complete

X = np.array( # complete
y = np.array( # complete

train_X, test_X, train_y, test_y = train_test_split( X, y,  # complete


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Solution 2c
Wow! Normalizing the features produces a huge ($\sim{35}\%$) increase in accuracy. Clearly, we should be using normalized magnitude features moving forward.
In addition to demonstrating the importance of feature engineering, this exercise teaches another important lesson: contextual features can be dangerous.
Contextual astronomical features can provide very strong priors: stars are more likely close to the galactic plane, supernovae occur next to/on top of galaxies, bluer stars have have lower metallicity, etc. Thus, including contextual information may improve overall model performance.
However, all astronomical training sets are heavily biased. Thus, the strong priors associated with contextual features can lead to severely biased model predictions.
Generally, I (AAM) remove all contextual features from my ML models for this reason. If you are building ML models, consider contextual information as it may help overall performance, but... be weary.
Worry about the data
Problem 3) Model Building
After the data have been properly curated, the next important choice in the ML workflow is the selection of ML algorithm. With experience, it is possible to develop intuition for the best ML algorithm given a specific problem.
Short of that? Try three (or four, or five) different models and choose whichever works the best.
For the star-galaxy problem, we will use the Random Forest (RF) algorithm (Breiman 2001) as implemented by scikit-learn.
RandomForestClassifier is part of the sklearn.ensemble module.
RF has a number of nice properties for working with astronomical data:

relative insensitivity to noisy or useless features
invariant response to highly non-gaussian feature distributions
fast, flexible and scales well to large data sets

which is why we will adopt it here.
Problem 3a
Build a RF model using the normalized features from the training set.
Include 25 trees in the forest using the n_estimators paramater in RandomForestClassifier.
Can you write Python code for it?

import # complete
rs = 626 # aread code for Pasadena

train_Xnorm = # complete

RFclf = # complete


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
Solution 5f
When building galaxy 2-point correlation functions it is very important to avoid including stars in the statistics as they will bias the final measurement.
Finally - always remember:
worry about the data
Challenge Problem) Taking the Plunge
Applying the model to field data
QSOs are unresolved sources that look like stars in optical imaging data. We will now download photometric measurements for 10k QSOs from SDSS and see how accurate the RF model performs for these sources.
Challenge 1
Calculate the accuracy with which the model classifies QSOs based on the 10k QSOs selected with the above command. How does that accuracy compare to that estimated by the test set?
Can you write Python code for it?


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
The simple SDSS model sets a high standard! A $\sim{96}\%$ accuracy following a single hard cut is a phenomenal  performance.
Problem 4b Using 10-fold cross validation, estimate the accuracy of the RF model.
Can you write Python code for it?

from sklearn.model_selection import # complete

RFpreds = # complete

print("The CV accuracy for the training set is {:.4f}".format( # complete


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
While (in this case) the affect is small, it is clear that $N_\mathrm{tree}$ affects the model output.
Now we will optimize the model over all tuning parameters. How does one actually determine the optimal set of tuning parameters?
Brute force.
This data set and the number of tuning parameters is small, so brute force is appropriate (alternatives exist when this isn't the case). We can optimize the model via a grid search that performs CV at each point in the 3D grid. The final model will adopt the point with the highest accuracy.
It is important to remember two general rules of thumb: (i) if the model is optimized at the edge of the grid, refit a new grid centered on that point, and (ii) the results should be stable in the vicinity of the grid maximum. If this is not the case the model is likely overfit.
Problem 5b
Use GridSearchCV to perform a 3-fold CV grid search to optimize the RF star-galaxy model. Remember the rules of thumb.
What are the optimal tuning parameters for the model?
Hint 1 - think about the computational runtime based on the number of points in the grid. Do not start with a very dense or large grid.
Hint 2 - if the runtime is long, don't repeat the grid search even if the optimal model is on an edge of the grid
Can you write Python code for it?

rs = 64 # average temperature in Los Angeles

from sklearn.model_selection import GridSearchCV

grid_results = # complete

print("The optimal parameters are:")
for key, item in grid_results.best_params_.items(): # warning - slightly different meanings in Py2 & Py3
    print("{}: {}".format(key, item))


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflow.ipynb".
The first task is:
While it is possible to look up each of the names of the $r$-band magnitudes in the SDSS PhotoObjAll schema, the schema list is long, and thus difficult to parse by eye. Fortunately, we can identify the desired columns using the database itself:
select COLUMN_NAME
from INFORMATION_SCHEMA.Columns
where table_name = 'PhotoObjAll' AND
COLUMN_NAME like '%Mag/_r' escape '/'

which returns the following list of columns: psfMag_r, fiberMag_r, fiber2Mag_r, petroMag_r, deVMag_r, expMag_r, modelMag_r, cModelMag_r.
We now select these magnitude measurements for 10000 stars and galaxies from SDSS. Additionally, we join these photometric measurements with the SpecObjAll table to obtain their spectroscopic classifications, which will serve as labels for the machine-learning model.
Note - the SDSS database contains duplicate observations, flagged observations, and non-detections, which we condition the query to exclude (as explained further below). We also exclude quasars, as the precise photometric classification of these objects is ambiguous: low-$z$ AGN have resolvable host galaxies, while high-$z$ QSOs are point-sources. Query conditions:

p.mode = 1 select only the primary photometric detection of a source
s.sciencePrimary = 1 select only the primary spectroscopic detection of a source (together with above, prevents duplicates)
p.clean = 1 the SDSS clean flag excludes flagged observations and sources with non-detections
s.class != 'QSO' removes potentially ambiguous QSOs from the training set
Can you write Python code for it?

sdss_query = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             ORDER BY p.objid ASC
               """
sdss_set = SDSS.query_sql(sdss_query)
sdss_set


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Now that the model is fully optimized - we are ready for the moment of truth!
Problem 5c
Using the optimized model parameters, train a RF model and estimate the model's generalization error using the test set.
How does this compare to the baseline model?
Can you write Python code for it?

RFopt_clf = # complete

test_preds = # complete

print('The optimized model produces a generalization error of {:.4f}'.format( # complete

RFopt_clf = RandomForestClassifier(n_estimators=30, max_features=3, min_samples_leaf=10)
RFopt_clf.fit(train_Xnorm, train_y)

test_Xnorm = test_X[:,0][:,np.newaxis] - test_X[:,1:]
test_preds = RFopt_clf.predict(test_Xnorm)

print('The optimized model produces a generalization error of {:.4f}'.format(1 - accuracy_score(test_y, test_preds)))