I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 1b
Based on your plots of the data, which feature do you think will be the most important for separating stars and galaxies? Why?
write your answer here - do not change it after later completing the problem
The final data preparation step it to create an independent test set to evalute the generalization error of the final tuned model. Independent test sets are generated by witholding a fraction of the training set. No hard and fast rules apply for the fraction to be withheld, though typical choices vary between $\sim{0.2}-0.5$.
sklearn.model_selection has a useful helper function train_test_split.
Problem 1c Split the 20k spectroscopic sources 70-30 into training and test sets. Save the results in arrays called: train_X, train_y, test_X, test_y, respectively. Use rs for the random_state in train_test_split.
Hint - recall that sklearn utilizes X, a 2D np.array(), and y as the features and labels arrays, respecitively.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
rs = 1851

feats = list(sdss_df.columns)
feats.remove('class')

X = np.array(sdss_df[feats])
y = np.array(sdss_df['class'])

train_X, test_X, train_y, test_y = train_test_split( X, y, test_size = 0.3, random_state = rs)


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 2b
Train a Random Forest (RF) model (Breiman 2001) on the training set. Include 50 trees in the forest using the n_estimators parameter. Again, set random_state = rs.
Hint - use the RandomForestClassifier object from the sklearn.ensemble module. Also - be sure to set n_jobs = -1 in every call of RandomForestClassifier.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

rf_clf = RandomForestClassifier(n_estimators=50, random_state=rs, n_jobs=-1)
rf_clf.fit(train_X, train_y)


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 3b
Use 10-fold cross validation to estimate the FoM for the $k$NN model. Take the mean value across all folds as the FoM estimate.
Hint - the cross_val_score function from the sklearn.model_selection module performs the necessary calculations.
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

knn_cv = cross_val_score(knn_clf, train_X, train_y, cv=10)

print('The kNN model FoM = {:.4f} +/- {:.4f}'.format(np.mean(knn_cv), np.std(knn_cv, ddof=1)))


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 5c
Calculate the confusion matrix for the test set. Is there symmetry to the misclassifications?
Hint - the confusion_matrix function in sklearn.metrics will help.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

print(confusion_matrix(test_y, test_preds))


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 6) Classify New Data
Run the cell below to load in some new data (which in this case happens to have known labels, but in practice this will almost never be the case...)
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
Problem 6b
Calculate the accuracy of the model predictions on the new data.
Can you write Python code for it?

new_preds = rf_clf.predict(new_X)
print("The model has an accuracy of {:.4f}".format(accuracy_score(new_y, new_preds)))


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
We will now ignore everything in the test set until we have fully optimized the machine learning model.
Problem 2) Model Building
After curating the data, you must select a specific machine learning algorithm. With experience, it is possible to develop intuition for the best ML algorithm given a specific problem.
Short of that? Try two (or three, or four, or five) different models and choose whichever works the best.
Problem 2a
Train a $k$-nearest neighbors model on the star-galaxy training set. Select $k$ = 25 for this model.
Hint - the KNeighborsClassifier object in the sklearn.neighbors module may be useful for this task.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

knn_clf = KNeighborsClassifier(n_neighbors=25)
knn_clf.fit(train_X, train_y)


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
write your answer here
Problem 5d
Calculate (and plot the region of interest) the ROC curve assumming that stars are the positive class.
Hint 1 - you will need to calculate probabilistic classifications for the test set using the predict_proba() method.
Hint 2 - the roc_curve function in the sklearn.metrics module will be useful.
Can you write Python code for it?

from sklearn.metrics import roc_curve

test_y_int = np.ones_like(test_y, dtype=int)
test_y_int[np.where(test_y == 'GALAXY')] = 0

test_preds_proba = rf_clf.predict_proba(test_X)

fpr, tpr, thresh = roc_curve(test_y_int, test_preds_proba[:,1])

fig, ax = plt.subplots()
ax.plot(fpr, tpr)
ax.set_xlabel('FPR')
ax.set_ylabel('TPR')
ax.set_xlim(2e-3,.2)
ax.set_ylim(0.3,1)


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
1. Load Data
We are ready to get started! :)  Start by loading the data and confirming that feats has the same number of columns as COL_NAMES.  Please note that the last columns is a class label with values {0, 1}, where 0=bogus, and 1=real.  Today we are doing unsupervised learning, but some clustering evaluation methods use labels to quantitatively measure the quality of the clustering result.
Can you write Python code for it?

meta = np.load(F_META)
feats = np.load(F_FEATS)

COL_NAMES = ['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap',
             'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 'sky',
             'magdiff', 'fwhm', 'classtar', 'mindtoedge', 'magfromlim', 'seeratio',
             'aimage', 'bimage', 'aimagerat', 'bimagerat', 'elong', 'nneg',
             'nbad', 'ssdistnr', 'ssmagnr', 'sumrat', 'magapbig', 'sigmagapbig',
             'ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 'scorr', 'label']

# INSTRUCTION: Verify that feats has the same number of columns as COL_NAMES
#

print("The number of rows, columns = {}, {}".format(feats.shape[0], feats.shape[1]))
print("The number of features + label = {}".format(len(COL_NAMES)))


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
2. Plot Features
We will perform K-means clustering using two features: 'chipsf' and 'elong'.  Chipsf is the uncertainty associated with performing PSF-fit photometry.  The higher the chi values, the more uncertainty associated with the source's PSF fit. Elong is a measure of how elongated the source is.  A transient point source should have a spherical point spread function.  An elongated point source may be a sign of a problem with image subtraction.
Extract features chipsf and along from the data.  Scatter plot them together, and also plot their histograms.
Question: What do you notice about these features?
Can you write Python code for it?

featnames_to_select = ['chipsf', 'elong']

# Extract the Correct Features
#
featidxs_to_select_indices = [ COL_NAMES.index(x) for x in featnames_to_select]
feats_selected = feats[:,featidxs_to_select_indices]

# Scatter Plot the Two Features
#
def plot_scatter(dat, xlabel, ylabel, xscale='linear', yscale='linear'):
    plt.plot(dat[:,0], dat[:,1], 'k.')
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.xscale(xscale)
    plt.yscale(yscale)
    plt.show()

# Scatter Plot the Two Features
#
def plot_histogram(dat, bins, title, xscale='linear', yscale='linear'):
    plt.hist(dat, bins)
    plt.xscale(xscale)
    plt.yscale(yscale)
    plt.title(title)
    plt.show()

# INSTRUCTION: Scatter Plot the Data
#
plot_scatter(feats_selected, featnames_to_select[0], featnames_to_select[1], 'log', 'linear')

# INSTRUCTION: Plot the Histograms for both features.  Hint, it may be helpful to plot some features on a log scale.
#
plot_histogram(feats_selected[:,0], np.logspace(np.log10(0.01),np.log10(10000.0), 50), featnames_to_select[0], 'log', 'linear')
plot_histogram(feats_selected[:,1], np.arange(1, 2, 0.01), featnames_to_select[1], 'linear', 'linear')


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
5. Quantitative Cluster Evaluation
So far, we've been visually verifying our clusters.  Let's use quantitative methods to verify our results.
The following is a score that does not require labels:
- inertia: "Sum of squared distances of samples to their closest cluster center."
- Silhouette coefficient: Measures minimal inertia in ratio to distance to next nearest cluster.  The score is higher are clusters become more compact and well-separated.
The following scores do require labels, and are documented here.

ARI, AMI measure the similarity between ground_truth labels and predicted_labels.  ARI measure similarity, and AMI measures in terms of mutual information. Random assignments score close to 0, correct assignments close to 1.
homogeneity: purity of the cluster (did all cluster members have the same label?). Scores in [0,1] where 0 is bad.
completeness: did all labels cluster together in a single cluster? Scores in [0,1] where 0 is bad.
Can you write Python code for it?

sample_size = 300

def bench_k_means(estimator, name, data, labels):
    t0 = time()
    estimator.fit(data)
    print('%-9s\t%.2fs\t%i\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=sample_size)))

labels = feats[:,-1]
print(82 * '_')
print('init\t\ttime\tinertia\thomo\tcompl\tv-meas\tARI\tAMI\tsilhouette')

# INSTRUCTIONS: Use the bench_k_means method to compare your clustering results
#
bench_k_means(kmeans_k2_scaled, 'K2 scaled', feats_selected_scaled, labels)
bench_k_means(kmeans_k5_scaled, 'K5 scaled', feats_selected_scaled, labels)


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
6.  Cluster Evaluation by Visual Inspection
This time with postage stamps!
It can be tempting to let yourself be guided by metrics alone, and the metrics are useful guideposts that can help determine whether you're moving in the right direction.  However, the goal of clustering is to reveal structure in your dataset.  Fortunately, because the features were extracted from sources that were extracted from images, we can view the cutouts from each source to visually verify whether our clusters contain homogeneous objects.
The display methods below give you an opportunity to display random candidates from each cluster, or the candidates that are closest to the cluster center.
Can you write Python code for it?

def display_stamps(candids, fig_title):

    # display five across
    num_per_row = 5

    for i, candid in enumerate(candids):
        f_stamp = glob.glob(os.path.join(D_STAMPS, 'candid{}*.png'.format(candid)))[0] # there should only be one file returned!
        if (i % num_per_row) == 0:
            fig = plt.figure(figsize=(18, 3))
            fig.suptitle(fig_title)

        ax = fig.add_subplot(1, num_per_row, i%num_per_row + 1)
        ax.set_axis_off()
        ax.set_title(candid)
        stamp = imread(f_stamp)
        imshow(stamp)
    return

def closest_to_centroid(centroid, cluster_feats, cluster_candids):

    dists = euclidean_distances(cluster_feats, centroid.reshape(1, -1))[:,0]
    closest_indices = np.argsort(dists)[:10]
    return cluster_candids[closest_indices]

def show_cluster_stamps(kmeans_res, displayMode='closest', num_to_display=10):
    # spits out a random selection of stamps from each cluster

    for i in range(kmeans_res.n_clusters):
        centroid = kmeans_res.cluster_centers_[i, :]
        mask = kmeans_res.labels_ == i
        cluster_candids = meta[mask]['candid']
        cluster_feats = feats_selected_scaled[mask]
        if displayMode == 'near_centroid':
            selected_candids = closest_to_centroid(centroid, cluster_feats, cluster_candids)
        if displayMode == 'random':
            np.random.shuffle(cluster_candids)
            selected_candids = cluster_candids[:num_to_display]
        display_stamps(selected_candids, 'Cluster {}'.format(i))

# INSTRUCTION: Use the show_cluster_stamps method to display cutouts associated with each cluster.
# Do you see similar objects in each cluster?
#
show_cluster_stamps(kmeans_k5_scaled, 'near_centroid')


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
7.  Clustering in a Dimensionally-Reduced Space
Given the tools seen above, starting clustering more than 2 features at a time.  This work is free-form.  I'll start you off with some suggested features.  After plotting the feature distributions, you may choose to down-select further.
Because we're now working with more than 2 features, use PCA to project the feature space onto its first two principal components.  You may use the methods above to run KMeans in that reduced feature space and evaluate your results.
Can you write Python code for it?

featnames_to_select = ['chipsf', 'elong', 'diffmaglim', 'magpsf', 'sigmapsf',
                       'chipsf', 'magap', 'sigmagap', 'sky', 'magdiff', 'fwhm',
                       'mindtoedge', 'magfromlim', 'seeratio', 'aimage', 'bimage',
                       'aimagerat', 'bimagerat', 'elong', 'nneg', 'nbad', 'sumrat', 'magapbig', 'sigmagapbig']

# INSTRUCTION: Visualize these features.  Discard any you consider to be problematic.

# INSTRUCTION: Filter the feature space

# INSTRUCTION: Run PCA on this feature space to reduce it to 2 principal components

# INSTRUCTION: Run KMeans on this 2-dimensional PCA space, and evaluate your results both quantatively and qualitatively.


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources-AnswerKey.ipynb".
The first task is:
Answer
The chipsf feature is creating a problem.  Its range of values, compared to the range of elong, is distorting the distance calculation.  KMeans is effectively partitioning along the value of the this feature alone.
4. Feature Scaling
We just discovered that distance metrics can be sensitive to the scale of your data (e.g., some features span large numeric ranges, but others don't).  For machine learning methods that calculate similiarty between feature vectors, it is important to normalize data within a standard range such as (0, 1) or with z-score normalization (scaling to unit mean and variance).  Fortunately, sklearn also makes this quite easy.  Please review sklearn's preprocessing module options, specifically StandardScaler which corresponds to z-score normalization and MinMaxScaler.  Please implement one.
After your data has been scaled, scatter plot your rescaled features, and run KMeans with the transformed data.  Compare the results on the transformed data with those above.
Can you write Python code for it?

# INSTRUCTION: Re-scale your data using either the MinMaxScaler or StandardScaler from sklearn
#
# feats_selected_scaled = MinMaxScaler().fit_transform(feats_selected)
feats_selected_scaled = StandardScaler().fit_transform(feats_selected)

# INSTRUCTION: Scatter plot your rescaled data
#
plot_scatter(feats_selected_scaled, 'chipsf', 'elong', 'linear', 'linear')

# INSTRUCTION: Retry KMeans with the same values of k used above.

kmeans_k2_scaled = runKMeans(feats_selected_scaled)
plotKMeans(kmeans_k2_scaled, feats_selected_scaled, featnames_to_select[0], featnames_to_select[1], 'linear', 'linear')
kmeans_k5_scaled = runKMeans(feats_selected_scaled, 5)
plotKMeans(kmeans_k5_scaled, feats_selected_scaled, featnames_to_select[0], featnames_to_select[1], 'linear', 'linear')


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
1. Load Data
We are ready to get started! :)  Start by loading the data and confirming that feats has the same number of columns as COL_NAMES.  Please note that the last columns is a class label with values {0, 1}, where 0=bogus, and 1=real.  Today we are doing unsupervised learning, but some clustering evaluation methods use labels to quantitatively measure the quality of the clustering result.
Can you write Python code for it?

meta = np.load(F_META)
feats = np.load(F_FEATS)

COL_NAMES = ['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap',
             'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 'sky',
             'magdiff', 'fwhm', 'classtar', 'mindtoedge', 'magfromlim', 'seeratio',
             'aimage', 'bimage', 'aimagerat', 'bimagerat', 'elong', 'nneg',
             'nbad', 'ssdistnr', 'ssmagnr', 'sumrat', 'magapbig', 'sigmagapbig',
             'ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 'scorr', 'label']

# INSTRUCTION: Verify that feats has the same number of columns as COL_NAMES
#


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
3. KMeans Using Two Features
We rarely ever cluster only two features from a dataset.  However, the advantage of doing so is that we can readily visualize two-dimensional data.  Let's start off by clustering features elong and chipsf with KMeans.  The plotKMeans function below implements a visualization of KMean's partitioning that was used in sklearn's KMean's demo.
Question: What do you think about the quality of the clusterings produced?
Can you write Python code for it?

def runKMeans(dat, n_clusters=2, seed=0):
        return KMeans(n_clusters, random_state=seed).fit(dat)

def plotKMeans(kmeans_res, reduced_dat, xlabel, ylabel, xscale='linear', yscale='linear'):

    # Plot the decision boundary. For that, we will assign a color to each
    h = .02     # point in the mesh [x_min, x_max]x[y_min, y_max].
    x_min, x_max = reduced_dat[:, 0].min() - 1, reduced_dat[:, 0].max() + 1
    y_min, y_max = reduced_dat[:, 1].min() - 1, reduced_dat[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Obtain labels for each point in mesh. Use last trained model.
    Z = kmeans_res.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.imshow(Z, interpolation='nearest',
               extent=(xx.min(), xx.max(), yy.min(), yy.max()),
               cmap=plt.cm.Paired,
               aspect='auto', origin='lower')
    plt.plot(reduced_dat[:,0], reduced_dat[:,1], 'k.')
    plt.scatter(kmeans_res.cluster_centers_[:, 0], kmeans_res.cluster_centers_[:, 1],
                marker='x', s=169, linewidths=3,
                color='w', zorder=10)
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.xscale(xscale)
    plt.yscale(yscale)
    plt.show()

# INSTRUCTION: Use the runKMeans and plotKMeans functions to cluster the data (feats_selected)
# with several values of k.


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
4. Feature Scaling
We just discovered that distance metrics can be sensitive to the scale of your data (e.g., some features span large numeric ranges, but others don't).  For machine learning methods that calculate similiarty between feature vectors, it is important to normalize data within a standard range such as (0, 1) or with z-score normalization (scaling to unit mean and variance).  Fortunately, sklearn also makes this quite easy.  Please review sklearn's preprocessing module options, specifically StandardScaler which corresponds to z-score normalization and MinMaxScaler.  Please implement one.
After your data has been scaled, scatter plot your rescaled features, and run KMeans with the transformed data.  Compare the results on the transformed data with those above.
Can you write Python code for it?

# INSTRUCTION: Re-scale your data using either the MinMaxScaler or StandardScaler from sklearn
#

# INSTRUCTION: Scatter plot your rescaled data
#

# INSTRUCTION: Retry KMeans with the same values of k used above.
#


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
5. Quantitative Cluster Evaluation
So far, we've been visually verifying our clusters.  Let's use quantitative methods to verify our results.
The following is a score that does not require labels:
- inertia: "Sum of squared distances of samples to their closest cluster center."
- Silhouette coefficient: Measures minimal inertia in ratio to distance to next nearest cluster.  The score is higher are clusters become more compact and well-separated.
The following scores do require labels, and are documented here.

ARI, AMI measure the similarity between ground_truth labels and predicted_labels.  ARI measure similarity, and AMI measures in terms of mutual information. Random assignments score close to 0, correct assignments close to 1.
homogeneity: purity of the cluster (did all cluster members have the same label?). Scores in [0,1] where 0 is bad.
completeness: did all labels cluster together in a single cluster? Scores in [0,1] where 0 is bad.
Can you write Python code for it?

sample_size = 300

def bench_k_means(estimator, name, data, labels):
    t0 = time()
    estimator.fit(data)
    print('%-9s\t%.2fs\t%i\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=sample_size)))

labels = feats[:,-1]
print(82 * '_')
print('init\t\ttime\tinertia\thomo\tcompl\tv-meas\tARI\tAMI\tsilhouette')

# INSTRUCTIONS: Use the bench_k_means method to compare your clustering results
#


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
6.  Cluster Evaluation by Visual Inspection
This time with postage stamps!
It can be tempting to let yourself be guided by metrics alone, and the metrics are useful guideposts that can help determine whether you're moving in the right direction.  However, the goal of clustering is to reveal structure in your dataset.  Fortunately, because the features were extracted from sources that were extracted from images, we can view the cutouts from each source to visually verify whether our clusters contain homogeneous objects.
The display methods below give you an opportunity to display random candidates from each cluster, or the candidates that are closest to the cluster center.
Can you write Python code for it?

def display_stamps(candids, fig_title):

    # display five across
    num_per_row = 5

    for i, candid in enumerate(candids):
        f_stamp = glob.glob(os.path.join(D_STAMPS, 'candid{}*.png'.format(candid)))[0] # there should only be one file returned!
        if (i % num_per_row) == 0:
            fig = plt.figure(figsize=(18, 3))
            fig.suptitle(fig_title)

        ax = fig.add_subplot(1, num_per_row, i%num_per_row + 1)
        ax.set_axis_off()
        ax.set_title(candid)
        stamp = imread(f_stamp)
        imshow(stamp)
    return

def closest_to_centroid(centroid, cluster_feats, cluster_candids):

    dists = euclidean_distances(cluster_feats, centroid.reshape(1, -1))[:,0]
    closest_indices = np.argsort(dists)[:10]
    return cluster_candids[closest_indices]

def show_cluster_stamps(kmeans_res, displayMode='closest', num_to_display=10):
    # spits out a random selection of stamps from each cluster

    for i in range(kmeans_res.n_clusters):
        centroid = kmeans_res.cluster_centers_[i, :]
        mask = kmeans_res.labels_ == i
        cluster_candids = meta[mask]['candid']
        cluster_feats = feats_selected_scaled[mask]
        if displayMode == 'near_centroid':
            selected_candids = closest_to_centroid(centroid, cluster_feats, cluster_candids)
        if displayMode == 'random':
            np.random.shuffle(cluster_candids)
            selected_candids = cluster_candids[:num_to_display]
        display_stamps(selected_candids, 'Cluster {}'.format(i))

# INSTRUCTION: Use the show_cluster_stamps method to display cutouts associated with each cluster.
# Do you see similar objects in each cluster?
#


I am working on the file "Sessions/Session07/Day2/Clustering-Astronomical-Sources.ipynb".
The first task is:
7.  Clustering in a Dimensionally-Reduced Space
Given the tools seen above, starting clustering more than 2 features at a time.  This work is free-form.  I'll start you off with some suggested features.  After plotting the feature distributions, you may choose to down-select further.
Because we're now working with more than 2 features, use PCA to project the feature space onto its first two principal components.  You may use the methods above to run KMeans in that reduced feature space and evaluate your results.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model-Answer-Key.ipynb".
The first task is:
1. Load Data
Please note that I have made some adjustments to the data.

I have created a dataframe (feats_df) from feats_np
I have also dropped columns from feats_df.  Some were irrelevant to classification, and some contained a lot of NaNs.
Can you write Python code for it?

meta_np = np.load(F_META)
feats_np = np.load(F_FEATS)

COL_NAMES = ['diffmaglim', 'magpsf', 'sigmapsf', 'chipsf', 'magap', 'sigmagap',
             'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr', 'sky',
             'magdiff', 'fwhm', 'classtar', 'mindtoedge', 'magfromlim', 'seeratio',
             'aimage', 'bimage', 'aimagerat', 'bimagerat', 'elong', 'nneg',
             'nbad', 'ssdistnr', 'ssmagnr', 'sumrat', 'magapbig', 'sigmagapbig',
             'ndethist', 'ncovhist', 'jdstarthist', 'jdendhist', 'scorr', 'label']

# NOTE FROM Umaa: I've decided to eliminate the following features. Dropping them.
#
COL_TO_DROP = ['ndethist', 'ncovhist', 'jdstarthist', 'jdendhist',
               'distnr', 'magnr', 'sigmagnr', 'chinr', 'sharpnr',
               'classtar', 'ssdistnr', 'ssmagnr', 'aimagerat', 'bimagerat',
               'magapbig', 'sigmagapbig', 'scorr']
feats_df = pd.DataFrame(data=feats_np, index=meta_np['candid'], columns=COL_NAMES)
print("There are {} columns left.".format(len(feats_df.columns)))
print("They are: {}".format(list(feats_df.columns)))
feats_df.drop(columns=COL_TO_DROP, inplace=True)
feats_df.describe()

# INSTRUCTION: How many real and bogus examples are in this labeled set
#
real_mask = feats_df['label'] == 1
bogus_mask = ~real_mask
print("Number of Real Examples: {}".format(np.sum(real_mask)))
print("Number of Bogus Examples: {}".format(np.sum(bogus_mask)))


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model-Answer-Key.ipynb".
The first task is:
2. Plot Features
Examine each individual feature. You may use the subroutine below, or code of your own devising. Note the features that are continuous vs categorial, and those that have outliers.  There are certain features that have sentinel values.  You may wish to view some features on a log scale.
Question: Which features seem to have pathologies? Which features should we exclude?
Can you write Python code for it?

# Histogram a Single Feature
#
def plot_rb_hists(df, colname, bins=100, xscale='linear', yscale='linear'):
    mask_real = feats_df['label'] == 1
    mask_bogus = ~mask_real
    plt.hist(df[colname][mask_real], bins, color='b', alpha=0.5, density=True)
    plt.hist(df[colname][mask_bogus], bins, color='r', alpha=0.5, density=True)
    plt.xscale(xscale)
    plt.yscale(yscale)
    plt.title(colname)
    plt.show()

# INSTRUCTION: Plot the individual features.
#
for col in feats_df.columns:
    if col in ['chipsf', 'sky', 'fwhm']:
        plot_rb_hists(feats_df, col, bins=np.logspace(np.log10(0.01),np.log10(10000.0), 50), xscale='log')
    else:
        plot_rb_hists(feats_df, col)


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model-Answer-Key.ipynb".
The first task is:
Answer:

'nbad' and 'nneg' are discrete features
'seeratio' has a sentinel value of -999
There are some features that have high ranges than others.  For classifiers that are sensitive to scaling, we will need to scale the data
There does not appear to be good separability between real and bogus on any individual features.

3. Curate a Test and Training Set
We need to reserve some of our labeled data for evaluation. This means we must split up the labeled data we have into the set used for training (training set), and the set used for evaluation (test set). Ideally, the distribution of real and bogus examples in both the training and test sets are roughly identical. One can use sklearn.model_selection.train_test_split and use the stratify option.
For ZTF data, we split the training and test data by date. That way repeat observations from the same night (which might be nearly identical) cannot be split into the training and test set, and artificially inflate test performance. Also, due to the change in survey objectives, it's possible that the test set features have drifted away from the training sets.
Provided is nid.npy which contains the Night IDs for ZTF. Split on nid=550 (July 5, 2018). This should leave you with roughly 500 examples in your test set.
Can you write Python code for it?

feats_plus_label = np.array(feats_df)
nids = meta_np['nid']

# INSTRUCTION: nid.npy contains the nids for this labeled data.
# Split the data into separate data structures for train/test data at nid=500.
# Verify that you have at least 500 reals in your test set.

nid_mask_train = nids <= 550
nid_mask_test = ~nid_mask_train

train_plus_label = feats_plus_label[nid_mask_train,:]
test_plus_label = feats_plus_label[nid_mask_test,:]

nreals_train = np.sum(train_plus_label[:,-1] == 1)
nbogus_train = np.sum(train_plus_label[:,-1] == 0)
nreals_test = np.sum(test_plus_label[:,-1] == 1)
nbogus_test = np.sum(test_plus_label[:,-1] == 0)

print("TRAIN Num Real={}, Bogus={}".format(nreals_train, nbogus_train))
print("TEST Num Real={}, Bogus={}".format(nreals_test, nbogus_test))


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model-Answer-Key.ipynb".
The first task is:
Part 2. Scaling Data
With missing values handled, you're closing to training your classifiers.  However, because distance metrics can be sensitive to the scale of your data (e.g., some features span large numeric ranges, but others don't), it is important to normalize data within a standard range such as (0, 1) or with z-score normalization (scaling to unit mean and variance).  Fortunately, sklearn also makes this quite easy.  Please review sklearn's preprocessing module options, specifically StandardScaler which corresponds to z-score normalization and MinMaxScaler.  Please implement one.
FYI - Neural networks and Support Vector Machines (SVM) are sensitive to the scale of the data.  Decision trees (and therefore Random Forests) are not (but it doesn't hurt to use scaled data).
Can you write Python code for it?

# INSTRUCTION: Re-scale your data using either the MinMaxScaler or StandardScaler from sklearn
#

scaler = StandardScaler()
train_feats = scaler.fit_transform(train_feats)
test_feats = scaler.fit_transform(test_feats)


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model.ipynb".
The first task is:
Answer:

'nbad' and 'nneg' are discrete features
'seeratio' has a sentinel value of -999
There are some features that have high ranges than others.  For classifiers that are sensitive to scaling, we will need to scale the data
There does not appear to be good separability between real and bogus on any individual features.

3. Curate a Test and Training Set
We need to reserve some of our labeled data for evaluation. This means we must split up the labeled data we have into the set used for training (training set), and the set used for evaluation (test set). Ideally, the distribution of real and bogus examples in both the training and test sets are roughly identical. One can use sklearn.model_selection.train_test_split and use the stratify option.
For ZTF data, we split the training and test data by date. That way repeat observations from the same night (which might be nearly identical) cannot be split into the training and test set, and artificially inflate test performance. Also, due to the change in survey objectives, it's possible that the test set features have drifted away from the training sets.
Provided is nid.npy which contains the Night IDs for ZTF. Split on nid=550 (July 5, 2018). This should leave you with roughly 500 examples in your test set.
Can you write Python code for it?

feats_plus_label = np.array(feats_df)
nids = meta_np['nid']

# INSTRUCTION: nid.npy contains the nids for this labeled data.
# Split the data into separate data structures for train/test data at nid=500.
# Verify that you have at least 500 reals in your test set.


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model.ipynb".
The first task is:
Part 2. Scaling Data
With missing values handled, you're closing to training your classifiers.  However, because distance metrics can be sensitive to the scale of your data (e.g., some features span large numeric ranges, but others don't), it is important to normalize data within a standard range such as (0, 1) or with z-score normalization (scaling to unit mean and variance).  Fortunately, sklearn also makes this quite easy.  Please review sklearn's preprocessing module options, specifically StandardScaler which corresponds to z-score normalization and MinMaxScaler.  Please implement one.
FYI - Neural networks and Support Vector Machines (SVM) are sensitive to the scale of the data.  Decision trees (and therefore Random Forests) are not (but it doesn't hurt to use scaled data).
Can you write Python code for it?

# INSTRUCTION: Re-scale your data using either the MinMaxScaler or StandardScaler from sklearn
#


I am working on the file "Sessions/Session07/Day3/Building-A-Supervised-Machine-Learning-Model.ipynb".
The first task is:
Part 3. Train Classifiers
Import a few classifiers and build models on your training data.  Some suggestions include a Support Vector Machine, Random Forest, Neural Net, NaiveBayes and K-Nearest Neighbor.
Can you write Python code for it?

from sklearn.neural_network import MLPClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB

knn3 = KNeighborsClassifier(3),
svml = SVC(kernel="linear", C=0.025, probability=True)
svmr = SVC(gamma=2, C=1, probability=True)
dtre = DecisionTreeClassifier()
rafo = RandomForestClassifier(n_estimators=100)
nnet = MLPClassifier(alpha=1)
naiv = GaussianNB()

# INSTRUCTION: Train at least three classifiers and run on your test data. Here's an example to get your started.
#

rafo.fit(train_feats, train_labels)
rafo_scores = rafo.predict_proba(test_feats)[:,1]

# INSTRUCTION: Print out the following metrics per classifier into a table: accuracy, auc, f1_score, etc.
#


I am working on the file "Sessions/Session07/Day3/Cleaning-Astronomical-Datasets.ipynb".
The first task is:
0b. Data Location
We will use the same data from the Day 2 clustering exercise (see that notebook to download the data).
Please specify paths for the following:
Can you write Python code for it?

F_META = '../Day2/dsfp_ztf_meta.npy'
F_FEATS = '../Day2/dsfp_ztf_feats.npy'
D_STAMPS = '../Day2/dsfp_ztf_png_stamps'


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
"Exerciiiiiiisee the neurons! This Network is cleeeah!"
(bonus points if you know the 90's movie, from which this is a near-quote.)

How well does the training go if we don't normalize the data to be between 0 and 1.0?
What is the effect of batch_size on how long it takes the network to achieve 80% accuracy?
What happens if you increase the validation set fraction (and thus reduce the training set size)? Try going really low on the training set size. Can you still get 80% accuracy?

Diagnostics!
Because neural networks are such complex objects, and in this era of experimentation, diagnostics (both standard and tailored) are critical for training NN models.
Evaluate overall model efficacy
Evaluate model on training and test data and compare. This provides summary values that are equivalent to the final value in the accuracy/loss history plots.
Can you write Python code for it?

loss_train, acc_train  = model.evaluate(x_train, y_train, verbose=False)
loss_test, acc_test  = model.evaluate(x_test, y_test, verbose=False)
print(f'Train acc/loss: {acc_train:.3}, {loss_train:.3}')
print(f'Test acc/loss: {acc_test:.3}, {loss_test:.3}')


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Create a Neural Net and train it!
Decide model format
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Define Confusion Matrix
Can you write Python code for it?

# Function: Convert from categorical back to numerical value
def convert_to_index(array_categorical):
  array_index = [np.argmax(array_temp) for array_temp in array_categorical]
  return array_index

def plot_confusion_matrix(cm,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function modified to plots the ConfusionMatrix object.
    Normalization can be applied by setting `normalize=True`.

    Code Reference :
    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html

    This script is derived from PyCM repository: https://github.com/sepandhaghighi/pycm

    """

    plt_cm = []
    for i in cm.classes :
        row=[]
        for j in cm.classes:
            row.append(cm.table[i][j])
        plt_cm.append(row)
    plt_cm = np.array(plt_cm)
    if normalize:
        plt_cm = plt_cm.astype('float') / plt_cm.sum(axis=1)[:, np.newaxis]
    plt.imshow(plt_cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(cm.classes))
    plt.xticks(tick_marks, cm.classes, rotation=45)
    plt.yticks(tick_marks, cm.classes)

    fmt = '.2f' if normalize else 'd'
    thresh = plt_cm.max() / 2.
    for i, j in itertools.product(range(plt_cm.shape[0]), range(plt_cm.shape[1])):
        plt.text(j, i, format(plt_cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if plt_cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('Actual')
    plt.xlabel('Predict')


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
For later: check out the keras documentation to see what other kinds of model formats are available. What might they be useful for?
Add layers to the model sequentially
Can you write Python code for it?

model.add(Dense(units=32, activation='sigmoid', input_shape=(img_size,))) # dense layer of 32 neurons
model.add(Dense(units=num_classes, activation='softmax')) # dense layer of 'num_classes' neurons, because these are the number of options for the classification
#model.add(BatchNormalization())
model.summary() # look at the network output


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Let's experiment with neural networks!
Learning Objectives

Gain familiarity with
Standard NN libraries: Keras and Tensorflow
One standard NN architecture: fully connected ('dense') networks
Two standard tasks performed with NNs: Binary Classification, Multi-class Classification
Diagnostics of NNs
        1.   History (Loss Function)
        2.   Receiver Operator Curve and Area Under the Curve

Experience fundamental considerations, pitfalls, and strategies when training NNs
Data set preparation (never underestimate the time required for this)
Training set size
Training speed and efficiency
Model fitting (training)

Begin connecting NN functionality to data set structure and problem of interest

Topics not covered

Class (im)balance
Training set diversity

Notes
Our main task will be to Classify Handwritten Digits (Is it a "zero" [0] or a "one" [1]?; ooooh, the suspense). This is very useful though, because it's an opportunity to separate yourself from the science of the data. MNIST is a benchmark data set for machine learning. Astronomy doesn't yet have machine learning-specific benchmark data sets.

Prepare the Data
For this MNIST set, this is the fastest it will probably ever take you to prepare a data set. Ye have been warn-ed.
Download the data
(ooh look it's all stored on Amazon's AWS!)
(pssst, we're in the cloooud, it's the future!)
Can you write Python code for it?

# import MNIST data
(x_train_temp, y_train_temp), (x_test_temp, y_test_temp) = mnist.load_data()


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Look at the data
(always do this so that you know what the structure is.)
Can you write Python code for it?

# Print the shapes
print("Train Data Shape:", x_train_temp.shape)
print("Test Data Shape:", x_test_temp.shape)
print("Train Label Shape:", y_train_temp.shape)
print("Test Label Shape:", y_test_temp.shape)


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Plot Confusion Matrix
Can you write Python code for it?

# apply conversion function to data
y_test_ind = convert_to_index(y_test)
y_pred_test_ind = convert_to_index(y_pred_test)

# compute confusion matrix
cm_test = ConfusionMatrix(y_test_ind, y_pred_test_ind)
np.set_printoptions(precision=2)

# plot confusion matrix result
plt.figure()
plot_confusion_matrix(cm_test,title='cm')


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Plot accuracy and loss as a function of epochs (equivalently training time)
Can you write Python code for it?

# set up figure
f = plt.figure(figsize=(12,5))
f.add_subplot(1,2, 1)

# plot accuracy as a function of epoch
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['training', 'validation'], loc='best')

# plot loss as a function of epoch
f.add_subplot(1,2, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['training', 'validation'], loc='best')
plt.show(block=True)


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Predict train and test data
Can you write Python code for it?

y_pred_train = model.predict(x_train, verbose=True)
y_pred_test = model.predict(x_test,verbose=True)


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Problems

Problem 1: Binary data

Problem 3a: Generate a binary data sets from the MNIST data set.
Problem 3b: Prepare the data set in the same way as the multi-class scenario, but apprioriately for only two classes.
Problem 3c: Re-train the neural network for the binary classification problem.
Problem 3d: Perform all the diagnostics again.
Problem 3e: Use sklearn to generate a Receiver Operator Curve (ROC) and the associated AUC.
Problem 3f: Create a new diagnostic that shows examples of the images and how they were classified. For a given architecture and training, plot four columns of images, five rows down. Each row is just another example of something that is classified. The four columns are
True positive
False positive
True negative
True postiive

So this will be a grid of images that are examples of classifications. This is important to give a sense of how well the network is classifying different types of objects.

Problem 2: How does the following metrics scale with the data set size? (Constructing a learning curve)

time required to perform 5 epochs.
accuracy and loss at final step

Some advice for this problem:
  * create a function or functions that can wrap around the NN architecture, and then output key results.
  * watch out for network initialization. If you don't redefine the network model each time, the weights will be preserved from the most recent training.
Sub-problems
1. Try training and evaluation for a number of well-spaced data set sizes and report on the pattern you see. Plot a "learning curve": the loss (or accuracy) as a function of the training set data size. This is an important step in judging what you need for your training set, and in assessing the feasibility of the application.
2. Do you see any change a function data size? If not, go back and train the original network for more epochs. When does the loss plateau? Use that epoch for the
3. Does keras have any model function fit parameters that automatically decide when it will stop? Try using that.

Problem 3: How do the...
Can you write Python code for it?

!gsutil cp -r gs://lsst-dsfp2018/stronglenses ./
!ls .


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Reshape the data arrays: flatten for ingestion into a Dense layer.
We're going to use a Dense Neural Network Architecture, which takes 1D data structures, not 2D images.
So we need to make the input shape appropriate, so we 'flatten' it.
Can you write Python code for it?

# Flatten the images
img_rows, img_cols = x_train[0].shape[0], x_train[0].shape[1]
img_size = img_rows*img_cols
x_train = x_train.reshape(x_train.shape[0], img_size)
x_test = x_test.reshape(x_test.shape[0], img_size)
print("New shape", x_train.shape)


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
Review the model summary, noting the major features --- the meaning of each column, the flow downward, the number of parameters at the bottom.
What's the math that leads to the number of parameters in each layer? This is a little bit of fun math to play with. In the long run it can help understand how to debug model compilation errors, and to design custom networks better. No need to dwell here too long, but give it a try.The ingredients are
image size
number of units in the layer
number of bias elements per unit

Compile the model
Select three key options
1.   optimizer: the method for optimizing the weights. "Stochastic Gradient Descent (SGD)" is the canonical method.
2.   loss function: the form of the function to encode the difference between the data's true label and the predict label.
3.   metric: the function by which the model is evaluated.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsGetNetworking.ipynb".
The first task is:
The optimizer is an element that I sometimes tune if I want more control over the training. Check the Keras docs for the optiosn available for the optimizer.
Fit (read: Train) the model
Can you write Python code for it?

# Training parameters
batch_size = 32 # number of images per epoch
num_epochs = 5 # number of epochs
validation_split = 0.8 # fraction of the training set that is for validation only

history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=num_epochs,
                    validation_split=validation_split,
                    verbose=True)


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Apply one-hot encoding to the data

Current encoding provides a literal label. For example, the label for "3"  is 3.
One-hot encoding places a "1" in an array at the appropriate location for that datum. For example, the label "3" becomes [0, 0, 0, 1, 0, 0, 0, 0, 0, 0]

This increases the efficiency of the matrix algebra during network training and evaluation.
Can you write Python code for it?

# One-hot encoding
num_classes = 10
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Can you see the pattern of the number in the array?
Can you write Python code for it?

# Plot the data!
f = plt.figure()
f.add_subplot(1,2, 1)
plt.imshow(x_train_temp[0])
f.add_subplot(1,2, 2)
plt.imshow(x_train_temp[1])
plt.show(block=True)


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Compile the autencoder
Can you write Python code for it?

autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
num_epochs = 10


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Create a separate model that is just the encoder
This will allow us to encode the images and look at what the encoding results in.
Can you write Python code for it?

encoder = Model(inputs=autoencoder.input, outputs=autoencoder.get_layer('flatten_8').output)
encoder.summary()


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Design Neural Network Architecture!
Select model format
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Look at the data
(always do this so that you know what the structure is.)
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Plot the input, output, and encoded images
Can you write Python code for it?

# set number of images to visualize
num_images = 10

# select random subsect to visualize
np.random.seed(42)
random_test_images = np.random.randint(x_test.shape[0], size=num_images)

# encode images
encoded_imgs = encoder.predict(x_test)

#decode encode AND decode images
decoded_imgs = autoencoder.predict(x_test)

# plot figure
plt.figure(figsize=(18, 4))

num_rows=4
num_pixel_x = 2
num_pixel_y = 4

for i, image_idx in enumerate(random_test_images):
    # plot original image
    ax = plt.subplot(4, num_images, i + 1)
    plt.imshow(x_test[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # plot encoded image
    ax = plt.subplot(num_rows, num_images, num_images + i + 1)

    plt.imshow(encoded_imgs[image_idx].reshape(num_pixel_x, num_pixel_y), interpolation=None, resample=None)
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # plot reconstructed image
    ax = plt.subplot(num_rows, num_images, 2*num_images + i + 1)
    plt.imshow(decoded_imgs[image_idx].reshape(28, 28))
    plt.gray()
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Predict train and test data
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Prepare the data
Data often need to be re-shaped and normalized for ingestion into the neural network.
Normalize the data
The images are recast as float and normalized to one for the network.
Can you write Python code for it?

print("Before:", np.min(x_train_temp), np.max(x_train_temp))
x_train = x_train_temp.astype('float32')
x_test = x_test_temp.astype('float32')
x_train /= 255
x_test /= 255
y_train = y_train_temp
y_test = y_test_temp
print("After:", np.min(x_train), np.max(x_train))


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Reshape the data arrays: set the input shape to be ready for a convolution [NEW]
We're going to use a Dense Neural Architecture, not as images, so we need to make the input shape appropriate.
Can you write Python code for it?

# read the dimensions from one example in the training set
img_rows, img_cols = x_train[0].shape[0], x_train[0].shape[1]

# Different NN libraries (e.g., TF) use different ordering of dimensions
# Here we set the "input shape" so that later the NN knows what shape to expect
if K.image_data_format() == 'channels_first':
    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Things to think about and notice:

How does the "output shape" column change as you go through the network? How does this relate to pictures of CNNs you've seen (or might find on google images, for example)?
What happens when you re-compile the cell where you add layers sequentially, without first compiling model-definition cell. Why does that happen?

Compile the model
Select three key options
1.   optimizer: the method for optimizing the weights. "Stochastic Gradient Descent (SGD)" is the canonical method.
2.   loss function: the form of the function to encode the difference between the data's true label and the predict label.
3.   metric: the function by which the model is evaluated.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day4/LetsHaveAConvo.ipynb".
The first task is:
Things to think about and notice:

How fast is this training compared to the Dense/Fully Connected Networks? What could be a causing a difference between these two networks?
Why is it taking a long time at the end of each epoch?

Diagnostics!
Evaluate overall model efficacy
Evaluate model on training and test data and compare. This provides summary values that are equivalent to the final value in the accuracy/loss history plots.
Can you write Python code for it?

loss_train, acc_train  = model.evaluate(x_train, y_train, verbose=False)
loss_test, acc_test  = model.evaluate(x_test, y_test, verbose=False)
print(f'Train acc/loss: {acc_train:.3}, {loss_train:.3}')
print(f'Test acc/loss: {acc_test:.3}, {loss_test:.3}')


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
And just like that - we have now created a new database intro.db, with which we can "store bytes" or later "retrieve bytes" once we have added some data to the database.
Aside - note that unlike many SQL libraries, SQLite does not require a server and creates an actual database file on your hard drive. This improves portability, but also creates some downsides as well.
Now we need to create a table and insert some data. We will interact with the database via the execute() method for the cursor object.
Recall that creating a table requires a specification of the table name, the columns in the table, and the data type for each column. Here's an example where I create a table to store info on my pets:
cur.execute("""create table PetInfo(
                                    Name text,
                                    Species text,
                                    Age tinyint,
                                    FavoriteFood text
                                    )""")

Problem 1a
Create a new table in the database called DSFPstudents with columns Name, Institution, and Year, where Year is the year in graduate school.
Can you write Python code for it?

cur.execute( # complete

cur.execute("""create table DSFPstudents(
                                    Name text,
                                    Institution text,
                                    Year tinyint
                                    )""")


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
Challenge Problem
To close the notebook we will perform a nested query. In brief, the idea is to join the results of one query with a separate query.
Here, we are going to attempt to identify bright AGN that don't have SDSS spectra. To do so we will need the photoObjAll table, the specObjAll table, and the rosat table, which includes all cross matches between SDSS sources and X-ray sources detected by the Rosat satellite.
Create a nested query that selects all Rosat sources that don't have SDSS spectra with cModelFlux_u + cModelFlux_g + cModelFlux_r + cModelFlux_i + cModelFlux_z > 10000 (this flux contraint ensures the source is bright without making any cuts on color) and type = 3, this last constraint means the source is extended in SDSS images.
Hint - you may run into timeout issues in which case you should run the query on CasJobs.
Can you write Python code for it?

SDSS.query_sql("""select rm.*
from
(select r.objid, r.sourcename, r.ra, r.dec, r.cps, r.hr1, r.hr2, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, cModelMag_z
from photoobjall p join rosat r on p.objid = r.objid
  where (cModelFlux_u + cModelFlux_g + cModelFlux_r + cModelFlux_i + cModelFlux_z > 10000)
    and p.type = 3) as rm
  left join specobjall p on rm.objid = p.bestobjid
  where p.bestobjid is null
                """)


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
Once a table is created, we can use the database to store bytes. If I were to populate my PetInfo table I would do the following:
cur.execute("""insert into PetInfo(Name, Species, Age, FavoriteFood)
                            values ("Rocky", "Dog", 12, "Bo-Nana")""")
cur.execute("""insert into PetInfo(Name, Species, Age, FavoriteFood)
                            values ("100 Emoji-Flames Emoji", "Red Panda", 2, "bamboo leaves")""")

Note - column names do not need to be explicitly specified, but for clarity this is always preferred.
Problem 1b
Insert data for yourself, and the two people sitting next to you into the database.
Can you write Python code for it?

cur.execute( # complete

cur.execute("""insert into DSFPstudents(Name, Institution, Year)
                            values ("Adam Miller", "Northwestern", 13)""")
cur.execute("""insert into DSFPstudents(Name, Institution, Year)
                            values ("Lucianne Walkowicz", "Adler", 14)""")


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
Problem 2c
Perform an identical query to the one above, but this time use a left outer join (or left join).
How do your results compare to the previous query?
Can you write Python code for it?

SDSS.query_sql("""select top 20 objid, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, cModelMag_z,
                         class
                  from photoobjall p
                      left outer join specobjall s on p.objid = s.bestobjid""")


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
Problem 2d
This time use a right outer join (or right join).
How do your results compare to the previous query?
Can you write Python code for it?

SDSS.query_sql("""select top 20 objid, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, cModelMag_z,
                         class
                  from photoobjall p
                      right outer join specobjall s on s.bestobjid = p.objid
                """)


I am working on the file "Sessions/Session08/Day1/IntroToSQLiteSolutions.ipynb".
The first task is:
That's more columns than we will likely ever need. Instead, let's focus on objID, a unique identifier, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, and cModelMag_z, the source magnitude in $u', g', r', i', z'$, respectively.
We will now introduce the concept of joining two tables.
The most common operation is known as an inner join (which is often referred to as just join). An inner join returns records that have matching sources in both tables in the join.
Less, but nevertheless still powerful, is the outer join. An outer join returns all records in either table, with NULL values for columns in a table in which the record does not exist.
Specialized versions of the outer join include the left join and right join, whereby all records in either the left or right table, respectively, are returned along with their counterparts.
Problem 2b
Select objid and $u'g'r'i'z'$ from PhotoObjAll and the corresponding class from specObjAll for 20 random sources.
There are multiple columns you could use to join the tables, in this case match objid to bestobjid from specObjAll and use an inner join.
Can you write Python code for it?

SDSS.query_sql( # complete

SDSS.query_sql("""select top 20 objid, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, cModelMag_z,
                         class
                  from photoobjall p
                      inner join specobjall s on p.objid = s.bestobjid""")


I am working on the file "Sessions/Session08/Day1/OOP Lecture.ipynb".
The first task is:
Instances
Instances are not the same thing as objects
Can you write Python code for it?

obs1 = Observation([0,1,2])
obs2 = Observation([4,5,6])

print("Avg 1 = {:e}; Avg 2 = {:e}".format(obs1.average(), obs2.average()))
print("Type of Avg 1 = {:}; Type of Avg 2 = {:}".format(type(obs1), type(obs2)))

print(obs1.data)
print(obs2.data)


I am working on the file "Sessions/Session08/Day1/OOP Lecture.ipynb".
The first task is:
Objects
Objects have

data called attributes
functions to act on their data called methods
Can you write Python code for it?

class Observation(): # "object" and "class" are interchangable!
    def __init__(self, data): # method
        self.data = data #attribute
    def average(self): # method
        dsum = 0
        for i,d in enumerate(self.data):
            dsum += d
        average=dsum/(i+1)
        return average


I am working on the file "Sessions/Session08/Day1/OOP Lecture.ipynb".
The first task is:
Objects in practice
In python everything is an object:
Can you write Python code for it?

print(print) #functions are objects!
dont_do_this = print # this is the object representing a function!
dont_do_this("dont do this!")


I am working on the file "Sessions/Session08/Day1/OOP_problem.ipynb".
The first task is:
Problem 1) Building a basic set of objects
Our first task is to map our problem onto a set of objects that we instantiate (that is, make instances of) in order to solve our problem.
Let's outline the scope of our problem.
A solar system exists in a Universe; here we can ignore the gravitational perturbation on the Solar System from the rest of the Universe. Our model will consist of a small number of bodies containing mass. It might also contain bodies without mass, so called "test particles".
The problem to be solved numerically is the gravitational N-body problem,
$$\ddot{\mathbf{r}}i = -G\sum{i \ne j} \frac{m_j \mathbf{r}{ij}}{r{ij}^3},$$
where $\mathbf{r}_{ij} \equiv \mathbf{r_i} - \mathbf{r_j}$. This task itself can be broken into two components:

the force calculation
the ODE integrator to advance $\mathbf{r}_i$ and $\dot{\mathbf{r}}_i$ forward in time

Problem 1a
In disucssion with a classmate, sketch out a set of classes that you will need to complete this project. Don't worry about things like numerical integrators yet.
Also, sketch out interfaces (start with the constructor), but don't worry about writing code right now.
Once you're done, find me and I'll give you the minimal list of objects.
Can you write Python code for it?

class MyFirstClass():
    pass

class MySecondClass():
    pass


I am working on the file "Sessions/Session08/Day1/OOP_problem.ipynb".
The first task is:
Problem 2
Now, we code the numerical algorithms. We're going to do the most simple things possible: a brute force ("direct N-Body" if you're feeling fancy) force calculation, and a leapfrog time integrator.
The leapfrog scheme is an explicit, second order scheme given by
$$r_{i+1} = r_{i} + v_{i} \Delta t + \frac{\Delta t^2}{2} a_{i}$$
$$v_{i+1} = v_{i} + \frac{\Delta t}{2} (a_{i} + a_{i+1}),$$
where $\Delta t$ is the time step (which we'll just keep constant), and the subscript refers to the iteration number $i$.
Note that this scheme requires a force update in between calculating $r_{i+1}$ and $v_{i+1}$.
Problem 2a
Write a method that implements the force integrator. Test it on simple cases:
 * two equal 1 $M_\odot$ objects in your universe, 1 AU apart
 * a $1\ M_\odot$ object and a $1\ M_{\oplus}$ object, 1 AU apart
Problem 2b
Write the leapfrog integration as a method in the Universe class. Test it on one particle with no force (what should it do?)
Problem 2c
Wire it all up! Try a 3-body calculation of the Earth-Sun-Moon system. Try the Earth-Jupiter-Sun system!
Challenge Problem

Construct a visualization method for the Universe class
Read about the Fast Multipole Method (FMM) here and implement one for the force calculation
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
Finally, we have only briefly discussed joining tables, but this is where relational databases really shine.
For this example we only have a single table, so we will exclude any examples of a pandas join, but there is functionality to join or merge dataframes in a fashion that is fully analogous to databases.
In summary, there are many different possible solutions for data storage and management.
For "medium" to "big" data that won't easily fit into memory ($\sim$16 GB), it is likely that a database is your best solution. For slightly smaller problems pandas provides a really nice, lightweight alternative to a full blown database that nevertheless maintains a lot of the same functionality and power.
Problem 4) We Aren't Done Talking About Your Laptop's Inadequacies
So far we have been focused on only a single aspect of computing: storage (and your laptop sucks at that).
But here's the thing - your laptop is also incredibly slow.
Supposing for a moment that you could hold all (or even a significant fraction) of the information from LSST in memory on your laptop, you would still be out of luck, as you would die before you could actually process the data and make any meaningful calculations.
(we argued that it would take $\sim$200 yr to process LSST on your laptop in Session 5)
You are in luck, however, as you need not limit yourself to your laptop. You can take advantage of multiple computers, also known  as parallel processing.
At a previous session, I asked Robert Lupton, one of the primary developers of the LSST photometric pipeline, "How many CPUs are being used to process LSST data?" To which he replied, "However many are needed to process everything within 1 month."
The critical point here is that if you can figure out how to split a calculation over multiple computers, then you can finish any calculation arbitrarily fast with enough processors (to within some limits, like the speed of light, etc)
We will spend a lot more time talking about both efficient algorithm design and parallel processing later...
Can you write Python code for it?

import time

pixel_data = np.random.rand(4000000)
photons = np.empty_like(pixel_data)

tstart = time.time()
for pix_num, pixel in enumerate(pixel_data):
    photons[pix_num] = pixel/8
trun = time.time() - tstart
print('It takes {:.6f} s to correct for the gain'.format(trun))


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
Pandas can also be used to aggregate the results of a search.
Problem 3d
How many extended sources (type = ext) have modelMag_i between 19 and 20? Use as few lines as possible.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
Problem 2c
Using the pandas read_csv() method, read in the data to a new variable, called s_read. Do you expect s_read and nums to be the same? Check whether or not your expectations are correct.
Hint - take the sum of the difference not equal to zero to identify if any elements are not the same.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
So, what is going on?
Sometimes, when you convert a number to ASCII (i.e. text) format, there is some precision that is lost in that conversion.
How do you avoid this?
One way is to directly write your files in binary. To do so has serveral advantages: it is possible to reproduce byte level accuracy, and, binary storage is almost always more efficient than text storage (the same number can be written in binary with less space than in ascii).
The downside is that developing your own procedure to write data in binary is a pain, and it places strong constraints on where and how you can interact with the data once it has been written to disk.
Fortuantely, we live in a world with pandas. All this hard work has been done for you, as pandas naturally interfaces with the hdf5 binary table format. (You may want to also take a look at pyTables)
(Historically astronomers have used FITS files as a binary storage solution)
Problem 2d
Repeat your procedure from above, but instead of writing to a csv file, use the pandas to_hdf() and read_df() method to see if there are any differences in s and s_read.
Hint - You will need to specify a name for the table that you have written to the hdf5 file in the call to to_hdf() as a required argument. Any string will do.
Hint 2 - Use s_read.values instead of s_read['nums'].values.
Can you write Python code for it?

s. # complete
s_read = # complete

# complete


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
This Laptop Is Inadequate:
An Aperitif for DSFP Session 8
Version 0.1
By AA Miller 2019 Mar 24
When I think about LSST there are a few numbers that always stick in my head:

37 billion (the total number of sources that will be detected by LSST)
10 (the number of years for the baseline survey)
1000 (~the number of observations per source)
37 trillion ($37 \times 10^9 \times 10^4$ = the total number of source observations)

These numbers are eye-popping, though the truth is that there are now several astronomical databases that have $\sim{10^9}$ sources (e.g., PanSTARRS-1, which we will hear more about later today).
A pressing question, for current and future surveys, is: how are we going to deal with all that data?
If you're anything like me - then, you love your laptop.
And if you had it your way, you wouldn't need anything but your laptop... ever.
But is that practical?
Problem 1) The Inadequacy of Laptops
Problem 1a
Suppose you could describe every source detected by LSST with a single number. Assuming you are on a computer with a 64 bit architecture, to within an order of magnitude, how much RAM would you need to store every LSST source within your laptop's memory?
Bonus question - can you think of a single number to describe every source in LSST that could produce a meaningful science result?
Take a minute to discuss with your partner
Solution 1a
As for a single number to perform useful science, I can think of two.
First - you could generate a heirarchical triangular mesh with enough trixels to characterize every LSST resolution element on the night sky. Then you could assign a number to each trixel, and describe the position of every source in LSST with a single number. Under the assumption that every source detected by LSST is a galaxy, this is not a terrible assumption, you could look at the clustering of these positions to (potentially) learn things about structure formation or galaxy formation (though without redshifts you may not learn all that much).
The other number is the flux (or magnitude)...
Can you write Python code for it?

nums = # complete
s = # complete
# complete


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
pandas also enables GROUP BY operations, where the data are split based on some criterion, a function is then applied to the groups, and the results are then combined back into a data structure.
Problem 3e
Group the data by their type and then report the minimum, median, and maximum redshift of each group. Can you immediately tell anything about these sources based on these results?
Hint - just execute the cell below.
Can you write Python code for it?

grouped = sdss_spec.groupby([sdss_spec.type])
print(grouped['z'].min())
print(grouped['z'].median())
print(grouped['z'].max())


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequate.ipynb".
The first task is:
pandas provides many different methods for selecting columns from the DataFrame. Supposing you wanted psfMag, you could use any of the following:
sdss_spec['psfMag_g']
sdss_spec[['psfMag_r', 'psfMag_z']]
sdss_spec.psfMag_g

(notice that selecting multiple columns requires a list within [])
Problem 3c
Plot a histogram of the psfMag_g - modelMag_g distribution for the data set (which requires a selection of those two columns).
Do you notice anything interesting?
Hint - you may want to use more than the default number of bins (=10).
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
1.5 s isn't too bad in the grand scheme of things.
Except that this example should make you cringe. There is absolutely no need to use a for loop for these operations.
This brings us to fast coding lesson number 1 - vectorize everything.
Can you write Python code for it?

photons = np.empty_like(pixel_data)

tstart = time.time()
photons = pixel_data/8
trun = time.time() - tstart
print('It takes {:.6f} s to correct for the gain'.format(trun))


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
By removing the for loop we improve the speed of this particular calculation by a factor of $\sim$125. That is a massive win.
Alternatively, we could have sped up the operations via the use of parallel programing. The multiprocessing library in python makes it relatively easy to implement parallel operations. There are many different ways to implement parallel processing in python, here we will just use one simple example.
(again, we will go over multiprocessing in far more detail later this week)
Can you write Python code for it?

from multiprocessing import Pool

def divide_by_gain(number, gain=8):
    return number/gain

pool = Pool()

tstart = time.time()
photons = pool.map(divide_by_gain, pixel_data)
trun = time.time() - tstart
print('It takes {:.6f} s to correct for the gain'.format(trun))


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
Finally, we have only briefly discussed joining tables, but this is where relational databases really shine.
For this example we only have a single table, so we will exclude any examples of a pandas join, but there is functionality to join or merge dataframes in a fashion that is fully analogous to databases.
In summary, there are many different possible solutions for data storage and management.
For "medium" to "big" data that won't easily fit into memory ($\sim$16 GB), it is likely that a database is your best solution. For slightly smaller problems pandas provides a really nice, lightweight alternative to a full blown database that nevertheless maintains a lot of the same functionality and power.
Problem 4) We Aren't Done Talking About Your Laptop's Inadequacies
So far we have been focused on only a single aspect of computing: storage.
(your laptop sucks at storage)
But here's the thing - your laptop is also incredibly slow.
Supposing for a moment that you could hold all (or even a significant fraction) of the information from LSST in memory on your laptop, you would still be out of luck, as you would die before you could actually process the data and make any meaningful calculations.
(in Session 5 we argued it would take $\sim$200 yr to process LSST on your laptop )
You are in luck, however, as you need not limit yourself to your laptop. You can take advantage of multiple computers, also known  as parallel processing.
At a previous session, I asked Robert Lupton, one of the primary developers of the LSST photometric pipeline, "How many CPUs are being used to process LSST data?" To which he replied, "However many are needed to process everything within 1 month."
The critical point here is that if you can figure out how to split a calculation over multiple computers, then you can finish any calculation arbitrarily fast with enough processors (to within some limits, like the speed of light, etc)
We will spend a lot more time talking about both efficient algorithm design and parallel processing later this...
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
So was all of this in service of a lie?
No. What I said before remains true - text files do not guarantee byte level precision, and they take more space on disk. Text files have some advantages:

anyone, anywhere, on any platform can easily manipulate text files
text files can be easily inspected (and corrected) if necessary
special packages are needed to read/write in binary
binary files, which are not easily interpretable, are difficult to use in version control (and banned by some version control platforms)

To summarize, here is my advice: think of binary as your (new?) default for storing data.
But, as with all things, consider your audience: if you are sharing/working with people that won't be able to deal with binary data, or, you have an incredibly small amount of data, csv (or other text files) should be fine.
Problem 3) Binary or ASCII - Doesn't Matter if You Aren't Organized
While the reproducibility of data is essential, ultimately, concerns about binary vs ascii are useless if you cannot access the data you need when you need it.
Your data are valuable (though cheaper to acquire than ever before), which means you need a good solution for managing that data, or else you are going to run into a significant loss of time and money.
Problem 3a
How would you organize the following: (a) 3 very deep images of a galaxy, (b) 4 nights of optical observations ($\sim$50 images night$^{-1}$) of a galaxy cluster in the $ugrizY$ filters, (c) images from a 50 night time-domain survey ($\sim$250 images night$^{-1}$) covering 1000 deg$^2$?
Similarly, how would you organize: (a) photometric information for your galaxy observations, (b) photometry for all the galaxies in your cluster field, (c) the observations/light curves from your survey?
Take a minute to discuss with your partner
Keeping in mind that there are several suitable answers to each of these questions, here are a few thoughts: (a) the 3 images should be kept together, probably in a single file directory. (b) With 200 images taken over the course of 4...
Can you write Python code for it?

sdss_spec = pd.read_csv("DSFP_SDSS_spec_train.csv")
sdss_spec.head()


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
So, if you are using pandas anyway, and if you aren't using pandas –– check it out!, then I strongly suggest removing csv files from your workflow to instead focus on binary hdf5 files. This requires typing the same number of characters, but it ensures byte level reproducibility.
And reproducibiliy is the pillar upon which the scientific method is built.
Is that the end of the story? ... No.
In the previous example, I was being a little tricky in order to make a point. It is in fact possible to create reproducible csv files with pandas. By default, pandas sacrifices a little bit of precision in order to gain a lot more speed. If you want to ensure reproducibility then you can specify that the float_precision should be round_trip, meaning you get the same thing back after reading from a file that you wrote to disk.
Can you write Python code for it?

s.to_csv('tmp.txt', header=True, index=False)

s_read = pd.read_csv('tmp.txt', float_precision='round_trip')

sum(nums - s_read['nums'].values != 0)


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
So, it turns out that $\sim{23}\%$ of the time, pandas does not in fact read in the same number that it wrote to disk.
The truth is that these differences are quite small (see next slide), but there are many mathematical operations (e.g., subtraction of very similar numbers) that may lead these tiny differences to compound over time such that your data are not, in fact, constant.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
So, what is going on?
Sometimes, when you convert a number to ASCII (i.e. text) format, there is some precision that is lost in that conversion.
How do you avoid this?
One way is to directly write your files in binary. To do so has serveral advantages: it is possible to reproduce byte level accuracy, and, binary storage is almost always more efficient than text storage (the same number can be written in binary with less space than in ascii).
The downside is that developing your own procedure to write data in binary is a pain, and it places strong constraints on where and how you can interact with the data once it has been written to disk.
Fortuantely, we live in a world with pandas. All this hard work has been done for you, as pandas naturally interfaces with the hdf5 binary table format. (You may want to also take a look at pyTables)
(Historically astronomers have used FITS files as a binary storage solution)
Problem 2d
Repeat your procedure from above, but instead of writing to a csv file, use the pandas to_hdf() and read_df() method to see if there are any differences in s and s_read.
Hint - You will need to specify a name for the table that you have written to the hdf5 file in the call to to_hdf() as a required argument. Any string will do.
Hint 2 - Use s_read.values instead of s_read['nums'].values.
Can you write Python code for it?

s.to_hdf('tmp.h5', 's', header=True)
s_read = pd.read_hdf('tmp.h5', 's')

sum(nums - s_read.values != 0)


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
This Laptop Is Inadequate:
An Aperitif for DSFP Session 8
Version 0.1
By AA Miller 2019 Mar 24
When I think about LSST there are a few numbers that always stick in my head:

37 billion (the total number of sources that will be detected by LSST)
10 (the number of years for the baseline survey)
1000 (~the number of observations per source)
37 trillion ($37 \times 10^9 \times 10^4$ = the total number of source observations)

These numbers are eye-popping, though the truth is that there are now several astronomical databases that have $\sim{10^9}$ sources (e.g., PanSTARRS-1, which we will hear more about later today).
A pressing question, for current and future surveys, is: how are we going to deal with all that data?
If you're anything like me - then, you love your laptop.
And if you had it your way, you wouldn't need anything but your laptop... ever.
But is that practical?
Problem 1) The Inadequacy of Laptops
Problem 1a
Suppose you could describe every source detected by LSST with a single number. Assuming you are on a computer with a 64 bit architecture, to within an order of magnitude, how much RAM would you need to store every LSST source within your laptop's memory?
Bonus question - can you think of a single number to describe every source in LSST that could produce a meaningful science result?
Take a minute to discuss with your partner
$$\frac{64 \, \mathrm{bit}}{1 \, \mathrm{source}} \times \frac{1 \,\mathrm{GB}}{8\times10^9 \,\mathrm{bit}} \times 3.7 \times 10^{10}\, \mathrm{sources} \approx 296 \, \mathrm{GB}$$
While there are specialized machines that have this amount of memory, this is unreasonable for any modern laptops.
As for a single number to perform useful science, I can think of two.
First - you could generate a heirarchical triangular mesh with enough trixels to characterize every LSST resolution element on the night sky. Then you could assign a number to each trixel, and describe the position of every source in LSST with a single number. Under the assumption that every source detected by...
Can you write Python code for it?

nums = np.random.rand(10000)
s = pd.Series(nums, name='nums')
s.to_csv('tmp.txt', header=True, index=False)


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
pandas also enables GROUP BY operations, where the data are split based on some criterion, a function is then applied to the groups, and the results are then combined back into a data structure.
Problem 3e
Group the data by their type and then report the minimum, median, and maximum redshift of each group. Can you immediately tell anything about these sources based on these results?
Hint - just execute the cell below.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
pandas can also be used to aggregate the results of a search.
Problem 3d
How many extended sources (type = ext) have modelMag_i between 19 and 20? Use as few lines as possible.
Can you write Python code for it?

len(sdss_spec[(sdss_spec.type == 'ext') &
              (sdss_spec.modelMag_i > 19) &
              (sdss_spec.modelMag_i < 20)])


I am working on the file "Sessions/Session08/Day1/ThisLaptopIsInadequateSolutions.ipynb".
The first task is:
pandas provides many different methods for selecting columns from the DataFrame. Supposing you wanted psfMag, you could use any of the following:
sdss_spec['psfMag_g']
sdss_spec[['psfMag_r', 'psfMag_z']]
sdss_spec.psfMag_g

(notice that selecting multiple columns requires a list within [])
Problem 3c
Plot a histogram of the psfMag_g - modelMag_g distribution for the data set (which requires a selection of those two columns).
Do you notice anything interesting?
Hint - you may want to use more than the default number of bins (=10).
Can you write Python code for it?

mag_diff = sdss_spec.psfMag_g - sdss_spec.modelMag_g

_ = plt.hist(mag_diff, bins=25)
plt.xlabel('psfMag_g - modelMag_g')
plt.ylabel('N')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
A 14% improvement is pretty good! But, we can wrangle even more out of the gender feature. Recall that kNN models measure the Euclidean distance between sources, meaning the scale of each feature really matters. Given that the fare ranges from 0 up to 512.3292, the kNN model will see this feature as far more important than gender, for no other reason than the units that have been adopted.
If women are far more likely to survive than men, then we want to be sure that gender is weighted at least the same as all the other features, which we can do with a minmax scaler. As a brief reminder - a minmax scaler scales all values of a feature to be between 0 and 1 by subtracting the minimum value of each feature and then dividing by the maximum minus the minimum.
Problem 3f
Scale all the features from the previous problem using a minmax scaler and evaluate the CV accuracy of the kNN model.
Hint - you may find MinMaxScaler helpful
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

knn_clf = # complete

cv_results = cross_val_score( # complete

print('The accuracy when scaling features = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
A benefit to using this method, as opposed to pandas, is that data typing and data units are naturally read from the IRSA table and included with the associated columns. Thus, if you are uncertain if some brightness measurement is in magnitudes or Janskys, the astropy Table can report on that information.
Unfortunately, astropy does not know about every strange formating decision that every astronomer has made at some point in their lives (as we are about to see...)
Problem 2) The Sins of Our Journals
Unlike IRSA/IPAC, which uses a weird but nevertheless consistent format for data tables, data retrieved from Journal articles essentially follows no rules. In principle, tables in Journal articles are supposed to be provided in a machine readable format. In practice, as we are about to see, this is far from the case.
For this particular wrangling case study we will focus on supernova light curves, a simple thing to report: time, filter, brightness, uncertainty on that brightness, that the community has nevertheless managed to mangle into some truly wild and difficult to parse forms.
(Sorry for the heavy emphasis on time-domain examples - I'm pulling straight from my own life today, but the issues described here are not perfectly addressed by any subfield within the astro umbrella)
Here is the LaTeX-formatted version of Table 4 from Miller et al. 2011:
<img style="display: block; margin-left: auto; margin-right: auto" src="images/Miller11_tbl4.png" align=\"middle">
That is a very simple table to interpret, no?
Have a look at the "machine-readible" file that ApJ provides for readers that might want to evaluate these photometric measurements.
Problem 2a
Download the ApJ version of Table 4 from Miller et al. 2011 and read that table into either a pandas DataFrame or an astropy Table.
Can you write Python code for it?

# read in the data from Miller et al 2011


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
As far as ages are concerned, imputation of the missing data does not significantly improve the model.
Which brings us to the concluding lesson in wrangling the Titanic data - not every piece of information is useful. This is critical to remember when building machine learning models.
(As a quick aside - it wouldn't be entirely fair to say there is no useful information in the age feature. It is clear, for example, that "children," i.e. those with Age < 10, had a much higher probability of survival than adults. Perhaps the creation of a child feature based on age would improve the model... or using the age in combination with other features, e.g., AgexPclass which will further highlight that 1st class passengers were more likely to survive than 3rd class passengers)
Finally - note that you can try to build a model and submit it to Kaggle to see how well you preform on blind data.
https://www.kaggle.com/c/titanic - the classifications are not revealed, but from the leaderboard it is clear that some people were able to build models that perfectly classified the blind data.
Problem 4) Wrangling an astro machine learning model
Now we will put together the above examples to work on an astronomical problem of great importance for LSST: photometric redshifts. As we have previously discussed, LSST will observe many, many, many, many more galaxies than we can possibly observe with spectroscopic instruments. As a result, the vast majority of the galaxies observed by LSST will not have precise redshift estimates, and instead we will need to estimate redshifts via photometry alone.
This week we have heard a lot about SDSS CasJobs, and for today's problem we will use 50,000 SDSS sources with spectroscopic measurements to train a machine learning model to estimate redshifts from photometry. The following query was used to generate this training set:
select top 50000 s.specObjID, s.z, s.type,
   psfMag_u, psfMag_g, psfMag_r, psfMag_i, psfMag_z,
   modelMag_u, modelMag_g, modelMag_r, modelMag_i, modelMag_z,
  ...
Can you write Python code for it?

sdss = pd.read_csv("DSFP_SDSS_spec_train.csv")
sdss[:5]


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Note - that should have failed! And for good reason - recall that kNN models measure the Euclidean distance between all points within the feature space. So, when considering the sex of a passenger, what is the numerical distance between male and female?
In other words, we need to wrangle this data before we can run the machine learning model.
Most of the features in this problem are non-numeric (i.e. we are dealing with categorical features), and therefore we need to figure out how to include them in the kNN model.
The first step when wrangling for machine learning is to figure out if anything can be thrown away. We certainly want to avoid including any uninformative features in the model.
If you haven't already, now would be a good time to create a new cell and examine the contents of the csv
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Now that we have dealt with missing and categorical data we can construct out machine learning model. We will use a $k$ nearest neighbors regression model to determine how well we can measure photometric redshifts.
The sdss dataframe now includes far more columns than are necessary for the model. Should any of these be excluded from the fitting?
Problem 4d
Build a scikit-learn appropriate X and y array for the features and labels (i.e. redshifts) of the photo-z training set.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Now that we have the data in the appropriate X and y arrays, estimate the accuracy with which a K nearest neighbors classification model can predict whether or not a passenger would survive the Titanic disaster. Use $k=10$ fold cross validation for the prediction.
Problem 3b
Train a $k=7$ nearest neighbors machine learning model on the Titanic training set.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn_clf = KNeighborsClassifier( # complete
knn_clf.fit( # complete


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Okay - there is nothing elegant about that particular solution. But it works, and wranglin' ain't pretty.
It is likely that you developed a solution that looks very different from this one, and that is fine. When data are provided in an unrulely format, the most important thing is to develop some method, any method, for converting the information into a useful format. Following whatever path you used above, it should now be easy to plot the light curve of SN 2008ha.
Problem 3) My Heart Will Go On
Sometimes there is no difficultly whatsoever in reading in the data (as was the case in Problems 1 and 2), but instead the difficultly lies in wranglin' the data to be appropriate for the model that you are building.
For the next problem we will work with the famous Titanic survival data set. If you haven't already, please download the csv file with the relevant information.
Briefly, the Titantic is a famous historical ship that was thought to be unsinkable. Spoiler alert it hit an iceberg and sank. The data in the Titanic data set includes information about 891 passengers from the Titanic, as well as whether or not they survived. The aim of this data set is to build a machine learning model to predict which passengers survived and which did not.
The features include:
|Feature    | Description |
|:---------:|:--------------------------------------:|
|PassengerId| Running index that describes the individual passengers|
|Pclass| A proxy for socio-economic status (1 = Upper class, 2 = Middle Class, 3 = Lower Class)|
|Name| The passenger's name|
|Sex | The passenger's sex|
|Age | The passenger's age - note age's ending in 0.5 are estimated |
|SibSp| The sum of the passenger's sibblings and spouces on board|
|Parch| The sum of the passenger's parents and children on board|
|Ticket| The ticket number for the passenger|
|Fare| The price paid for the ticket by th passenger|
|Cabin| The Cabin in which the passenger stayed|
|Embarked| The point of Origin for the Passenger: C = Cherbourg, S = Southampton, Q = Queenstown|
And...
Can you write Python code for it?

titanic_df = pd.read_csv( # complete

feat_list = list(titanic_df.columns)
label = 'Survived'
feat_list.remove(label)
X = # complete
y = # complete


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 3c
Are there any features that obviously will not help with this classification problem?
If you answer yes - ignore those features moving forward
Solution 3c
write your solution here
Given that we have both categorical and numeric features, let's start with the numerical features and see how well they can predict survival on the Titanic.
One note - for now we are going to exclude Age, because as you saw when you examined the data, there are some passengers that do not have any age information. This problem, known as "missing data" is one that we will deal with before the end of this problem.
Problem 3d
How accurately can the numeric features, Pclass, SibSp, Parch, and Fare predict survival on the Titanic? Use a $k = 7$ Nearest Neighbors model, and estimate the model accuracy using 10-fold cross validation.
Hint 1 - you'll want to redefine your features vector X
Hint 2 - you may find cross_val_score from scikit-learn helpful
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

X = # complete

knn_clf = # complete

cv_results = cross_val_score( # complete

print('The accuracy from numeric features = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 3i
Use the create_bin_cat_feats function to convert the Embarked and Sex, yes we need to do this for Sex as well where we otherwise previously introduced order, categorical features to a numeric representation. Add these features to the titanic_df data frame.
Can you write Python code for it?

gender_dict = create_bin_cat_feats( # complete
porigin_dict = create_bin_cat_feats( # complete

for feat in gender_dict.keys():
    titanic_df[feat] = gender_dict[feat]

for feat in porigin_dict.keys():
    titanic_df[feat] = porigin_dict[feat]