I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
Problem 2) $k$-means clustering
As a subfield of unsupervised learning, clustering aims to group/separate sources in the multidimensional feature space. The "unsupervised" comes from the fact that there are no target labels provided to the algorithm, so the machine is asked to cluster the data "on its own." The lack of labels means there is no (simple) method for validating the accuracy of the solution provided by the machine (though sometimes simple examination can show the results are terrible).
For this reason, "classic" unsupervised methods are not particularly useful for astronomy.$^\dagger$ Supposing one did find some useful clustering structure, an adversarial researcher could always claim that the current feature space does not accurately capture the physics of the system and as such the clustering result is not interesting or, worse, erroneous.
$^\dagger$This is my (AAM) opinion and there are many others who disagree.
We start today with the most famous, and simple, clustering algorithm: $k$-means. $k$-means clustering looks to identify $k$ convex clusters, where $k$ is a user defined number. And here-in lies the rub: if we truly knew the number of clusters in advance, we likely wouldn't need to perform any clustering in the first place. This is the major downside to $k$-means.
As a reminder from lecture, the pseudocode for $k$-means:
initiate search by identifying k points (i.e. the cluster centers)
loop
    assign each point in the data set to the closest cluster center
    calculate new cluster centers based on mean position of all points within each cluster
    if diff(new center - old center) < threshold:
        stop (i.e. clusters are defined)

The threshold is defined by the user, though in some cases the total number of iterations can also be used as a stopping criteria. An advantage of $k$-means is that the solution will always converge, though the solution may only be a local minimum. Disadvantages include the assumption of convexity, i.e. difficult to capture complex geometry, and...
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
Problem 2b
Fit a $k = 2$, $k$-means model to the iris data. Plot the resulting clusters in the sepal length-sepal width plane.
Can you write Python code for it?

Kcluster = # complete
Kcluster.fit( # complete

fig, ax = plt.subplots()
ax.scatter( # complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
Problem 2c
Fit a $k = 3$, $k$-means model to the iris data. Plot the resulting clusters in the sepal length-sepal width plane.
Can you write Python code for it?

Kcluster = # complete
# complete

fig, ax = plt.subplots()
ax.scatter( # complete


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
Problem 3b
Adjust the tuning parameters to see how they affect the final clustering results. How does the use of DBSCAN compare to $k$-means? Can you obtain 3 clusters with DBSCAN? If not, given the knowledge that the iris dataset has 3 classes - does this invalidate DBSCAN as a viable algorithm?
Can you write Python code for it?

dbs = DBSCAN( # complete
dbs.fit(# complete

fig, ax = plt.subplots()
ax.scatter( # complete


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
Problem 4b
Using the SDSS galaxy data, identify interesting clusters within the data. This question is intentionally very open ended. If you uncover anything especially exciting you'll have a chance to share it with the group. Feel free to use the algorithms discussed above, or any other packages available via sklearn. Can you make sense of the clusters in the context of galaxy evolution?
Hint - don't fret if you know nothing about galaxy evolution (neither do I!). Just take a critical look at the clusters that are identified
Can you write Python code for it?

# complete
# complete
# complete
# complete
# complete

# complete
# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
That doesn't look right at all!
So in addition to not knowing the correct number of clusters in the data, we see that the results are also sensitive to how the cluster positions are initiated.
$k$-means evaluates the Euclidean distance between individual sources and cluster centers, thus, the magnitude of the individual features has a strong effect on the final clustering outcome.
Problem 2f
Calculate the mean, standard deviation, min, and max of each feature in the iris data set. Based on these summaries, which feature is most important for clustering?
Can you write Python code for it?

# complete
# complete
# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
write your answer here
Problem 4) Cluster SDSS Galaxy Data
The following query will select 10k likely galaxies from the SDSS database and return the results of that query into an astropy.Table object. (For now, if you are not familiar with the SDSS DB schema, don't worry about this query, just know that it returns a bunch of photometric features.)
from astroquery.sdss import SDSS  # enables direct queries to the SDSS database

GALquery = """SELECT TOP 5000
         p.dered_u - p.dered_g as ug, p.dered_g - p.dered_r as gr,
         p.dered_g - p.dered_i as gi, p.dered_g - p.dered_z as gz,
         p.petroRad_i, p.petroR50_i, p.deVAB_i, p.fracDev_i
         FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
         WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND p.type = 3
         AND p.deVAB_i > -999 AND p.petroRad_i > -999 AND p.petroR50_i > -999 AND p.dered_r < 20
           """
SDSSgals = SDSS.query_sql(GALquery)
SDSSgals

Problem 4a
Download the SDSS galaxy data
Read in the file galaxy_clustering.csv, and convert the data into a feature array X.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/Clustering.ipynb".
The first task is:
write your answer here
Since $k$-means is built on Euclidean distance measures in the feature space, it can be really useful to re-scale all the features prior to applying the clustering algorithm.
(Two notes – (a) some algorithms are extremely sensitive to feature scaling so this is a always a good thing to keep in mind, and (b) the iris data set is small and of relatively similar scale so the effects will not be that dramatic)
Imagine you are classifying stellar light curves: the data set will include binary white dwarf stars with periods of $\sim 0.01 \; \mathrm{d}$ and Mira variables with periods of $\sim 1000 \; \mathrm{d}$. Without re-scaling, this feature that covers 6 orders of magnitude! Without rescaling all other features will add little weight to any final clustering solution.
The two most common forms of re-scaling are to rescale to a guassian with mean $= 0$ and variance $= 1$, or to rescale the min and max of the feature to $[0, 1]$. The best normalization is problem dependent. The sklearn.preprocessing module makes it easy to re-scale the feature set.$^\dagger$
$\dagger$ For supervised methods, it is essential that the same scaling used for the training set be used for all other data run through the model. The testing, validation, and field observations cannot be re-scaled independently. This would result in meaningless final classifications/predictions.
Problem 2g
Re-scale the features to normal distributions, and perform $k$-means clustering on the iris data. How do the results compare to those obtained earlier?
Hint - you may find 'StandardScaler()' within the sklearn.preprocessing module useful.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit( # complete

Kcluster = # complete
# complete

fig, ax = plt.subplots()
ax.scatter( # complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
As we can see in the above plot, every galaxy with $\mathrm{mass} \gtrsim 78$ is classified as normal.
It is also clear from the plot that linear regression is not a particularly good approach to this problem.
Problem 3c
Build a classifier via "hard cut" and assess it's accuracy.
Find the maximum mass for dwarf galaxies in the data set. Classify all sources with a mass less than this as dwarf galaxies.
Can you write Python code for it?

mass_cut = # complete

y_pred = # complete
# complete

# complete

# complete

# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values.$^\dagger$  In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
$^\dagger$In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, so ...
read the docs!
Problem 1a What is the pythonic type of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
In the previous cell we have simulated 200 galaxies, most of which are normal with masses > 60. We use this information to define our class vector y, such that all normal galaxies have a class of 1 and all dwarf galaxies have a class of 0. Finally, we simulate our observations X, which adds Gaussian noise to the true mass measurements.
Problem 3a
Build a classifier using LinearRegression(). Find the best fit line for the data. All sources with an extrapolated best-fit value > 0.5 are classified as normal, and everything else is considered a dwarf.
From the training data, how accurate is this model?
Can you write Python code for it?

reg_model = # complete
reg_model.fit( # complete

y_pred = # complete

n_incorrect = np.sum((y_pred - y)**2)

accuracy = (n_obs - n_incorrect)/n_obs

print(f'This model has an accuracy of {accuracy:.3f}')


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Introduction to scikit-learn
Version 0.1

By AA Miller (Northwestern/CIERA)
28 Feb 2022
Broadly speaking, machine-learning methods constitute a diverse collection of data-driven algorithms designed to classify/characterize/analyze sources in multi-dimensional spaces. The topics and studies that fall under the umbrella of machine learning is growing, and there is no good catch-all definition. We cannot cover all possible algorithms.
Today we will focus on the scikit-learn python library, which provides a nice interface to build a wide variety of machine learning models. As we will see, scikit-learn makes machine learning "easy."
Problem 1) Data with scikit-learn
In 4 lines we can build a complex classifier for the famous iris data set.
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)

Those 4 lines of code have constructed a model that is superior to any system of hard cuts that we could have encoded while looking at the multidimensional space.
It's also fast – execute the next cell.
Can you write Python code for it?

# execute example code here

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?

print(np.shape(# complete
print(# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 1e
What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 1f
What are the names of class 0, 1, and 2 in the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 2b
Plot the simulated data. Overplot the true relation from which the data are drawn.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 2d
Fit the model to the training data x and y_obs.
Hint - the standard feature array in scikit-learn is 2D, so you will need to convert x to a 2D array.
Can you write Python code for it?

X = # complete
reg_model.fit( # complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 2e
Output the best-fit parameters from the model (these are stored in the .coef_ and .intercept_ attributes).
Overplot the best-fit line on top of the observations.
Can you write Python code for it?

print(f'The best fit is y = ' # complete

fig, ax = plt.subplots()
ax.plot( # complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 2f
Fit for the model parameters using some other method (e.g., numpy, scipy, linear algebra), and compare the model parameters to those found with scikit-learn.
Can you write Python code for it?

# complete

print('The best-fit model is: y = ' # complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
Problem 3b
To understand the short-comings of this model, plot the data. Overplot the best-fit linear regression model.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot( # complete
# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
The results are identical!
Under the hood, polynomial regression is just linear algebra, so it doesn't matter which library you use to solve the problem. Both numpy and scikit-learn can handle multi-dimensional input and even account for uncertainties on the observations via input weights. For this problem, it does not matter which library you use.
Problem 3) Linear Classification
We will now examine why linear regression does not work for classification.
Suppose we have a method to measure the mass of a galaxy (but this method has noise). We would like to separate the galaxies into two classes "normal" and "dwarf", where dwarf galaxies are those with a mass below 60. We could simulate such a dataset as follows.
Note - this is very much a "toy" data set to be used purely for illustration. While we use some familiar terms, nothing about this is truly physical, which is why we do not include units, etc.
Can you write Python code for it?

np.random.seed(1938)
n_obs = 200
mass = np.random.normal(65, 20, size=n_obs)
mass[mass > 70] *= (mass[mass > 70]-60)/60*5
y = (mass > 60).astype(bool)
obs_mass = mass + np.random.normal(0, 7, size=n_obs)
X = obs_mass.reshape(-1,1)


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
The sklearn.linear_model has a LinearRegression class.
The "standard" procedure in scikit-learn is to generate the model by creating an instance of the relevant class object (e.g., reg_model = LinearRegression()).
The model is then trained using the aforementioned training and label arrays, X and y, with the .fit() method.
Finally, predictions can be made using the .predict() method (note - the precise syntax for this can vary depending on the model being used).
Problem 2c
Generate an instance of the LinearRegression class. Call this instance reg_model.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

reg_model = # complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
This model shows significant improvement over our alternative methods!
Problem 3g
Overplot the best fit logistic regression model on the data.
Hint – the .intercept_ and .coef_ attributes can be used to determine the argument for the logistic function.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot( # complete

# complete

# complete

# complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
This represents a slight improvement over the linear regression model. (Note - if you did the reverse and used the normal galaxies to determine the mass cut, you would get similar results)
At this point you must be thinking - "There has to be a better way!"
Fortunately, there is.
Logistic Regression - the "hello, world" of machine learning classification.
(and also one of the absolutely worst named algorithms ever, as logistic regression is used for classification and not regression)
Briefly, logistic regression is used to understand the relationship between a dependent variable (normal vs. dwarf) and one or more independent variables (mass, but could also be mass, star formation rate, metallicity, and so on) by estimating probabilities via the logistic function.
The logistic function:
$$ p(x) = \frac{1}{1 + e^{-(x - \mu)/s}}$$
can be rewritten as:
$$ p(x) = \frac{1}{1 + e^{-(B_0 + B_1 x)}}$$
which recasts the problem as having a slope and intercept, similar to the linear regression problem that we worked on previously.
Problem 3d
Execute the cell below to see the logistic function.
How might this be useful for our galaxy classification problem?
Can you write Python code for it?

x = np.linspace(-10,10,1000)
p_x = 1/(1 + np.exp(-x))
fig, ax = plt.subplots()
ax.plot(x, p_x)
ax.set_xlabel('x', fontsize=14)
ax.set_ylabel('p(x)', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
We can see that the logistic regression model is clearly superior to the linear regression model. The model is not perfect, but no model could be in the presence of noise.
We also see that the probability cut ($p \approx 0.5 \;\mathrm{at\;mass} \approx 61.5$) is much closer to the true answer of 60.
The previous example used a simple toy data set that was easy to visualize. But logistic regression can be extended to include multiple features, and more than two classes (in which case the problem is a multinomial one rather than a binomial one).
This optimization happens entirely under the hood with scikit-learn.
Problem 3f
Fit a logistic regression model to the iris data set.
Assess the accuracy of this model.
Hint – accuracy_score from sklearn.metrics will simplify your calculations.
Can you write Python code for it?

from sklearn.metrics import accuracy_score

logreg = LogisticRegression( # complete
# complete
# complete

print('The accuracy of this model is {}' # complete


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
We will return to the iris data set later in this notebook.
Problem 2) Linear Regression
At its core, scitkit-learn is designed to help you, the user, easily fit models to data.
To demonstrate this we will start with a familiar example - linear regression.
Problem 2a
Simulate data drawn from a linear relationship with Gaussian scatter.
Can you write Python code for it?

np.random.seed(2012)
n_obs = 25
x = np.random.uniform(0,100, n_obs)
y_true = 2.3*x + 14
y_obs = y_true + np.random.normal(0, 15, n_obs)


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearn.ipynb".
The first task is:
You likely haven't encountered a scikit-learn Bunch before. It's essentially the same as a dictionary.
Problem 1b What are the keys of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
An important lesson from this week - worry about the data!
If you are worried about the data, then you should look at the data. This is actually an important aspect of applying machine learning models.
Problem 1g
Make a scatter plot showing sepal length vs. sepal width for the iris data set. Color the points according to their respective classes.
Can you write Python code for it?

print(iris.feature_names)  # shows that sepal length is first feature and sepal width is second feature

fig, ax = plt.subplots()
ax.scatter(iris.data[:,0], iris.data[:,1],
           c = iris.target, s = 30,
           edgecolor = "None", cmap = "viridis")
ax.set_xlabel('sepal length (cm)')
ax.set_ylabel('sepal width (cm)')
fig.tight_layout()


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
As we can see in the above plot, every galaxy with $\mathrm{mass} \gtrsim 78$ is classified as normal.
It is also clear from the plot that linear regression is not a particularly good approach to this problem.
Problem 3c
Build a classifier via "hard cut" and assess it's accuracy.
Find the maximum mass for dwarf galaxies in the data set. Classify all sources with a mass less than this as dwarf galaxies.
Can you write Python code for it?

mass_cut = np.max(X[y == 0])

y_pred = np.zeros_like(mass).astype(int)
y_pred[mass > mass_cut] = 1

n_incorrect = np.sum((y_pred - y)**2)

accuracy = (n_obs - n_incorrect)/n_obs

print(f'This model has an accuracy of {accuracy:.3f}')


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values.$^\dagger$  In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
$^\dagger$In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, so ...
read the docs!
Problem 1a What is the pythonic type of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
In the previous cell we have simulated 200 galaxies, most of which are normal with masses > 60. We use this information to define our class vector y, such that all normal galaxies have a class of 1 and all dwarf galaxies have a class of 0. Finally, we simulate our observations X, which adds Gaussian noise to the true mass measurements.
Problem 3a
Build a classifier using LinearRegression(). Find the best fit line for the data. All sources with an extrapolated best-fit value > 0.5 are classified as normal, and everything else is considered a dwarf.
From the training data, how accurate is this model?
Can you write Python code for it?

reg_model = LinearRegression()
reg_model.fit(X, y)

y_pred = (reg_model.predict(X) > 0.5).astype(int)

n_incorrect = np.sum((y_pred - y)**2)

accuracy = (n_obs - n_incorrect)/n_obs

print(f'This model has an accuracy of {accuracy:.3f}')


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Problem 1e
What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Problem 1f
What are the names of class 0, 1, and 2 in the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Problem 2d
Fit the model to the training data x and y_obs.
Hint - the standard feature array in scikit-learn is 2D, so you will need to convert x to a 2D array.
Can you write Python code for it?

X = x.reshape(-1,1)
reg_model.fit(X, y_obs)


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Problem 3b
To understand the short-comings of this model, plot the data. Overplot the best-fit linear regression model.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(X, y, 'o', mfc="None")
model = np.poly1d([ reg_model.coef_[0], reg_model.intercept_])
ax.plot([0, 400], model([0,400]))
ax.plot([0,625], [0.5, 0.5], '0.8', lw=3)
ax.set_xlabel('mass')
ax.set_ylabel('class')
fig.tight_layout()

(0.5 - reg_model.intercept_)/reg_model.coef_[0]


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
Problem 3f
Assess the accuracy of the Logistic Regression model.
Can you write Python code for it?

y_pred = logreg.predict(X).astype(int)

n_incorrect = np.sum((y_pred - y)**2)

accuracy = (n_obs - n_incorrect)/n_obs

print(f'This model has an accuracy of {accuracy:.3f}')


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
This model shows significant improvement over our alternative methods!
Problem 3g
Overplot the best fit logistic regression model on the data.
Hint – the .intercept_ and .coef_ attributes can be used to determine the argument for the logistic function.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(X, y, 'o', mfc="None")
model = np.poly1d([ reg_model.coef_[0], reg_model.intercept_])
ax.plot([0, 300], model([0,300]))
ax.plot([0,625], [0.5, 0.5], '0.8', lw=3)

dat_grid = np.linspace(0, 625, 1000)
logmodel = 1/(1 + np.exp(-logreg.intercept_ - logreg.coef_[0]*dat_grid))
ax.plot(dat_grid, logmodel)
ax.set_xlabel('mass')
ax.set_ylabel('class')
fig.tight_layout()


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
This represents a slight improvement over the linear regression model. (Note - if you did the reverse and used the normal galaxies to determine the mass cut, you would get similar results)
At this point you must be thinking - "There has to be a better way!"
Fortunately, there is.
Logistic Regression - the "hello, world" of machine learning classification.
(and also one of the absolutely worst named algorithms ever, as logistic regression is used for classification and not regression)
Briefly, logistic regression is used to understand the relationship between a dependent variable (normal vs. dwarf) and one or more independent variables (mass, but could also be mass, star formation rate, metallicity, and so on) by estimating probabilities via the logistic function.
The logistic function:
$$ p(x) = \frac{1}{1 + e^{-(x - \mu)/s}}$$
can be rewritten as:
$$ p(x) = \frac{1}{1 + e^{-(B_0 + B_1 x)}}$$
which recasts the problem as having a slope and intercept, similar to the linear regression problem that we worked on previously.
Problem 3d
Execute the cell below to see the logistic function.
How might this be useful for our galaxy classification problem?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
We will return to the iris data set later in this notebook.
Problem 2) Linear Regression
At its core, scitkit-learn is designed to help you, the user, easily fit models to data.
To demonstrate this we will start with a familiar example - linear regression.
Problem 2a
Simulate data drawn from a linear relationship with Gaussian scatter.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/IntroToScikitLearnSolutions.ipynb".
The first task is:
write your answer here
Problem 3e
Create an instance of the LogisticRegression class from the sklearn.linear_model module.
Fit the model to the simulated galaxy data.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression()
logreg.fit(X, y)


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 1) Examine the Training Data
For this problem the training set, i.e. sources with known labels, includes stars and galaxies that have been confirmed with spectroscopic observations. The machine learning model is needed because there are $\gg 10^8$ sources with photometric observations in SDSS, and only $4 \times 10^6$ sources with spectroscopic observations. The model will allow us to translate our knowledge from the spectroscopic observations to the entire data set. The features include each $r$-band magnitude measurement made by SDSS (don't worry if you don't know what this means...). This yields 8 features to train the models (significantly fewer than the 454 properties measured for each source in SDSS).
If you are curious (and it is fine if you are not) this training set was constructed by running the following query on the SDSS database:
SELECT TOP 20000
p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
s.class
FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
ORDER BY p.objid ASC

First download the training set and the blind test set for this problem.
Problem 1a
Visualize the training set data. The data have 8 features ['psfMag_r', 'fiberMag_r', 'fiber2Mag_r', 'petroMag_r', 'deVMag_r', 'expMag_r', 'modelMag_r', 'cModelMag_r'], and a 9th column ['class'] corresponding to the labels ('STAR' or 'GALAXY' in this case).
Hint - just execute the cell below.
Can you write Python code for it?

sdss_df = pd.read_hdf("sdss_training_set.h5")
sns.pairplot(sdss_df, hue = 'class', diag_kind = 'hist')


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 1b
Based on your plots of the data, which feature do you think will be the most important for separating stars and galaxies? Why?
write your answer here - do not change it after later completing the problem
The final data preparation step it to create an independent test set to evalute the generalization error of the final tuned model. Independent test sets are generated by witholding a fraction of the training set. No hard and fast rules apply for the fraction to be withheld, though typical choices vary between $\sim{0.2}-0.5$.
sklearn.model_selection has a useful helper function train_test_split.
Problem 1c Split the 20k spectroscopic sources 70-30 into training and test sets. Save the results in arrays called: train_X, train_y, test_X, test_y, respectively. Use rs for the random_state in train_test_split.
Hint - recall that sklearn utilizes X, a 2D np.array(), and y as the features and labels arrays, respecitively.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
rs = 1851

# complete

X = # complete
y = # complete

train_X, test_X, train_y, test_y = # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 3b
Use 10-fold cross validation to estimate the FoM for the $k$NN model. Take the mean value across all folds as the FoM estimate.
Hint - the cross_val_score function from the sklearn.model_selection module performs the necessary calculations.
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

knn_cv = cross_val_score( # complete

print('The kNN model FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 3d
Do the machine-learning models outperform the SDSS photometric classifier?
write your answer here
Problem 4) Model Optimization
While the "off-the-shelf" model provides an improvement over the SDSS photometric classifier, we can further refine and improve the performance of the machine learning model by adjusting the model tuning parameters. A process known as model optimization.
All machine-learning models have tuning parameters. In brief, these parameters capture the smoothness of the model in the multidimentional-feature space. Whether the model is smooth or coarse is application dependent -- be weary of over-fitting or under-fitting the data. Generally speaking, RF (and most tree-based methods) have 3 flavors of tuning parameter:

$N_\mathrm{tree}$ - the number of trees in the forest n_estimators (default: 10) in sklearn
$m_\mathrm{try}$ - the number of (random) features to explore as splitting criteria at each node max_features (default: sqrt(n_features)) in sklearn
Pruning criteria - defined stopping criteria for ending continued growth of the tree, there are many choices for this in sklearn (My preference is min_samples_leaf (default: 1) which sets the minimum number of sources allowed in a terminal node, or leaf, of the tree)

Just as we previously evaluated the model using CV, we must optimize the tuning paramters via CV. Until we "finalize" the model by fixing all the input parameters, we cannot evalute the accuracy of the model with the test set as that would be "snooping."
Before globally optimizing the model, let's develop some intuition for how the tuning parameters affect the final model predictions.
Problem 4a
Determine the 10-fold cross validation accuracy for $k$NN models with $k$ = 1, 10, 100.
How do you expect changing the number of neighbors to affect the results?
Can you write Python code for it?

for k in [1,10,100]:
    # complete

    print('With k = {:d}, the kNN FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 5c
Calculate the confusion matrix for the test set. Is there symmetry to the misclassifications?
Hint - the confusion_matrix function in sklearn.metrics will help.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

print(confusion_matrix( # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 5e
Suppose that (like me) you really care about supernovae. In this case you want a model that correctly classifies 99% of all stars, so that stellar flares do not fool you into thinking you have found a new supernova.
What classification threshold should be adopted for this model?
What fraction of galaxies does this model misclassify?
Can you write Python code for it?

tpr_99_thresh = # complete

print('This model requires a classification threshold of {:.4f}'.format(tpr_99_thresh))

fpr_at_tpr_99 = # complete

print('This model misclassifies {:.2f}% of galaxies'.format(fpr_at_tpr_99*100))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 6) Classify New Data
Run the cell below to load in some new data (which in this case happens to have known labels, but in practice this will almost never be the case...)
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
Problem 6a
Create a feature and label array for the new data.
Hint - copy the code you developed above in Problem 2.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
write your answer here
Problem 3) Model Evaluation
To evaluate the performance of the model we establish a baseline (or figure of merit) that we would like to exceed. For our current application we want to maximize the accuracy of the model.
If the model does not improve upon the baseline (or reach the desired figure of merit) then one must iterate on previous steps (feature engineering, algorithm selection, etc) to accomplish the desired goal.
The SDSS photometric pipeline uses a simple parametric model to classify sources as either stars or galaxies. If we are going to the trouble of building a complex ML model, then it stands to reason that its performance should exceed that of the simple model. Thus, we adopt the SDSS photometric classifier as our baseline.
The SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag_r} - \mathtt{cModelMag_r} > 0.145.$$
Sources that satisfy this criteria are considered galaxies.
Problem 3a
Determine the baseline figure of merit by measuring the accuracy of the SDSS photometric classifier on the training set.
Hint - the accuracy_score function in the sklearn.metrics module may be useful.
Can you write Python code for it?

from sklearn.metrics import accuracy_score

phot_y = # complete
# complete
# complete
# complete

print("The baseline FoM = {:.4f}".format( # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
write your answer here
Problem 4b
Determine the 10-fold cross validation accuracy for RF models with $N_\mathrm{tree}$ = 1, 10, 30, 100, and 300.
How do you expect changing the number of trees to affect the results?
Can you write Python code for it?

for ntree in [1,10,30,100,300]:
    # complete

    print('With {:d} trees the FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxies.ipynb".
The first task is:
write your answer here
Problem 5d
Calculate (and plot the region of interest) the ROC curve assumming that stars are the positive class.
Hint 1 - you will need to calculate probabilistic classifications for the test set using the predict_proba() method.
Hint 2 - the roc_curve function in the sklearn.metrics module will be useful.
Can you write Python code for it?

from sklearn.metrics import roc_curve

test_y_int = # complete
# complete

test_preds_proba = rf_clf.predict_proba( # complete

fpr, tpr, thresh = roc_curve( # complete

fig, ax = plt.subplots()
ax.plot( # complete
ax.set_xlabel('FPR')
ax.set_ylabel('TPR')


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
A nice property of RF, relative to $k$NN, is that RF naturally provides an estimate of the most important features in a model.
RF feature importance is measured by randomly shuffling the values of a particular feature, and measuring the decrease in the model's overall accuracy. The relative feature importances can be accessed using the .feature_importances_ attribute associated with the RandomForestClassifer() object. The higher the value, the more important feature.
Problem 2c
Calculate the relative importance of each feature.
Which feature is most important? Does this match your answer from 1c?
Can you write Python code for it?

feat_str = ',\n'.join(['{}'.format(feat) for feat in np.array(feats)[np.argsort(rf_clf.feature_importances_)[::-1]]])

print('From most to least important: \n{}'.format(feat_str))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 3b
Use 10-fold cross validation to estimate the FoM for the $k$NN model. Take the mean value across all folds as the FoM estimate.
Hint - the cross_val_score function from the sklearn.model_selection module performs the necessary calculations.
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

knn_cv = cross_val_score(knn_clf, train_X, train_y, cv=10)

print('The kNN model FoM = {:.4f} +/- {:.4f}'.format(np.mean(knn_cv), np.std(knn_cv, ddof=1)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 3c
Use 10-fold cross validation to estimate the FoM for the random forest model.
Can you write Python code for it?

rf_cv = cross_val_score(rf_clf, train_X, train_y, cv=10)

print('The RF model FoM = {:.4f} +/- {:.4f}'.format(np.mean(rf_cv), np.std(rf_cv, ddof=1)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 3d
Do the machine-learning models outperform the SDSS photometric classifier?
write your answer here
Problem 4) Model Optimization
While the "off-the-shelf" model provides an improvement over the SDSS photometric classifier, we can further refine and improve the performance of the machine learning model by adjusting the model tuning parameters. A process known as model optimization.
All machine-learning models have tuning parameters. In brief, these parameters capture the smoothness of the model in the multidimentional-feature space. Whether the model is smooth or coarse is application dependent -- be weary of over-fitting or under-fitting the data. Generally speaking, RF (and most tree-based methods) have 3 flavors of tuning parameter:

$N_\mathrm{tree}$ - the number of trees in the forest n_estimators (default: 10) in sklearn
$m_\mathrm{try}$ - the number of (random) features to explore as splitting criteria at each node max_features (default: sqrt(n_features)) in sklearn
Pruning criteria - defined stopping criteria for ending continued growth of the tree, there are many choices for this in sklearn (My preference is min_samples_leaf (default: 1) which sets the minimum number of sources allowed in a terminal node, or leaf, of the tree)

Just as we previously evaluated the model using CV, we must optimize the tuning paramters via CV. Until we "finalize" the model by fixing all the input parameters, we cannot evalute the accuracy of the model with the test set as that would be "snooping."
Before globally optimizing the model, let's develop some intuition for how the tuning parameters affect the final model predictions.
Problem 4a
Determine the 10-fold cross validation accuracy for $k$NN models with $k$ = 1, 10, 100.
How do you expect changing the number of neighbors to affect the results?
Can you write Python code for it?

for k in [1,10,100]:
    knn_cv = cross_val_score(KNeighborsClassifier(n_neighbors=k), train_X, train_y, cv=10)

    print('With k = {:d}, the kNN FoM = {:.4f} +/- {:.4f}'.format(k, np.mean(knn_cv), np.std(knn_cv, ddof=1)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 5e
Suppose that (like me) you really care about supernovae. In this case you want a model that correctly classifies 99% of all stars, so that stellar flares do not fool you into thinking you have found a new supernova.
What classification threshold should be adopted for this model?
What fraction of galaxies does this model misclassify?
Can you write Python code for it?

tpr_99_thresh = thresh[np.argmin(np.abs(0.99 - tpr))]

print('This model requires a classification threshold of {:.4f}'.format(tpr_99_thresh))

fpr_at_tpr_99 = fpr[np.argmin(np.abs(0.99 - tpr))]

print('This model misclassifies {:.2f}% of galaxies'.format(fpr_at_tpr_99*100))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 6) Classify New Data
Run the cell below to load in some new data (which in this case happens to have known labels, but in practice this will almost never be the case...)
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 6a
Create a feature and label array for the new data.
Hint - copy the code you developed above in Problem 2.
Can you write Python code for it?

new_X = np.array(new_data_df[feats])
new_y = np.array(new_data_df['class'])


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 6b
Calculate the accuracy of the model predictions on the new data.
Can you write Python code for it?

new_preds = rf_clf.predict(new_X)
print("The model has an accuracy of {:.4f}".format(accuracy_score(new_y, new_preds)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
Problem 6c
Can you explain why the accuracy for the new data is significantly lower than what you calculated previously?
If you can build and train a better model (using the trianing data) for classifying the new data - I will be extremely impressed.
write your answer here
Challenge Problem) Full RF Optimization
Now we will optimize the model over all tuning parameters. How does one actually determine the optimal set of tuning parameters? Brute force.
We will optimize the model via a grid search that performs CV at each point in the 3D grid. The final model will adopt the point with the highest accuracy.
It is important to remember two general rules of thumb: (i) if the model is optimized at the edge of the grid, refit a new grid centered on that point, and (ii) the results should be stable in the vicinity of the grid maximum. If this is not the case the model is likely overfit.
Use GridSearchCV to perform a 3-fold CV grid search to optimize the RF star-galaxy model. Remember the rules of thumb.
What are the optimal tuning parameters for the model?
Hint 1 - think about the computational runtime based on the number of points in the grid. Do not start with a very dense or large grid.
Hint 2 - if the runtime is long, don't repeat the grid search even if the optimal model is on an edge of the grid
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

grid_results = GridSearchCV(RandomForestClassifier(n_jobs=-1),
                            {'n_estimators': [30, 100, 300],
                             'max_features': [1, 3, 7],
                             'min_samples_leaf': [1,10,30]},
                            cv = 3)
grid_results.fit(train_X, train_y)

print('The best model has {}'.format(grid_results.best_params_))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
We will now ignore everything in the test set until we have fully optimized the machine learning model.
Problem 2) Model Building
After curating the data, you must select a specific machine learning algorithm. With experience, it is possible to develop intuition for the best ML algorithm given a specific problem.
Short of that? Try two (or three, or four, or five) different models and choose whichever works the best.
Problem 2a
Train a $k$-nearest neighbors model on the star-galaxy training set. Select $k$ = 25 for this model.
Hint - the KNeighborsClassifier object in the sklearn.neighbors module may be useful for this task.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

knn_clf = KNeighborsClassifier(n_neighbors=25)
knn_clf.fit(train_X, train_y)


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
write your answer here
Now you are ready for the moment of truth!
Problem 5) Model Predictions
Problem 5a
Calculate the FoM for the SDSS photometric model on the test set.
Can you write Python code for it?

phot_y = np.empty_like(test_y)
phot_gal = np.logical_not(test_X[:,0] - test_X[:,-1] < 0.145)
phot_y[phot_gal] = 'GALAXY'
phot_y[~phot_gal] = 'STAR'

print("The baseline FoM = {:.4f}".format(accuracy_score(test_y, phot_y)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
write your answer here
Problem 3) Model Evaluation
To evaluate the performance of the model we establish a baseline (or figure of merit) that we would like to exceed. For our current application we want to maximize the accuracy of the model.
If the model does not improve upon the baseline (or reach the desired figure of merit) then one must iterate on previous steps (feature engineering, algorithm selection, etc) to accomplish the desired goal.
The SDSS photometric pipeline uses a simple parametric model to classify sources as either stars or galaxies. If we are going to the trouble of building a complex ML model, then it stands to reason that its performance should exceed that of the simple model. Thus, we adopt the SDSS photometric classifier as our baseline.
The SDSS photometric classifier uses a single hard cut to separate stars and galaxies in imaging data:
$$\mathtt{psfMag_r} - \mathtt{cModelMag_r} > 0.145.$$
Sources that satisfy this criteria are considered galaxies.
Problem 3a
Determine the baseline figure of merit by measuring the accuracy of the SDSS photometric classifier on the training set.
Hint - the accuracy_score function in the sklearn.metrics module may be useful.
Can you write Python code for it?

from sklearn.metrics import accuracy_score

phot_y = np.empty_like(train_y)
phot_gal = np.logical_not(train_X[:,0] - train_X[:,-1] < 0.145)
phot_y[phot_gal] = 'GALAXY'
phot_y[~phot_gal] = 'STAR'

print("The baseline FoM = {:.4f}".format(accuracy_score(train_y, phot_y)))


I am working on the file "Sessions/Session14/Day1/SeparatingStarsAndGalaxiesSolutions.ipynb".
The first task is:
write your answer here
Problem 4b
Determine the 10-fold cross validation accuracy for RF models with $N_\mathrm{tree}$ = 1, 10, 30, 100, and 300.
How do you expect changing the number of trees to affect the results?
Can you write Python code for it?

for ntree in [1,10,30,100,300]:
    rf_cv = cross_val_score(RandomForestClassifier(n_estimators=ntree), train_X, train_y, cv=10)

    print('With {:d} trees the FoM = {:.4f} +/- {:.4f}'.format(ntree, np.mean(rf_cv), np.std(rf_cv, ddof=1)))


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Classification with a Perceptron
Version 0.1

By AA Miller (Northwestern/CIERA)
21 February 2022
Perceptrons are a type of articifical neuron. We will construct a basic neuron in pure python today and use it to classify data in a "simple" two class problem.
Consider the following image - what do you see?

 (data credit: MNIST)

Without hesitation, I am certain that you were able to identify the above image as the number 8.
Breakout Problem 1
Using everything we have learned this week about machine learning – devise how to use the Random Forest algorithm to build a binary classifier to separate the number 8 from other handwritten digits.
Take a few minutes to discuss with your partner
There are many possible approaches -

build an algorithm that identifies "circles"
  $~~~~$(number 3 does not have fully closed circles)

build an algorithm that indentifies lines
  $~~~~$(number 4 is only straight lines)

examine only the bottom half of the image
  $~~~~$(tops of 8 and 9 are similar, but bottoms different)

At this point we only have a few features, they are all very complex to derive, and it isn't at all clear that we would successfully separate 8 and 0.
In sum, we are on the road towards success.
And yet,
when you looked at the image of the 8, you immediately recognized the contents of the image at the start of this lecture.
We each have a super computer in our heads. One trained over many, many generations of evolution to immediately recognize what is right in front of our eyes. Constructing "rules" (especially very precise rules) to teach a computer to make the same recognition is extremely challenging.
For complex tasks, like computer vision/classifying handwritten digits, we need a new type of machine learning relative to what we discussed earlier this week.
(We will need deep learning to accomplish this task. We will develop...
Can you write Python code for it?

def walk_dog( # complete
    '''Perceptron to calculate whether we should walk the dog

    Parameters
    ----------
    questions : array-like, size = 3
    weights : array-lik, optional (default = np.array([-2, -1, 5]))
    threshold : float, optional (default = 2.5)
        decision threshold for whether to walk the dog or not

    Returns
    -------
    walk : bool
        Boolean variable to indicate whether to walk the dog
    '''

    # complete
    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Problem 1b
Use the newly created perceptron to determine whether to walk the dog if:
  1. it is raining outside
  2. the dog is asleep
  3. the dog last went out 2.5 hours ago
Hint - recall that the inputs should be binary.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Problem 1c
After sleeping for four hours, it is still raining, but the dog wakes up (i.e., it has been 6.5 hr since the dog last went out). Should you walk the dog?
Hint - recall that the inputs should be binary.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Problem 2) Generic Perceptron
We can simplify our representation of the perceptron by replacing the sum with a dot product, $w_n \cdot x_n = \sum_n w_n x_n$, and we can move the threshold to the other side of the inequality, which we will now refer to as the bias $b$.
$$\mathrm{output} = \left{ \begin{array}{lcr}
0 & \mathrm{if} \; w_n \cdot x_n + b & \le 0 \
1 & \mathrm{if} \; w_n \cdot x_n + b & > 0
\end{array}\right.
$$
(in this notation, the bias can be thought of how easy it is for the neuron to be activated)
Now, build a generic perceptron that can take any collection of input signals and weights, as well as a bias, to determine the binary output from the artificial neuron.
(this will prove useful for more than just walking the dog)
Problem 2a
Write a generic function perceptron that takes as input arrays called signals and weights as well as a float called bias. The function should return a boolean indicating whether or not the perceptron is "activated".
Can you write Python code for it?

def perceptron(# complete
    '''Generic perceptron function

    Parameters
    ----------
    signals : array-like
        the input signals for the perceptron

    weights : array-like
        the weight applied to each input

    bias : float
        the value required for activation

    Returns
    -------
    activated : bool
        whether or not the perceptron is activated
    '''

    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Problem 2b
Is the perceptron activated if the signal = [2.3, 5.3, 1.2, 3.4], the weights = [-3, 2, 0.5, -1], and no bias (i.e., bias = 0)?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
Problem 3) Perceptrons for classification
Perceptrons can be used for binary classification problems!
(perhaps this is no surprise given that this session is about machine learning...)
To demonstrate how this works we will generate some synthetic two-dimensional data, but the principle can easily be scaled to an arbitrarily large number of dimensions.
We use scikit-learn to simulate two classes in a 2d plane using the make_blobs() function. We will only include 30 samples so the data are easy to visualize.
Can you write Python code for it?

X, y = make_blobs(n_samples=30, centers=2, n_features=2,
                  center_box = (0,4), random_state=1885)

fig, ax = plt.subplots()
activated = y == 1
ax.plot(X[activated,0], X[activated,1], 'o',
        ms = 9, mew=2, mfc='None')
ax.plot(X[~activated,0], X[~activated,1], '+',
        ms=15, mew=2)
ax.set_xlabel('X1', fontsize=15)
ax.set_ylabel('X2', fontsize=15)
fig.tight_layout()


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassification.ipynb".
The first task is:
We see that the accuracy slowly improves over the course of the 20 epochs.
In other words, the machine... IT IS LEARNING.
Problem 3c
Adjust the weights, or bias, or number of epochs, or learning rate, or all of them, to see how the changes affect the output of the perceptron.
What do you notice as these changes are made?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
How can we use a perceptron to classify this data?
In this case we have two inputs, and thus two weights, plus the bias to determine whether the perceptron is activated (class = 1, the open circles in the previous plot).
We also have 30 observations that we can use to train the algorithm.
For a perceptron, training means updating the weights to better reflect the training data.
Here's the pseudo-code:
  1. Apply the perceptron to one of the data points
  2. Adjust the weights if the perceptron makes an incorrect classification
  3. Repeat this procedure over all N datapoints
  4. Repeat this procedure for M iterations
How do we adjust the weights? For every sample we evaluate the model error (is the classification correct or not). We then adjust the weight to reduce the error for the following prediction.
For a perceptron, these updates can be calculated simply as:
$$w_\mathrm{updated} = w_\mathrm{current} + \eta\,\, (y_\mathrm{true} - y_\mathrm{pred})\,\, x,$$
where $w_\mathrm{updated}$ is the new value for the weight, $w_\mathrm{current}$ is the current value for the weight, $\eta$ is the called the learning rate, $x$ is the value of the input signal, and $(y_\mathrm{true} - y_\mathrm{pred})$ captures whether or not the classification was correct.
The learning rate is a small number that adjusts the weight in the direction of being more accurate. It is selected by the user, though familiar tricks like cross validation can be used to identify an optimal size.
To train the perceptron, we need to decide the total number of iterations $M$ to pass through the training data. These iterations are called epochs. Within each epoch, we update the weights and bias to improve our predictions on the generated data.
Updating the bias is similar to updating the weights, but we exclude the value of the input $x$ as this does not affect the bias.
Problem 3a
Write a function train_perceptron that accepts as input X, y, weights, bias, epochs, and learning rate. The function should train the perceptron for $M$ epochs....
Can you write Python code for it?

def train_perceptron(X, y, weights, bias=0, epochs=5, learning_rate=0.01):
    '''Train a perceptron to classify binary labels via numerical features

    Parameters
    ----------
    X : array-like
        Feature array for the data, in the style of scikit-learn
    y : array-like, type = bool
        Label array for the data
    weights : array-like
        Weights for the input signals to the perceptron
    bias : array-like
        Bias value for the perceptron
    epochs : int
        Number of instances for training the perceptron
    learning_rate : float
        Relative step size for tuning the weights and bias
    '''
    w_updated = weights
    bias_updated = bias
    misclassified = 0
    for signal, true in zip(X, y):
        activated = perceptron(signal, w_updated, bias_updated)
        error = true - activated
        misclassified += error**2
    print(f'For initial weights, accuracy = {(len(y)-misclassified)/len(y):.4f}')
    for epoch in range(epochs):
        misclassified = 0
        for signal, true in zip(X, y):
            activated = perceptron(signal, w_updated, bias_updated)
            error = true - activated
            misclassified += error**2
            for i in range(len(weights)):
                w_updated[i] = w_updated[i] + learning_rate*(true - activated)*signal[i]
            bias_updated = bias_updated + learning_rate*(true - activated)

        print(f'For epoch {epoch}, accuracy = {(len(y)-misclassified)/len(y):.4f}')


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
Problem 2) Generic Perceptron
We can simplify our representation of the perceptron by replacing the sum with a dot product, $w_n \cdot x_n = \sum_n w_n x_n$, and we can move the threshold to the other side of the inequality, which we will now refer to as the bias $b$.
$$\mathrm{output} = \left{ \begin{array}{lcr}
0 & \mathrm{if} \; w_n \cdot x_n + b & \le 0 \
1 & \mathrm{if} \; w_n \cdot x_n + b & > 0
\end{array}\right.
$$
(in this notation, the bias can be thought of how easy it is for the neuron to be activated)
Now, build a generic perceptron that can take any collection of input signals and weights, as well as a bias, to determine the binary output from the artificial neuron.
(this will prove useful for more than just walking the dog)
Problem 2a
Write a generic function perceptron that takes as input arrays called signals and weights as well as a float called bias. The function should return a boolean indicating whether or not the perceptron is "activated".
Can you write Python code for it?

def perceptron(signals, weights, bias):
    '''Generic perceptron function

    Parameters
    ----------
    signals : array-like
        the input signals for the perceptron

    weights : array-like
        the weight applied to each input

    bias : float
        the value required for activation

    Returns
    -------
    activated : bool
        whether or not the perceptron is activated
    '''

    activated = True if np.dot(signals, weights) + bias > 0 else False
    return activated


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
Problem 2c
What if the signal and weights do not change but the bias = -2?
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
Problem 3) Perceptrons for classification
Perceptrons can be used for binary classification problems!
(perhaps this is no surprise given that this session is about machine learning...)
To demonstrate how this works we will generate some synthetic two-dimensional data, but the principle can easily be scaled to an arbitrarily large number of dimensions.
We use scikit-learn to simulate two classes in a 2d plane using the make_blobs() function. We will only include 30 samples so the data are easy to visualize.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
Problem 3b
Train the perceptron. Use weights of [.1, 1], a bias of 0, train for 20 epochs, with a learning rate $\eta = 0.005$.
Note – as we will see below the perceptron is highly sensitive to the initial guess for weights and biases.
Can you write Python code for it?

train_perceptron(X, y, np.array([.1, 1]), bias=0, epochs=20, learning_rate=0.005)


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
The above plot shows that the sigmoid function is a "smooth" version of the perceptron. We can exploit this smoothness to achieve better learning outcomes. Small changes in the weights and biases will produce small changes in the sigmoid function, enabling gradual improvement, which cannot be said for the perceptron near the region of activation.
This will prove important in the next lecture.
Challenge Problem
Train a perceptron to classify the number 8 in the handwritten digits data set.
Hint - you will need to convert the 2d data into a vector format.
Appendix
Functions to load plots shown in the notebook.
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import fetch_openml

train_samples = 5000

# Load data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True, as_frame=False)

fig, ax = plt.subplots(figsize=(4,4))

ax.imshow(X[41].reshape(28,28)[3:-3,3:-3],
          cmap='binary')
ax.set_xticks([])
ax.set_yticks([])
fig.tight_layout()
fig.savefig('./images/number8.png')


I am working on the file "Sessions/Session14/Day2/BuildingPerceptronsForClassificationSolutions.ipynb".
The first task is:
write answer here
It is possible to build a perceptron that gets worse at classification. This can happen if the learning rate is extremely large, essentially flipping on or off the activation for every source during each epoch.
It is also possible to build a perceptron that effectively never learns anything if the initial weights are extremely far from the optimal solution. The weights effectively define a line (or hyperplane in more than two dimensions) to separate the classes. If the line is very distant from the data itself, then the weights cannot be easily updated to improve the classification.
You have now built a perceptron. You have also seen it's limitations.
The strength of machine learning solutions lies in their ability to identify and capture non-linear structure within data.
But the perceptron is almost too non-linear. For inputs that are close to the activation boundary, very small changes in the weights can rapidly lead to an extreme difference in outcome.
It would be better if our adjustments led to more gradual changes, so that, if possible, the quality of the model improved during each epoch of the classifier. This can be achieved with a different model for our artificial neuron.
Consider, for instance, a neuron that is activated via the sigmoid function:
$$\sigma(z) \equiv \frac{1}{1 + e^{-z}}$$
where $z$ is the previously defined activation for a neuron: $w \cdot x - b$
We can visually show the difference between the perceptron and sigmoid neurons.
Can you write Python code for it?

fig, ax = plt.subplots()
ax. plot([-10, 0, 0, 10], [0,0,1,1], lw=3, label='Perceptron')
xgrid = np.linspace(-10, 10, 1000)
ax.plot(xgrid, 1/(1 + np.exp(-xgrid)), lw=2, label='Sigmoid')
ax.set_xlim(-7.5, 7.5)
ax.legend()
ax.set_xlabel('input*weights - bias', fontsize=14)
ax.set_ylabel('output', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session14/Day2/DeeplearningBlank.ipynb".
The first task is:
Problem 2b Make a histogram showing the fraction of each class
Keep only the top two classes (i.e., the classes with the most galaxies)
Can you write Python code for it?

images_top_two = ...
labels_top_two = ...


I am working on the file "Sessions/Session14/Day2/DeeplearningBlank.ipynb".
The first task is:
Problem 2c Split the data into a training and test set (66/33 split) using the train_test_split function from sklearn
Can you write Python code for it?

from sklearn.model_selection import train_test_split


I am working on the file "Sessions/Session14/Day2/DeeplearningBlank.ipynb".
The first task is:
The next block of code will show how one can train the model for 100 epochs. Note that we use the binary cross-entropy as our objective function and stochastic gradient descent as our optimization method.
Problem 3b Edit the code so that the function plots the loss for the training and test loss for each epoch.
Can you write Python code for it?

# train the model
def train_model(training_data,training_labels, test_data,test_labels, model):
  # define the optimization
  criterion = torch.nn.BCELoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=0.1,momentum=0.9)
  # Increase the number of epochs for your "final" run
  for epoch in range(10):
    # clear the gradient
    optimizer.zero_grad()
    # compute the model output
    myoutput = model(training_data)
    # calculate loss
    loss = criterion(myoutput, training_labels)
    # credit assignment
    loss.backward()
    # update model weights
    optimizer.step()

    # ADD PLOT


I am working on the file "Sessions/Session14/Day2/DeeplearningBlank.ipynb".
The first task is:
The next block trains the code, assuming a hidden layer size of 100 neurons.
Problem 3c Change the learning rate lr to minimize the cross entropy score
Can you write Python code for it?

model = MLP(np.shape(images_train[0])[0],100)
train_model(images_train, labels_train, images_test, labels_test, model)


I am working on the file "Sessions/Session14/Day2/DeeplearningBlank.ipynb".
The first task is:
The next cell will outline how one can make a MLP with pytorch.
Problem 3a Talk to a partner about how this code works, line by line. Add another hidden layer which is the same size as the first hidden layer. Choose an appropriate final nonlinear layer for this classification problem. Choose the appropriate number of outputs.
Can you write Python code for it?

class MLP(torch.nn.Module):
      # this defines the model
        def __init__(self, input_size, hidden_size):
            super(MLP, self).__init__()
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.hiddenlayer = torch.nn.Linear(self.input_size, self.hidden_size)
            self.outputlayer = torch.nn.Linear(self.hidden_size, HOW_MANY_OUTPUTS)
            # some nonlinear options
            self.sigmoid = torch.nn.Sigmoid()
            self.softmax = torch.nn.Softmax()
            self.relu = torch.nn.ReLU()
        def forward(self, x):
            layer1 = self.hiddenlayer(x)
            activation = self.sigmoid(layer1)
            layer2 = self.outputlayer(activation)
            output = self.NONLINEAR(layer2)
            return output


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
Challenge Problem Add a third class to your classifier and begin accounting for uneven classes. There are several steps to this:

Edit the neural network to output 3 classes
Change the criterion to a custom criterion function, such that the entropy of each class is weighted by the inverse fraction of each class size (e.g., if the galaxy class breakdowns are 1:2:3, the weights would be 6:3:2).
Can you write Python code for it?

class MLP_new(torch.nn.Module):
      # this defines the model
        def __init__(self, input_size, hidden_size):
            super(MLP_new, self).__init__()
            print(input_size,hidden_size)
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.hiddenlayer = torch.nn.Linear(self.input_size, self.hidden_size)
            self.outputlayer = torch.nn.Linear(self.hidden_size, 3)
            self.sigmoid = torch.nn.Sigmoid()
            self.softmax = torch.nn.Softmax()
        def forward(self, x):
            layer1 = self.hiddenlayer(x)
            activation = self.sigmoid(layer1)
            layer2 = self.outputlayer(activation)
            activation2 = self.sigmoid(layer1)
            layer3 = self.outputlayer(activation2)
            output = self.softmax(layer3)
            return output

#Only work with 0,1,2
gind = np.where((labels_original==0) | (labels_original==1) | (labels_original==2))
images_top_three = images[gind]
labels_top_three = labels_original[gind]

x,counts = np.unique(labels_top_three,return_counts=True)
print(counts)

torch.set_default_dtype(torch.float)
labels_top_three_one_hot = F.one_hot(torch.tensor(labels_top_three - np.min(labels_top_three)).long(), num_classes=3)
images_top_three = torch.tensor(images_top_three).float()
labels_top_three_one_hot = labels_top_three_one_hot.float()
# we're going to flatten the images for our MLP
images_top_three_flat = images_top_three.reshape(len(images_top_three),-1)

#Normalize the flux of the images here
images_top_three_flat = (images_top_three_flat  - torch.mean(images_top_three_flat))/torch.std(images_top_three_flat)
images_train, images_test, labels_train, labels_test = train_test_split(
    images_top_three_flat, labels_top_three_one_hot, test_size=0.33, random_state=42)

# train the model
def train_model(training_data,training_labels, test_data,test_labels, model):
  # define the optimization
  criterion = torch.nn.CrossEntropyLoss(weight=torch.Tensor(np.sum(counts)/counts))
 ...


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
Problem 1: Understanding the Data
For this problem set, we will use the Galaxy10 dataset made available via the astroNN module. This dataset is made up of 17736 images of galaxies which have been labelled by hand. See this link for more information.
First we will visualize our data.
Problem 1a Show one example of each class as an image.
Can you write Python code for it?

from astroNN.datasets import load_galaxy10
images, labels_original = load_galaxy10()
from astroNN.datasets.galaxy10 import galaxy10cls_lookup
%matplotlib inline

# Plot an example image from each class

# First, find an example of each class
uclasses, counts = np.unique(labels_original,return_counts=True)
print(len(labels_original))
for i, uclass in enumerate(uclasses):
  print(uclass,counts[i])
  first_example = np.where(labels_original==uclass)[0][0]
  plt.imshow(images[first_example])
  plt.title(galaxy10cls_lookup(uclass))
  plt.show()


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
Problem 2b Make a histogram showing the fraction of each class
Keep only the top two classes (i.e., the classes with the most galaxies)
Can you write Python code for it?

plt.hist(labels_original)
plt.xlabel('Class Label')
plt.show()

#Only work with 1 and 2
gind = np.where((labels_original==1) | (labels_original==2))
images_top_two = images[gind]
labels_top_two = labels_original[gind]


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
Problem 2c Split the data into a training and test set (66/33 split) using the train_test_split function from sklearn
Can you write Python code for it?

from sklearn.model_selection import train_test_split
images_train, images_test, labels_train, labels_test = train_test_split(
    images_top_two_flat, labels_top_two_one_hot, test_size=0.33, random_state=42)

np.shape(images_train)


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
The next block of code will show how one can train the model for 100 epochs. Note that we use the binary cross-entropy as our objective function and stochastic gradient descent as our optimization method.
Problem 3b Edit the code so that the function plots the loss for the training and test loss for each epoch.
Can you write Python code for it?

# train the model
def train_model(training_data,training_labels, test_data,test_labels, model):
  # define the optimization
  criterion = torch.nn.BCELoss()
  optimizer = torch.optim.SGD(model.parameters(), lr=0.007,momentum=0.9)
  for epoch in range(100):
    # clear the gradient
    optimizer.zero_grad()
    # compute the model output
    myoutput = model(training_data)
    # calculate loss
    loss = criterion(myoutput, training_labels)
    # credit assignment
    loss.backward()
    # update model weights
    optimizer.step()

    # STUDENTS ADD THIS PART
    output_test = model(test_data)
    loss_test = criterion(output_test, test_labels)
    plt.plot(epoch,loss.detach().numpy(),'ko')
    plt.plot(epoch,loss_test.detach().numpy(),'ro')
    print(epoch,loss.detach().numpy())
  plt.show()


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
The next block trains the code, assuming a hidden layer size of 100 neurons.
Problem 3c Change the learning rate lr to minimize the cross entropy score
Can you write Python code for it?

model = MLP(np.shape(images_train[0])[0],50)
train_model(images_train, labels_train, images_test, labels_test, model)


I am working on the file "Sessions/Session14/Day2/DeeplearningSolutions.ipynb".
The first task is:
The next cell will outline how one can make a MLP with pytorch.
Problem 3a Talk to a partner about how this code works, line by line. Add another hidden layer which is the same size as the first hidden layer.
Can you write Python code for it?

class MLP(torch.nn.Module):
      # this defines the model
        def __init__(self, input_size, hidden_size):
            super(MLP, self).__init__()
            print(input_size,hidden_size)
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.hiddenlayer = torch.nn.Linear(self.input_size, self.hidden_size)
            self.outputlayer = torch.nn.Linear(self.hidden_size, 2)
            self.sigmoid = torch.nn.Sigmoid()
            self.softmax = torch.nn.Softmax()
        def forward(self, x):
            layer1 = self.hiddenlayer(x)
            activation = self.sigmoid(layer1)
            layer2 = self.outputlayer(activation)
            activation2 = self.sigmoid(layer1)
            layer3 = self.outputlayer(activation2)
            output = self.softmax(layer3)
            return output


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
After you have read that data you must calculate features on the test set using exactly the same method that you employed for the training set.
Note - if you created a new calc_feature script for every feature that you calculated, this should be straightforward.
Can you write Python code for it?

### Calculate features for the test set here


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Features as a Representation of Time Series for Classification
version 0.1

By AA Miller (Northwestern CIERA/Adler Planetarium)
10 June 2019
This lecture is about machine learning...
But honestly, this lecture isn't really about machine learning...
This lecture is about the classification of variable sources in astronomical survey data. There are many different ways to approach such a classification problem, and today we will use a machine leaning approach to accomplish this task.
As a(n incredibly) brief reminder, machine learning algorithms use a training set with known labels$^1$ to develop a mapping between the data and the labels. You can, and should, think of this mapping as a black box. The mapping can occur between the raw data and the labels (e.g., neural net classification of images) or between representative features$^2$ and the labels.
$^1$ Labels are the parameters of interest to be estimated (a variable star classification in this case).
$^2$Features = measured properties of the sources.
Once the mapping between the data and the labels has been learned from the training set, new classifications can be obtained by applying the machine learning model to sources where the labels are unknown.
Break Out Problem 1
Why would it be useful to measure features from astronomical light curves in order to classify them in an automated fashion?
Solution to Break Out 1
Write your answer here
The peculiarities of astronomical light curves (observational gaps, heteroskedastic uncertainties, etc) makes it difficult to compare any 2 random sources. For example, the cadence of observations in one portion of the sky will ultimately be very different from any other point on the sky separated by an appreciable distance ($\sim 100^\circ$ for LSST).
The use of features allows us to place all sources on the same basis. In this way it then becomes possible to make 1 to 1 comparisons between sources with different observing sequences.
Problem 1) The ML Training Set
Here we are going to define some helper functions that...
Can you write Python code for it?

def lc_plot(t, m, m_unc, period=0.0):
    if period == 0.0:
        fig, ax = plt.subplots()
        ax.errorbar(t, m, m_unc,
                    fmt='o', color='MediumAquaMarine',
                    mec="0.2",mew=0.5)
        ax.set_xlabel('HJD (d)')
        ax.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        fig.gca().invert_yaxis()
    elif period != 0.0:
        fig = plt.figure()
        gs = GridSpec.GridSpec(5, 1)
        ax_full = plt.subplot(gs[:2, :])
        ax_full.errorbar(t, m, m_unc,
                         fmt='o', color='MediumAquaMarine',
                         mec="0.2",mew=0.5)
        ax_full.set_xlabel('HJD (d)')
        ax_full.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        plt.gca().invert_yaxis()

        ax_phase = plt.subplot(gs[2:, :])
        for repeat in [-1, 0, 1]:
            ax_phase.errorbar(t/period % 1 + repeat, m, m_unc,
                             fmt='o', color='MediumAquaMarine',
                             mec="0.2",mew=0.5)
        ax_phase.axvline(x=0, ls='--', color='0.8', lw=1, zorder=3)
        ax_phase.axvline(x=1, ls='--', color='0.8', lw=1, zorder=3)
        ax_phase.set_xlim(-0.2, 1.2)
        ax_phase.set_xlabel('Phase')
        ax_phase.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        plt.gca().invert_yaxis()

    plt.tight_layout()

def read_lc(filename):
    hjd, mag, mag_unc = np.loadtxt(filename, unpack=True)
    return hjd, mag, mag_unc


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
If you did not already have the training set, download and unpack the tarball.
%> tar -zxvf feature_engineering.tar.gz
We will be working with data from the ASAS survey, and I have already curated a training set that only includes stars in 1 of 7 classes: Mira variables, RR Lyrae stars, detatched eclipsing binaries, semi-detatched eclipsing binaries, W UMa binaries, Cepheids, and R Cor Bor stars.
[If you don't know what any of these things are, don't worry, we have examples below.]
Problem 1a
Plot an example Mira light curve.
Hint - just execute the cell.
Can you write Python code for it?

# Mira example
t, m, m_unc = read_lc("./training_lcs/181637+0341.6")
lc_plot(t, m, m_unc, period=150.461188)


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1e
Plot an example W UMa EB light curve.
Can you write Python code for it?

# WU example
t, m, m_unc = read_lc("./training_lcs/193546-1136.3")
lc_plot(t, m, m_unc, period=0.424015)


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1g
Plot an example R Cor Bor star light curve.
Can you write Python code for it?

# R Cor Bor example
t, m, m_unc = read_lc("./training_lcs/163242-5315.6")
lc_plot(t, m, m_unc, period=0.0)


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 2) Worry About the Data
Feature engineering is all about domain expertise. Before you begin the process of adding, creating, and removing features, it is important to develop some intuition for what features might be helpful (or in other words... worry about the data).
Problem 2a
Examine the light curves of at least two sources from each source in the training set. In the text cell below, write characteristics of the different classes that you notice which may be helpful for classification.
The class of every source in the training set is listed in training_sources.csv. The helper functions in Problem 1 can be used to examine the light curves.
Hint – if you want to examine phase-folded light curves, such as those shown above, you will need to measure the period for each source. Check your notes from Session 13 if you don't remember how to do this.
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 4) Feature Engineering
It should now be clear why this is not a machine learning problem. We have, in this case, provided all the machine learning code that you will need.
Instead, this is a feature engineering problem. Feature engineering requires the utilization of domain knowledge to create new features for a data set to improve the performance of machine learning algorithms.
Add new features - if necessary
  - Utilize domain knowledge to create/compute new features
  - Combine features or represent them in an alternative fashion
Remove noisy/uniformative features - if necessary
  - Determine feature importance (RF)
  - Forward/backward selection to iteratively remove features
Below I have provided a partial function calc_feature which you can alter to calculate new features for the data set.
Can you write Python code for it?

def calc_feature(df, train=True):

    if train==True:
        lc_dir = "./training_lcs/"
    else:
        lc_dir = "./test_lcs/"

    feature = np.empty(len(df))
    for source_num, asas_id in enumerate(df["ASAS_ID"]):
        t, m, mu = read_lc(lc_dir+asas_id)
        # feature calculations
        # feature calculations
        # feature calculations

        feature[source_num] = feat_val

    return feature


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 5a
Calculate the accuracy of your model via an analysis of the independent test set. A helper function has been provided for you to do this below.
Can you write Python code for it?

from sklearn.metrics import accuracy_score
def calc_model_acc(X_train, y_train, X_test, y_test):
    '''
    Parameters
    ----------

    X_train - arr_like, (n_source, n_feat) shape
        Feature set for the training set. A 2D array
        containing one row for every source, and one
        column for every feature in the training set.
    y_train - arr_like, (n_source,) shape
        Labels for the training set, with one label
        per source.
    X_test - arr_like, (n_source, n_feat) shape
        Feature set for the test set. A 2D array
        containing one row for every source, and one
        column for every feature in the training set.
    y_test - arr_like, (n_source,) shape
        Labels for the test set, with one label
        per source.

    '''
    rf_clf = RandomForestClassifier(n_estimators=150, min_samples_leaf=1)
    rf_clf.fit(X_train, y_train)
    y_pred = rf_clf.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    print("Your model accuracy is: {:.2f}".format(accuracy*100))


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
The provided training set comes with 3 features: i) the mean magnitude of the observations, ii) the total number of observations obtained, and iii) the duration of the observations.
Problem 3b
Using the helper function provided below, calculate the 10-fold cross-validation accuracy of a random forest machine learning model using the 3 features provided above.
Note - do not adjust any part of calc_cv_score throughout this exercise.
Can you write Python code for it?

def calc_cv_score(X, y):
    rf_clf = RandomForestClassifier(n_estimators=150, min_samples_leaf=1)

    cv_score = cross_val_score(rf_clf, X, y, cv=10, n_jobs=-1)

    print("These features have CV accuracy = {:.4f} +/- {:.4f}".format(np.mean(cv_score), np.std(cv_score, ddof=1)))

calc_cv_score( # complete


I am working on the file "Sessions/Session14/Day2/FeatureEngineering.ipynb".
The first task is:
write your answer here
Problem 3) Machine Learning Classification
To classify newly observed light curves we need a machine learning model.
Previously I said this is not a machine learning problem, and that is because we will all use the same pre-specified model. I have provided a file training_sources.csv which includes the name of the sources, along with some features, and their classification.
Problem 3a
Read in the training set file, and create a feature vector X and label array y.
Can you write Python code for it?

train_df = pd.read_csv("training_sources.csv")

X_train = np.array(train_df[["mean", "nobs", "duration"]])
y_train = np.array(train_df["Class"])


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Bonus Problem 5a Playing with the Latent Space
Create a random forest classifier to classiy each galaxy using only your latent space.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, ConfusionMatrixDisplay
clf = RandomForestClassifier(max_depth=20, random_state=0)
yhat, train_encodings = model(images_train)

clf.fit(train_encodings.detach().numpy(), labels_train)
new_labels = clf.predict(test_encodings.detach().numpy())

cm = confusion_matrix(labels_test,new_labels,normalize='true')
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot()
plt.show()


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Problem 1b.
Split the training and test set with a 66/33 split.
Can you write Python code for it?

images_train, images_test, labels_train, labels_test = train_test_split(
    images, labels, test_size=0.33, random_state=42)


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Problem 2: Understanding the Autoencoder
Below is sample of an autoencoder, built in Pytorch. Describe the code line-by-line with a partner. Add another hidden layer before and after the encoded (latent) layer (this will be a total of 2 new layers). Choose the appropriate activation function for this regression problem. Make all of the activation functions the same.
Can you write Python code for it?

class Autoencoder(torch.nn.Module):
      # this defines the model
        def __init__(self, input_size, hidden_size, hidden_inner, encoded_size):
            super(Autoencoder, self).__init__()
            print(input_size,hidden_size,encoded_size)
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.encoded_size = encoded_size
            self.hidden_inner = hidden_inner
            self.hiddenlayer1 = torch.nn.Linear(self.input_size, self.hidden_size)
            self.hiddenlayer2 = torch.nn.Linear(self.hidden_size, self.hidden_inner)
            self.encodedlayer = torch.nn.Linear(self.hidden_inner, self.encoded_size)
            self.hiddenlayer3 = torch.nn.Linear(self.encoded_size, self.hidden_inner)
            self.hiddenlayer4 = torch.nn.Linear(self.hidden_inner, self.hidden_size)
            self.outputlayer = torch.nn.Linear(self.hidden_size, self.input_size)
            # some nonlinear options
            self.sigmoid = torch.nn.Sigmoid()
            self.softmax = torch.nn.Softmax()
            self.relu = torch.nn.ReLU()
        def forward(self, x):
            layer1 = self.hiddenlayer1(x)
            activation1 = self.relu(layer1)
            layer2 = self.hiddenlayer2(activation1)
            activation2 = self.relu(layer2)
            layer3 = self.encodedlayer(activation2)
            activation3 = self.relu(layer3)
            layer4 = self.hiddenlayer3(activation3)
            activation4 = self.relu(layer4)
            layer5 = self.hiddenlayer4(activation4)
            activation5 = self.relu(layer5)
            layer6 = self.outputlayer(activation5)
            output = self.relu(layer6)
            return output, layer3


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Problem 3. Training
This is going to be a lot of guess-and-check. You've been warned. In this block, we will train the autoencoder. Add a plotting function into the training.
Note that instead of cross-entropy, we use the "mean-square-error" loss. Switch between SGD and Adam optimized. Which seems to work better? Optimize the learning-rate parameter and do not change other parameters, like momentum.
Write a piece of code to run train_model for 10 epochs. Play with the size of each hidden layer and encoded layer. When you feel you've found a reasonable learning rate, up this to 100 (or even 500 if you're patient) epochs. Hint: You want to find MSE~0.25.
Can you write Python code for it?

# train the model
def train_model(training_data,test_data, model):
  # define the optimization
  criterion = torch.nn.MSELoss()

  # Choose between these two optimizers
  #optimizer = torch.optim.SGD(model.parameters(), lr=0.005)
  optimizer = torch.optim.Adam(model.parameters(), lr=0.005,weight_decay=1e-6)

  for epoch in range(500):
    # clear the gradient
    optimizer.zero_grad()
    # compute the model output
    myoutput, encodings_train = model(training_data)
    # calculate loss
    loss = criterion(myoutput, training_data)
    # credit assignment
    loss.backward()
    # update model weights
    optimizer.step()
    print(epoch,loss)
    # STUDENTS ADD THIS PART
    output_test, encodings_test = model(test_data)
    loss_test = criterion(output_test, test_data)
    plt.plot(epoch,loss.detach().numpy(),'ko')
    plt.plot(epoch,loss_test.detach().numpy(),'ro')
  plt.show()

model = Autoencoder(np.shape(images_train[0])[0],50,50,10)
train_model(images_train, images_test, model)


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Problem 4a. Understand our Results
Plot an image (remember you will need to reshape it to a 14x14 grid) with imshow, and plot the autoencoder output for the same galaxy. Try plotting the difference between the two. What does your algorithm do well reconstructing? Are there certain features which it fails to reproduce?
Can you write Python code for it?

yhat, test_encodings = model(images_test)
yhat = yhat.detach().numpy()
yhat = np.reshape(yhat,(len(images_test),12,12))
image_reshaped = np.reshape(images_test,(len(images_test),12,12))
rand_int = int(np.random.choice(np.arange(len(yhat))))
plt.imshow(yhat[rand_int])
plt.show()

plt.imshow(image_reshaped[rand_int])
plt.show()

plt.imshow(yhat[rand_int] - image_reshaped[rand_int].detach().numpy())
plt.show()


I am working on the file "Sessions/Session14/Day3/Autoencoders.ipynb".
The first task is:
Problem 4b.
Make a scatter plot of two of the 10 latent space dimensions. Do you notice any interesting correlations between different subsets of the latent space? Any interesting clustering?
Try color coding each point by the galaxy label using plt.scatter
Can you write Python code for it?

plt.scatter(test_encodings[0:500,1].detach().numpy(),
            test_encodings[0:500,3].detach().numpy(),
            c = labels_test[0:500],alpha=0.5)
plt.show()


I am working on the file "Sessions/Session14/Day3/AutoencodersBlank.ipynb".
The first task is:
Bonus Problem 5b Playing with the Latent Space
Create an isolation forest to find the most anomalous galaxies. Made a cumulative distribution plot showing the anomaly scores of each class of galaxies. Which ones are the most anomalous? Why do you think that is?
Can you write Python code for it?

clf = IsolationForest(...).fit(encodings)
scores = -clf.score_samples(encodings) #I am taking the negative because the lowest score is actually the weirdest, which I don't like...

#Plot an image of the weirdest galazy!

#This plots the cumulative distribution
def cdf(x, label='',plot=True, *args, **kwargs):
    x, y = sorted(x), np.arange(len(x)) / len(x)
    return plt.plot(x, y, *args, **kwargs, label=label) if plot else (x, y)

ulabels = np.unique(labels)
for ulabel in ulabels:
  gind = np.where(labels==ulabel)
  cdf(...)


I am working on the file "Sessions/Session14/Day3/AutoencodersBlank.ipynb".
The first task is:
Problem 1a: Understanding our dataset...again
Our data is a little too big for us to train an autoencoder in ~1 minute. Let's lower the resolution of our images and only keep one filter. Plot an example of the lower resolution galaxies.
Next, flatten each image into a 1D array. Then rescale the flux of the images such that the mean is 0 and the standard deviation is 1.
Can you write Python code for it?

# Readin the data
images, labels = load_galaxy10()
labels = labels.astype(np.float32)
images = images.astype(np.float32)
images = torch.tensor(images)
labels = torch.tensor(labels)
# Cut down the resolution of the images!!! What is this line doing in words?
images = images[:,::6,::6,1]

#Plot an example image here

#Flatten images here

#Normalize the flux of the images here


I am working on the file "Sessions/Session14/Day3/AutoencodersBlank.ipynb".
The first task is:
Problem 1b.
Split the training and test set with a 66/33 split.
Problem 2: Understanding the Autoencoder
Below is sample of an autoencoder, built in Pytorch. Describe the code line-by-line with a partner. Add another hidden layer before and after the encoded (latent) layer (this will be a total of 2 new layers). Choose the appropriate activation function for this regression problem. Make all of the activation functions the same.
Can you write Python code for it?

class Autoencoder(torch.nn.Module):
      # this defines the model
        def __init__(self, input_size, hidden_size, hidden_inner, encoded_size):
            super(Autoencoder, self).__init__()
            print(input_size,hidden_size,encoded_size)
            self.input_size = input_size
            self.hidden_size  = hidden_size
            self.encoded_size = encoded_size
            self.hidden_inner = hidden_inner
            self.hiddenlayer1 = torch.nn.Linear(self.input_size, self.hidden_size)
            # ADD A LAYER HERE
            self.encodedlayer = torch.nn.Linear(self.hidden_inner, self.encoded_size)
            self.hiddenlayer3 = torch.nn.Linear(self.encoded_size, self.hidden_inner)
            # ADD A LAYER HERE
            self.outputlayer = torch.nn.Linear(self.hidden_size, self.input_size)
            # some nonlinear options
            self.sigmoid = torch.nn.Sigmoid()
            self.softmax = torch.nn.Softmax()
            self.relu = torch.nn.ReLU()
        def forward(self, x):
            layer1 = self.hiddenlayer1(x)
            activation1 = self.ACTIVATION?(layer1)
            layer2 = self.hiddenlayer2(activation1)
            activation2 = self.ACTIVATION?(layer2)
            layer3 = self.encodedlayer(activation2)
            activation3 = self.ACTIVATION?(layer3)
            layer4 = self.hiddenlayer3(activation3)
            activation4 = self.ACTIVATION?(layer4)
            layer5 = self.hiddenlayer4(activation4)
            activation5 = self.ACTIVATION?(layer5)
            layer6 = self.outputlayer(activation5)
            output = self.ACTIVATION?(layer6)

            # Why do I have two outputs?
            return output, layer3