I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 3j
Use the newly created female, male, S, Q, and C features in combination with the Pclass, SibSp, Parch, and Fare features to estimate the classification accuracy of a $k = 7$ nearest neighbors model with 10-fold cross validation.
How does the addition of the point of origin feature affect the final model output?
Hint - don't forget to scale the features in the model
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

X = # complete

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

cv_results = cross_val_score( # complete

print('The accuracy with categorical features = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 3n
Use the age regression model to predict the ages for passengers with missing data.
Can you write Python code for it?

missing_ages = np.where(np.isnan(titanic_df['Age']))[0]

impute_X_missing = # complete

X_missing_minmax = scaler.transform(impute_X_missing)

age_preds = lr_age.predict(X_missing_minmax)


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 4b
Are there categorical features in the dataset? If yes, convert the categorical features for use in machine learning models.
Hint - recall Problem 3h and 3i.
Can you write Python code for it?

type_dict = create_bin_cat_feats( # complete

# complete
# complete


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 4c
Is there missing data in the training set? If yes, how will you deal these data?
Hint - you were already told there is missing data.
Can you write Python code for it?

# WISE non-detections have SNR < 2

for wsnr in ['w1snr', 'w2snr', 'w3snr', 'w4snr']:
    frac_missing = # complete
    print('{:.2f}% of the obs in {} are non-detections'.format(100*frac_missing, wsnr[0:2]))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 4e
Estimate the RMSE for a KNeighborsRegression via 10 fold CV.
Hint - use cross_val_predict in this case so you can plot $z_\mathrm{SDSS}$ vs. $z_\mathrm{kNN}$.
Hint 2 - make the plot in a separate cell so you can adjust it without needing to re-run the CV.
Can you write Python code for it?

# cross validation goes here
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import mean_squared_error

knn_reg = # complete
y_preds = cross_val_predict( # complete
print('The RMSE = {}'.format(np.sqrt(mean_squared_error(y,y_preds))))

# plotting goes here

fig, ax = plt.subplots(figsize=(7,7))
ax.scatter( # complete
ax.plot([0,6],[0,6], 'Crimson')
ax.set_xlabel('z_SDSS')
ax.set_ylabel('z_kNN')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Problem 4g
How many catastrophic outliers did your previous predictions have?
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Scaling the features leads to further improvement!
Now turn your attention to another categorical feature, Embarked, which includes the point of origin for each passenger and has three categories, S, Q, and C. We need to convert these values to a numeric representation for inclusion in the model.
Problem 3g
Convert the categorical feature Embarked to a numeric representation, and add it to the titanic_df.
Can you write Python code for it?

# following previous example, set C = 0, S = 1, Q = 2

porigin = # complete
# complete
# complete
# complete

titanic_df['porigin'] = porigin


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
That wasn't too terrible. But what if we consider a more typical light curve table, where there are loads of missing data, such as Table 2 from Foley et al. 2009:
<img style="display: block; margin-left: auto; margin-right: auto" src="images/Foley09_tbl2.png" align=\"middle">
Again, this table is straightforward to read, and it isn't hard to imagine how one could construct a machine-readable csv or other file from this information. But alas, this is not what is available from ApJ. So, we will need to figure out how to deal with both the missing data, "...", and the weird convention that many astronomers use where the uncertainties are (a) not reported in their own column, and (b) are not provided in the same units as the measurement itself. I can understand the former, but the later is somewhat baffling...
Problem 2b
Download the ApJ version of Table 2 from Foley et al. 2009 and read that table into either a pandas DataFrame or an astropy Table.
Can you write Python code for it?

# read in the data from Foley et al 2009


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
The MinMax scaler didn't help at all!
Perhaps we need a different kind of wrangling for this data set - feature engineering. (For example, does it make sense to include extinction as a feature, or should we instead combine the extinction with the observed magnitudes to get extinction corrected brightness measurements?)
Finally - we close with a machine learning question, not a data wrangling question. Does it actually make sense to use a kNN model for datasets that have categorical features?
Problem 4i
Build a better machine learning model, possibly with the use of different features, to improve the model classification performance.
Hint - there was a hint about this in yesterday's lecture.
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor

# complete
# complete
# complete
# complete

# plotting goes here

fig, ax = plt.subplots(figsize=(8,8))
ax.scatter( # complete
ax.plot([0,6],[0,6], 'Crimson')
ax.set_xlim(-0.1,4)
ax.set_ylim(-0.1,4)
ax.set_xlabel('z_SDSS')
ax.set_ylabel('z_kNN')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
The accuracy of the model hasn't improved by adding the age information (even though we know children were more likely to survive than adults).
Given that the missing ages don't have meaning, we need to develop alternative strategies for "imputing" the missing data. The most simple approach in this regard is to replace the missing values with the mean value of the feature distribution for sources that do have measurements (use the median if the distribution has significant outliers).
Problem 3l
Replace the unknown ages with the mean age of passengers, and estiamte the accuracy of the model via 10-fold cross validation.
Can you write Python code for it?

age_impute = # complete
# complete

titanic_df['age_impute'] = age_impute

X = # complete

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

cv_results = cross_val_score( # complete

print('The accuracy with the mean for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
The last thing we'd like to add to the model is the Age feature. Unfortunately, for 177 passengers we do not have a reported value for their age. This is a standard issue when building models known as "missing data" and this happens in astronomy all the time (for example, LSST is going to observe millions of L and T dwarfs that are easily detected in the $y$-band, but which do not have a detection in $u$-band).
There are several different strategies for dealing with missing data. The first and most straightforward is to simply remove observations with missing data (note - to simplify this example I already did this by removing the 2 passengers from the training set that did not have an entry for Embarked).
This strategy is perfectly fine if only a few sources have missing information (2/891 for Embarked - and none of the test set sources are missing Embarked). If, however, a significant fraction are missing data, this strategy would remove a lot of useful data from the model.
If you cannot remove the sources with missing data, then it is essential to ask the following question:
Does the missing information have meaning?
In the LSST L/T dwarf example, the lack of a $u$-band detection is meaningful: these stars are too faint for LSST. When this is the case, an indicator value (e.g., -999) allows the model to recognize the non-detection.
For the Titanic data, the lack of age information is not meaningful. Simply put, there are some passengers that did not have recorded ages. We will now show this to be the case.
Problem 3k
Replace the unknown ages with a value of -999, and estimate the accuracy of the model via 10-fold cross validation.
Can you write Python code for it?

age_impute = # complete
# complete

titanic_df['age_impute'] = age_impute

X = # complete

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

cv_results = cross_val_score( # complete

print('The accuracy with -999 for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyData.ipynb".
The first task is:
Wranglin' –– Corraling Unruly Data
One bit at a time
Version 0.1
By AA Miller 2019 Mar 26
Webster's Dictionary$^\ast$ defines wrangler as:
wrangler noun
wran·gler | raŋ-g(ə-)lər
(short for horse-wrangler, probably partial translation of Mexican Spanish caballerango groom): a ranch hand who takes care of the saddle horses broadly : cowboy
$^\ast$actually https://www.merriam-webster.com/dictionary/ - Webster's didn't define wrangler in the way I wanted
How then, as astronomers, are we all like cowhands?
Data are often like horses in that: they all differ, rarely conform to a single standard set of behavior, and they love to eat hay.$^\dagger$
$^\dagger$I made that last one up.
Thus, in our efforts to better understand the Universe, we must often manipulate, coax, and, in some cases, force our data to "behave." This involves a variety of tasks, such as: gathering, cleaning, matching, restructuring, transforming, filtering, combining, merging, verifying, and fixing data.
Here is a brief and unfortunate truth, there isn't a single person in the entire world that would organize data in exactly the same way that you would.
As a result, you may find that data that are useful to you are not organized in an optimal fashion for use in your workflow/software.
Hence: the need to wrangle.
There is one important and significant way in which our lives as astronomers are much better than the average data scientist: even though our data are "worthless," virtually all of it is numbers.
Furthermore, I contend that most astronomical data can easily be organized into a simple tabular structure.
Yesterday we joked that you could write a 10 page text document describing every individual galaxy in a galaxy cluster with 5000 members, but no one would ever do that.
Nevertheless, as you will see during the exercises, even with relatively simple, small numerical data sets there is a need for wrangling.
And wrangling brings up a lot of issues...
Consider the following data set that contains the street names for my best friends from...
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
An accuracy of 68% isn't particularly inspiring. But, there's a lot of important information that we are excluding. As far as the Titanic is concerned, Kate and Leo taught us that female passengers are far more likely to survive, while male passengers are not. So, if we can include gender in the model then we may be able to achieve more accurate predictions.
Problem 3e
Create a new feature called gender that equals 1 for male passengers and 2 for female passengers. Add this feature to your dataframe, and include it in a kNN model with the other numeric features. Does the inclusion of this feature improve the 10-fold CV accuracy?
Can you write Python code for it?

gender = np.ones(len(titanic_df['Sex']))
gender[np.where(titanic_df['Sex'] == 'female')] = 2

titanic_df['gender'] = gender

X = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare', 'gender']]

knn_clf = KNeighborsClassifier(n_neighbors=7)

cv_results = cross_val_score(knn_clf, X, y, cv=10)

print('The accuracy when including gender = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
As far as ages are concerned, imputation of the missing data does not significantly improve the model.
Which brings us to the concluding lesson in wrangling the Titanic data - not every piece of information is useful. This is critical to remember when building machine learning models.
(As a quick aside - it wouldn't be entirely fair to say there is no useful information in the age feature. It is clear, for example, that "children," i.e. those with Age < 10, had a much higher probability of survival than adults. Perhaps the creation of a child feature based on age would improve the model... or using the age in combination with other features, e.g., AgexPclass which will further highlight that 1st class passengers were more likely to survive than 3rd class passengers)
Finally - note that you can try to build a model and submit it to Kaggle to see how well you preform on blind data.
https://www.kaggle.com/c/titanic - the classifications are not revealed, but from the leaderboard it is clear that some people were able to build models that perfectly classified the blind data.
Problem 4) Wrangling an astro machine learning model
Now we will put together the above examples to work on an astronomical problem of great importance for LSST: photometric redshifts. As we have previously discussed, LSST will observe many, many, many, many more galaxies than we can possibly observe with spectroscopic instruments. As a result, the vast majority of the galaxies observed by LSST will not have precise redshift estimates, and instead we will need to estimate redshifts via photometry alone.
This week we have heard a lot about SDSS CasJobs, and for today's problem we will use 50,000 SDSS sources with spectroscopic measurements to train a machine learning model to estimate redshifts from photometry. The following query was used to generate this training set:
select top 50000 s.specObjID, s.z, s.type,
   psfMag_u, psfMag_g, psfMag_r, psfMag_i, psfMag_z,
   modelMag_u, modelMag_g, modelMag_r, modelMag_i, modelMag_z,
  ...
Can you write Python code for it?

sdss = pd.read_csv("DSFP_SDSS_spec_train.csv")
sdss[:5]


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
But wait! Does this actually make sense?
Our "numerification" has now introduced order where there previously was none. We are effectively telling the model that Cherbourg and Queenstown are far apart (not in distance but in terms of the similarity of the passengers that boarded the ship in each location), while each are equally close to Southampton. Is there actually any evidence to support this conclusion?
By definition categorical features do not have order (e.g., cat, dog, horse, elephant), and therefore we should not impose any when converting these features to numeric values for inclusion in our model. Instead, we should be creating a new set of binary features for every category within the feature set. Thus, Embarked will now need to be represented by 3 different features, where the feature Queenstown equals one for passengers that boarded there and zero for everyone else.
Problem 3h
Complete the function below that will automatically create binary arrays for a categorical feature.
Can you write Python code for it?

def create_bin_cat_feats(feature_array):
    categories = np.unique(feature_array)
    feat_dict = {}
    for cat in categories:
        exec('{} = np.zeros(len(feature_array)).astype(int)'.format(cat))
        exec('{0}[np.where(feature_array == "{0}")] = 1'.format(cat))
        exec('feat_dict["{0}"] = {0}'.format(cat))

    return feat_dict


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Earlier we saw that the performance of a kNN model greatly improves with feature scaling.
Problem 4h
Use a MinMax scaler to scale the features, performs 10 fold cross-validation, and estimate the catastrophic outlier fraction.
Can you write Python code for it?

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

knn_reg = KNeighborsRegressor(n_neighbors=7)
y_preds = cross_val_predict(knn_reg, Xminmax, y, cv=10)
print('The RMSE = {}'.format(np.sqrt(mean_squared_error(y,y_preds))))
print('{} are catastrophic'.format(catastrophic_fraction(y, y_preds)))

# plotting goes here

fig, ax = plt.subplots()
ax.scatter(y, y_preds, alpha=0.02)
ax.plot([0,6],[0,6], 'Crimson')
ax.set_xlabel('z_SDSS')
ax.set_ylabel('z_kNN')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
If you made a plot of redshift vs. predictions, you likely saw that there are many sources that are far from the 1 to 1 line. These are called catastrophic outliers, and they are a serious problem for science programs that rely on photometric redshifts.
Problem 4f
Write a function that takes input arrays ground_truth and predictions and determines the fraction of the dataset that is catastrophic outliers. For today's purposes we will say a catastrophic outlier is one where the prediction differs by 20%.
Can you write Python code for it?

def catastrophic_fraction(ground_truth, predictions, threshold=0.2):
    '''Function to calculate fraction of predictions that are catastrophic outliers

    Parameter
    ---------
    ground_truth : array-like
        Correct labels for the model sources

    predictions : array-like
        Predictions for the model sources

    threshold : float (optional, default=0.2)
        The threshold to determine if a "miss" is catastrophic or not

    Returns
    -------
    oh_nos : float
        Fractional number of catastrophic outliers
    '''
    n_outliers = len(np.where(np.abs(ground_truth - predictions)/ground_truth > threshold)[0])
    oh_nos = n_outliers/len(ground_truth)

    return oh_nos


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Note - that should have failed! And for good reason - recall that kNN models measure the Euclidean distance between all points within the feature space. So, when considering the sex of a passenger, what is the numerical distance between male and female?
In other words, we need to wrangle this data before we can run the machine learning model.
Most of the features in this problem are non-numeric (i.e. we are dealing with categorical features), and therefore we need to figure out how to include them in the kNN model.
The first step when wrangling for machine learning is to figure out if anything can be thrown away. We certainly want to avoid including any uninformative features in the model.
If you haven't already, now would be a good time to create a new cell and examine the contents of the csv
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Now that we have dealt with missing and categorical data we can construct out machine learning model. We will use a $k$ nearest neighbors regression model to determine how well we can measure photometric redshifts.
The sdss dataframe now includes far more columns than are necessary for the model. Should any of these be excluded from the fitting?
Problem 4d
Build a scikit-learn appropriate X and y array for the features and labels (i.e. redshifts) of the photo-z training set.
Can you write Python code for it?

X = np.array(sdss[['psfMag_u', 'psfMag_g', 'psfMag_r',
       'psfMag_i', 'psfMag_z', 'modelMag_u', 'modelMag_g', 'modelMag_r',
       'modelMag_i', 'modelMag_z', 'extinction_u', 'extinction_g',
       'extinction_r', 'extinction_i', 'extinction_z','ext', 'ps',
       'w1det', 'w1mag', 'w2det', 'w2mag', 'w3det', 'w3mag', 'w4det', 'w4mag']])

y = np.array(sdss['z'])


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Now that we have the data in the appropriate X and y arrays, estimate the accuracy with which a K nearest neighbors classification model can predict whether or not a passenger would survive the Titanic disaster. Use $k=10$ fold cross validation for the prediction.
Problem 3b
Train a $k=7$ nearest neighbors machine learning model on the Titanic training set.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier
knn_clf = KNeighborsClassifier(n_neighbors=7)
knn_clf.fit(X, y)


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Okay - there is nothing elegant about that particular solution. But it works, and wranglin' ain't pretty.
It is likely that you developed a solution that looks very different from this one, and that is fine. When data are provided in an unrulely format, the most important thing is to develop some method, any method, for converting the information into a useful format. Following whatever path you used above, it should now be easy to plot the light curve of SN 2008ha.
Problem 3) My Heart Will Go On
Sometimes there is no difficultly whatsoever in reading in the data (as was the case in Problems 1 and 2), but instead the difficultly lies in wranglin' the data to be appropriate for the model that you are building.
For the next problem we will work with the famous Titanic survival data set. If you haven't already, please download the csv file with the relevant information.
Briefly, the Titantic is a famous historical ship that was thought to be unsinkable. Spoiler alert it hit an iceberg and sank. The data in the Titanic data set includes information about 891 passengers from the Titanic, as well as whether or not they survived. The aim of this data set is to build a machine learning model to predict which passengers survived and which did not.
The features include:
|Feature    | Description |
|:---------:|:--------------------------------------:|
|PassengerId| Running index that describes the individual passengers|
|Pclass| A proxy for socio-economic status (1 = Upper class, 2 = Middle Class, 3 = Lower Class)|
|Name| The passenger's name|
|Sex | The passenger's sex|
|Age | The passenger's age - note age's ending in 0.5 are estimated |
|SibSp| The sum of the passenger's sibblings and spouces on board|
|Parch| The sum of the passenger's parents and children on board|
|Ticket| The ticket number for the passenger|
|Fare| The price paid for the ticket by th passenger|
|Cabin| The Cabin in which the passenger stayed|
|Embarked| The point of Origin for the Passenger: C = Cherbourg, S = Southampton, Q = Queenstown|
And...
Can you write Python code for it?

titanic_df = pd.read_csv('titanic_kaggle_training_set.csv', comment='#')

feat_list = list(titanic_df.columns)
label = 'Survived'
feat_list.remove(label)
X = titanic_df[feat_list].values
y = titanic_df[label]


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Problem 3c
Are there any features that obviously will not help with this classification problem?
If you answer yes - ignore those features moving forward
Solution 3c
write your solution here
The Name of the passenger will not be useful for this classification task. In particular, the useful information that might be learned from the name, such as the sex (e.g., Mr. vs. Mrs.), or the age (e.g., Mr. vs. Master), are already summarized elsewhere in the data.
It is also highly likely that the PassengerId, which is just a running index for each person in the dataset, is unlikely to be useful when classifying this data.
Given that we have both categorical and numeric features, let's start with the numerical features and see how well they can predict survival on the Titanic.
One note - for now we are going to exclude Age, because as you saw when you examined the data, there are some passengers that do not have any age information. This problem, known as "missing data" is one that we will deal with before the end of this problem.
Problem 3d
How accurately can the numeric features, Pclass, SibSp, Parch, and Fare predict survival on the Titanic? Use a $k = 7$ Nearest Neighbors model, and estimate the model accuracy using 10-fold cross validation.
Hint 1 - you'll want to redefine your features vector X
Hint 2 - you may find cross_val_score from scikit-learn helpful
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

X = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare']]

knn_clf = KNeighborsClassifier(n_neighbors=7)

cv_results = cross_val_score(knn_clf, X, y, cv=10)

print('The accuracy from numeric features = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Problem 3i
Use the create_bin_cat_feats function to convert the Embarked and Sex, yes we need to do this for Sex as well where we otherwise previously introduced order, categorical features to a numeric representation. Add these features to the titanic_df data frame.
Can you write Python code for it?

gender_dict = create_bin_cat_feats(titanic_df['Sex'])
porigin_dict = create_bin_cat_feats(titanic_df['Embarked'])

for feat in gender_dict.keys():
    titanic_df[feat] = gender_dict[feat]

for feat in porigin_dict.keys():
    titanic_df[feat] = porigin_dict[feat]


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Problem 3j
Use the newly created female, male, S, Q, and C features in combination with the Pclass, SibSp, Parch, and Fare features to estimate the classification accuracy of a $k = 7$ nearest neighbors model with 10-fold cross validation.
How does the addition of the point of origin feature affect the final model output?
Hint - don't forget to scale the features in the model
Can you write Python code for it?

from sklearn.preprocessing import MinMaxScaler

X = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare', 'female', 'male', 'S', 'Q', 'C']]

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

knn_clf = KNeighborsClassifier(n_neighbors=7)

cv_results = cross_val_score(knn_clf, Xminmax, y, cv=10)

print('The accuracy with categorical features = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Problem 3o
Use the imputed age estimates to predict the passenger survival via 10-fold cross validation.
Can you write Python code for it?

age_impute = titanic_df['Age'].copy().values
age_impute[missing_ages] = age_preds

titanic_df['age_impute'] = age_impute

X = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare', 'female', 'male', 'S', 'Q', 'C', 'age_impute']]

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

knn_clf = KNeighborsClassifier(n_neighbors=7)

cv_results = cross_val_score(knn_clf, Xminmax, y, cv=10)

print('The accuracy with the mean for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Problem 4e
Estimate the RMSE for a KNeighborsRegression via 10 fold CV.
Hint - use cross_val_predict in this case so you can plot $z_\mathrm{SDSS}$ vs. $z_\mathrm{kNN}$.
Hint 2 - make the plot in a separate cell so you can adjust it without needing to re-run the CV.
Can you write Python code for it?

# cross validation goes here
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import mean_squared_error

knn_reg = KNeighborsRegressor(n_neighbors=7)
y_preds = cross_val_predict(knn_reg, X, y, cv=10)
print('The RMSE = {}'.format(np.sqrt(mean_squared_error(y,y_preds))))

# plotting goes here

fig, ax = plt.subplots()
ax.scatter(y, y_preds, alpha=0.02)
ax.plot([0,6],[0,6], 'Crimson')
ax.set_xlabel('z_SDSS')
ax.set_ylabel('z_kNN')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Scaling the features leads to further improvement!
Now turn your attention to another categorical feature, Embarked, which includes the point of origin for each passenger and has three categories, S, Q, and C. We need to convert these values to a numeric representation for inclusion in the model.
Problem 3g
Convert the categorical feature Embarked to a numeric representation, and add it to the titanic_df.
Can you write Python code for it?

# following previous example, set C = 0, S = 1, Q = 2

porigin = np.empty(len(titanic_df['Sex'])).astype(int)
porigin[np.where(titanic_df['Embarked'] == 'C')] = 0
porigin[np.where(titanic_df['Embarked'] == 'S')] = 1
porigin[np.where(titanic_df['Embarked'] == 'Q')] = 2

titanic_df['porigin'] = porigin


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
That pure python solution is a bit annoying as it requires a for loop with a break, and specific knowledge about how IRSA tables handler data headers (hence the use of linenum + 4 for skiprows). Alternatively, one could  manipulate the data file to read in the data.
Can you write Python code for it?

# solution 2 - edit the text file
# !cp irsa_catalog_WISE_iPTF14jg_search_results.tbl tmp.tbl
### delete lines 1-89, and 90-92
### replace whitespace with commas (may require multiple operations)
### replace '|' with commas
### replace ',,' with single commas
### replace ',\n,' with '\n'
### delete the comma at the very beginning and very end of the file

tedit_tbl = pd.read_csv('tmp.tbl')
tedit_tbl.head(5)


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
That truly wasn't all that better - as it required a bunch of clicks/text editor edits. (There are programs such as sed and awk that could be used to execute all the necessary edits from the command line, but that too is cumbersome and somewhat like the initial all python solution).
If astronomers are creating data in a "standard" format, then it ought to be easy for other astronomers to access that data.
Fortunately, in this particular case, there is an easy solution - astropy Tables.
IRSA tables are so commonly used throughout the community, that the folks at astropy have created a convenience method for all of us to read in tables created in that particular (unusual?) format.
Problem 1b
Use Table.read() to read in irsa_catalog_WISE_iPTF14jg_search_results.tbl to an astropy Table object.
Can you write Python code for it?

from astropy.table import Table

Table.read('irsa_catalog_WISE_iPTF14jg_search_results.tbl', format='ipac')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
The MinMax scaler didn't help at all!
Perhaps we need a different kind of wrangling for this data set - feature engineering. (For example, does it make sense to include extinction as a feature, or should we instead combine the extinction with the observed magnitudes to get extinction corrected brightness measurements?)
Finally - we close with a machine learning question, not a data wrangling question. Does it actually make sense to use a kNN model for datasets that have categorical features?
Problem 4i
Build a better machine learning model, possibly with the use of different features, to improve the model classification performance.
Can you write Python code for it?

from sklearn.ensemble import RandomForestRegressor

for filt in ['u', 'r', 'i', 'z']:
    sdss['psf_g-{}'.format(filt)] = sdss['psfMag_g'] - sdss['psfMag_{}'.format(filt)]
    sdss['model_g-{}'.format(filt)] = sdss['modelMag_g'] - sdss['modelMag_{}'.format(filt)]

X = np.array(sdss[['ps', 'ext',
                   'w1det', 'w1mag', 'w2det', 'w2mag', 'w3det', 'w3mag', 'w4det', 'w4mag',
                   'psf_g-u', 'psf_g-r', 'psf_g-i', 'psf_g-z',
                   'model_g-u', 'model_g-r', 'model_g-i', 'model_g-z']])

rf_reg = RandomForestRegressor(n_estimators=100)
y_preds = cross_val_predict(rf_reg, X, y, cv=10)
print('The RMSE = {}'.format(np.sqrt(mean_squared_error(y,y_preds))))
print('{} are catastrophic'.format(catastrophic_fraction(y, y_preds)))

# plotting goes here

fig, ax = plt.subplots(figsize=(8,8))
ax.scatter(y, y_preds, alpha=0.02)
ax.plot([0,6],[0,6], 'Crimson')
ax.set_xlim(-0.1,4)
ax.set_ylim(-0.1,4)
ax.set_xlabel('z_SDSS')
ax.set_ylabel('z_kNN')


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
The accuracy of the model hasn't improved by adding the age information (even though we know children were more likely to survive than adults).
Given that the missing ages don't have meaning, we need to develop alternative strategies for "imputing" the missing data. The most simple approach in this regard is to replace the missing values with the mean value of the feature distribution for sources that do have measurements (use the median if the distribution has significant outliers).
Problem 3l
Replace the unknown ages with the mean age of passengers, and estiamte the accuracy of the model via 10-fold cross validation.
Can you write Python code for it?

age_impute = titanic_df['Age'].copy().values
age_impute[np.isnan(age_impute)] = np.mean(age_impute[np.isfinite(age_impute)])

titanic_df['age_impute'] = age_impute

X = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare', 'female', 'male', 'S', 'Q', 'C', 'age_impute']]

scaler = MinMaxScaler()
scaler.fit(X)
Xminmax = scaler.transform(X)

knn_clf = KNeighborsClassifier(n_neighbors=7)

cv_results = cross_val_score(knn_clf, Xminmax, y, cv=10)

print('The accuracy with the mean for missing ages = {:.2f}%'.format(100*np.mean(cv_results)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Using the mean age for missing values provides a marginal improvement over the models with no age information. Is there anything else we can do? Yes - we can build a machine learning model to predict the values of the missing data. So there will be a machine learning model within the final machine learning model. In order to predict ages, we will need to build a regression model. Simple algorithms include Linear or Logistic Regression, while more complex examples include kNN or random forest regression.
I quickly tested the above four methods, and found that the scikit-learn LinearRegression performs best when using the model defaults.
Probem 3m
Build a LinearRegression model to predict a passenger's age based on the Pclass, SibSp, Parch, Fare, female, male, S, Q, C features. The model should be trained with passengers that have known ages. Use 10-fold cross validation to determine the performance of this model.
Hint - note that for regression models the typical metric of evaluation is the mean squared error, and that for consistency within the scikit-learn API, the negative mean squared error is returned rather than the mean squared error.
Can you write Python code for it?

from sklearn.linear_model import LinearRegression

has_ages = np.where(np.isfinite(titanic_df['Age']))[0]

impute_X_train = titanic_df[['Pclass', 'SibSp', 'Parch', 'Fare', 'female', 'male', 'S', 'Q', 'C']].iloc[has_ages]
impute_y_train = titanic_df['Age'].iloc[has_ages]

scaler = MinMaxScaler()
scaler.fit(impute_X_train)
Xminmax = scaler.transform(impute_X_train)

lr_age = LinearRegression().fit(Xminmax, impute_y_train)

cv_results = cross_val_score(LinearRegression(), Xminmax, impute_y_train, cv=10, scoring='neg_mean_squared_error')

print('Missing ages have RMSE = {:.2f}'.format(np.mean((-1*cv_results)**0.5)))


I am working on the file "Sessions/Session08/Day2/CorrallingUnrulyDataSolutions.ipynb".
The first task is:
Wranglin' –– Corraling Unruly Data
One bit at a time
Version 0.1
By AA Miller 2019 Mar 26
Webster's Dictionary$^\ast$ defines wrangler as:
wrangler noun
wran·gler | raŋ-g(ə-)lər
(short for horse-wrangler, probably partial translation of Mexican Spanish caballerango groom): a ranch hand who takes care of the saddle horses broadly : cowboy
$^\ast$actually https://www.merriam-webster.com/dictionary/ - Webster's didn't define wrangler in the way I wanted
How then, as astronomers, are we all like cowhands?
Data are often like horses in that: they all differ, rarely conform to a single standard set of behavior, and they love to eat hay.$^\dagger$
$^\dagger$I made that last one up.
Thus, in our efforts to better understand the Universe, we must often manipulate, coax, and, in some cases, force our data to "behave." This involves a variety of tasks, such as: gathering, cleaning, matching, restructuring, transforming, filtering, combining, merging, verifying, and fixing data.
Here is a brief and unfortunate truth, there isn't a single person in the entire world that would organize data in exactly the same way that you would.
As a result, you may find that data that are useful to you are not organized in an optimal fashion for use in your workflow/software.
Hence: the need to wrangle.
There is one important and significant way in which our lives as astronomers are much better than the average data scientist: even though our data are "worthless," virtually all of it is numbers.
Furthermore, I contend that most astronomical data can easily be organized into a simple tabular structure.
Yesterday we joked that you could write a 10 page text document describing every individual galaxy in a galaxy cluster with 5000 members, but no one would ever do that.
Nevertheless, as you will see during the exercises, even with relatively simple, small numerical data sets there is a need for wrangling.
And wrangling brings up a lot of issues...
Consider the following data set that contains the street names for my best friends from...
Can you write Python code for it?

# Solution 1 - pure python solution with pandas

with open('irsa_catalog_WISE_iPTF14jg_search_results.tbl') as f:
    ll = f.readlines()
    for linenum, l in enumerate(ll):
        if l[0] == '|':
            header = l.replace('|', ',').replace(' ', '')
            header = list(header[1:-2].split(','))
            break

irsa_tbl = pd.read_csv("irsa_catalog_WISE_iPTF14jg_search_results.tbl",
            skiprows=linenum+4, delim_whitespace=True,
            header=None, names=header)
irsa_tbl.head(5)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
And now that you have results, drop the rawPhot table if it already exists, and load the observations into the database as before.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
As we discussed in the lecture, one challenge for this is problem is that individual observations are largely spaced out, and have very large uncertainties. The features that we have currently adopted (which are really just moments of the flux distribution) aren't robust to outliers, and are subject to the precise cadence of the observations.
One way to "correct" for the observational gaps and heteroskedastic uncertainties is to interpolate a model between the observations. A method that has become really popular recently is the use of Gaussian process regression (there will be more along these lines in the next homework assignment, but here we will provide a preview).
Can you write Python code for it?

import sklearn.gaussian_process as gp

# we'll create a function that just fits the g band lightcurve
def calculate_gpfit(filename):
    """
    Do a quick gaussian process regression of just g-band lightcurve to get a smooth representation
    This should be more robust to outliers
    """
    var = Variable(filename)
    var.filter_flux()
    t = var._g_time.values
    y = var._g_flux.values
    dy = var._g_flux_unc.values
    t = t.reshape(-1, 1)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    # a Gaussian Process is defined by a correlation function that relates each point to each other
    # I've chosen a simple common kernel - the Matern kernel.
    # It has a nice property in that it is 'stationary' - the covariance between points
    # only depends on their separation

    base_kernel = gp.kernels.Matern(length_scale=10.,nu=2.5, length_scale_bounds=(1.,20.))
    gkernel = 0.5*(np.median(dy)**2.)*base_kernel

    # the second thing you need to define a Gaussian Process is a mean function, which in our case is
    # implictly zero (this is fine for transients - they spend most of eternity at the background!)
    gband = gp.GaussianProcessRegressor(kernel=gkernel, alpha=dy**2.).fit(t, y)

    # now lets use the Gaussian process to predict the lightcurve on an evenly sampled grid
    # this is appropriate for Wavelets, or generally robust feature extraction

    newtime = np.linspace(t.min(), t.max(), 200, endpoint=True)
    newtime = newtime.reshape(-1, 1)
    gnew = gband.predict(newtime, return_std=True)
    return (t, y , dy, newtime, gnew)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
End of Warning
Later in the notebook we will attempt to speed up our processing via a few parallel operations, the next few slides set that up.
Can you write Python code for it?

# first we need to construct a client that will interface with our cluster
from ipyparallel import Client, require
worker = Client()

# once we create a client, we can decide how to allocate tasks across the cluster
# we've got however many 'engines' you started in the cluster
# lets just use all of them

lview = worker[:]

# now if you want to import packages, you can import them across all the 'engines'
with lview.sync_imports():
    import numpy as np
    import scipy.stats as spstat
    import pandas as pd
    import os

# there's not much point in creating plots on the engines - we want to actually see them presumably
%matplotlib inline
import matplotlib.pyplot as plt

# If everything has worked so far, you should see a list of worker IDs, and a corresponding list of process IDs.
# You can verify it with a `ps`
ar = lview.apply_async(os.getpid)
print("Engine ID:PID mapping: ", ar.get_dict())


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Following the sqlite3 import, we must first connect to the database. If we attempt a connection to a database that does not exist, then a new database is automatically created. Here we will create a new database file, called MiniBroker.db.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
For now - don't worry about what the labels mean (though if you inspect the light curves you may be able to figure this out...)
Problem 4a
Query the database to select features and labels for the light curves in your training set. Store the results of these queries in numpy arrays, X and y, respectively, which are suitable for the various scikit-learn machine learning algorithms.
Hint - recall that databases do not store ordered results.
Hint 2 - recall that scikit-learn expects y to be a 1d array. You will likely need to convert a 2d array to 1d.
Can you write Python code for it?

cur.execute( # complete
y = np.array(cur.fetchall()).ravel()

cur.execute(# complete - note objID is NOT a feature
X = np.array(cur.fetchall())


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
It turns out that the classification performance in this case isn't particularly good.
For the test set, the actual class labels were sneakily hidden in the file headers, and if you execute the cell below you will discover how many sources are in each class.
Can you write Python code for it?

! grep label test_set_for_LSST_DSFP/*dat | sed -e 's/label:/ /' | awk '{print $2+1}' | sort | uniq -c


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Lets look at one fit to see what's happening:
Can you write Python code for it?

ftest = filenames[0:20]
gtime, gflux, gdflux, gnewtime, gnewflux = calculate_gpfit(ftest[6])

fig, ax = plt.subplots(figsize=(7,7))
ax.errorbar(gtime, gflux, gdflux, fmt = 'o', color = '#78A5A3')
ax.plot(gnewtime, gnewflux[0], 'k-')
ax.fill(np.concatenate([gnewtime, gnewtime[::-1]]),
        np.concatenate([gnewflux[0] - 1.9600 * gnewflux[1],
                        (gnewflux[0] + 1.9600 * gnewflux[1])[::-1]]),
         alpha=.5, fc='0.8', ec='None', label='95% confidence interval')
ax.set_xlabel(r"$\mathrm{MJD}$")
ax.set_ylabel(r"$\mathrm{flux}$")
fig.tight_layout()


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Load all of the raw photometric observations into the rawPhot table in the database.
Problem 3b
Load all of the raw photometric observations into the rawPhot table in the database.
Hint - you can use glob to select all of the files being loaded.
Hint 2 - you have already loaded the data from FAKE001 into the table.
Can you write Python code for it?

# build your list of filenames here
import glob
filenames = glob.glob("training_set_for_LSST_DSFP/FAKE*.dat")

for # complete in complete:
    # complete
    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Not only does GP regression provide an interpolation between observations, but it returns an uncertainty about that interpolation as well. Thus, it's possible to measure features on interpolated light curves that are observed with precisely the same cadence for all sources.
But here's the rub - can we practically run this on 10^4--5 alerts every 37 seconds for LSST?
This is one reason parallelization is so important - as long as we have enough computers, we can easily keep up with the LSST data rate even with complex features (e.g., periodicity search, GP regression, etc).
Problem 6) Anomaly Detection
As we learned earlier - one of the primary goals of LSST brokers is to reduce the stream of 10 million alerts on any given night to the single (or 10, or 100) most interesting objects. One possible definition of "interesting" is rarity - in which case it would be useful to add some form of anomaly detection to the pipeline. scikit-learn has several different algorithms that can be used for anomaly detection. Here we will employ isolation forest which has many parallels to random forests, which we have previously learned about.
In brief, isolation forest builds an ensemble of decision trees where the splitting parameter in each node of the tree is selected randomly. In each tree the number of branches necessary to isolate each source is measured - outlier sources will, on average, require fewer splittings to be isolated than sources in high-density regions of the feature space. Averaging the number of branchings over many trees results in a relative ranking of the anomalousness (yes, I just made up a word) of each source.
Problem 6a
Using IsolationForest in sklearn.ensemble - determine the 10 most isolated sources in the data set.
Hint - for IsolationForest you will want to use the decision_function() method rather than predict_proba(), which is what we have previously used with sklearn.ensemble models to get relative rankings from the model.
Can you write Python code for it?

from sklearn.ensemble import IsolationForest

isoF_clf = # complete
# complete
# complete

print("The 10 most anomalous sources are: {}".format( # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Now that we have a cursor object, we can populate the database. As an example we will start by creating a table to hold all the raw photometry (though ultimately we will not use this table for analysis).
Note - there are many cursor methods capable of interacting with the database. The most common, execute, takes a single SQL command as its argument and executes that command. Other useful methods include executemany, which is useful for inserting data into the database, and executescript, which take an SQL script as its argument and executes the script.
In many cases, as below, it will be useful to use triple quotes in order to improve the legibility of your code.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if is already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 2) Data Preparation
While we could create a database table that includes every single photometric measurement made by LSST, this ~37 trillion row db would be enormous without providing a lot of added value beyond the raw flux measurements [while this table is necessary, alternative tables may provide more useful information]. Furthermore, extracting individual light curves from such a database will be slow. Instead, we are going to develop summary statistics for every source which will make it easier to select individual sources and develop classifiers to identify objects of interest.
Below we will create a new class, Variable, which will inherit the properties of the Source class. The Variable class includes several additional methods so we can (eventually) store summary statistics in a database table. In the interest of time, we limit the summary statistics to a relatively small list all of which have been shown to be useful for classification (see Richards et al. 2011 for further details). The statistics that we include (for now) are:

Std -- the standard deviation of the flux measurements
Amp -- the amplitude of flux deviations
MAD -- the median absolute deviation of the flux measurements
beyond1std -- the fraction of flux measurements beyond 1 standard deviation
skew -- the skewness of a data set
the mean $g' - r'$, $r' - i'$, and $i' - z'$ color

Note - while the magnitude system used in astronomy is stupid, and as a reminder:
$$\mathrm{mag} = -2.5\log{\mathrm{flux}} + C$$
where $C$ is a constant. Comparisons of $\Delta \mathrm{flux}$ for different stars encodes both the brightness variations and the distance to the star in question. $\Delta \mathrm{mag}$, on the other hand, normalizes the brightness variations to provide a more apples to apples comparison (however the use of a logarithmic basis results in a biased apples to apples comparison).
All of this is a long way of saying - when you calculate the Std, Amp, and MAD below you want to normalize these measurements by dividing each by...
Can you write Python code for it?

class Variable( # complete
    '''Variable subclass of fake sources'''

    def __init__( # complete
        # complete

    def filter_flux(self):
        '''Store individual passband fluxes as object attributes'''

        for filt in self._filters:
            filt_obs = np.where(self._lc_df['pb'] == filt)
            setattr(self, '_{0}_time'.format(filt),
                    self._lc_df['t'].iloc[filt_obs])
            setattr(self, '_{0}_flux'.format(filt),
                    self._lc_df['flux'].iloc[filt_obs])
            setattr(self, '_{0}_flux_unc'.format(filt),
                    self._lc_df['dflux'].iloc[filt_obs])

    def weighted_mean_flux(self):
        '''Measure (SNR weighted) mean flux in griz'''

        if not hasattr(self, '_g_flux'):
            self.filter_flux()

        weighted_mean = lambda flux, dflux: np.sum(flux*(flux/dflux)**2)/np.sum((flux/dflux)**2)

        for filt in self._filters:
            filt_flux = getattr(self, '_{0}_flux'.format(filt))
            filt_dflux = getattr(self, '_{0}_flux_unc'.format(filt))
            setattr(self, '_{0}_mean'.format(filt),
                    weighted_mean(filt_flux, filt_dflux))

    def normalized_flux_std(self):
        # complete
        # complete
        # complete
        # complete

    def normalized_amplitude(self):
        # complete
        # complete
        # complete
        # complete

    def normalized_MAD(self):
        # complete
        # complete
        # complete
        # complete

    def beyond_1std(self):
        # complete
        # complete
        # complete
        # complete

    def skew(self):
        # complete
        # complete
        # complete
        # complete

    def mean_colors(self):
        # complete
        # complete
        # complete
        # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 3) Part 2 -- Calculate and store the features in the same database
Now that we have loaded the raw observations, we need to create a new table to store summary statistics for each object. This table will include the Variable class attributes.
Can you write Python code for it?

cur.execute("""drop table if exists lcFeats""") # drop the table if it already exists
cur.execute("""create table lcFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES rawPhot(objId)
                )
""")


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 3e Confirm that reading the data in parallel is faster than the seriel operations in 3d using the %%timeit function as before.
Can you write Python code for it?

tmp_conn3 = # complete
# complete

# complete
# complete
# complete
# complete

%%timeit
# time how long it takes to read files in parallel

result = lview.map(load_data, filenames)
all_data = result.get()

for data in all_data:
    tmp_cur3.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 3h
Confirm that the data loaded correctly by counting the number of sources with gAmp > 2.
How many sources have gMinusR = -999?
Hint - you should find 9 and 2, respectively.
Can you write Python code for it?

cur.execute( # complete

nAmp2 = cur.fetchone()[0]

cur.execute( # complete
nNoColor = cur.fetchone()[0]

print("There are {:d} sources with gAmp > 2".format(nAmp2))
print("There are {:d} sources with no measured i' - z' color".format(nNoColor))


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 4) Build a Classification Model
One of the primary goals for LSST Brokers is to separate the Wheat from the Chaff. In other words, given that ~10 million alerts will be issued by LSST on a nightly basis, what is the single (or 10, or 100) most interesting alerts.
Here we will build on the skills developed during the previous homework to construct a machine-learning model to classify new light curves.
Fortunately - the data that has already been loaded to MiniBroker.db is a suitable training set for the classifier (we simply haven't provided you with labels just yet). Execute the cell below to add a new table to the database which includes the appropriate labels.
Can you write Python code for it?

cur.execute("""drop table if exists lcLabels""") # drop the table if it already exists
cur.execute("""create table lcLabels(
               objId int,
               label int,
               foreign key(objId) references rawPhot(objId)
               )""")

labels = np.zeros(100)
labels[20:60] = 1
labels[60:] = 2

data = np.append(np.arange(1,101)[np.newaxis].T, labels[np.newaxis].T, axis = 1)
tup_data = [tuple(x) for x in data]

cur.executemany("""insert into lcLabels(objId, label) values (?,?)""", tup_data)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 4b
Train a SVM model (SVC in scikit-learn) using a radial basis function (RBF) kernel with penalty parameter, $C = 1$, and kernel coefficient, $\gamma = 0.1$.
Evaluate the accuracy of the model via $k = 5$ fold cross validation.
Hint - you may find the cross_val_score module helpful.
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score( # complete

print("The SVM model produces a CV accuracy of {:.4f}".format(np.mean(cv_scores)))


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 4d
Does the model you selected perform better than the SVM model? Perform a $k = 5$ fold cross validation to determine which model provides superior accuracy.
Can you write Python code for it?

cv_scores = cross_val_score( # complete

print("My model produces a CV accuracy of {:.4f}".format( # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 4e
Which model are you going to use in your Broker? Justify your answer.
Write solution to 4e here
Problem 5) Class Predictions for New Sources
Now that we have developed a basic infrastructure for dealing with streaming data, we may reap the rewards of our efforts. We will use our ANTARES-like software to classify newly observed sources.
Problem 5a
Load the light curves for the new observations (found in full_testset_for_LSST_DSFP) into the a table in the database.
Hint - ultimately it doesn't matter much one way or another, but you may choose to keep new observations in a table separate from the training data. I put these observations into a new testPhot database. Up to you.
Can you write Python code for it?

# create new tables testPhot and testFeats

new_obs_filenames = glob.glob("test_set_for_LSST_DSFP/FAKE*.dat")

result = lview.map( # complete

for data in all_data:
    # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 5b
Calculate features for the new observations and insert those features into a database table.
Hint - again, you may want to create a new table for this, up to you. I'm using the testFeats table.
Can you write Python code for it?

result = lview.map(calculate_features, new_obs_filenames)
all_feats = result.get()

for feats in all_feats:
        # complete
            # complete
                # complete
                    # complete


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
Problem 5c
Train the model that you adopted in 4e on the training set, and produce predictions for the newly observed sources.
What is the class distribution for the newly detected sources?
Hint - the training set was constructed to have a nearly uniform class distribution, that may not be the case for the actual observed distribution of sources.
Can you write Python code for it?

clf = # complete

cur.execute( # complete
X_new = np.array(cur.fetchall())

y_preds = # complete

print("""There are {:d}, {:d}, and {:d} sources
         in classes 1, 2, 3, respectively""".format(*list(np.bincount(y_preds)))) # be careful using bincount


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
There are two things to highlight above: (1) we do not specify an id for the data as this is automatically generated, and (2) the data insertion happens via a tuple. In this case, we are taking advantage of the fact that a Python tuple can be concatenated:
(objId,) + tuple(lc10.DFlc.ix[0]))

While the above example demonstrates the insertion of a single row to the database, it is far more efficient to bulk load the data. To do so we will delete, i.e. DROP, the rawPhot table and use some pandas manipulation to load the contents of an entire file at once via executemany.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")

# next 3 lines are already in name space; repeated for clarity
filename = "training_set_for_LSST_DSFP/FAKE001.dat"
var = Variable(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

data = [(objId,) + tuple(x) for x in var._lc_df.values] # array of tuples

cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
We now have a database connection object, conn. To interact with the database (create tables, load data, write queries) we need a cursor object.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day3/miniBroker.ipynb".
The first task is:
While we cannot load the light curves to the database in parallel, using the concept of MapReduce we can use the iPython "cluster" that we created to map the I/O process (typically lots of overhead) to multiple machines, and then we can reduce their output to a single object to be loaded into the database.
Of course, before you parallelize, you profile.
Problem 3d
Earlier it was claimed that bulk loading the data is faster than loading it line by line. Prove this assertion, use %%timeit to "profile" the two different options (bulk load with executemany and loading one photometric measurement at a time via for loop).
Compare both serial solutions, to load the data in parallel.
Hint - to avoid corruption of your current working database, MiniBroker.db, create temporary databases for the pupose of running this test. Note - the creation and connection to the database should happen outside your %%timeit magic function. This means your solution will have (at least) 4 separate cells.
Can you write Python code for it?

tmp_conn1 = sqlite3.connect("tmp1.db")
tmp_cur1 =  # complete

tmp_cur1.execute("""drop table if exists rawPhot""") # drop the table if it already exists
tmp_cur1.execute( # create the table

%%timeit
# time how long it takes to load every observation individually

for filename in filenames:
    # complete
    # complete
    # complete
    # complete
    # complete

tmp_conn2 = # complete
# complete

# complete
# complete

%%timeit
# time how long it takes to load all observations of a single source at once

# complete
# complete
# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
And now that you have results, drop the rawPhot table if it already exists, and load the observations into the database as before.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
                """)

for data in all_data:
    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
As we discussed in the lecture, one challenge for this is problem is that individual observations are largely spaced out, and have very large uncertainties. The features that we have currently adopted (which are really just moments of the flux distribution) aren't robust to outliers, and are subject to the precise cadence of the observations.
One way to "correct" for the observational gaps and heteroskedastic uncertainties is to interpolate a model between the observations. A method that has become really popular recently is the use of Gaussian process regression (there will be more along these lines in the next homework assignment, but here we will provide a preview).
Can you write Python code for it?

import sklearn.gaussian_process as gp

# we'll create a function that just fits the g band lightcurve
def calculate_gpfit(filename):
    """
    Do a quick gaussian process regression of just g-band lightcurve to get a smooth representation
    This should be more robust to outliers
    """
    var = Variable(filename)
    var.filter_flux()
    t = var._g_time.values
    y = var._g_flux.values
    dy = var._g_flux_unc.values
    t = t.reshape(-1, 1)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    # a Gaussian Process is defined by a correlation function that relates each point to each other
    # I've chosen a simple common kernel - the Matern kernel.
    # It has a nice property in that it is 'stationary' - the covariance between points
    # only depends on their separation

    base_kernel = gp.kernels.Matern(length_scale=10.,nu=2.5, length_scale_bounds=(1.,20.))
    gkernel = 0.5*(np.median(dy)**2.)*base_kernel

    # the second thing you need to define a Gaussian Process is a mean function, which in our case is
    # implictly zero (this is fine for transients - they spend most of eternity at the background!)
    gband = gp.GaussianProcessRegressor(kernel=gkernel, alpha=dy**2.).fit(t, y)

    # now lets use the Gaussian process to predict the lightcurve on an evenly sampled grid
    # this is appropriate for Wavelets, or generally robust feature extraction

    newtime = np.linspace(t.min(), t.max(), 200, endpoint=True)
    newtime = newtime.reshape(-1, 1)
    gnew = gband.predict(newtime, return_std=True)
    return (t, y , dy, newtime, gnew)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
As we have many light curve files (in principle as many as 37 billion...), we will define a Source() class to ease our handling of the data.
Problem 1b
Complete the plotting method in the Source class defined below.
Hint - a for loop can be helpful for dealing with the different filters
Can you write Python code for it?

class Source():
    '''Astronomical source object for NOAO formatted light curve

    Parameters
    ----------
    filename : str
        path to space delimited file including each time,
        filter, flux, and flux_uncertainty brightness
        measurement on each line. A header line must be
        included in the file with 't pb flux dflux'

    Attributes
    ----------
    _filename : str
        the filename parameter

    _lc_df : pandas DataFrame
        light curve data read into a pandas dataframe

    _filters : arr-like
        numpy array storing each of the unique passbands
        in the light curve file
    '''

    def __init__(self, filename):
        lc_df = pd.read_csv(filename, delim_whitespace=True, comment = '#')
        u_obs = np.where(lc_df['pb'] == 'u')
        if len(u_obs[0]) > 0:
            lc_df.drop(lc_df.index[u_obs], inplace=True)
        self._lc_df = lc_df
        self._filename = filename
        self._filters = np.unique(self._lc_df['pb'])

    def plot_multicolor_lc(self):
        '''Plot the 4 band light curve'''
        fig, ax = plt.subplots(figsize=(7,7))
        color_dict = {'g': '#78A5A3',
                      'r': '#CE5A57',
                      'i': '#E1B16A',
                      'z': '#444C5C', }

        for filt in self._filters:
            filt_obs = np.where(self._lc_df['pb'] == filt)
            color = color_dict[filt]
            ax.errorbar(self._lc_df['t'].iloc[filt_obs],
                        self._lc_df['flux'].iloc[filt_obs],
                        self._lc_df['dflux'].iloc[filt_obs],
                        fmt = 'o', color = color_dict[filt],
                        label = "${0}$".format(filt))
        ax.legend(fancybox = True)
        ax.set_xlabel(r"$\mathrm{MJD}$")
        ax.set_ylabel(r"$\mathrm{flux}$")
        fig.tight_layout()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Following the sqlite3 import, we must first connect to the database. If we attempt a connection to a database that does not exist, then a new database is automatically created. Here we will create a new database file, called MiniBroker.db.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
It turns out that the classification performance in this case isn't particularly good.
For the test set, the actual class labels were sneakily hidden in the file headers, and if you execute the cell below you will discover how many sources are in each class.
Can you write Python code for it?

! grep label test_set_for_LSST_DSFP/*dat | sed -e 's/label:/ /' | awk '{print $2+1}' | sort | uniq -c


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Let's unpack everything that happened in these two commands. First - if the table rawPhot already exists, we drop it to start over from scratch. (this is useful here, but should not be adopted as general practice)
Second - we create the new table rawPhot, which has 6 columns: id - a running index for every row in the table, objId - an ID to identify which source the row belongs to, t - the time of observation in MJD, pb - the passband of the observation, flux the observation flux, and dflux the uncertainty on the flux measurement. In addition to naming the columns, we also must declare their type. We have declared id as the primary key, which means this value will automatically be assigned and incremented for all data inserted into the database. We have also declared pb as a variable character of length 1, which is more useful and restrictive than simply declaring pb as text, which allows any freeform string.
Now we need to insert the raw flux measurements into the database. To do so, we will use the BrokerLc class that we created earlier. As an initial example, we will insert the first 3 observations from the source FAKE010.
Can you write Python code for it?

filename = "training_set_for_LSST_DSFP/FAKE001.dat"
var = Variable(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(var._lc_df.iloc[0])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(var._lc_df.iloc[1])))
cur.execute("""insert into rawPhot(objId, t, pb, flux, dflux) values {}""".format((objId,) + tuple(var._lc_df.iloc[2])))


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Lets look at one fit to see what's happening:
Can you write Python code for it?

ftest = filenames[0:20]
gtime, gflux, gdflux, gnewtime, gnewflux = calculate_gpfit(ftest[6])

fig, ax = plt.subplots(figsize=(7,7))
ax.errorbar(gtime, gflux, gdflux, fmt = 'o', color = '#78A5A3')
ax.plot(gnewtime, gnewflux[0], 'k-')
ax.fill(np.concatenate([gnewtime, gnewtime[::-1]]),
        np.concatenate([gnewflux[0] - 1.9600 * gnewflux[1],
                        (gnewflux[0] + 1.9600 * gnewflux[1])[::-1]]),
         alpha=.5, fc='0.8', ec='None', label='95% confidence interval')
ax.set_xlabel(r"$\mathrm{MJD}$")
ax.set_ylabel(r"$\mathrm{flux}$")
fig.tight_layout()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Load all of the raw photometric observations into the rawPhot table in the database.
Problem 3b
Load all of the raw photometric observations into the rawPhot table in the database.
Hint - you can use glob to select all of the files being loaded.
Hint 2 - you have already loaded the data from FAKE001 into the table.
Can you write Python code for it?

# build your list of filenames here
import glob
filenames = glob.glob("training_set_for_LSST_DSFP/FAKE*.dat")

for filename in filenames[1:]:
    var = Variable(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])

    data = [(objId,) + tuple(x) for x in var._lc_df.values] # array of tuples

    cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Now that we have a cursor object, we can populate the database. As an example we will start by creating a table to hold all the raw photometry (though ultimately we will not use this table for analysis).
Note - there are many cursor methods capable of interacting with the database. The most common, execute, takes a single SQL command as its argument and executes that command. Other useful methods include executemany, which is useful for inserting data into the database, and executescript, which take an SQL script as its argument and executes the script.
In many cases, as below, it will be useful to use triple quotes in order to improve the legibility of your code.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if is already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Now, we use the previously defined lview variable to achieve DirectView access to the cluster to map the load_data function on to all the filenames, and get the results.
Can you write Python code for it?

result = lview.map(load_data, filenames)
all_data = result.get()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
One thing that we brushed over previously is that the brightness measurements have units of flux, rather than the traditional use of magnitudes. The reason for this is that LSST will measure flux variations via image differencing, which will for some sources in some filters result in a measurement of negative flux. (You may have already noticed this in 1a.) Statistically there is nothing wrong with such a measurement, but it is impossible to convert a negative flux into a magnitude. Thus we will use flux measurements throughout this exercise.
Using flux measurements will allow us to make unbiased measurements of the statistical distributions of the variations of the sources we care about.
Problem 1d
What is FAKE010 the source that is plotted above?
Hint - ask Griffin or Rob...
Solution 1d
FAKE010 is a transient, as can be seen by the rapid rise followed by a gradual decline in the light curve. In this particular case, we can further guess that FAKE010 is a Type Ia supernova due to the secondary maxima in the $i'$ and $z'$ light curves. These secondary peaks are not present in any other known type of transient.
Problem 1e
To get a better sense of the data, plot the multiband light curves for sources FAKE060 and FAKE095.
Can you write Python code for it?

lc60 = Source("training_set_for_LSST_DSFP/FAKE060.dat")
lc60.plot_multicolor_lc()

lc95 = Source("training_set_for_LSST_DSFP/FAKE095.dat")
lc95.plot_multicolor_lc()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 1c
Confirm the corrections made in 1b by plotting the multiband light curve for the source FAKE010.
Can you write Python code for it?

lc = Source('training_set_for_LSST_DSFP/FAKE010.dat')

lc.plot_multicolor_lc()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 2) Data Preparation
While we could create a database table that includes every single photometric measurement made by LSST, this ~37 trillion row db would be enormous without providing a lot of added value beyond the raw flux measurements [while this table is necessary, alternative tables may provide more useful information]. Furthermore, extracting individual light curves from such a database will be slow. Instead, we are going to develop summary statistics for every source which will make it easier to select individual sources and develop classifiers to identify objects of interest.
Below we will create a new class, Variable, which will inherit the properties of the Source class. The Variable class includes several additional methods so we can (eventually) store summary statistics in a database table. In the interest of time, we limit the summary statistics to a relatively small list all of which have been shown to be useful for classification (see Richards et al. 2011 for further details). The statistics that we include (for now) are:

Std -- the standard deviation of the flux measurements
Amp -- the amplitude of flux deviations
MAD -- the median absolute deviation of the flux measurements
beyond1std -- the fraction of flux measurements beyond 1 standard deviation
skew -- the skewness of a data set
the mean $g' - r'$, $r' - i'$, and $i' - z'$ color

Note - while the magnitude system used in astronomy is stupid, and as a reminder:
$$\mathrm{mag} = -2.5\log{\mathrm{flux}} + C$$
where $C$ is a constant. Comparisons of $\Delta \mathrm{flux}$ for different stars encodes both the brightness variations and the distance to the star in question. $\Delta \mathrm{mag}$, on the other hand, normalizes the brightness variations to provide a more apples to apples comparison (however the use of a logarithmic basis results in a biased apples to apples comparison).
All of this is a long way of saying - when you calculate the Std, Amp, and MAD below you want to normalize these measurements by dividing each by...
Can you write Python code for it?

class Variable(Source):
    '''Class to hold variable light curves and features

    Parameters
    ----------
    filename : str
        path to space delimited file including each time,
        filter, flux, and flux_uncertainty brightness
        measurement on each line. A header line must be
        included in the file with 't pb flux dflux'

    Attributes
    ----------
    _g_time : arr-like
        times of the g-band observations
    _g_flux : arr-like
        flux measurements of the g-band observations
    _g_flux_unc : arr-like
        uncertainties of the flux measurements of the g-band observations
    _r_time : arr-like
        times of the r-band observations
    _r_flux : arr-like
        flux measurements of the r-band observations
    _r_flux_unc : arr-like
        uncertainties of the flux measurements of the r-band observations
    _i_time : arr-like
        times of the i-band observations
    _i_flux : arr-like
        flux measurements of the i-band observations
    _i_flux_unc : arr-like
        uncertainties of the flux measurements of the i-band observations
    _z_time : arr-like
        times of the z-band observations
    _z_flux : arr-like
        flux measurements of the z-band observations
    _z_flux_unc : arr-like
        uncertainties of the flux measurements of the z-band observations
    _g_mean : float
        weighted mean of the g-band observations
    _r_mean : float
        weighted mean of the r-band observations
    _i_mean : float
        weighted mean of the i-band observations
    _z_mean : float
        weighted mean of the z-band observations
    _g_std : float
        normalized standard deviation of g-band flux measurements
    _r_std : float
        normalized standard deviation of r-band flux measurements
    _i_std : float
        normalized standard deviation of i-band flux measurements
    _z_std : float
        normalized standard deviation of z-band flux measurements
    _g_amp : float
        normalized amplitude of g-band flux...


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 2b
Confirm your solution to 2a by measuring the mean colors of source FAKE010. Does your measurement make sense given the plot you made in 1c?
Can you write Python code for it?

var = Variable('training_set_for_LSST_DSFP/FAKE010.dat')
var.mean_colors()

print("The g'-r', r'-i', and 'i-z' colors are:\
    {:.3f}, {:.3f}, and {:.3f}, respectively.". format(var._g_minus_r, var._r_minus_i, var._i_minus_z))


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 3) Part 2 -- Calculate and store the features in the same database
Now that we have loaded the raw observations, we need to create a new table to store summary statistics for each object. This table will include the Variable class attributes.
Can you write Python code for it?

cur.execute("""drop table if exists lcFeats""") # drop the table if it already exists
cur.execute("""create table lcFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES rawPhot(objId)
                )
""")


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 3) Store the sources in a database
Building (and managing) a database from scratch is a challenging task. For (very) small projects one solution to this problem is to use SQLite, which is a self-contained, publicly available SQL engine. One of the primary advantages of SQLite is that no server setup is required, unlike other popular tools such as postgres and MySQL. In fact, SQLite is already integrated with python so everything we want to do (create database, add tables, load data, write queries, etc.) can be done within Python.
Without diving too deep into the details, here are situations where SQLite has advantages and disadvantages according to their own documentation:
Advantages

Situations where expert human support is not needed
For basic data analysis (SQLite is easy to install and manage for new projects)
Education and training

Disadvantages

Client/Server applications (SQLite does not behave well if multiple systems need to access db at the same time)
Very large data sets (SQLite stores entire db in a single disk file, other solutions can store data across multiple files/volumes)
High concurrency (Only 1 writer allowed at a time for SQLite)

From the (limited) lists above, you can see that while SQLite is perfect for our application right now, if you were building an actual ANTARES-like system a more sophisticated database solution would be required.
Problem 3a
Import sqlite3 into the notebook.
Hint - if this doesn't work, you may need to run
conda install sqlite3
within your IDEAS conda environment.
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 3c
To ensure the data have been loaded properly, select the $g'$ light curve for source FAKE010 from the rawPhot table and plot the results. Does it match the plot from 1c?
Can you write Python code for it?

cur.execute("""select t, flux, dflux
               from rawPhot
               where objId = 10 and pb = 'g'""")

data = cur.fetchall()
data = np.array(data)

fig, ax = plt.subplots(figsize=(7,7))
ax.errorbar(data[:,0], data[:,1], data[:,2], fmt = 'o', color = '#78A5A3')
ax.set_xlabel(r"$\mathrm{MJD}$")
ax.set_ylabel(r"$\mathrm{flux}$")
fig.tight_layout()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 4) Build a Classification Model
One of the primary goals for LSST Brokers is to separate the Wheat from the Chaff. In other words, given that ~10 million alerts will be issued by LSST on a nightly basis, what is the single (or 10, or 100) most interesting alerts.
Here we will build on the skills developed during the previous homework to construct a machine-learning model to classify new light curves.
Fortunately - the data that has already been loaded to MiniBroker.db is a suitable training set for the classifier (we simply haven't provided you with labels just yet). Execute the cell below to add a new table to the database which includes the appropriate labels.
Can you write Python code for it?

cur.execute("""drop table if exists lcLabels""") # drop the table if it already exists
cur.execute("""create table lcLabels(
               objId int,
               label int,
               foreign key(objId) references rawPhot(objId)
               )""")

labels = np.zeros(100)
labels[20:60] = 1
labels[60:] = 2

data = np.append(np.arange(1,101)[np.newaxis].T, labels[np.newaxis].T, axis = 1)
tup_data = [tuple(x) for x in data]

cur.executemany("""insert into lcLabels(objId, label) values (?,?)""", tup_data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 4b
Train a SVM model (SVC in scikit-learn) using a radial basis function (RBF) kernel with penalty parameter, $C = 1$, and kernel coefficient, $\gamma = 0.1$.
Evaluate the accuracy of the model via $k = 5$ fold cross validation.
Hint - you may find the cross_val_score module helpful.
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score(SVC(C = 1.0, gamma = 0.1, kernel = 'rbf'), X, y, cv = 5)

print("The SVM model produces a CV accuracy of {:.4f}".format(np.mean(cv_scores)))


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 4d
Does the model you selected perform better than the SVM model? Perform a $k = 5$ fold cross validation to determine which model provides superior accuracy.
Can you write Python code for it?

cv_scores = cross_val_score(rf_clf, X, y, cv = 5)

print("The RF model produces a CV accuracy of {:.4f}".format(np.mean(cv_scores)))


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 4e
Which model are you going to use in your Broker? Justify your answer.
Write solution to 4e here
In this case we are going to adopt the SVM model as it is a factor of $\sim$50 times faster than RF, while providing nearly identical performance from an accuracy stand point.
Problem 5) Class Predictions for New Sources
Now that we have developed a basic infrastructure for dealing with streaming data, we may reap the rewards of our efforts. We will use our ANTARES-like software to classify newly observed sources.
Problem 5a
Load the light curves for the new observations (found in full_testset_for_LSST_DSFP) into the a table in the database.
Hint - ultimately it doesn't matter much one way or another, but you may choose to keep new observations in a table separate from the training data. I put these observations into a new testPhot database. Up to you.
Can you write Python code for it?

cur.execute("""drop table if exists testPhot""") # drop the table if is already exists
cur.execute("""create table testPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")
cur.execute("""drop table if exists testFeats""") # drop the table if it already exists
cur.execute("""create table testFeats(
                id integer primary key,
                objId int,
                gStd float,
                rStd float,
                iStd float,
                zStd float,
                gAmp float,
                rAmp float,
                iAmp float,
                zAmp float,
                gMAD float,
                rMAD float,
                iMAD float,
                zMAD float,
                gBeyond float,
                rBeyond float,
                iBeyond float,
                zBeyond float,
                gSkew float,
                rSkew float,
                iSkew float,
                zSkew float,
                gMinusR float,
                rMinusI float,
                iMinusZ float,
                FOREIGN KEY(objId) REFERENCES testPhot(objId)
                )
""")

new_obs_filenames = glob.glob("test_set_for_LSST_DSFP/FAKE*.dat")

result = lview.map(load_data, new_obs_filenames)
all_data = result.get()

for data in all_data:
    cur.executemany("""insert into testPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 5c
Train the model that you adopted in 4e on the training set, and produce predictions for the newly observed sources.
What is the class distribution for the newly detected sources?
Hint - the training set was constructed to have a nearly uniform class distribution, that may not be the case for the actual observed distribution of sources.
Can you write Python code for it?

svm_clf = SVC(C=1.0, gamma = 0.1, kernel = 'rbf').fit(X, y)

cur.execute("""select gStd, rStd, iStd, zStd,
                      gAmp,  rAmp,  iAmp,  zAmp,
                      gMAD, rMAD, iMAD, zMAD,
                      gBeyond, rBeyond, iBeyond, zBeyond,
                      gSkew, rSkew, iSkew, zSkew,
                      gMinusR, rMinusI, iMinusZ
               from testFeats
               order by objId asc""")
X_new = np.array(cur.fetchall())

y_preds = svm_clf.predict(X_new)

print("""There are {:d}, {:d}, and {:d} sources
         in classes 1, 2, 3, respectively""".format(*list(np.bincount(y_preds)))) # be careful using bincount


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
Problem 6b
Plot the light curves of the 2 most anomalous sources.
Can you identify why these sources have been selected as outliers? Are these sources in fact outliers?
Can you write Python code for it?

lc491 = Variable("test_set_for_LSST_DSFP/FAKE00491.dat")
lc491.plot_multicolor_lc()

lc2047 = Variable("test_set_for_LSST_DSFP/FAKE02030.dat")
lc2047.plot_multicolor_lc()


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
That isn't quite a factor of 4x speed up, but it represents a significant improvement, and recall that writing to the database must happen as a serial process because we are using SQLite.
Problem 3f
Once again -- ensure the data have been loaded properly by plotting the $g'$ light curve for source FAKE010 from the rawPhot table and plot the results. Does it match your earlier plots?
Can you write Python code for it?


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
The SVM model does a decent job of classifying the data. However - we are going to have 10 million alerts every night. Therefore, we need something that runs quickly. For most ML models the training step is slow, while predictions are (relatively) fast.
Problem 4c
Pick any other classification model from scikit-learn, and "profile" the time it takes to train that model vs. the time it takes to train an SVM model (by using the %%timeit magic function).
Is the model that you have selected faster than SVM?
Hint - you should import the model outside your timing loop as we only care about the training step in this case.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
rf_clf = RandomForestClassifier(n_estimators=25)
svm_clf = SVC(C = 1.0, gamma = 0.1, kernel = 'rbf')

%%timeit
# timing solution for RF model
rf_clf.fit(X,y)

%%timeit
# timing solution for SVM model
svm_clf.fit(X,y)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
The above procedure should look familiar to above, with one exception: the addition of the foreign key in the lcFeats table. The inclusion of the foreign key ensures a connected relationship between rawPhot and lcFeats. In brief, a row cannot be inserted into lcFeats unless a corresponding row, i.e. objId, exists in rawPhot. Additionally, rows in rawPhot cannot be deleted if there are dependent rows in lcFeats.
Problem 3g
In parallel - calculate all the BrokerLc features for every source in rawPhot. Then, bulkload these features into the lcFeats table. Use the same map reduce procedure as above. When calculating the features create a tuple called feats with entries corresponding to the lcFeats table, in the exact order of the columns in that table.
Hint - it might be easier to write a seriel solution first and then introduce the DirectView implemenation. Feature calculation requires significant resources, so accomplishing this in parallel will save lots of time.
Hint 2 - technically, LSST light curves will be "generated" from the rawPhot table. However, skip that step for now as the BrokerLc class reads in data from a file, so loop over those files to populate the lcFeats table.
Can you write Python code for it?

# as before, define a function to calculate the features for one lightcurve

@require(Source, Variable, 'pandas as pd', 'numpy as np', 'scipy.stats as spstat')
def calculate_features(filename):
    var = Variable(filename)
    objId = int(filename.split('FAKE')[1].split(".dat")[0])
    var.weighted_mean_flux()
    var.normalized_flux_std()
    var.normalized_amplitude()
    var.normalized_MAD()
    var.beyond_1std()
    var.skew()
    var.mean_colors()
    feats = (objId, var._g_std, var._r_std, var._i_std, var._z_std,
            var._g_amp,  var._r_amp,  var._i_amp,  var._z_amp,
            var._g_mad, var._r_mad, var._i_mad, var._z_mad,
            var._g_beyond, var._r_beyond, var._i_beyond, var._z_beyond,
            var._g_skew, var._r_skew, var._i_skew, var._z_skew,
            var._g_minus_r, var._r_minus_i, var._i_minus_z)
    return feats

# and then lets map the function onto all the data
result = lview.map(calculate_features, filenames)

# and get the result
all_feats = result.get()

# and now load it all into our table
for feats in all_feats:
    cur.execute("""insert into lcFeats(objId,
                                       gStd, rStd, iStd, zStd,
                                       gAmp,  rAmp,  iAmp,  zAmp,
                                       gMAD, rMAD, iMAD, zMAD,
                                       gBeyond, rBeyond, iBeyond, zBeyond,
                                       gSkew, rSkew, iSkew, zSkew,
                                       gMinusR, rMinusI, iMinusZ) values {}""".format(feats))


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
There are two things to highlight above: (1) we do not specify an id for the data as this is automatically generated, and (2) the data insertion happens via a tuple. In this case, we are taking advantage of the fact that a Python tuple can be concatenated:
(objId,) + tuple(lc10.DFlc.ix[0]))

While the above example demonstrates the insertion of a single row to the database, it is far more efficient to bulk load the data. To do so we will delete, i.e. DROP, the rawPhot table and use some pandas manipulation to load the contents of an entire file at once via executemany.
Can you write Python code for it?

cur.execute("""drop table if exists rawPhot""") # drop the table if it already exists
cur.execute("""create table rawPhot(
                id integer primary key,
                objId int,
                t float,
                pb varchar(1),
                flux float,
                dflux float)
""")

# next 3 lines are already in name space; repeated for clarity
filename = "training_set_for_LSST_DSFP/FAKE001.dat"
var = Variable(filename)

objId = int(filename.split('FAKE')[1].split(".dat")[0])

data = [(objId,) + tuple(x) for x in var._lc_df.values] # array of tuples

cur.executemany("""insert into rawPhot(objId, t, pb, flux, dflux) values (?,?,?,?,?)""", data)


I am working on the file "Sessions/Session08/Day3/miniBrokerSolutions.ipynb".
The first task is:
We now have a database connection object, conn. To interact with the database (create tables, load data, write queries) we need a cursor object.
Can you write Python code for it?