I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Phew! Our hard work to build a machine learning model has been rewarded, by creating an improved model: $\sim{96.9}\%$ accuracy vs. $\sim{96.4}\%$.
[But - was our effort worth only a $0.5\%$ improvement in the model?]
Problem 5) Model Optimization
While the "off-the-shelf" model provides an improvement over the SDSS photometric classifier, we can further refine and improve the performance of the machine learning model by adjusting the model tuning parameters. A process known as model optimization.
All machine-learning models have tuning parameters. In brief, these parameters capture the smoothness of the model in the multidimentional-feature space. Whether the model is smooth or coarse is application dependent -- be weary of over-fitting or under-fitting the data. Generally speaking, RF (and most tree-based methods) have 3 flavors of tuning parameter:

$N_\mathrm{tree}$ - the number of trees in the forest n_estimators (default: 10) in sklearn
$m_\mathrm{try}$ - the number of (random) features to explore as splitting criteria at each node max_features (default: sqrt(n_features)) in sklearn
Pruning criteria - defined stopping criteria for ending continued growth of the tree, there are many choices for this in sklearn (My preference is min_samples_leaf (default: 1) which sets the minimum number of sources allowed in a terminal node, or leaf, of the tree)

Just as we previously evaluated the model using CV, we must optimize the tuning paramters via CV. Until we "finalize" the model by fixing all the input parameters, we cannot evalute the accuracy of the model with the test set as that would be "snooping."
On Tuesday we were introduced to GridSearchCV, which is an excellent tool for optimizing model parameters.
Before we get to that, let's try to develop some intuition for how the tuning parameters affect the final model predictions.
Problem 5a
Determine the 5-fold CV accuracy for models with $N_\mathrm{tree}$ = 1, 10, 100.
How do you expect changing the number of trees to affect the results?
Can you write Python code for it?

rs = 1936 # year JPL was founded

CVpreds1 = # complete

# complete

# complete

print("The CV accuracy for 1, 10, 100 trees is {:.4f}, {:.4f}, {:.4f}".format( # complete

rs = 1936 # year JPL was founded

CVpreds1 = cross_val_predict(RandomForestClassifier(n_estimators = 1, random_state=rs),
                           train_Xnorm, train_y, cv = 5)

CVpreds10 = cross_val_predict(RandomForestClassifier(n_estimators = 10, random_state=rs),
                            train_Xnorm, train_y, cv = 5)

CVpreds100 = cross_val_predict(RandomForestClassifier(n_estimators = 100, random_state=rs),
                             train_Xnorm, train_y, cv = 5)

print("The CV accuracy for 1, 10, 100 trees is {:.4f}, {:.4f}, {:.4f}".format(accuracy_score(train_y, CVpreds1),
                                                                              accuracy_score(train_y, CVpreds10),
                                                                              accuracy_score(train_y, CVpreds100)))


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Problem 1) Obtain and Examine Training Data
As a reminder, for supervised-learning problems we use a training set, sources with known labels, i.e. they have been confirmed as normal stars, QSOs, or galaxies, to build a model to classify new observations where we do not know the source label.
The training set for this exercise uses Sloan Digital Sky Survey (SDSS) data. For features, we will start with each $r$-band magnitude measurement made by SDSS. This yields 8 features (twice that of the Iris data set, but significantly fewer than the 454 properties measured for each source in SDSS).
Step 1 in the ML workflow is data preparation - we must curate the training set. As a reminder:
A machine-learning model is only as good as its training set.
This point cannot be emphasized enough. Machine-learning models are data-driven, they do not capture any physical theory, and thus it is essential that the training set satisfy several criteria.
Two of the most important criteria for a good training set are:

the training set should be unbiased [this is actually really hard to achieve in astronomy since most surveys are magnitude limited]
the training set should be representative of the (unobserved or field) population of sources [a training set with no stars will yield a model incapable of discovering point sources]

So, step 1 (this is a must), we are going to examine the training set to see if anything suspicious is going on. We will use astroquery to directly access the SDSS database, and store the results in an astropy Table.
Note The SDSS API for astroquery is not standard for the package, which leads to a warning. This is not, however, a problem for our purposes.
Can you write Python code for it?

from astroquery.sdss import SDSS  # enables direct queries to the SDSS database


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Problem 2a
Train a $k$ Nearest Neighbors model with $k = 11$ neighbors on the 10k source training set. Note - for this particular problem, the number of neighbors does not matter much.
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

feats = # complete

bright_X = # complete
bright_y = # complete

KNNclf = # complete

from sklearn.neighbors import KNeighborsClassifier

feats = list(bright_set.columns)
feats.remove('class')

bright_X = np.array(bright_set[feats].to_pandas())
bright_y = np.array(bright_set['class'])

KNNclf = KNeighborsClassifier(n_neighbors = 11)
KNNclf.fit(bright_X, bright_y)


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Problem 5f
Suppose you want a model that only misclassifies 1% of stars as galaxies.
What classification threshold should be adopted for this model?
What fraction of galaxies does this model miss?
Can you think of a reason to adopt such a threshold?
Can you write Python code for it?

# complete

fpr01_idx = (np.abs(fpr-0.01)).argmin()

tpr01 = tpr[fpr01_idx]
threshold01 = thresholds[fpr01_idx]

print("To achieve FPR = 0.01, a decision threshold = {:.4f} must be adopted".format(threshold01))
print("This threshold will miss {:.4f} of galaxies".format(1 - tpr01))


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Solution 2b
Based on the pair plots generated above - stars and galaxies appear highly distinct based on their SDSS $r'$-band measurements, thus, this model likely exhibits poor performance. [we will see if we can confirm this]
Leveraging the same domain knowledge discussed above, namely that galaxies cannot be modeled with a PSF, we can "normalize" the magnitude measurements by taking their difference relative to psfMag_r. This normalization has the added advantage of removing any knowledge of the apparent brightness of the sources, which should help when comparing independent bright and faint sets.
Problem 2c
Normalize the feature vector relative to psfMag_r, and refit the $k$NN model to the 7 newly engineered features.
Does the accuracy improve when predicting the class of sources in the faint test set?
Hint - be sure you apply the eaxct same normalization to both the training and test set
Can you write Python code for it?

bright_Xnorm = # complete

KNNclf = # complete

faint_predsNorm = # complete

print("The normalized features produce an accuracy ~{:.4f}".format( # complete

bright_Xnorm = bright_X[:,0][:,np.newaxis] - bright_X[:,1:]
faint_Xnorm = faint_X[:,0][:,np.newaxis] - faint_X[:,1:]

KNNclf = KNeighborsClassifier(n_neighbors = 11)
KNNclf.fit(bright_Xnorm, bright_y)

faint_predsNorm = KNNclf.predict(faint_Xnorm)

print("The normalized features produce an accuracy ~{:.4f}".format(accuracy_score(faint_y, faint_predsNorm)))


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Solution 2c
Wow! Normalizing the features produces a huge ($\sim{35}\%$) increase in accuracy. Clearly, we should be using normalized magnitude features moving forward.
In addition to demonstrating the importance of feature engineering, this exercise teaches another important lesson: contextual features can be dangerous.
Contextual astronomical features can provide very strong priors: stars are more likely close to the galactic plane, supernovae occur next to/on top of galaxies, bluer stars have have lower metallicity, etc. Thus, including contextual information may improve overall model performance.
However, all astronomical training sets are heavily biased. Thus, the strong priors associated with contextual features can lead to severely biased model predictions.
Generally, I (AAM) remove all contextual features from my ML models for this reason. If you are building ML models, consider contextual information as it may help overall performance, but... be weary.
Worry about the data
Problem 3) Model Building
After the data have been properly curated, the next important choice in the ML workflow is the selection of ML algorithm. With experience, it is possible to develop intuition for the best ML algorithm given a specific problem.
Short of that? Try three (or four, or five) different models and choose whichever works the best.
For the star-galaxy problem, we will use the Random Forest (RF) algorithm (Breiman 2001) as implemented by scikit-learn.
RandomForestClassifier is part of the sklearn.ensemble module.
RF has a number of nice properties for working with astronomical data:

relative insensitivity to noisy or useless features
invariant response to highly non-gaussian feature distributions
fast, flexible and scales well to large data sets

which is why we will adopt it here.
Problem 3a
Build a RF model using the normalized features from the training set.
Include 25 trees in the forest using the n_estimators paramater in RandomForestClassifier.
Can you write Python code for it?

import # complete
rs = 626 # aread code for Pasadena

train_Xnorm = # complete

RFclf = # complete

from sklearn.ensemble import RandomForestClassifier
rs = 626 # aread code for Pasadena

train_Xnorm = train_X[:,0][:,np.newaxis] - train_X[:,1:]

RFclf = RandomForestClassifier(n_estimators = 25, random_state = rs)
RFclf.fit(train_Xnorm, train_y)


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Solution 5c
The optimized model provides a $\sim{0.6}\%$ improvement over the baseline model.
We will now examine the performance of the model using some alternative metrics.
Note - if these metrics are essential for judging the model performance, then they should be incorporated to the workflow in the evaluation stage, prior to examination of the test set.
Problem 5d
Calculate the confusion matrix for the model, as determined by the test set.
Is there symmetry to the misclassifications?
Can you write Python code for it?

from sklearn.metrics import # complete

# complete

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(test_y, test_preds)
print(cm)


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Solution 5d
Adopting galaxies as the positive class, the TPR = 96.7%, while the TNR = 97.1%. Thus, yes, these is ~symmetry to the classifications.
Problem 5e
Calculate and plot the ROC curves for both stars and galaxies.
Hint - you'll need probabilities in order to calculate the ROC curve.
Can you write Python code for it?

from sklearn.metrics import roc_curve

test_preds_proba = # complete
# complete

fpr, tpr, thresholds = roc_curve( # complete
plt.plot( # complete

plt.legend()

from sklearn.metrics import roc_curve, roc_auc_score

test_preds_proba = RFopt_clf.predict_proba(test_Xnorm)
test_y_stars = np.zeros(len(test_y), dtype = int)
test_y_stars[np.where(test_y == "STAR")] = 1
test_y_galaxies = test_y_stars*-1. + 1

fpr, tpr, thresholds = roc_curve(test_y_stars, test_preds_proba[:,1])
plt.plot(fpr, tpr, label = r'$\mathrm{STAR}$', color = "MediumAquaMarine")

fpr, tpr, thresholds = roc_curve(test_y_galaxies, test_preds_proba[:,0])
plt.plot(fpr, tpr, label = r'$\mathrm{GALAXY}$', color = "Tomato")

plt.legend()


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
Solution 5f
When building galaxy 2-point correlation functions it is very important to avoid including stars in the statistics as they will bias the final measurement.
Finally - always remember:
worry about the data
Challenge Problem) Taking the Plunge
Applying the model to field data
QSOs are unresolved sources that look like stars in optical imaging data. We will now download photometric measurements for 10k QSOs from SDSS and see how accurate the RF model performs for these sources.
Can you write Python code for it?

QSO_query = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class = 'QSO'
             ORDER BY s.specobjid ASC
               """
QSO_set = SDSS.query_sql(QSO_query)


I am working on the file "Sessions/Session02/Day5/PracticalMachLearnWorkflowSolutions.ipynb".
The first task is:
While it is possible to look up each of the names of the $r$-band magnitudes in the SDSS PhotoObjAll schema, the schema list is long, and thus difficult to parse by eye. Fortunately, we can identify the desired columns using the database itself:
select COLUMN_NAME
from INFORMATION_SCHEMA.Columns
where table_name = 'PhotoObjAll' AND
COLUMN_NAME like '%Mag/_r' escape '/'

which returns the following list of columns: psfMag_r, fiberMag_r, fiber2Mag_r, petroMag_r, deVMag_r, expMag_r, modelMag_r, cModelMag_r.
We now select these magnitude measurements for 10000 stars and galaxies from SDSS. Additionally, we join these photometric measurements with the SpecObjAll table to obtain their spectroscopic classifications, which will serve as labels for the machine-learning model.
Note - the SDSS database contains duplicate observations, flagged observations, and non-detections, which we condition the query to exclude (as explained further below). We also exclude quasars, as the precise photometric classification of these objects is ambiguous: low-$z$ AGN have resolvable host galaxies, while high-$z$ QSOs are point-sources. Query conditions:

p.mode = 1 select only the primary photometric detection of a source
s.sciencePrimary = 1 select only the primary spectroscopic detection of a source (together with above, prevents duplicates)
p.clean = 1 the SDSS clean flag excludes flagged observations and sources with non-detections
s.class != 'QSO' removes potentially ambiguous QSOs from the training set
Can you write Python code for it?

sdss_query = """SELECT TOP 10000
             p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
             p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
             s.class
             FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
             WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
             ORDER BY p.objid ASC
               """
sdss_set = SDSS.query_sql(sdss_query)
sdss_set


I am working on the file "Sessions/Session03/Day1/ReIntroToDatabasesSolutions.ipynb".
The first task is:
As we round out this brief review of databases, recall that the quality of databases relies on ACID properties:

Atomicity
Consistency
Isolation
Durability

As we round out this brief review of databases, recall that the success of databases relies on ACID properties:

Atomicity - all parts of transaction succeed, or rollback state of database
Consistency
Isolation
Durability

As we round out this brief review of databases, recall that the success of databases relies on ACID properties:

Atomicity - all parts of transaction succeed, or rollback state of database
Consistency - data always meets validation rules
Isolation
Durability

As we round out this brief review of databases, recall that the success of databases relies on ACID properties:

Atomicity - all parts of transaction succeed, or rollback state of database
Consistency - data always meets validation rules
Isolation - no interference across transactions (even if concurrent)
Durability

As we round out this brief review of databases, recall that the success of databases relies on ACID properties:

Atomicity - all parts of transaction succeed, or rollback state of database
Consistency - data always meets validation rules
Isolation - no interference across transactions (even if concurrent)
Durability - a committed transaction remains committed (even if there's a power outage, etc)

Problem 2) Complex Queries with SDSS
Above we looked at the most basic operations possible with a database (recall - databases are unnecessary, and possibly cumbersome, with small data sets). A typical database consists of many tables, and these tables may be joined together to unlock complex questions for the data.
As a reminder on (some of) this functionality, we are now going to go through some problems using the SDSS database. The full SDSS schema explains all of the tables, columns, views and functions for querying the database. We will keep things relatively simple in that regard.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/ReIntroToDatabasesSolutions.ipynb".
The first task is:
At the most basic level - databases store your bytes, and later return those bytes (or a subset of them) when queried.
They provide a highly efficient means for filtering your bytes (there are many different strategies that the user can employ).
The backend for most databases is the Structured Query Language or SQL, which is a standard declarative language.
There are many different libraries that implement SQL: MySQL, PostgreSQL, Greenplum, Microsoft SQL server, IBM DB2, Oracle Database, etc.
Problem 1) Basic SQL Operations with SQLite
The most basic implementation is SQLite a self-contained, SQL database engine. We will discuss SQLite further later in the week, but in brief - it is a nice stand alone package that works really well for small problems (such as the example that we are about to encounter).
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/ReIntroToDatabasesSolutions.ipynb".
The first task is:
Now that we have bytes in the database, we can retrieve those bytes with one (or several) queries. There are 3 basic building blocks to a query:
SELECT...
FROM...
WHERE...

Where SELECT specifies the information we want to retrieve from the database, FROM specifies the tables being queried in the database, and WHERE specifies the conditions for the query.
Problem 1c
Select the institutions for all students in the DSFPstudents table who have been in grad school for more than 2 years.
Hint - to display the results of your query run cur.fetchall().
Can you write Python code for it?

cur.execute( # complete
cur.fetchall()

cur.execute("""select Institution from DSFPstudents where year > 2""")
cur.fetchall()


I am working on the file "Sessions/Session03/Day1/ReIntroToDatabasesSolutions.ipynb".
The first task is:
Problem 2d
This time use a right outer join (or right join).
How do your results compare to the previous query?
Can you write Python code for it?

SDSS.query_sql("""select top 20 objid, cModelMag_u, cModelMag_g, cModelMag_r, cModelMag_i, cModelMag_z,
                         class
                  from photoobjall p
                      right outer join specobjall s on s.bestobjid = p.objid
                """)


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
(The -r is normally unnecessary, but we need it here because we're using the "testing" PyPI)
4c: Build a "source" version of your package
Check out the PyPI page for your package.  You'll see it now has the info from your setup.py but there's no package.  Again, distutils provides a tool to do this automatically - you take the source distribution that was created, and upload it:
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
0d: Change directory to your new directory
One thing about shell commands is that they always start wherever you started your IPython instance.  So doing cd as a shell command only changes things temporarily (i.e. within that shell command).  IPython provides a %cd magic that makes this change last, though. Use this to %cd into the directory you just created, and then use the pwd shell command to ensure this cd "stuck" (You can also try doing cd as a shell command to prove to yourself that it's different from the %cd magic.)
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
1b: Convert the directory into a git repo
Make that code into a git repository by doing git init in the directory you created,  then git add and git commit.
Can you write Python code for it?

%cd # complete

!git init

!git add code.py
!git commit -m #complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
1c: Create a repository for your code in Github
Go to github's web site in your web browser.   If you do not have a github account, you'll need to create one (follow the prompts on the github site).
Once you've got an account, you'll need to make sure your git client can authenticate with github.  If you're using a GUI, you'll have to figure it out (usually it's pretty easy). On the command line you have two options:
* The simplest way is to connect to github using HTTPS.  This requires no initial setup, but git will prompt you for your github username and password every so often.
* If you find that annoying (I do...), you can set up your system to use SSH to talk to github. Look for the "SSH and GPG keys" section of your settings on github's site, or if you're not sure how to work with SSH keys, check out github's help on the subject.
Once you've got github set up to talk to your computer, you'll need to create a new repository for the code you created.  Hit the "+" in the upper-right, create a "new repository" and fill out the appropriate details (don't create a README just yet).
To stay sane, I recommend using the same name for your repository as the local directory name you used... But that is not a requirement, just a recommendation.
Once you've created the repository, connect your local repository to github and push your changes up to github.
Can you write Python code for it?

!git remote add <yourgithubusername> <the url github shows you on the repo web page> #complete

!git push <yourgithubusername> master -u


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
1f: Choose a License
A bet you didn't expect to be reading legalese today... but it turns out this is important.  If you do not explicitly license your code, in most countries (including the US and EU) it is technically illegal for anyone to use your code for any purpose other than just looking at it.
(Un?)Fortunately, there are a lot of possible open source licenses out there.  Assuming you want an open license, the best resources is to use the "Choose a License" website. Have a look over the options there and decide which you think is appropriate for your code.
Once you've chosen a License, grab a copy of the license text, and place it in your repository as a file called LICENSE (or LICENSE.md or the like).  Some licenses might also suggest you place the license text or just a copyright notice in the source code as well, but that's up to you.
Once you've done that, do as we've done before: push all your additions up to github.  If you've done it right, github will automatically figure out your license and show it in the upper-right corner of your repo's github page.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
2c: create a branch for your change
You're going to make some changes to their code, but who knows... maybe they'll spend so long reviewing it that you want to do another.  So it's always best to make changes in a specific "branch" for that change.  So to do this we need to make a github branch.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
2c: modify the code
Make some change to their code repo.  Usually this would be a new feature or a bug fix or documentation clarification or the like... But it's up to you.
Once you've done that, be sure to commit the change locally.
Can you write Python code for it?

!git add <files modified>#complete
!git commit -m ""#complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
3b: Test your package
You should now be able to import your package and the code inside it as though it were some installed package like numpy, astropy, pandas,  etc.
Can you write Python code for it?

from <yourpkgname> import code#complete

#if your code.py has a function called `do_something` as in the example above, you can now run it like:
code.do_something()


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
3c: Apply packaging tricks
One of the nice things about packages is that they let you hide the implementation of some part of your code in one place while exposing a "cleaner" namespace to the users of your package.  To see a (trivial) example, of this, lets pull a function from your code.py into the base namespace of the package.  In the below make the __init__.py have one line: from .code import do_something.  That places the do_something() function into the package's root namespace.
Can you write Python code for it?

%%file <yourpkgname>/__init__.py

#complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
3d: Create a setup.py file
Ok, that's great in a pinch, but what if you want your package to be available from other directories?  If you open a new terminal somewhere else and try to import <yourpkgname> you'll see that it will fail, because Python doesn't know where to find your package.  Fortunately, Python (both the language and the larger ecosystem) provide built-in tools to install packages. These are built around creating a setup.py script that controls installation of a python packages into a shared location on your machine.  Essentially all Python packages are installed this way, even if it happens silently behind-the-scenes.
Below is a template bare-bones setup.py file.  Fill it in with the relevant details for your package.
Can you write Python code for it?

%%file /Users/erik/tmp/lsst-test/setup.py
#!/usr/bin/env python

from distutils.core import setup

setup(name='<yourpkgname>',
      version='0.1dev',
      description='<a description>',
      author='<your name>',
      author_email='<youremail>',
      packages=['<yourpkgname>'],
     ) #complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
4b: Register your package on PyPI
distutils has built-in functionality for interacting with PyPI.  This includes the ability to register your package directly from the command line, automatically filling out the details you provided in your setup.py.
Hint: You'll  want to make sure your package version is something you want to release before executing the register command.  Released versions can't be duplicates of existing versions, and shouldn't end in "dev" or "b" or the like.
Can you write Python code for it?

!python setup.py register  -r https://testpypi.python.org/pypi


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
BUT you will probably get an error here.  That's because Python is smart about imports: once it's imported a package once it won't re-import it later.  Usually that saves time, but here it's a hassle.  Fortunately, we can use the reload function to get around this:
Can you write Python code for it?

from importlib import reload  #not necessary on Py 2.x, where reload() is built-in

reload(<yourpkgname>)#complete
<yourpkgname>.do_something()#complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Code Repositories
Version 0.1
The notebook contains problems oriented around building a basic Python code repository and making it public via Github.  Of course there are other places to put code repositories, with complexity ranging from services comparable to github to simple hosting a git server on your local machine.  But this focuses on git and github as a ready-to-use example with plenty of additional resources to be found online.
Note that these problems assum you are using the Anaconda Python distribution.  This is particular useful for these problems because it makes it very easy to install testing packages in virtual environments quickly and with little wasted disk space. If you are not using anaconda, you can either use an alternative virtual environment scheme (e.g. in Py 3, the built-in venv), or just install pacakges directly into your default python (and hope for the best...).
For git interaction, this notebook also uses the git command line tools directly.  There are a variety of GUI tools that make working with git more visually intuitive (e.g. SourceTree, gitkraken, or the github desktop client), but this notebook uses the command line tools as the lowest common denominator.  You are welcome to try to reproduce the steps with your client, however - feel free to ask your neighbors or instructors if you run into trouble there.
As a final note, this notebook's examples assume you are using a system with a unix-like shell (e.g. macOS, Linux, or Windows with git-bash or the Linux subsystem shell).

E Tollerud (STScI)
Problem 0: Using Jupyter as a shell
As an initial step before diving into code repositories, it's important to understand how you can use Jupyter as a shell.  Most of the steps in this notebook require interaction with the system that's easier done with a shell or editor rather than using Python code in a notebook. While this could be done by opening up a terminal beside this notebook, to keep most of your work in the notebook itself, you can use the capabilities Jupyter + IPython...
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Don't forget to add and commit via git and push up to github...
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Final note: %cd -0 is a convenient shorthand to switch back to the initial directory.
Problem 1: Creating a bare-bones repo and getting it on Github
Here we'll create a simple (public) code repository with a minimal set of content, and publish it in github.
1a: Create a basic repository locally
Start by creating the simplest possible code repository, composed of a single code file.  Create a directory (or use the one from 0c), and place a code.py file in it, with a bit of Python code of your choosing. (Bonus points for witty or sarcastic code...)  You could even use non-Python code if you desired, although Problems 3 & 4 feature Python-specific bits so I wouldn't recommend it.
To make the file from the notebook, the %%file <filename> magic is a convenient way to write the contents of a notebook cell to a file.
Can you write Python code for it?

!mkdir #complete only if you didn't do 0c, or want a different name for your code directory

%%file <yourdirectory>/code.py

def do_something():
    # complete
    print(something)# this will make it much easier in future problems to see that something is actually happening


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Hopefully they are now satisfied and are willing to hit the merge button.
2g: Get the updated version
Now you should get the up-to-date version from the original owner of the repo, because that way you'll have both your changes and any other changes they might have made in the meantime.  To do this you'll need to connect your local copy to your nieghbor's github repo (not your fork).
Can you write Python code for it?

!git remote add <neighbors-username> <url-from-neighbors-github-repo> #complete
!git fetch <neighbors-username> #complete
!git branch --set-upstream-to=<neighbors-username>/master master
!git checkout master
!git pull


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
If for some reason this fails (which does happen for unclear reasons on occasion), you can usually just directly upload the .tar.gz file from the web interface without too much trouble.
4e: Install your package with pip
The pip tool is a convenient way to install packages on PyPI.  Again, we use Anaconda to create a testing environment to make sure everything worked correctly.
(Normally the -i wouldn't be necessary - we're using it here only because we're using the "testing" PyPI)
Can you write Python code for it?

%%sh

conda create -n test_pypi_<yourpkgname> anaconda #complete
source activate test_pypi_<yourpkgname> #complete
pip install -i https://testpypi.python.org/pypi <yourpkgname>

%%sh

cd $HOME
source activate test_pypi_<yourpkgname> #complete
python -c "import <yourpkgname>;<yourpkgname>.do_something()" #complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Now if you look at the local repo, it should include your changes.
Suggestion To stay sane, you might change the "origin" remote to your username.  E.g. git remote rename origin <yourusername>.  To go further, you might even delete your fork's master branch, so that only your neighbor's master exists.  That might save you headaches in the long run if you were to ever access this repo again in the future.
2h: Have them reciprocate
Science (Data or otherwise) and open source code is a social enterprise built on shared effort, mutual respect, and trust.  So ask them to issue a PR aginst your code, too.  The more we can stand on each others' shoulders, the farther we will all see.
Hint: Ask them nicely.  Maybe offer a cookie or something?
Problem 3: Setting up a bare-bones Python Package
Up to this point we've been working on the simplest possible shared code: a single file with all the content.  But for most substantial use cases this isn't going to cut it.  After all, Python was designed around the idea of namespaces that let you hide away or show code to make writing, maintaining, and versioning code much easier.  But to make use of these, we need to deploy the installational tools that Python provides.  This is typically called "packaging".  In this problem we will take the code you just made it and build it into a proper python package that can be installed and then used anywhere.
For more background and detail (and the most up-to-date recommendations) see the Python Packaging Guide.
3a: Set up a Python package structure for your code
First we adjust the structure of your code from Problem 1 to allow it to live in a package structure rather than as a stand-alone .py file.  All you need to do is create a directory, move the code.py file into that directory, and add a file (can be empty) called __init__.py into the directory.
You'll have to pick a name for the package, which is usually the same as the repo name (although that's not strictly required).
Hint: don't forget to switch back to your code repo...
Can you write Python code for it?

!mkdir <yourpkgname>#complete
!git mv code.py <yourpkgname>#complete

#The "touch" unix command simply creates an empty file if there isn't one already.
#You could also use an editor to create an empty file if you prefer.

!touch <yourpkgname>/__init__.py#complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Now we can try running the package from anywhere (not just the source code directory), as long as we're in the same environment that we installed the package in.
Can you write Python code for it?

%%sh

cd $HOME
source activate test_<yourpkgname> #complete
python -c "import <yourpkgname>;<yourpkgname>.do_something()" #complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Problem 2: Collaborating with others' repos
There's not much point in having open source code if no one else can look at it or use it.  So now we'll have you try modify your neighbors' project using github's Pull Request feature.
2a: Get (git?) your neighbor's code repo
Find someone sitting near you who has gotten through Problem 1.  Ask them their github user name and the name of their repository.
Once you've got the name of their repo, navigate to it on github.  The URL pattern is always "https://www.github.com/theirusername/reponame". Use the github interface to "fork" that repo, yielding a "yourusername/reponame" repository.  Go to that one, take note of the URL needed to clone it (you'll need to grab it from the repo web page, either in "HTTPS" or "SSH" form, depending on your choice in 1a).  Then clone that onto your local machine.
Can you write Python code for it?

# Don't forget to do this cd or something like it... otherwise you'll clone *inside* your repo
%cd -0

!git clone <url from github>#complete
%cd <reponame>#complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Problem 4: Publishing your package on (fake) PyPI
Now that your package can be installed by anyone who comes across it on github.  But it tends to scare some people that they need to download the source code and know git to use your code.  The Python Package Index (PyPI), combined with the pip tool (now standard in Python) provides a much simpler way to distribute code.  Here we will publish your code to a testing version of PyPI.
4a: Create a PyPI account
First you'll need an account on PyPI to register new packages.  Go to the testing PyPI, and register. You'll also need to supply your login details in the .pypirc directory in your home directory as shown below. (If it were the real PyPI you'd want to be more secure and not have your password in plain text.  But for the testing server that's not really an issue.)
Can you write Python code for it?

%%file ~/.pypirc

[distutils]
index-servers=
    testpypi

[testpypi]
repository = https://testpypi.python.org/pypi
username = <your user name goes here>
password = <your password goes here>


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
To test that it built sucessfully, the easiest thing to do is cd into the build/lib.X-Y-Z directory ("X-Y-Z" here is OS and machine-specific).  Then you should be able to import <yourpkgname>.  It's usually best to do this as a completely independent process in python.  That way you can be sure you aren't accidentally using an old import as we saw above.
Can you write Python code for it?

%%sh

cd build/lib.X-Y-Z #complete
python -c "import <yourpkgname>;<yourpkgname>.do_something()" #complete


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
Verify that there is a <yourpkg>-<version>.tar.gz file in the dist directory.  It should have all of the source code necessary for your package.
4d: Upload your package to PyPI
Check out the PyPI page for your package.  You'll see it now has the info from your setup.py but there's no package.  Again, distutils provides a tool to do this automatically - you take the source distribution that was created, and upload it:
Can you write Python code for it?

!python setup.py sdist upload  -r https://testpypi.python.org/pypi


I am working on the file "Sessions/Session03/Day1/SoftwareRepositories.ipynb".
The first task is:
and push it up (to a branch on your github fork).
Can you write Python code for it?

!git push origin <name-of-branch>#complete


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
1b: Ensure your repo has code suitable for unit tests
Depending on what your code actually does, you might need to modify it to actually perform something testable.  For example, if all it does is print something, you might find it difficult to write an effective unit test.  Try adding a function that actually performs some operation and returns something different depending on various inputs.  That tends to be the easiest function to unit-test: one with a clear "right" answer in certain situations.
Also be sure you have cded to the root of the repo for pytest to operate correctly.
1c: Add a test file with a test function
The test must be part of the package and follow the convention that the file and the function begin with test to get picked up by the test collection machinery.  Inside the test function, you'll need some code that fails if the test condition fails.  The easiest way to do this is with an assert statement, which raises an error if its first argument is False.
Hint: remember that to be a valid python package, a directory must have an __init__.py
Can you write Python code for it?

!mkdir #complete
!touch #complete

%%file <yourpackage>/tests/test_something.py

def test_something_func():
    assert #complete


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
1d: Run the test directly
While this is not how you'd ordinarily run the tests, it's instructive to first try to execute the test directly, without using any fancy test framework.  If your test function just runs, all is good.  If you get an exception, the test failed (which in this case might be good).
Hint: you may need to use reload or just re-start your notebook kernel to get the cell below to recognize the changes.
Can you write Python code for it?

from <yourpackage>.tests import test_something

test_something.test_something_func()


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
1e: Run the tests with py.test
Once you have an example test, you can try invoking py.test, which is how you should run the tests in the future.  This should yield a report that shows a dot for each test. If all you see are dots, the tests ran sucessfully.  But if there's a failure, you'll see the error, and the traceback showing where the error happened.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
1f: Make the test fail (or succeed...)
If your test failed when you ran it, you should now try to fix the test (or the code...) to make it work.  Try running
(Modify your test to fail if it succeeded before, or vice versa)
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
1g: Check coverage
The coverage plugin we installed will let you check which lines of your code are actually run by the testing suite.
Can you write Python code for it?

!py.test --cov=<yourproject> tests/ #complete


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
2b
This test has an intentional bug... but depending how you right the test you might not catch it...  Use unit tests to find it! (and then fix it...)
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

def function_b(value):
    if value < 0:
        return value - 1
    else:
        value2 = subfunction_b(value + 1)
        return value + value2

def subfunction_b(inp):
    vals_to_accum = []
    for i in range(10):
        vals_to_accum.append(inp ** (i/10))
    if vals_to_accum[-1] > 2:
        vals.append(100)
    # really you would use numpy to do this kind of number-crunching... but we're doing this for the sake of example right now
    return sum(vals_to_accum)


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
3b: Set up an account on travis
This turns out to be quite convenient.  If you go to the Travis web site, you'll see a "Sign in with GitHub" button.  You'll need to authorize Travis, but once you've done so it will automatically log you in and know which repositories are yours.
3c: Create a minimal .travis.yml file.
Before we can activate travis on our repo, we need to tell travis a variety of metadata about what's in the repository and how to run it.  The template below should be sufficient for the simplest needs.
Can you write Python code for it?

%%file .travis.yml

language: python
python:
  - "3.6"
# command to install dependencies
#install: "pip install numpy"  #uncomment this if your code depends on numpy or similar
# command to run tests
script: pytest


I am working on the file "Sessions/Session03/Day1/TestingandCI.ipynb".
The first task is:
Code Testing and CI
Version 0.1
The notebook contains problems about code testing and continuous integration.

E Tollerud (STScI)
Problem 1: Set up py.test in you repo
In this problem we'll aim to get the py.test testing framework up and running in the code repository you set up in the last set of problems.  We can then use it to collect and run tests of the code.
1a: Ensure py.test is installed
Of course py.test must actually be installed before you can use it. The commands below should work for the Anaconda Python Distribution, but if you have some other Python installation you'll want to install pytest (and its coverage plugin) as directed in the install instructions for py.test.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
1d: Run the test directly
While this is not how you'd ordinarily run the tests, it's instructive to first try to execute the test directly, without using any fancy test framework.  If your test function just runs, all is good.  If you get an exception, the test failed (which in this case might be good).
Hint: you may need to use reload or just re-start your notebook kernel to get the cell below to recognize the changes.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
2b
This test has an intentional bug... but depending how you right the test you might not catch it...  Use unit tests to find it! (and then fix it...)
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
2c
There are (at least) two significant bugs in this code (one fairly apparent, one much more subtle).  Try to catch them both, and write a regression test that covers those cases once you've found them.
One note about this function: in real code you're probably better off just using the Angle object from astropy.coordinates.  But this example demonstrates one of the reasons why that was created, as it's very easy to write a buggy version of this code.
Hint: you might find it useful to use astropy.coordinates.Angle to create test cases...
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

import math

# know that to not have to worry about this, you should just use `astropy.coordinates`.
def angle_to_sexigesimal(angle_in_degrees, decimals=3):
    """
    Convert the given angle to a sexigesimal string of hours of RA.

    Parameters
    ----------
    angle_in_degrees : float
        A scalar angle, expressed in degrees

    Returns
    -------
    hms_str : str
        The sexigesimal string giving the hours, minutes, and seconds of RA for the given `angle_in_degrees`

    """
    if math.floor(decimals) != decimals:
        raise ValueError('decimals should be an integer!')

    hours_num = angle_in_degrees*24/180
    hours = math.floor(hours_num)

    min_num = (hours_num - hours)*60
    minutes = math.floor(min_num)

    seconds = (min_num - minutes)*60

    format_string = '{}:{}:{:.' + str(decimals) + 'f}'
    return format_string.format(hours, minutes, seconds)


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
2d
Hint: numpy has some useful functions in numpy.testing for comparing arrays.
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

import numpy as np

def function_d(array1=np.arange(10)*2, array2=np.arange(10), operation='-'):
    """
    Makes a matrix where the [i,j]th element is array1[i] <operation> array2[j]
    """
    if operation == '+':
        return array1[:, np.newaxis] + array2
    elif operation == '-':
        return array1[:, np.newaxis] - array2
    elif operation == '*':
        return array1[:, np.newaxis] * array2
    elif operation == '/':
        return array1[:, np.newaxis] / array2
    else:
        raise ValueError('Unrecognized operation "{}"'.format(operation))


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
3b: Set up an account on travis
This turns out to be quite convenient.  If you go to the Travis web site, you'll see a "Sign in with GitHub" button.  You'll need to authorize Travis, but once you've done so it will automatically log you in and know which repositories are yours.
3c: Create a minimal .travis.yml file.
Before we can activate travis on our repo, we need to tell travis a variety of metadata about what's in the repository and how to run it.  The template below should be sufficient for the simplest needs.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
Solution (one of many...)
Can you write Python code for it?

def test_decimals():
    assert angle_to_sexigesimal(0) == '0:0:0.000'
    assert angle_to_sexigesimal(0, decimals=5) == '0:0:0.00000'

def test_qtrs():
    assert angle_to_sexigesimal(90, decimals=0) == '6:0:0'
    assert angle_to_sexigesimal(180, decimals=0) == '12:0:0'
    assert angle_to_sexigesimal(270, decimals=0) == '18:0:0'
    assert angle_to_sexigesimal(360, decimals=0) == '24:0:0'
    # this reveals the major bug that the 180 at the top should be 360

def test_350():
    assert angle_to_sexigesimal(350, decimals=0) == '23:20:00'
    # this fails, revealing that sometimes the values round

def test_neg():
    assert angle_to_sexigesimal(-7.5, decimals=0) == '-0:30:0'
    assert angle_to_sexigesimal(-20, decimals=0) == angle_to_sexigesimal(340, decimals=0)
    # these fail, revealing a "debatable" bug: that negative degrees give negative RAs that are
    # nonsense.  You could always tell users not to give negative values... but users, particularly
    # future you, probably won't listen.

def test_neg_decimals():
    import pytest

    with pytest.raises(ValueError):
        angle_to_sexigesimal(10, decimals=-2)


def test_minus():
    array1 = np.arange(10)*2
    array2 = np.arange(10)

    func_mat =  function_d(array1, array2, operation='-')

    for i, val1 in enumerate(array1):
        for j, val2 in enumerate(array2):
            assert func_mat[i, j] == val1 - val2

def test_plus():
    array1 = np.arange(10)*2
    array2 = np.arange(10)

    func_mat =  function_d(array1, array2, operation='+')

    for i, val1 in enumerate(array1):
        for j, val2 in enumerate(array2):
            assert func_mat[i, j] == val1 + val2

def test_times():
    array1 = np.arange(10)*2
    array2 = np.arange(10)

    func_mat =  function_d(array1, array2, operation='*')

    for i, val1 in enumerate(array1):
        for j, val2 in enumerate(array2):
            assert func_mat[i, j] == val1 * val2

def test_div():
    array1 = np.arange(10)*2
    array2 = np.arange(10)

    func_mat =  function_d(array1, array2, operation='/')

    for i, val1 in enumerate(array1):
        for j, val2 in enumerate(array2):
            assert func_mat[i, j] == val1 / val2
    #GOTCHA! This doesn't work because of floating point differences between numpy and python scalars
    # This is where that numpy stuff is handy - see the next function

def test_div_npt():
    from numpy import testing as npt

    array1 = np.arange(10)*2
    array2 = np.arange(10)

    func_mat =  function_d(array1, array2, operation='/')

    test_mat = np.empty(((len(array1), len(array2))))
    for i, val1 in enumerate(array1):
        for j, val2 in enumerate(array2):
            test_mat[i, j] =  val1 / val2

    npt.assert_array_almost_equal(func_mat, test_mat)


I am working on the file "Sessions/Session03/Day1/TestingandCI_solutions.ipynb".
The first task is:
This should yield a report, which you can use to decide if you need to add more tests to acheive complete coverage.  Check out the command line arguments to see if you can get a more detailed line-by-line report.
Problem 2: Implement some unit tests
The sub-problems below each contain different unit testing complications.  Place the code from the snippets in your repository (either using an editor or the %%file trick), and write tests to ensure the correctness of the functions.  Try to achieve 100% coverage for all of them (especially to catch some hidden bugs!).
Also, note that some of these examples are not really practical - that is, you wouldn't want to do this in real code because there's better ways to do it.  But because of that, they are good examples of where something can go subtly wrong... and therefore where you want to make tests!
2a
When you have a function with a default, it's wise to test both the with-default call (function_b()), and when you give a value (function_b(1.2))
Hint: Beware of numbers that come close to 0... write your tests to accomodate floating-point errors!
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

# `math` here is for *scalar* math... normally you'd use numpy but this makes it a bit simpler to debug

import math

inf = float('inf')  # this is a quick-and-easy way to get the "infinity" value

def function_a(angle=180):
    anglerad = math.radians(angle)
    return math.sin(anglerad/2)/math.sin(anglerad)


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Data Management Part 1
For April 25, 2017

Yusra AlSayyad (Princeton University)
This excerise demonstrates the power of spatial indexes in Databases and non-database SQL implementations.
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.spatial import Delaunay
# from matplotlib.collections import LineCollection
import os
import copy

%matplotlib inline


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Now use groupby again to make a histogram of sources in integer bins of psfMag for only the primary sources. Plot the log number counts vs. psfMag you get. What's the approximate depth of this dataset in mags?
Can you write Python code for it?

grouped = df[# COMPLETE
counts = grouped.deepSourceId.# COMPLETE
plt.plot(counts.index, # COMPLETE

plt.xlabel("PsfMag")
plt.ylabel("log10 Counts")


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Now we want to find the boundary of the whole structure.  How do we know if an edge is an outer edge or inner edge? Outer edges  will only appear once in the bag. MOST of the edges were part of 2 triangles and will appear twice.
We don't care about direction. So edge 4 -> 8 is the same as 8 -> 4
Find all the edges that only appear ONCE in this bag of edges. There are many ways to do this. In the spirit of SQL, this sounds like a: ... GROUP BY HAVING COUNT(*) = 1.  Pandas will do that for you.
Can you write Python code for it?

dfEdges = pd.DataFrame(edges)

# COMPLETE

singleEdges = #

# Shapely can turn a bag of edges into a polygon.
# If you haven't installed shapely, do that now.

from shapely.geometry import MultiLineString
lines = MultiLineString([((line[0][0], line[0][1]),
                          (line[1][0], line[1,1])) for line in allpoints[singleEdges]])
polygonList = list(polygonize(lines))
print("The polygon's area is: %s" % (polygonList[0].area))


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Problem 2a Projection and Selection
Recall that in SQL, we can SELECT particular columns and particular rows.
In pandas there are different interfaces for this: http://pandas.pydata.org/pandas-docs/stable/indexing.html#
This includes some syntax thats intuitive for numpy. For example, you can select columns like:
df['psfMag']
df[['psfMag', 'psfMagSigma']] # or even
df.psfMag
and rows like:
df[df.psfMag < 18]
Make a scatter plot of ra and dec with two separate colors for detect_is_primary 0 or 1.
Note: The photometry pipeline deblended all multipeak detections (parents) into its component peaks (children). If a source was deblended, it was marked detect_is_primary = 0. The children flux measurements were entered into the table, detect_is_primary marked 1, and the deepSourceId of their parent put in the parentDeepSourceId column. All deepSourceIds in the parentDeepSourceId column are also in deepSourceId column (but with detect_is_primary=0).
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Problem 2b Selection and aggregation
How many "primary" sources (that is rows  with detect_is_primary = 1) have a psfMag between 19 and 20?
Can you do this in one line?
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Problem 2c Group By
Pandas also provides grouping: http://pandas.pydata.org/pandas-docs/stable/groupby.html
Count the number of primary vs. non-primary sources using groupby.
Can you write Python code for it?

grouped = df.groupby(#COMPLETE
grouped.#COMPLETE


I am working on the file "Sessions/Session03/Day2/Databases.ipynb".
The first task is:
Problem 2d Join
Backstory: analysis has revealed that the photometry is bad for any children of parent sources that are brighter than 18th magnitude.  Too much scattered light from the background. We want to remove all sources that have a parent brighter than 18th mag.
Fortunately, pandas provides joins. Pandas DataFrames have two methods merge and join which both implement joins. There is also a static pd.merge().  Use any of these to join the DataFrame df to itself, in order to add the psfMag of the parent to the row. Call it "parent_psfMag". The result should be a new DataFrame dfJoined.
What type of Join? We want the resulting table to have the same number of rows as df regardlesss if a source has a parent or not.
HINT: In pandas, deepSourceId isn't a column, but an "index." As a result, the arguments to merge or join are not as symmetric as writing SQL. If you use pd.merge, user right_index=True.
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
1b: Access the docstring from within Python
It turns out that docstrings are more than just strings sitting un-used inside a function: they are associated with and carried around as a part of the function.  This means you can access them in useful ways even if you don't have the code right in front of you.  Explore some of the ways available to look at your function's docstring. Try the various ways below and consider which seems most useful in various contexts.
Can you write Python code for it?

do_something.__doc__

help(do_something)

do_something?

# this one does more than just show the docstring, but is useful to know nonetheless
do_something??


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
1c: Share your docstring
Using either the function you just wrote or a new one (if you've already shown the function to your neighbor), try to communicate the essentials of your function enough for your neighbor to use the function.  That is, have your neighbor run your function (and do something with the output) using only the information in the docstring.
No peeking at the code!  One way to ensure this is to put the function in your github repo and have your neighbor pull down the updates and import the code directly.  Or you can just type your function higher in the notebook and scroll down without your neighbor looking.
Rinse and repeat for you looking at your neighbor's code.
Can you write Python code for it?

from your_neighbors_package import your_neighbors_code # complete

your_neighbors_code? # complete

... = your_neighbors_code(...)
...  # complete


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
1e: Write a docstring for your modules (and package)
Add a docstring to the module and packages in your repository.  This is usually just free-form text (not as much structure as a function or class), although you might include some structure like section headings or bullet-pointed lists.  Once you've done that (and reloaded or restarted the kernel), the commands below should pop up your documentation.
Hint: remember that a package's __init__.py file acts sort of like the "package.py" file for the package
Can you write Python code for it?

import <mypackage>  #complete

<mypackage>?  #complete

from <mypackage> import <mymodule>  #complete

<mymodule>?  #complete


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
2b: Create a directory for the docs
It's good practice to keep the narrative documentation (i.e., the non-docstring part) and the code in separate places.  To do that you'll need to create a new directory for the docs.
Can you write Python code for it?

%cd <yourpackage> #complete

!mkdir docs
%cd docs


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
2c: Set up the standard sphinx documentation layout
Sphinx has a standard layout of files that it uses, and even provides a tool for doing this called sphinx-quickstart.  Use that command to create a sphinx repo.
The invocation of sphinx-quickstart below just gives you the defaults for everything without prompting. If you want to see all the options, you can run this tool in a terminal inside your docs directory - by default it prompts you for lots of information, though, and we can't respond to those in the notebook.  If you do that, be sure to answer "yes" to the question about "autodoc" (more on that in 2g).
Can you write Python code for it?

!ls
# should be empty...

!sphinx-quickstart -a "<yourname>" -p <yourpackagename> -v <version> --ext-autodoc -q #complete

!ls


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
2f: Commit the doc files and push them up to github
If you haven't been doing so, now's a good time to add all the new content of the docs directory to git and push it up to github.
Important gotcha to watch out for: if you just git add docs, you'll probably get both your docs and the stuff in the _build directory.  In general you never want to include generated files in your github repository, because it confuses users (and is nearly-impossible to keep up-to-date, anyway).  To keep yourself from getting confused, you can create a .gitignore file that is aware of all the generated-file directories.  That prevents them from getting added by git accidentally.  An example is shown below that should be appropriate for your repo if you've followed the other notebooks.
Can you write Python code for it?

%cd ..  #or whatever you need to do to get back to the base of your repository

%%file .gitignore

docs/_build/*
build
dist

!git add .gitignore docs
!git commit -m #complete


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
4b: Render a docstring in sphinx
Another important feature of Sphinx is the ability to include docstrings in the documentation.  The example below shows some of that functionality.  Add a file like that, add it to your table of contents (just like for second_doc), and then re-build and examine your docs.
Hint: you need to have your package accessible from python for this to work.  So if you haven't done a python setup.py install on your package yet, you'll need to do that.  If this bothers you, the challenge problem shows how to set up machinery that does not require installing the package to generate the docs.
Can you write Python code for it?

%%file api_docs.rst

API Documentation
=================

This package has two modules, detailed below.

Also, after doing this, you can mention some of the functions from *anywhere* in the docs by doing :func:`<yourpackage>.<module>.<function_you_want_documented>`.

<yourpackagename>
-----------------

.. automodule:: <yourpackage>

<your modulename>
-----------------

.. automodule:: <yourpackage>.<module>

.. autofunction:: <yourpackage>.<module>.<function_you_want_documented>
.. autofunction:: <yourpackage>.<module>.<function_you_want_documented2>


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
Problem 2: Building your Docs with Sphinx
Intro
2a: Make sure Sphinx is installed
You may have sphinx installed already, but if not, you'll want to install it.  The invocation below is appropriate for the Anaconda Python Distribution (but change it from conda to pip to install from pip).
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day2/Documentation.ipynb".
The first task is:
You may not need multiple automodule calls if uo on how you structured things.  Also you might be able to avoid the autofunction directives depending on how you laid things out, by adding :members: undert he automodule directive (properly indented.  The above explicit approach is what Sphinx recommends, but see the challenge problem for a tool that makes this all quite a bit simpler.
4c: Make a class inheritance diagram
As an example of some of the extensions that Sphinx provides, there's a neat tool to generate class "inheritance diagrams" - basically these are diagrams that show which classes are subclass of what other classes.  For a complex example of a diagram that justifies this feature, see here.
For this problem you should add the set of (do-nothing) classes shown below.  Then, you'll need to add the string 'sphinx.ext.inheritance_diagram' to the extensions list in your conf.py.  Then somewhere in your .rst file, add .. inheritance-diagram:: yourmodule.class_heirarchy.  Then once you build it with sphinx, you should see the diagram at that point in the docs.
Can you write Python code for it?

%%file <yourpackage>/class_heirarchy.py #complete

class A():  # this needs to be "A(object)" in py 2.x
    pass

class B(A):
    pass

class C(A):
    pass

class D(B,C):
    pass


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 1) Create a galaxy class
We will begin our experience with object-oriented code by creating a class to represent a galaxy.  In order to do this, we must define the class, give it some default methods, and then define some attributes to describe each instance of a galaxy.
Problem 1a
What do you think should be the attributes and characteristics that should be tracked for a galaxy object?  Think critically and make a list of at least six different quantities that would make sense if this were a research project.  Please do this prior to starting (or looking at) the example below. Discuss with your code partner.
Provide your answer to Problem 1a here
Problem 1b
Define a galaxy class, and give it an initializer class method.  Have your initializer accept arguments for total_mass, cold_gas_mass, and stellar_mass.  Have it also accept keyword arguments for age with a default set to 0.  Maybe also include a SFR and color in your attributes, but initialize it to 0 and 'red' respectively.  I'll include docstrings for you to help guide you along the way.
Hint: Don't forget to use self to indicate an object of this class.
Can you write Python code for it?

class Galaxy():
    """
    Galaxy class for simply representing a galaxy.
    """
    def __init__( #complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 1d
So Python probably didn't do a very good internal representation of your galaxy.  Remember, classes have a few built-in class methods to be defined.  One is the __init__ method, so that we know how to initialize the class.  Another very useful class method is the __repr__, which tells us how to represent a class object when we try to print it out.  It's very useful for debugging.  Please copy your class definition from above, and add the __repr__ method to return a string somehow identifying a galaxy based on its attributes.  Now recreate your favorite galaxy, and print it out again and see if it is a bit more readable than before.
Can you write Python code for it?

class Galaxy():
    """
    Galaxy class for simply representing a galaxy.
    """
    def __init__(#complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 2) Make a more interesting galaxy class that can evolve with time
Now let's build on our basic galaxy class with some relevant class methods specific to galaxies that enables them to change with time.  Obviously, galaxies have a huge number of traits, and we don't currently understand all the ways in which these variables interrelate.  But we can make a toy model here just to get the idea of how they might evolve.  This is largely the premise of semi-analytic models that people use to study how galaxies might evolve with time.
Problem 2a
Think about what methods you would add to allow your galaxy to change with time using the galaxy attributes defined above and/or other galaxy attributes.  Come up with at least three and then discuss with your partner before continuing.
Provide your answer to Problem 2a here
Problem 2b
We want to enhance our general Galaxy class to enable galaxies to actually evolve with time.  Rather than just copying all of the code from our Galaxy class above, we can subclass the Galaxy class to inherit all of its attributes, methods, etc!  Create a new class named EvolvingGalaxy below, which uses the Galaxy class as its "parent" or "super" class.  Remember, because you already defined __init__ and __repr__ methods for this class, they are inherited and you don't need to redefine them, unless you want to.
Now try creating an instance of EvolvingGalaxy and printing it out, just like you did for your favorite galaxy in problem 1c.
Can you write Python code for it?

class EvolvingGalaxy( #complete
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """

#complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 2c
Let's start by making an evolve method for our Galaxy class in order to evolve the galaxy forward in time.  evolve should expect an argument for time in years.  And at least initially, let's just make this a simple method that just changes the age of the galaxy by the appropriate amount.  We'll add more on to it in a second.
Because we're going to be following the evolution of this galaxy, we should create a current_state method, which returns a tuple of the attributes of the EvolvingGalaxy object at that moment.  Now as our EvolvingGalaxy changes with time, we will be able to track it and save that information somewhere.
Can you write Python code for it?

class EvolvingGalaxy( #complete
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """
    #complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 2d
Now that we can evolve our EvolvingGalaxy and check on its current_state over time, let's run it forward in time and see the results!
Make a new function (not a class method) called integrate_time that accepts an EvolvingGalaxy object, a timestep (some number of years long), and a number of timesteps over which to evolve the galaxy.  Step the EvolvingGalaxy forward in time using your for loop, one timestep at a time, and store its current_state at each timestep.  You can store this current_state information however you'd like, but I find it easiest to create a NumPy array of size [5, n_timesteps], and fill it in using the for loop.
Similarly, make a new function called plot_galaxy_evolution that accepts the output from your integrate_time function and uses Matplotlib's pyplot module to plot up the evolution of the EvolvingGalaxy's properties versus its age.
Finally, create an instance of EvolvingGalaxy, and plug it into your new integrate_time and plot_galaxy_evolution functions to integrate for a gigayear, and plot how the EvolvingGalaxy quantities change with age.  Not much going on--sad!
Can you write Python code for it?

def integrate_time( #complete
    """
    Integrate the time forward for a galaxy and record its state at each timestep; return as array
    """
    #complete

def plot_galaxy_evolution( #complete
    """
    Plot the evolution of a galaxy from its input data array
    """
    #complete

#complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 2e
How do galaxies evolve with time?  They do in lots of ways, but let's make a few specific rules.
Galaxies tend to accrete gas from the intergalactic medium over time.  We'll modify the code from above for your EvolvingGalaxy class to include a accrete_gas_from_IGM method, which adds to the cold_gas_mass and total_mass of the galaxy with time.  Let's give it a random component too.
Galaxies form stars, converting cold_gas_mass to stellar_mass according to their star formation rate (SFR).  But there must be sufficient cold_gas_mass to consume, so make sure there is enough for the SFR to operate for the alloted time, otherwise the SFR should drop to 0.
Finally, we haven't actually calculated a SFR, so let's just create a method, calculate_star_formation_rate, to choose a random value every time it is called.  We'll also scale it in some way by the total_mass of the galaxy and use the Milky Way as a guide.  The Milky Way forms about 1 Msun/year and it has a total_mass of 1e12 Msun.
Note: You can use the random.random() function to get a random number between 0 and 1.
Include calls to these methods in our evolve method, and re-generate a galaxy evolution plot like you did in Problem 2d.  Since there is a random component, you can try re-running it a few times and see how the behavior changes.
Can you write Python code for it?

class EvolvingGalaxy(Galaxy):
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """
    def current_state( #complete
        """
        Return a tuple of the galaxy's total_mass, cold_gas_mass, stellar_mass, age, and SFR
        """
        #complete

    def calculate_star_formation_rate( #complete
        """
        Calculate the star formation rate by taking a random number between 0 and 1
        normalized by the galaxy total mass / 1e12;

        Also updates the galaxy's color to blue if SFR > 0.01, otherwise color = red
        """
        #complete

    def accrete_gas_from_IGM( #complete
        """
        Allow the galaxy to accrete cold gas from the IGM at a variable rate normalized to
        the galaxy's mass
        """
        #complete

    def form_stars( #complete
        """
        Form stars according to the current star formation rate and time available
        If unable cold gas, then shut off star formation
        """
        #complete

    def evolve( #complete
        """
        Evolve this galaxy forward for a period time
        """
        #complete

#complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 2f
Based on your knowledge of how galaxies evolve, do these results make sense?  Discuss with your coding partner if so/not, and ways in which you might tweak it to work better.
Provide your answer to Problem 2f here
Problem 3) Make a galaxy class that can interact with other galaxies (challenging!)
Most galaxies do not evolve in isolation, so let's make a galaxy class that enables galaxies to actually interact with each other.  We can re-use much of the code from our previous examples through inheritance!  Yeah!
Problem 3a
Let us create a new subclass of EvolveGalaxy, so that it inherits most of the traits of our previous work.  But we will add some new features including position and velocity values in the x-y plane, so that our galaxy can actually move over time.
Create a new class MovingGalaxy that is a subclass of EvolveGalaxy.  Make a new __init__ method with all of the arguments from it's parent class EvolveGalaxy, but add the additional arguments of x_position, y_position, x_velocity, y_velocity, and id_num.
Note: all arguments must come before any keyword arguments, so the age kwarg has to come at the very end.
Also, make a new __repr__ method for this class that just includes it's idnum, x_position, and y_position.
To assure your code works, create an instance of a MovingGalaxy, and print it out.
Can you write Python code for it?

class MovingGalaxy(EvolvingGalaxy):
    """
    Galaxy class that can evolve and move in the x,y plane
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, #complete

        # Replace self with super to activate the superclass's methods
        super().__init__(total_mass, cold_gas_mass, stellar_mass)

        #complete

    def __repr__(self):
        #complete

#complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 3b
In order for our MovingGalaxy to move, we have to give it a few more methods.  Copy your class definition from above, and add three more methods.
Create a move method which accepts a time argument and updates the MovingGalaxy's x_position, and y_position by accounting for the motion from its x_velocity and y_velocity.
Create a calculate_momentum method that returns a tuple of (x_momentum, y_momentum) for a galaxy.
Finally, create an evolve method, which accepts a time argument, and just executes your move method before calling the superclass's evolve method (i.e. EvolvingGalaxy's).
Can you write Python code for it?

class MovingGalaxy(EvolvingGalaxy):
    """
    This galaxy can move over time in the x,y plane
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, x_position, y_position, x_velocity, y_velocity, idnum, age=0):

        # Replace self with super to activate the superclass's methods
        super().__init__(total_mass, cold_gas_mass, stellar_mass)

        self.x_position = x_position
        self.y_position = y_position
        self.x_velocity = x_velocity
        self.y_velocity = y_velocity
        self.idnum = idnum

    def __repr__(self):
        return "Galaxy %i (x = %.0f; y = %.0f)" % (self.idnum, self.x_position, self.y_position)

    def move( #complete

    def calculate_momentum( #complete

    def evolve( #complete


I am working on the file "Sessions/Session03/Day2/OOP.ipynb".
The first task is:
Problem 3d
Initialize a Universe with several galaxies in it. Run your Universe forward in time, and plot the resulting state of the Universe over time to files called frame%04i.png.  But be careful to not use too many galaxies or to evolve the Universe forward, as it may be really expensive for your computer.  Start small and build from there!
Can you write Python code for it?


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 1) Create a galaxy class
We will begin our experience with object-oriented code by creating a class to represent a galaxy.  In order to do this, we must define the class, give it some default methods, and then some attributes to describe each instance of a galaxy.
Problem 1a
What do you think should be the attributes and characteristics that should be tracked for a galaxy object?  Think critically and make a list of at least six different quantities that would make sense if this were a research project.  Please do this prior to starting (or looking at) the example below. Discuss with your code partner.
Possible Answers: Mass, Luminosity, SFR, Metallicity, Color, Age, Central Black Hole Mass, Disk?, Type (Sab, E0, etc.), Spin
Problem 1b
Define a galaxy class, and give it an initializer class method.  Have your initializer accept arguments for total_mass, cold_gas_mass, and stellar_mass.  Have it also accept keyword arguments for age with a default set to 0.  Maybe also include a SFR and color in your attributes, but initialize it to 0 and 'red' respectively.  I'll include docstrings for you to help guide you along the way.
Hint: Don't forget to use self to indicate an object of this class.
Can you write Python code for it?

class Galaxy():
    """
    Galaxy class for simply representing a galaxy.
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, age=0):
        self.total_mass = total_mass
        self.cold_gas_mass = cold_gas_mass
        self.stellar_mass = stellar_mass
        self.age = age
        self.SFR = 0
        self.color = 'red'


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 1c
Try creating a galaxy instance of your galaxy class!  Create a galaxy object, named after your favorite galaxy, with appropriate values for your arguments and keyword arguments (or as best as you can approximate).  There isn't much you can do with it right now, but at least you can be assured that your existing code works.  Try printing your object to see what python thinks of it.
Can you write Python code for it?

milky_way = Galaxy(1e12, 1e8, 1e10, age=5e9)
print(milky_way)


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 1d
So Python probably didn't do a very good internal representation of your galaxy.  Remember, classes have a few built-in class methods to be defined.  One is the __init__ method, so that we know how to initialize the class.  Another very useful class method is the __repr__, which tells us how to represent a class object when we try to print it out.  It's very useful for debugging.  Please copy your class definition from above, and add the __repr__ method to return a string somehow identifying a galaxy based on its attributes.  Now recreate your favorite galaxy, and print ut out again and see if it is a bit more readable than before.
Can you write Python code for it?

class Galaxy():
    """
    Galaxy class for simply representing a galaxy.
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, age=0):
        self.total_mass = total_mass
        self.cold_gas_mass = cold_gas_mass
        self.stellar_mass = stellar_mass
        self.age = age
        self.SFR = 0
        self.color = 'red'

    def __repr__(self):
        return "Galaxy (m_total = %.1g; m_cold = %.1g; m_stars = %.1g; age = %.1g; SFR = %0.2f)" % \
                (self.total_mass, self.cold_gas_mass, self.stellar_mass, self.age, self.SFR)

milky_way = Galaxy(1e12, 1e8, 1e10, age=5e9)
print(milky_way)


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 2) Make a more interesting galaxy class that can evolve with time
Now let's build on our basic galaxy class with some relevant class methods specific to galaxies that enables them to change with time.  Obviously, galaxies have a huge number of traits, and we don't currently understand all the ways in which these variables interrelate.  But we can make a toy model here just to get the idea of how they might evolve.  This is largely the premise of semi-analytic models that people use to study how galaxies might evolve with time.
Problem 2a
Think about what methods you would add to allow your galaxy to change with time using the galaxy attributes defined above and/or other galaxy attributes.  Come up with at least three and then discuss with your partner before continuing.
Possible Answers: evolve(time), cold_mode_accretion(time), hot_mode_accretion(time), absorb_galaxy(galaxy), star_burst(), agn_feedback(), change_color(), change_type()
Problem 2b
We want to enhance our general Galaxy class to enable galaxies to actually evolve with time.  Rather than just copying all of the code from our Galaxy class above, we can subclass the Galaxy class to inherit all of its attributes, methods, etc!  Create a new class named EvolvingGalaxy below, which uses the Galaxy class as its "parent" or "super" class.  Remember, because you already defined __init__ and __repr__ methods for this class, they are inherited and you don't need to redefine them, unless you want to.
Now try creating an instance of EvolvingGalaxy and printing it out, just like you did for your favorite galaxy in problem 1c.
Can you write Python code for it?

class EvolvingGalaxy(Galaxy):
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """

milky_way = EvolvingGalaxy(1e12, 1e8, 1e10, age=5e9)
print(milky_way)


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 2c
Let's start by making an evolve method for our Galaxy class in order to evolve the galaxy forward in time.  evolve should expect an argument for time in years.  And at least initially, let's just make this a simple method that just changes the age of the galaxy by the appropriate amount.  We'll add more on to it in a second.
Because we're going to be following the evolution of this galaxy, we should create a current_state method, which returns a tuple of the attributes of the EvolvingGalaxy object at that moment.  Now as our EvolvingGalaxy changes with time, we will be able to track it and save that information somewhere.
Can you write Python code for it?

class EvolvingGalaxy(Galaxy):
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """
    def evolve(self, time):
        """
        Evolve this galaxy forward for a period time
        """
        self.age += time

    def current_state(self):
        return (self.total_mass, self.cold_gas_mass, self.stellar_mass, self.age, self.SFR)


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 2d
Now that we can evolve our EvolvingGalaxy and check on its current_state over time, let's run it forward in time and see the results!
Make a new function (not a class method) called integrate_time that accepts an EvolvingGalaxy object, a timestep (some number of years long), and a number of timesteps over which to evolve the galaxy.  Step the EvolvingGalaxy forward in time using your for loop, one timestep at a time, and store its current_state at each timestep.  You can store this current_state information however you'd like, but I find it easiest to create a NumPy array of size [5, n_timesteps], and fill it in using the for loop.
Similarly, make a new function called plot_galaxy_evolution that accepts the output from your integrate_time function and uses Matplotlib's pyplot module to plot up the evolution of the EvolvingGalaxy's properties versus its age.
Finally, create an instance of EvolvingGalaxy, and plug it into your new integrate_time and plot_galaxy_evolution functions to integrate for a gigayear, and plot how the EvolvingGalaxy quantities change with age.  Not much going on--sad!
Can you write Python code for it?

def integrate_time(galaxy, timestep, n_timesteps):
    """
    Integrate the time forward for a galaxy and record its state at each timestep; return as array
    """
    data_arr = np.empty([5, n_timesteps])
    for i in range(n_timesteps):
        galaxy.evolve(timestep)
        data_arr[:,i] = galaxy.current_state()
    return data_arr

def plot_galaxy_evolution(data_arr):
    """
    Plot the evolution of a galaxy from its input data array
    """
    plt.clf()
    plt.semilogy(data[3], data[0], color='k', label='Total')
    plt.semilogy(data[3], data[1], color='b', label='Gas')
    plt.semilogy(data[3], data[2], color='r', label='Stars')
    plt.semilogy(data[3], data[4], color='g', label='SFR')
    plt.xlabel('Age')
    plt.ylabel('Mass')
    plt.legend(loc=1)
    plt.show()

milky_way = EvolvingGalaxy(1e12, 1e8, 1e10, age=5e9)
data = integrate_time(milky_way, 1e6, 1000)
plot_galaxy_evolution(data)


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 2e
How do galaxies evolve with time?  They do in lots of ways, but let's make a few specific rules.
Galaxies tend to accrete gas from the intergalactic medium over time.  We'll modify the code from above for your EvolvingGalaxy class to include a accrete_gas_from_IGM method, which adds to the cold_gas_mass and total_mass of the galaxy with time.  Let's give it a random component too.
Galaxies form stars, converting cold_gas_mass to stellar_mass according to their star formation rate (SFR).  But there must be sufficient cold_gas_mass to consume, so make sure there is enough for the SFR to operate for the alloted time, otherwise the SFR should drop to 0.
Finally, we haven't actually calculated a SFR, let's just create a method, calculate_star_formation_rate, choose a random value every time it is called.  We'll also scale it in some way by the total_mass of the galaxy and use the Milky Way as a guide.  The Milky Way forms about 1 Msun/year and it has a total_mass of 1e12 Msun.
Include these functions in our evolve method, and re-generate a galaxy evolution plot like you did in Problem 2d.  Since there is a random component, you can try re-running it a few times and see how the behavior changes.
Can you write Python code for it?

class EvolvingGalaxy(Galaxy):
    """
    Galaxy class for representing a galaxy that can evolve over time.
    """
    def current_state(self):
        """
        Return a tuple of the galaxy's total_mass, cold_gas_mass, stellar_mass, age, and SFR
        """
        return (self.total_mass, self.cold_gas_mass, self.stellar_mass, self.age, self.SFR)

    def calculate_star_formation_rate(self):
        """
        Calculate the star formation rate by taking a random number between 0 and 1
        normalized by the galaxy total mass / 1e12;

        Also updates the galaxy's color to blue if SFR > 0.01, otherwise color = red
        """
        self.SFR = random.random() * (self.total_mass / 1e12)
        if self.SFR > 0.01:
            self.color = 'blue'
        else:
            self.color = 'red'

    def accrete_gas_from_IGM(self, time):
        """
        Allow the galaxy to accrete cold gas from the IGM at a variable rate normalized to
        the galaxy's mass
        """
        cold_gas_accreted = random.random() * 0.1 * time * (self.total_mass / 1e12)
        self.cold_gas_mass += cold_gas_accreted
        self.total_mass += cold_gas_accreted

    def form_stars(self, time):
        """
        Form stars according to the current star formation rate and time available
        If unable cold gas, then shut off star formation
        """
        if self.cold_gas_mass > self.SFR * time:
            self.cold_gas_mass -= self.SFR * time
            self.stellar_mass += self.SFR * time
        else:
            self.SFR = 0
            self.color = 'red'

    def evolve(self, time):
        """
        Evolve this galaxy forward for a period time
        """
        if random.random() < 0.01:
            self.calculate_star_formation_rate()
        self.accrete_gas_from_IGM(time)
        self.form_stars(time)
        self.age += time

milky_way = EvolvingGalaxy(1e12, 1e8, 1e10, age=5e9)
data = integrate_time(milky_way, 1e6,...


I am working on the file "Sessions/Session03/Day2/OOP_solns.ipynb".
The first task is:
Problem 2f
Based on your knowledge of how galaxies evolve, do these results make sense?  Discuss with your coding partner if so/not, and ways in which you might tweak it to work better.
Possible Answers: For the most part, yes.  The galaxy consumes its cold gas and creates stars, and when it runs out of gas, it shuts off star formation for a time.  But after time, it accretes enough cold gas to allow it to form stars again, so it's in sort of loop.  But it's not really realistic because it needs to interact with its environment and other galaxies through galaxy mergers, gas accretion, gas stripping, etc.
Problem 3) Make a galaxy class that can interact with other galaxies (challenging!)
Most galaxies do not evolve in isolation, so let's make a galaxy class that enables galaxies to actually interact with each other.  We can re-use much of the code from our previous examples through inheritance!  Yeah!
Problem 3a
Let us create a new subclass of EvolveGalaxy, so that it inherits most of the traits of our previous work.  But we will add some new features including position and velocity values in the x-y plane, so that our galaxy can actually move over time.
Create a new class MovingGalaxy that is a subclass of EvolveGalaxy.  Make a new __init__ method with all of the arguments from it's parent class EvolveGalaxy, but add the additional arguments of x_position, y_position, x_velocity, y_velocity, and id_num.
Note: all arguments must come before any keyword arguments, so the age kwarg has to come at the very end.
Also, make a new __repr__ method for this class that just includes it's idnum, x_position, and y_position.
To assure your code works, create an instance of a MovingGalaxy, and print it out.
Can you write Python code for it?

class MovingGalaxy(EvolvingGalaxy):
    """
    Galaxy class that can evolve and move in the x,y plane
    """
    def __init__(self, total_mass, cold_gas_mass, stellar_mass, x_position, y_position, x_velocity, y_velocity, idnum, age=0):

        # Replace self with super to activate the superclass's methods
        super().__init__(total_mass, cold_gas_mass, stellar_mass)

        self.x_position = x_position
        self.y_position = y_position
        self.x_velocity = x_velocity
        self.y_velocity = y_velocity
        self.idnum = idnum

    def __repr__(self):
        return "Galaxy %i (x = %.0f; y = %.0f)" % (self.idnum, self.x_position, self.y_position)

milky_way = MovingGalaxy(1e12, 1e8, 1e10, 0, 0, 0, 0, 0)
print(milky_way)


I am working on the file "Sessions/Session03/Day3/MapReduce.ipynb".
The first task is:
Probelm 2a)
Create an Inverted index. Given a set of documents, an inverted index is a dictionary where each word is associated with a list of the document identifiers in which that word appears.
Mapper Input
The input is a 2 element list: [document_id, text], where document_id is a string representing a document identifier and text is a string representing the text of the document. The document text may have words in upper or lower case and may contain punctuation. You should treat each token as if it was a valid word; that is, you can just use value.split() to tokenize the string.
Reducer Output
The output should be a (word, document ID list) tuple where word is a String and document ID list is a list of Strings like:
["all", ["milton-paradise.txt", "blake-poems.txt", "melville-moby_dick.txt"]]
["Rossmore", ["edgeworth-parents.txt"]]
["Consumptive", ["melville-moby_dick.txt"]]
["forbidden", ["milton-paradise.txt"]]
["child", ["blake-poems.txt"]]
["eldest", ["edgeworth-parents.txt"]]
["four", ["edgeworth-parents.txt"]]
["Caesar", ["shakespeare-caesar.txt"]]
["winds", ["whitman-leaves.txt"]]
["Moses", ["bible-kjv.txt"]]
["children", ["edgeworth-parents.txt"]]
["seemed", ["chesterton-ball.txt", "austen-emma.txt"]]
etc...
Can you write Python code for it?

mr = MapReduce()

def mapper(record):
    # COMPELTE

def reducer(key, list_of_values):
    # COMPLETE

inputdata = open(os.path.join(DATA_DIR, "books.json"))
mr.execute(inputdata, mapper, reducer)


I am working on the file "Sessions/Session03/Day3/MapReduce.ipynb".
The first task is:
Problem 1a) Re-write the mapper and reducer to return the maximum number in all 3 lists.
Can you write Python code for it?

def mapper(arr):
    # COMPLETE

def reducer(x, y):
    # COMPLETE

intermediate = map(mapper, inputData)
reduce(reducer, intermediate)


I am working on the file "Sessions/Session03/Day3/MapReduce.ipynb".
The first task is:
Problem 1b)
How would you use this to compute the MEAN of the input data.
Problem 1c)
Think about how you would adapt this this to compute the MEDIAN of the input data. Do not implement it today! If it seems hard, it is because it is.
What special properties do SUM, MAX, MEAN have that make it trivial to represent in MapReduce?
Problem 2)
Let's go through a more complete example. The following MapReduce class faithfully implements the MapReduce programming model, but it executes entirely on one processor -- it does not involve parallel computation.
Setup
First, download the data:
$ curl -O https://lsst-web.ncsa.illinois.edu/~yusra/escience_mr/books.json
$ curl -O https://lsst-web.ncsa.illinois.edu/~yusra/escience_mr/records.json
Can you write Python code for it?

DATA_DIR = './data' # Set your path to the data files

import json
import sys

class MapReduce:
    def __init__(self):
        self.intermediate = {}
        self.result = []

    def emit_intermediate(self, key, value):
        self.intermediate.setdefault(key, [])
        self.intermediate[key].append(value)

    def emit(self, value):
        self.result.append(value)

    def execute(self, data, mapper, reducer):
        for line in data:
            record = json.loads(line)
            mapper(record)

        for key in self.intermediate:
            reducer(key, self.intermediate[key])

        jenc = json.JSONEncoder()
        for item in self.result:
            print(jenc.encode(item))