I am working on the file "Labs/Lab8/Lab 8.ipynb".
The first task is:
Numpy has a really handy built-in function for padding images called pad. The required inputs are the array to be padded, the size of the pad region (in each direction), and the nature of the padding (lots of interpolation options are available, but we just need a constant value). The default for a constant value is to propagate the values at the edges outward, but we want in this case to draw a clear distinction between where we have data and where we don't, so we can set the padded region values to some specific constant value. Often astronomers pad arrays with either zeros or NaNs, but NaNs can be problematic when executing the next step (shifting the images) and zero is too close to the actual data values in our sample image below to stand out, so I've chosen an arbitrary value below based solely on its ability to stand out relative to the rest of the colorbar so that I can tell where the padded regions is.
Can you write Python code for it?

newarr = np.pad(gauss,5,'constant', constant_values=-0.001)
newarr.shape

plt.imshow(newarr)
plt.colorbar()


I am working on the file "Labs/Lab8/Lab 8.ipynb".
The first task is:
The first two lines compute the r and phi coordinates in a polar coordinate system. The second two lines define two functions that we'll play with. The function z1 is just the sinc function in 2D, where we take sinc(r) instead of sinc(x). We've put a factor of 2 inside the sin to change the period from 2 pi to pi. The second function z2 just takes z1 and multiplies it by a factor that varies between 1.5 and 0.5 depending on the angle phi, with a period of pi/2 radians in the phi direction.
1.2 Contour plots
How can we represent data like this? One simple way is through a contour plot. The archetypical contour plot is a topographic map: it shows curves of constant height, or more generally equal z value, as a function of (x,y) position. Matplotlib produces contour plots using the contour() command. Basic usage is simple: contour(xx, yy, z). The first two arguments are the x and y coordinates, and these can be either 1D arrays whose size matches the corresponding size of the z array, or 2D arrays whose shape matches that of the z array. Let's see what happens when we do this:
Can you write Python code for it?


I am working on the file "Labs/Lab8/Lab 8.ipynb".
The first task is:
The next thing that we'd want to do with padded images is to shift the image to some common center so that it can be combined with other images.
Can you write Python code for it?

import scipy.ndimage.interpolation as interp

newarr2 = interp.shift(newarr,(2.5,4.5), cval=-0.001)

plt.imshow(newarr2)
plt.colorbar()


I am working on the file "Labs/Lab8/Lab 8.ipynb".
The first task is:
The output looks like this:

This is a contour plot of the data. One minor annoyance is that the spacing on the x and y axes isn't equal, so things look stretched. We can fix that by changing the aspect ratio of the axes. To do that, we use the commands
Can you write Python code for it?

plt.gca().set_aspect('equal')
plt.contour(xx, yy, z1)


I am working on the file "Labs/Lab8/Lab 8.ipynb".
The first task is:
We can also label the contours using the clabel() command. Let's remake this plot using some labelled contours, placing the contours starting at -0.4 and going in intervals of 0.4.
Can you write Python code for it?

plt.gca().set_aspect('equal')
cs = plt.contour(xx, yy, z1, arange(-0.4, 2.0, 0.4))
plt.clabel(cs)


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
2.2 The describe() method
There are also a whole bunch of built in functions that can operate on a pandas dataframe that become available once you've defined it. To see a full list type data. in an empty frame and then hit tab.
An especially useful one is dataframe.describe() method, which creates a summary table with some common statistics for all of the columns in the dataframe.
In our case here there are a number of NaNs in our table (cases where an answer was left blank), and the describe method ignores them for mean, standard deviation (std), min and max. However, there is a known bug in the pandas module that cause NaNs to break the quartiles in the describe method, so these will always be NaN for any column that has a NaN anywhere in it, rendering them mostly useless here. Still, this is a nice quick way to get descriptive statistics for a table.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
2.3. Computing Descriptive Statistics
You can also of course compute descriptive statistics for columns in a pandas dataframe individually. Examples of each one applied to a single column - student scores on the assessment (PRE_SCORE) are shown below.
Can you write Python code for it?

np.mean(data["PRE_SCORE"])

#or
data["PRE_SCORE"].mean()

np.nanmedian(data["PRE_SCORE"])

#or
data["PRE_SCORE"].median()

data["PRE_SCORE"].max()

data["PRE_SCORE"].min()

data["PRE_SCORE"].mode()
#where first number is the index (should be zero unless column has multiple dimensions
# and second number is the mode
#not super useful for continuous variables for example, if you put in a continuous variable (like ZPR_1) it won't
#return anything because there are no repeat values

#perhaps equally useful is the value_counts method, which will tell you how many times each value appears int he column
data["PRE_SCORE"].value_counts()

#and to count all of the non-zero values
data["PRE_SCORE"].count()

#different generally from len(dataframe["column name]) because len will count NaNs
# but the Score column has no NaNs, so swap this cell and the one before our with
#a column that does have NaNs to verify
len(data["PRE_SCORE"])

#standard deviation
data["PRE_SCORE"].std()

#variance
data["PRE_SCORE"].var()

#verify relationship between variance and standard deviation
np.sqrt(data["PRE_SCORE"].var())

#quantiles
data["PRE_SCORE"].quantile(0.5) # should return the median!

data["PRE_SCORE"].quantile(0.25)

data["PRE_SCORE"].quantile(0.75)

#interquartile range
data["PRE_SCORE"].quantile(0.75)-data["PRE_SCORE"].quantile(0.25)

data["PRE_SCORE"].skew()

data["PRE_SCORE"].kurtosis()


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
2.4. Creating Statistical Graphics
<div class=hw>
### Exercise 2 - Summary plots for distributions

*Warning: Although you will be using QuaRCS data to investigate and experiment with each type of plot below, when you write up your descriptions, they should refer to the **general properties** of the plots, and not to the QuaRCS data specifically. In other words, your descriptions should be general descriptions of the plot types that could be applied to any dataset.*

### 2a - Histogram
The syntax for creating a histogram for a pandas dataframe column is:

dataframe["Column Name"].hist(bins=nbins)

Play around with the column name and bins and refer to the docstring as needed until you understand thoroughly what is being shown. Describe what this ***type of plot*** (not any individual plot that you've made) shows in words and describe when you think it might be useful.

Play around with inputs (e.g. column name) until you find a case (dataframe column) where you think the histogram tells you something important and use it as an example to inform your answer. Inputs that do not produce informative histograms should also help to inform your answer. Save a couple of representative histograms (good and bad, use plt.savefig("figure name")) and integrate them into your written (markdown) explanation to support your argument.
Can you write Python code for it?

#this cell is for playing around with histograms


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
<div class=hw>
### Exercise 1
------------------

Choose one categorical (answer to any demographic or attitudinal question) and one continuous variable (e.g. PRE_TIME, ZPR_1) and compute all of the statistics from the list above ***in one code cell*** (use print statements) for each variable. Write a paragraph describing all of the statistics that are informative for that variable in words. An example is given below for PRE_SCORE. Because score is numerical ***and*** discrete, all of the statistics above are informative. In your two cases, fewer statistics will be informative, so your explanations may be shorter, though you should challenge yourselves to go beyond merely reporting the statistcs, and should interpret them as well, as below.

*QuaRCS score can take discrete integer values between 0 and 25. The minimum score for this dataset is 1 and the maximum is 25. There are 2,777 valid entries for score in this QuaRCS dataset, for which the mean is 13.9 and the median is 14 (both 56\% of the maximum score). These are very close together, suggesting a reasonably centrally-concentrated score distrubution, and the low skewness value of 0.1 supports this. The kurtosis of the distribution is negative (platykurtic), which tells us that the distribution of scores is flat rather than peaky.   The most common score ("mode") is 10, with 197 (~7%) of participants getting this score, however all score values from 7-21 have counts of greater than 100, supporting the flat nature of the distribution suggested by the negative kurtosis. The interquartile range (25-75 percentiles) is 8 points, and the standard deviation is 5.3. These represent a large fraction (20 and 32\%) of the entire available score range, respectively, making the distribution quite wide.

*Your description of categorical distribution here*

*Your description of continuous distribution here*
Can you write Python code for it?

#your code computing all descriptive statistics for your categorical variable here

#your code computing all descriptive statistics for your categorical variable here


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
<div class=hw>
### Exercise 4
------------------

Choose a categorical variable (any demographic or attitudinal variable) that you find interesting and that has at least four possible values and calculate the condifence intervals on the mean score for each group. Then write a paragraph describing the results. Are the differences between the groups significant according to your data? Would they still be significant if you were to compute the 98% (3-sigma) confidence intervals?
Can you write Python code for it?

#code to filter data and compute confidence intervals for each answer choice


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
If you get to this point during lab time on Tuesday, stop here
3. Testing Differences Between Datasets
3.1 Computing Confidence Intervals
Now that we have a mechanism for filtering the dataset, we can test differences between groups with confidence intervals. The syntax for computing the confidence interval on a mean for a given variable is as follows.
variable1 = st.t.interval(conf_level,n,loc=np.nanmean(variable2), scale=st.sem(variable2))
where conf_level is the confidence level you with to calculate (e.g. 0.95 is 95% confidence, 0.98 is 98%, etc.)
n is the number of samples and should generally be set to the number of valid entries in variable2 -1.
An example can be found below.
Can you write Python code for it?

## apply filter to select only men from data, and pull the scores from this group into a variable
df2=filter(data,'PRE_GENDER',1)
men_scores=df2['PRE_SCORE']

#compute 95% confidence intervals on the mean (low and high)
men_conf=st.t.interval(0.95, len(men_scores)-1, loc=np.mean(men_scores), scale=st.sem(men_scores))
men_conf


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Once a dataset has been read in as a pandas dataframe, several useful built-in pandas methods are made available to us. Recall that you call methods with data.method. Check out each of the following
Can you write Python code for it?

# the * is a trick to print without the ...s for an ordinary python object
print(*data.columns)

data.dtypes


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Read in the QuaRCS data as a pandas dataframe called "data".
Can you write Python code for it?

data=pd.read_csv('AST200_data_anonymized.csv', encoding="ISO-8859-1")
mask = np.where(data == 999)
data = data.replace(999,np.nan)


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Your explanation here
2.5.  Selecting a Subset of Data
<div class=hw>
### Exercise 3
--------------

Write a function called "filter" that takes a dataframe, column name, and value for that column as input and returns a new dataframe containing only those rows where column name = value. For example filter(data, "PRE_GENDER", 1) should return a dataframe about half the size of the original dataframe where all values in the PRE_GENDER column are 1.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Your explanation here
<div class=hw>
### 2c - Pie Chart

The format for making the kind of pie chart that might be useful in this context is as follows:
newdataframe = dataframe["column name"].value()counts
newdataframe.plot.pie(figsize=(6,6))

Play around with the column and refer to the docstring as needed until you understand thoroughly what is being shown. Describe what this ***type of plot*** (not any individual plot that you've made) shows in words and describe when you think it might be useful. In your explanation here, focus on how a bar chart compares to a histogram, and when you think one or the other might be useful.

Play around with inputs (e.g. column names) until you find a case that you think is well-described by a pie chart and use it as an example to inform your answer. Inputs that do not produce informative pie charts should also help to inform your answer. Save a couple of representative pie charts (good and bad) and integrate them into your written explanation.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Your explanation here
<div class=hw>
### 2d - Scatter Plot
The syntax for creating a scatter plot is:

dataframe.plot.scatter(x='column name',y='column name')

Play around with the column and refer to the docstring as needed until you understand thoroughly what is being shown. Describe what this ***type of plot*** (not any individual plot that you've made) shows in words and describe when you think it might be useful.

Play around with inputs (e.g. column names) until you find a case that you think is well-described by a scatter plot and use it as an example to inform your answer. Inputs that do not produce informative scatter plots should also help to inform your answer. Save a couple of representative pie charts (good and bad) and integrate them into your written explanation.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
Your explanation here, with figures
<div class=hw>
### 2b - Box plot

The syntax for creating a box plot for a pair of pandas dataframe columns is:

dataframe.boxplot(column="column name 1", by="column name 2")

Play around with the column and by variables and refer to the docstring as needed until you understand thoroughly what is being shown. Describe what this ***type of plot*** (not any individual plot that you've made) shows in words and describe when you think it might be useful.

Play around with inputs (e.g. column names) until you find a case that you think is well-described by a box and whisker plot and use it as an example to inform your answer. Inputs that do not produce informative box plots should also help to inform your answer. Save a couple of representative box plots (good and bad) and integrate them into your written explanation.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab 9.ipynb".
The first task is:
explanatory text
3.2 Visualizing Differences with Overlapping Plots
<div class=hw>
### Exercise 5
---------------

Make another dataframe consisting only of students who "devoted effort" to the assessment, meaning their answer for PRE_EFFORT was EITHER a 4 or a 5 (you may have to modify your filter function to accept more than one value for "value").

Make overlapping histograms showing (a) scores for the entire student population and (b) scores for this "high effort" subset. The "alpha" keyword inside the plot commands will set the transparency of your histogram so that you can see both. Play around with it until it looks good. Make sure your chart includes a legend, and describe what conclusions you can draw from the result in a paragraph below the final chart.
Can you write Python code for it?

#modified filter function here

#define your new high effort dataframe using the filter

#plot two overlapping histograms


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
2. Problem 2 explanation and images go here
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
3. Problem 3 explanation and images go here
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
9.3 - Aperture Photometry
The next step is to actually extract the photometry, and here too there is some art to choosing parameters. Although photutils will extract the photometry for each star in an automated way, you need to intelligently choose the parameters based on your data.
The tunable parameters are:
1. the aperture radius inside of which to count the flux from the star
2. the inner and outer radius of the sky aperture. The annulus defined by these two numbers needs to be large enough to get a good measurement of the background level, but small enough to generally avoid confusion with other nearby sources.
We'll start with some potentially reasonable values for these parameters.
Can you write Python code for it?


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
Exercise 1. Use the resources (docstrings, wikipedia, other functions) at your disposal to answer the following questions in the cell below.

What does the mad_std function do?
How is it different from the more typical np.std function? How different are the answers in these two cases, and why?

Your answers go here
9.2 - Extracting Stars
The star finder requires two parameters to extract stars in the image:
1) The threshhold, which we will define as some number of background levels, above which we will call something a star
2) An estimate for the FWHM of point-sources (stars) in the image
Exercise 2
To start, estimate the FWHM of the stars in your image using pyraf's imexam functions, as you did in Lab 8. Measure the FWHM for at least 10 stars and average them. In the cell below, paste the imexam output and calculate the average of the measurements in the cell below that. Insert your calculated average FWHM in the third cell below.
insert imexam output here
Can you write Python code for it?

#insert calculation of average FWHM here

#FWHM= this is a placeholder. INSERT YOUR VALUE IN PLACE OF 10 BELOW
FWHM=10


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
Exercise 4
For each of the next two cells, write a comment describing what each line is doing in the line above it.
Can you write Python code for it?

#add comments to this cell
starapertures = CircularAperture((xpos,ypos),r=aprad)
skyannuli = CircularAnnulus((xpos,ypos),r_in=skyin,r_out=skyout)
phot_apers = [starapertures, skyannuli]
phot_table = aperture_photometry(image,phot_apers)
phot_table

#add comments to this cell
bkg_mean = phot_table['aperture_sum_1']/skyannuli.area()
bkg_starap_sum = bkg_mean * starapertures.area()
final_sum = phot_table['aperture_sum_0']-bkg_starap_sum
phot_table['background subtracted star counts'] = final_sum
phot_table


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
Exercise 5 -
Spend the rest of the lab time investigating what changes about the photometric measurement (background subtracted sky counts caluclated column) when you adjust the tunable parameters (aperture radius and inner/outer sky annulus) and report your findings below. You may wish to examine only a handful of stars in the table and avoid pr (to print just a few rows of a table, see the example below), but make sure the rows you select include stars with a range of brightnesses. You may also wish to make separate versions/copies of the table with different aperture parameters so that you can compare without overwriting. Think in particular about crowded fields and see if you can derive the best parameters for this case as well (identify things in the table with very close values for xcenter and ycenter to find these crowded regions).
Can you write Python code for it?

#example of printing just a few rows in a table - chosen to be one bright star, one faint
phot_table[6:8]


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
We also need to set the threshhold (described above) for star finder, which we define as some multiple (nsigma) of the background level. To start, let's set nsigma to 10, meaning that in order for somehting to be considered a star, it must have at least 10x the detector counts of the background level.
The next several lines below set up the parameters for the star finder (by specifying the FWHM and threshhold parameters), apply the star finder to the image, and then extract and print the number of sources.
Can you write Python code for it?

nsigma=10
daofind = DAOStarFinder(fwhm=FWHM, threshold=nsigma*bkg_sigma)
sources = daofind(image)
nstars=len(sources)
nstars


I am working on the file "Labs/Lab9/Lab9.ipynb".
The first task is:
We will be using a python package called photutils, which is based on an old IRAF package of the same name. One of the key functions within this package is the DAOStarFinder function, which we'll just call "the star finder" here.
The star finder will extract sources, defined as some multiple (which you provide) of the background/sky level, so first we need to get a reasonable estimate of the background level. This is done using the function mad_std from the astropy.stats package, as below.
Can you write Python code for it?

from astropy.stats import mad_std
bkg_sigma = mad_std(image)
print(bkg_sigma)

print(np.std(image))


I am working on the file "Labs/Lab9/MAT245 Lab 9.ipynb".
The first task is:
Goals (1):

Split the breast cancer data into 70% training and 30% validation sets.
Write a python implementation of the logistic regression function $(\theta, x) \mapsto h_\theta(x)$.
Implement the stochastic gradient ascent (SGA) algorithm described above to choose the best parameter $\theta$ for the hypothesis function $h_\theta$. How do different learning rates affect convergence? Typical choices are in the range 0.001 - 0.1.
Validate your model's classification accuracy on the validation set (the sklearn.metrics.accuracy_score function may come in handy here).
How many iterations of SGA do you need to consistently get >85% classification accuracy on the validation set?

Principal component analysis
Background
Principal component analysis (PCA) is a dimensionality reduction technique. The idea is to project the data down to lower dimension by 'dropping' those directions/dimensions that don't contain much variance. For instance, consider the following sample of data points in 2D:
<img src="pca.svg" alt="Gaussian data in 2D" style="width: 300px;"/>
The goal of a PCA in this case would be to project all of the data points onto the axis spanned by the longer arrow; since the short arrow is orthogonal to the large one, it would be ideal if we could project along the short arrow. The new dataset will be 1-dimensional, and since most of the variation in the data was along the direction spanned by the long arrow, hopefully we haven't lost much information.
For more details about the mathematics of PCA, see Andrew Ng's great notes here.
Identifying digits with PCA and k-Nearest Neighbors.
The sklearn digits dataset contains images of handwritten digits, much like the famous MNIST dataset. Here's a sample:
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np

digits = datasets.load_digits()
samples, targets = digits.data, digits.target

%matplotlib inline
plt.imshow(np.reshape(samples[0], (8,8)), cmap='Greys')


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Contour Plot
Can you write Python code for it?

' Prepare grid interval and theta interval  '

plt.close('all')

interval_0 = np.linspace(-10, 10, 100)
interval_1 = np.linspace(-1, 4, 100)

(t0, t1) = np.meshgrid(interval_0, interval_1, indexing='xy')

' Filling costs '
j_history = np.zeros(shape=t0.shape)

' Generating bidimensional array '
for (i, j), element in np.ndenumerate(j_history):
    theta = np.array([t0[i,j], t1[i,j]])
    j_history[i,j] = cost_function(x, y, theta)

fig1 = plt.figure(figsize=(6,5))
ax = plt.contour(t0, t1, j_history, np.logspace(-2, 3, 20), cmap=plt.cm.jet)
plt.scatter(t[0], t[1], c='r')

plt.xlabel('Theta 0')
plt.ylabel('Theta 1')

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Cost Function
Can you write Python code for it?

def cost_function(x, y, theta):
    m = y.size
    regression = x.dot(theta)
    regressionError = np.sum((regression - y)**2)
    return ((1./(2*m))*regressionError)


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Data Visualization
A linear decision boundary can bee seen from the below plot. However, it is possible to verify that there is a slight slope, the line curves a bit, that prevents us from using the linear regression model to elaborate the model.
The first step to output our model is to implement the sigmoid function.
Can you write Python code for it?

plt.close('all')

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(admitted[:, 0], admitted[:,1], s=50, c='black', marker='x', label='Admitted')
ax.scatter(notadmitted[:, 0], notadmitted[:,1], s=50, c='yellow', marker='o', label='Not Admitted')

ax.legend()
ax.set_xlabel('Exam 1 Score')
ax.set_ylabel('Exam 2 Score')

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Feature scaling or feature normalization
Feature scaling is a method used to standardize the range of independent variables or features of data. In data processing, it is also known as data normalization and is generally performed during the data preprocessing step.
Can you write Python code for it?

def standardization(x):
    mean_residual = []
    std_residual  = []
    standardization = x

    dimension = x.shape[1]
    for i in range(dimension):
        mean = np.mean(x[:, i])
        std  = np.std(x[:, i])

        mean_residual.append(mean)
        std_residual.append(std)

        standardization[:, i] = (1./std)*(standardization[:, i] - mean)

    return standardization


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
For the data under study, the cost function should return an approximate value of 32,072 for the initial values of θ. Let's test the function return to θ at zero
Can you write Python code for it?

' Update y in order to perform good matrices operations '
y = data[:,1]
' Insert ones in order to perform good matrices operations '
x = np.insert(x, 0, np.ones(x.size), axis=1)
theta = np.array([0., 0.])
cost_function(x,y,theta)


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Gradient Descent
In order to find the better fit for our prediction model, lets implement the Gradient Descent algorithm and minimize our J(θ). We will perform little steps across theta values and control the convergence, in order to ensure  that the final θ are what we are looking for.
Can you write Python code for it?

def gradient_descent(x, y, theta, alpha, tolerance, log = False):
    m = y.size
    episilon  = 0.000001
    residual  = np.ones(tolerance)
    interact = 0
    converge = False
    while not converge:
        ' step 1 : Start adjusting theta values '
        hypothesis   = x.dot(theta)
        error        = (hypothesis - y)
        gradient     = (1./m) * (alpha) * (x.T.dot(error))

        tmpTheta = theta
        theta = theta - gradient

        ' step 2 : Run the cost function over the data with new theta '
        residual[interact] = cost_function(x, y, theta)

        if interact % 100 == 0 and log:
            print('Interaction ' + str(interact) + ' - Residual cost '+ str(residual[interact]))

        ' step 3 : Verify convergence over the given episilon and residual given step, also verify tolerance tries '

        step = abs(np.linalg.norm(theta) - np.linalg.norm(tmpTheta))
        converge = (step <= episilon)

        if (interact + 1) == tolerance:
            print('Caution! GD has reached the tolerance. Results may not converge.')
            converge = True

        interact = interact + 1

    return theta, residual, interact

(t,r,i) = gradient_descent(x, y, theta, 0.01, 4000)

print('Last interaction was '+ str(i) + ' and the output for θ is: ')
print(t)


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
How to use the model and perform a prediction
Using the obtained model to perform a prediction is a very easy task when working with one variable linear regression. We will use matrice operations to obtain the prediction values for cities with 35.000 and 70.000 population
Can you write Python code for it?

' Predicting values for 35.000 population '
p = np.array([1., 35.000]).dot(t)
print('Predicting values for 35.000 population : ' + str(p))

' Predicting values for 70.000 population '
p = np.array([1., 70.000]).dot(t)
print('Predicting values for 35.000 population : ' + str(p))


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Prediction Function
Can you write Python code for it?

def predict(target, theta):

    m = target.shape[0]
    p = np.zeros(m)

    param = target.dot(theta)

    h = sigmoid_function(param)

    for i in range(h.shape[0]):
        if h[i] > 0.5:
            p[i] = 1
        else:
            p[i] = 0

    return p


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Regression Line
Finally we can evaluate if the obtained model has a good fit to the data.
Can you write Python code for it?

plt.close('all')

plt.figure()
plt.plot(x[:,1],y,'ro',markersize=5)
plt.tick_params(axis='both', direction='in', width=2, length=7,bottom='on', top='on', left='on', right='on')

plt.ylabel('Profit in $10,000s')
plt.xlabel('Population of City in 10,000s')

plt.xticks(np.arange(4, 24, 2))
plt.ylim(-5, 25)
plt.xlim(4, 24)

plt.plot(x[:, 1], x.dot(t), '-', label='Linear regression')
plt.legend(loc='lower right', shadow=True, fontsize='x-large', numpoints=1)

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Sigmoid Function
The Sigmoid Function converts a continuous input into a value between zero and one. It helps us to work in the LOG domain when performing the Cost Function and Gradient Descent.
Can you write Python code for it?

def sigmoid_function(z):
    return 1 / (1 + np.exp(-z))


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Surface Plot
Can you write Python code for it?

plt.close('all')

fig = plt.figure()
ax = fig.gca( projection='3d')

surf = ax.plot_surface(t0, t1, j_history, rstride=1, cstride=1, alpha=1, cmap=plt.cm.coolwarm)

# Customize the z axis
zmin = j_history.min()
zmax = j_history.max()

ax.set_zlim(zmin, zmax)
ax.view_init(elev=15, azim=190)

plt.xlabel('Theta 0')
plt.ylabel('Theta 1')

fig.colorbar(surf, shrink=0.5, aspect=5)

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
Testing values for α
α = 0.001
Can you write Python code for it?

(t, j, i, r) = gradient_descent(x, y, theta, 0.001, 6000)

print('Iteraction reached : ' + str(i))
print('Theta : ' + str(t))

' Verify error rate behavior '

plt.close('all')

plt.figure(figsize=(10,6))

axes = plt.gca()
axes.set_xlim([0,i])

plt.plot(j)

plt.show()

' Verify step rate behavior '

plt.close('all')

plt.figure(figsize=(10,6))

axes = plt.gca()
axes.set_xlim([0,i])

plt.plot(r)

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
The above figure allows us to note that the value of the step given by theta does not change over the iterations until the execution tolerance is reached and the operation aborted. Therefore, there is no convergence for this alpha value.
α = 0.002
Can you write Python code for it?

(t, j, i, r) = gradient_descent(x, y, theta, 0.002, 6000)

print('Iteraction reached : ' + str(i))
print('Theta : ' + str(t))

' Verify error rate behavior '

plt.close('all')

plt.figure(figsize=(10,6))

axes = plt.gca()
axes.set_xlim([0,i])

plt.plot(j)

plt.show()

' Verify step rate behavior '

plt.close('all')

plt.figure(figsize=(10,6))

axes = plt.gca()
axes.set_xlim([0,i])

plt.plot(r)

plt.show()


I am working on the file "Labs/Sample Python Regression Analysis.ipynb".
The first task is:
This time, our Gradient Descent has converged and stoped before interaction counter reaches the tolerance limit. Lets visualize the error rate behavior while performing the task and make sure it deacrases until the max interaction.
Can you write Python code for it?

plt.close('all')

plt.figure(figsize=(10,6))

axes = plt.gca()
axes.set_xlim([0,i])

plt.plot(r)

plt.show()


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Below we load the graph and extract the relevant tensors using get_tensor_by_name. These tensors reflect the input and outputs of the graph, or least the ones we care about for detecting objects.
Can you write Python code for it?

detection_graph = load_graph(SSD_GRAPH_FILE)
# detection_graph = load_graph(RFCN_GRAPH_FILE)
# detection_graph = load_graph(FASTER_RCNN_GRAPH_FILE)

# The input placeholder for the image.
# `get_tensor_by_name` returns the Tensor with the associated name in the Graph.
image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')

# Each box represents a part of the image where a particular object was detected.
detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')

# Each score represent how level of confidence for each of the objects.
# Score is shown on the result image, together with the class label.
detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')

# The classification of the object (integer id).
detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Exercise 4 - Model Tradeoffs
Download a few models from the model zoo and compare the timings.
Detection on a Video
Finally run your pipeline on this short video.
Can you write Python code for it?

# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML

HTML("""
<video width="960" height="600" controls>
  <source src="{0}" type="video/mp4">
</video>
""".format('driving.mp4'))


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Exercise 5 - Object Detection on a Video
Run an object detection pipeline on the above clip.
Can you write Python code for it?

clip = VideoFileClip('driving.mp4')

# TODO: Complete this function.
# The input is an NumPy array.
# The output should also be a NumPy array.
def pipeline(img):
    pass


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
MobileNets
MobileNets, as the name suggests, are neural networks constructed for the purpose of running very efficiently (high FPS, low memory footprint) on mobile and embedded devices. MobileNets achieve this with 3 techniques:

Perform a depthwise convolution followed by a 1x1 convolution rather than a standard convolution. The 1x1 convolution is called a pointwise convolution if it's following a depthwise convolution. The combination of a depthwise convolution followed by a pointwise convolution is sometimes called a separable depthwise convolution.
Use a "width multiplier" - reduces the size of the input/output channels, set to a value between 0 and 1.
Use a "resolution multiplier" - reduces the size of the original input, set to a value between 0 and 1.

These 3 techiniques reduce the size of cummulative parameters and therefore the computation required. Of course, generally models with more paramters achieve a higher accuracy. MobileNets are no silver bullet, while they perform very well larger models will outperform them.  MobileNets are designed for mobile devices, NOT cloud GPUs. The reason we're using them in this lab is automotive hardware is closer to mobile or embedded devices than beefy cloud GPUs.
Convolutions
Vanilla Convolution
Before we get into the MobileNet convolution block let's take a step back and recall the computational cost of a vanilla convolution. There are $N$ kernels of size $D_k * D_k$. Each of these kernels goes over the entire input which is a $D_f * D_f * M$ sized feature map or tensor (if that makes more sense). The computational cost is:
$$
D_f * D_f * M * N * D_k * D_k
$$
Let $D_g * D_g$ be the size of the output feature map. Then a standard convolution takes in a $D_f * D_f * M$ input feature map and returns a $D_g * D_g * N$ feature map as output.

Depthwise Convolution
A depthwise convolution acts on each input channel separately with a different kernel. $M$ input channels implies there are $M$ $D_k * D_k$ kernels. Also notice this results in $N$ being set to 1. If...
Can you write Python code for it?

def vanilla_conv_block(x, kernel_size, output_channels):
    """
    Vanilla Conv -> Batch Norm -> ReLU
    """
    x = tf.layers.conv2d(
        x, output_channels, kernel_size, (2, 2), padding='SAME')
    x = tf.layers.batch_normalization(x)
    return tf.nn.relu(x)

# TODO: implement MobileNet conv block
def mobilenet_conv_block(x, kernel_size, output_channels):
    """
    Depthwise Conv -> Batch Norm -> ReLU -> Pointwise Conv -> Batch Norm -> ReLU
    """
    pass


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Run detection and classification on a sample image.
Can you write Python code for it?

# Load a sample image.
image = Image.open('./assets/sample1.jpg')
image_np = np.expand_dims(np.asarray(image, dtype=np.uint8), 0)

with tf.Session(graph=detection_graph) as sess:
    # Actual detection.
    (boxes, scores, classes) = sess.run([detection_boxes, detection_scores, detection_classes],
                                        feed_dict={image_tensor: image_np})

    # Remove unnecessary dimensions
    boxes = np.squeeze(boxes)
    scores = np.squeeze(scores)
    classes = np.squeeze(classes)

    confidence_cutoff = 0.8
    # Filter boxes with a confidence score less than `confidence_cutoff`
    boxes, scores, classes = filter_boxes(confidence_cutoff, boxes, scores, classes)

    # The current box coordinates are normalized to a range between 0 and 1.
    # This converts the coordinates actual location on the image.
    width, height = image.size
    box_coords = to_image_coords(boxes, height, width)

    # Each class with be represented by a differently colored box
    draw_boxes(image, box_coords, classes)

    plt.figure(figsize=(12, 8))
    plt.imshow(image)


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Sample solution
Can you write Python code for it?

with tf.Session(graph=detection_graph) as sess:
    image_tensor = sess.graph.get_tensor_by_name('image_tensor:0')
    detection_boxes = sess.graph.get_tensor_by_name('detection_boxes:0')
    detection_scores = sess.graph.get_tensor_by_name('detection_scores:0')
    detection_classes = sess.graph.get_tensor_by_name('detection_classes:0')

    new_clip = clip.fl_image(pipeline)

    # write to file
    new_clip.write_videofile('result.mp4')

HTML("""
<video width="960" height="600" controls>
  <source src="{0}" type="video/mp4">
</video>
""".format('result.mp4'))


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Timing Detection
The model zoo comes with a variety of models, each its benefits and costs. Below you'll time some of these models. The general tradeoff being sacrificing model accuracy for seconds per frame (SPF).
Can you write Python code for it?

def time_detection(sess, img_height, img_width, runs=10):
    image_tensor = sess.graph.get_tensor_by_name('image_tensor:0')
    detection_boxes = sess.graph.get_tensor_by_name('detection_boxes:0')
    detection_scores = sess.graph.get_tensor_by_name('detection_scores:0')
    detection_classes = sess.graph.get_tensor_by_name('detection_classes:0')

    # warmup
    gen_image = np.uint8(np.random.randn(1, img_height, img_width, 3))
    sess.run([detection_boxes, detection_scores, detection_classes], feed_dict={image_tensor: gen_image})

    times = np.zeros(runs)
    for i in range(runs):
        t0 = time.time()
        sess.run([detection_boxes, detection_scores, detection_classes], feed_dict={image_tensor: image_np})
        t1 = time.time()
        times[i] = (t1 - t0) * 1000
    return times

with tf.Session(graph=detection_graph) as sess:
    times = time_detection(sess, 600, 1000, runs=10)

# Create a figure instance
fig = plt.figure(1, figsize=(9, 6))

# Create an axes instance
ax = fig.add_subplot(111)
plt.title("Object Detection Timings")
plt.ylabel("Time (ms)")

# Create the boxplot
plt.style.use('fivethirtyeight')
bp = ax.boxplot(times)


I am working on the file "Labs/Term3/Lesson 10 - 11/CarND-Object-Detection-Lab.ipynb".
The first task is:
Your solution should show the majority of the parameters in MobileNet block stem from the pointwise convolution.
MobileNet SSD
In this section you'll use a pretrained MobileNet SSD model to perform object detection. You can download the MobileNet SSD and other models from the TensorFlow detection model zoo. Paper describing comparing several object detection models.
Alright, let's get into SSD!
Single Shot Detection (SSD)
Many previous works in object detection involve more than one training phase. For example, the Faster-RCNN architecture first trains a Region Proposal Network (RPN) which decides which regions of the image are worth drawing a box around. RPN is then merged with a pretrained model for classification (classifies the regions). The image below is an RPN:

The SSD architecture is a single convolutional network which learns to predict bounding box locations and classify the locations in one pass. Put differently, SSD can be trained end to end while Faster-RCNN cannot. The SSD architecture consists of a base network followed by several convolutional layers:

NOTE: In this lab the base network is a MobileNet (instead of VGG16.)
Detecting Boxes
SSD operates on feature maps to predict bounding box locations. Recall a feature map is of size $D_f * D_f * M$. For each feature map location $k$ bounding boxes are predicted. Each bounding box carries with it the following information:

4 corner bounding box offset locations $(cx, cy, w, h)$
$C$ class probabilities $(c_1, c_2, ..., c_p)$

SSD does not predict the shape of the box, rather just where the box is. The $k$ bounding boxes each have a predetermined shape. This is illustrated in the figure below:

The shapes are set prior to actual training. For example, In figure (c) in the above picture there are 4 boxes, meaning $k$ = 4.
Exercise 2 - SSD Feature Maps
It would be a good exercise to read the SSD paper prior to a answering the following questions.
Q: Why does SSD use several differently sized feature maps to predict detections?
A: Your answer...
Can you write Python code for it?

# Frozen inference graph files. NOTE: change the path to where you saved the models.
SSD_GRAPH_FILE = 'ssd_mobilenet_v1_coco_11_06_2017/frozen_inference_graph.pb'
RFCN_GRAPH_FILE = 'rfcn_resnet101_coco_11_06_2017/frozen_inference_graph.pb'
FASTER_RCNN_GRAPH_FILE = 'faster_rcnn_inception_resnet_v2_atrous_coco_11_06_2017/frozen_inference_graph.pb'


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
Compute correlation coefficient (on Annual mean values)
Can you write Python code for it?

cor = numpy.corrcoef(df_glrm_subset.resample("A").mean().interpolate().tolist(),
                     df_modis_subset.resample("A").mean().interpolate().tolist())
print("correlation coefficient is: " , cor[0][1])


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
Create subsets of both vectors (altimetry and surface area) for the overlapping period
Can you write Python code for it?

df_glrm_daily = df["lake_height"].resample("D").mean().interpolate()
df_modis_daily = df_modis["Area"].resample("D").mean().interpolate()
start = max(df_glrm_daily.index[0], df_modis_daily.index[0])
end =  min(df_glrm_daily.index[-1],  df_modis_daily.index[-1])
print start, end
df_glrm_subset = df_glrm_daily[(df_glrm_daily.index >= start) & (df_glrm_daily.index <= end)]
df_modis_subset = df_modis_daily[(df_modis_daily.index >= start) & (df_modis_daily.index <= end)]
print len(df_glrm_subset), len(df_modis_subset)

fig, ax1 = plt.subplots()
t = df_glrm_subset.resample("W").mean().interpolate().index
s1 = df_glrm_subset.resample("W").mean().interpolate().tolist()
ax1.plot(t, s1, 'b-'); ax1.set_xlabel('time (s)')
# Make the y-axis label, ticks and tick labels match the line color.
ax1.set_ylabel('GLRM_10 Altimetry', color='b')
ax1.tick_params('y', colors='b'); ax2 = ax1.twinx()
s2 = df_modis_subset.resample("W").mean().interpolate().tolist()
ax2.plot(t, s2, 'r-'); ax2.set_ylabel('MODIS Surface Area', color='r')
ax2.tick_params('y', colors='r'); fig.tight_layout(); plt.title( "GRLM & MODIS Overlapping period (Weekly)"); plt.show()


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
GRLM Altimetry data from July 22 2008 to September 3, 2016
Create new columns of year, month, day in a convenient format
Can you write Python code for it?

df_grlm = pd.read_csv(GRLM, skiprows=43, delim_whitespace=True, names="mission,cycle,date,hour,minute,lake_height,error,mean(decibels),IonoCorrection,TropCorrection".split(","), engine='python', index_col=False)
def get_year(date): return int(str(date)[0:4])
def get_month(date): return int(str(date)[4:6])
def get_day(date): return int(str(date)[6:])
df_grlm['year'] = df_grlm['date'].apply(get_year)
df_grlm['month'] = df_grlm['date'].apply(get_month)
df_grlm['day'] = df_grlm['date'].apply(get_day)
df_grlm = df_grlm.where(df_grlm.minute < 61 ) # remove lines that do not have time
df_grlm = df_grlm.where(df_grlm.lake_height < 900 ) # remove entries that do not have lake-height

df_grlm.lake_height.plot(); plt.title("Actual data without resampling"); plt.ylabel("Variation (m)")


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
MODIS data
Can you write Python code for it?

"""import scipy.io
mat = scipy.io.loadmat('/home2/svimal/Github/Nina/data/lake_610/ts_GLWD1_241_h22v04.mat')
print mat.keys()
import numpy as np
np.shape(mat["miss_ts"].T), np.shape(mat["ts_water"])
miss_ts = np.squeeze(mat["miss_ts"].T)
ts_water = np.squeeze(mat["ts_water"])
miss_ts[0:3], ts_water[0:3]
#mat["miss_ts"][0][0:5], mat["ts_water"][0:5]"""


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
Observations:
Clearly from the above plot, we can see that the minima around 2011 and 2012 should perhaps correspond to the same time, so the correlation would improve if we use the actual dates. The general trend seems to be correlated well.
Compute correlation coefficient (on weekly mean values)
Can you write Python code for it?

cor = numpy.corrcoef(df_glrm_subset.resample("W").mean().interpolate().tolist(),
                     df_modis_subset.resample("W").mean().interpolate().tolist())
print "correlation coefficient is: " , cor[0][1]

fig, ax1 = plt.subplots()
t = df_glrm_subset.resample("A").mean().interpolate().index
s1 = df_glrm_subset.resample("A").mean().interpolate().tolist()
ax1.plot(t, s1, 'b-'); ax1.set_xlabel('time (s)')
# Make the y-axis label, ticks and tick labels match the line color.
ax1.set_ylabel('GLRM_10 Altimetry', color='b')
ax1.tick_params('y', colors='b'); ax2 = ax1.twinx()
s2 = df_modis_subset.resample("A").mean().interpolate().tolist()
ax2.plot(t, s2, 'r-'); ax2.set_ylabel('MODIS Surface Area', color='r')
ax2.tick_params('y', colors='r'); fig.tight_layout(); plt.title( "GRLM & MODIS Overlapping period"); plt.show()


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake610.ipynb".
The first task is:
This part has to be corrected with actual dates.
But in the meanwhile, I am assuming the dates are equally spaced at 10 days to somewhat match the area array length
One other way to do it would be to look at the number of images per month and interpolating inbetween, that would improve the accuracy much, but its better to use the actual dates
Can you write Python code for it?

df_modis = pd.read_csv(MODIS, names=["Dates", "Area"], sep="\t",engine='python', index_col=False)
#print df_modis.Dates.head(10),df_modis.Dates.tail(10)

print pd.date_range('1993-01-01', periods=len(df_modis), freq='10D')
print df_modis.Dates.head(1), df_modis.Dates.tail(1)
df_modis.index = pd.date_range('1993-01-01', periods=len(df_modis), freq='10D')
print df_modis.head(2)
df_modis.Area.plot(); plt.title("MODIS data from file " +str(MODIS).split("/")[-1]); plt.ylabel("MODIS Area")


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake_345.ipynb".
The first task is:
Add time information to the dataframe
Can you write Python code for it?

df = df_grlm
df[["year", "month", "day", "hour", "minute"]] = df[["year", "month", "day", "hour", "minute"]].fillna(0).astype(int)
df['Time'] = df.year.astype(str).str.cat(df.month.astype(str).astype(str), sep='-').str.cat(df.day.astype(str), sep='-')\
                .str.cat(df.hour.astype(str).astype(str), sep='-').str.cat(df.minute.astype(str).astype(str), sep='-')
df = df.where(df.year>10) # to ger rid of all the nan values
df.index = pd.to_datetime(pd.Series(df["Time"]), format="%Y-%m-%d-%H-%M");
print df.index[0:3], df.index[-3:]


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake_345.ipynb".
The first task is:
Create subsets of both vectors (altimetry and surface area) for the overlapping period
Can you write Python code for it?

df_glrm_subset = df["lake_height"].resample("D").mean().interpolate()
df_glrm_subset = df_glrm_subset[(df_glrm_subset.index > '2008-07-22') & (df_glrm_subset.index <= '2015-08-13')]
df_glrm_subset.plot(); plt.legend(); plt.title("Subset of Altimetry"); plt.ylabel("Variation (m)")
df_glrm_subset.index

df_modis_daily = df_modis["Area"].resample("D").mean().interpolate()
df_modis_subset = df_modis_daily[(df_modis_daily.index > '2008-07-22') & (df_modis_daily.index <= '2015-08-13')]
df_modis_subset.plot()
df_modis_subset.index

# QA: Create a time series of time alone, to check the number of data points that we should have for days.
#Note the vaiable called length
print pd.date_range('22/07/2008', periods=len(df_modis_subset), freq='D')

# Check if the two vectors are of the same length
print len(df_glrm_subset.tolist()), len(df_modis_subset.tolist())


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake_345.ipynb".
The first task is:
GRLM Altimetry data from July 22 2008 to September 3, 2016
Create new columns of year, month, day in a convenient format
Can you write Python code for it?


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake_345.ipynb".
The first task is:
Interpolate the missing data points
Can you write Python code for it?

df_grlm.lake_height.interpolate().plot(); plt.title("Interpolated Actual data without resampling"); plt.ylabel("Variation (m)")


I am working on the file "LakeArea_Altimetry/Altimetry_MODIS_SurfaceArea_lake_345.ipynb".
The first task is:
Resample the data to get monthly and annual variation in lake height
Can you write Python code for it?

df["lake_height"].resample("M").mean().plot(); plt.title("Mean Monthly Altimetry"); plt.ylabel("Variation (m)")

df["lake_height"].resample("A").mean().plot(); plt.title("Mean Annual Altimetry"); plt.ylabel("Variation (m)")


I am working on the file "Lambda Functions.ipynb".
The first task is:
Any / All
Can you write Python code for it?

lst = [True, True, False, True]
all(lst)

any(lst)


I am working on the file "Lambda Functions.ipynb".
The first task is:
Complex
complex() returns a complex number with the value real + imag*1j or converts a string or number to a complex number.
Can you write Python code for it?

complex()  # if no arguments are given results in 0j

complex(2,4)

complex('2') # if imag part is omitted substitutes with 0j

complex('2','3') # this will create an error as the second parameter cannot be a string.

complex('1+2j')

complex('3',2)  # cannot take a secord argument if first is a strink


I am working on the file "Lambda Functions.ipynb".
The first task is:
Enumerate Function
Return an enumerate object. iterable must be a sequence, an iterator, or some other object which supports iteration. Returns a tuple containing a count (from start which defaults to 0) and the values obtained from iterating over iterable.
Can you write Python code for it?

colors = ['Blue','Black','White','Red']
list(enumerate(colors))

list(enumerate(colors,1))

for index,item in enumerate(colors):
    print(index)
    print(item)

for i,j in enumerate(colors):
    if i == 2:
        break
    print(j)


I am working on the file "Lambda Functions.ipynb".
The first task is:
Filter Function
The function filter(function,list) offers an elegant way to filter out all the elements of a list, for which the function function returns True.
The function filter(f,1) needs a function f as its first argument. f returns a Boolean value i.e. either True or False. This function will be applied to every element of the list l. Only if f returns True will the element of the list included in the resultset.
Can you write Python code for it?

nums = [2,3,4,7,9,10]
evens = list(filter(lambda x: x%2==0,nums))
print(evens)


I am working on the file "Lambda Functions.ipynb".
The first task is:
Reduce Function
reduce(func,seq)
If seq = [s1,s2,s3,...,sn], calling reduce(func,seq) works like this:
* At first the two elements of seq will be applied to func, i,e func(s1,s2). The list on which reduce() works looks now like this: [func(s1,s2),s3,...,sn]
* In the next step func will be applied on the previous result and the third element on the list, i.e.func(func(s1,s2),s3). The list looks like this now: [func(func(s1,s2),s3),...,sn]
* Continue like this until just one element is left and return this element as the result of reduce()
Can you write Python code for it?

from functools import reduce
reduce(lambda x,y: x+y, [47,23,11,34])

from functools import reduce
reduce(lambda x,y: x if (x > y) else y, [47,12,33,95])

from functools import reduce
reduce(lambda x,y: x * y, range(1,10,2))


I am working on the file "Lambda Functions.ipynb".
The first task is:
This makes sense because simply iterating through the dictionaries will result in just the keys. We would have to call methods to mix keys and values
Can you write Python code for it?

def swapdic(d1,d2):
    dout = {}

    for d1key,d2val in zip(d1,d2.values()):
        dout[d1key] = d2val

    return dout

swapdic(d1,d2)


I am working on the file "Lambda Functions.ipynb".
The first task is:
Zip with Dictionary
Can you write Python code for it?


I am working on the file "Land cover representation.ipynb".
The first task is:
Calculate area percentages
Can you write Python code for it?

area_percentages = pandas.DataFrame({'name': [r['name'].lower() for r in LandCoverType.objects.filter(id__lte=15).values()],
                                     'study_area': get_percentages(study_landcover)}, index=range(1, 16))

for i, radius in enumerate(radii):
    area_percentages["{0}m".format(radius)] = get_percentages(location_landcover_list[i])

area_percentages

area_percentages.to_csv(path.join(output_filepath, "area_percentages.csv"))


I am working on the file "Land cover representation.ipynb".
The first task is:
query database
Can you write Python code for it?

location_landcover_list = []
for location_area, radius in zip(location_area_list, radii):
    location_landcover = get_landcover(location_area)
    location_landcover_list.append(location_landcover)

    # export result to a GeoJSON file
    #location_landcover.to_file(filename=path.join(output_filepath, "location_landcover_{0}m.geojson".format(radius)),
    #                           driver='GeoJSON', schema=schema)


I am working on the file "LandauPole.ipynb".
The first task is:
Automatic search and plotting
Can you write Python code for it?

def find_pole(mchi=375, msig=10, mphi=None, mTHD=750, upto=1e10, invpole=1e-4, npts=3000, twoloop=True):
    """Return Landau pole and  RGE curves for theory."""
    if not mphi:
        mphi = mchi
    threshs, cRGE = precalcRGE(thresholds=[(mchi, RealScalar(1,7,0)), (mphi, ComplexScalar(1,5,-2)), (mTHD, ComplexScalar(1,2,1)), (msig*1e3, Weyl(1,5,0,ng))])
    def func(y, t):
        """Derivative of inverse alpha.
        d inv_alpha/dt = c0 + c1/inv_alpha
        """
        c0, c1 = cRGE[np.searchsorted(threshs, t)-1]
        if twoloop:
            return c0 + c1.dot(1/y)
        else:
            return c0
    ystart = y0
    ts = np.linspace(0, t(upto), npts)
    while np.any(abs(ystart) > invpole):
        ys = odeint(func, y0, ts)
        try:
            last = np.where(ys<invpole)[0][0]-1   # note that negative will also satisfy here
        except IndexError:
            break
        ystart = ys[last]
        ts = np.linspace(ts[last], ts[last+1], npts)
    #print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[last])))
    pole = mu(ts[last])
    ts = np.linspace(0, ts[last], npts)
    ys = odeint(func, y0, ts)
    return pole #, ts, ys


I am working on the file "LandauPole.ipynb".
The first task is:
Landau poles for various models
Definitions and initial values
To find pole we search for zero of $\alpha_{i}^{-1}$, using  formula
$$ \frac{d\alpha_{i}^{-1}}{d t} = - a_i + \frac{b_{ij}}{4\pi\alpha_{j}^{-1}}$$
Proper procedure to numerically find zero:
https://mail.scipy.org/pipermail/scipy-user/2005-March/004248.html
We'll use more primitive one, just graphically checking that abscissa is crossed and then looking for smallest alpha.
Can you write Python code for it?

# Starting point for all evolutions
MZ = 91.188
alpha3_0 = 0.1173
alpha2_0 = 0.03374
alpha1_0 = 0.016923
y0 = np.array([1/alpha1_0, 1/alpha2_0, 1/alpha3_0])

def t(mu):
    """Evolution variable t in terms of mu [Gev]."""
    return math.log(mu/MZ)/(2*pi)

def mu(t):
    """Inverse of t(mu)"""
    return MZ*math.exp(2*pi*t)

def precalcRGE(thresholds=[]):
    """Precalculates RGE coeffs.

    Element of thresholds list: (mu, [irrep1, irrep2, ...])
    with irreps entering evolution at scale mu [GeV].
    mu's should increase along the list.
    Returns: array of t-thresholds and corresponding
             arrays of RGE coeffs -a_i and
             matrices b_ij / (4pi).
    """
    ts = [t(MZ)]
    cRGE = [(-rge1LM(), -rge2LM()/(4*pi))]
    active_BSM_fermions = []
    active_BSM_scalars = []
    thresholds.sort(key=lambda it: it[0])
    for mu, reps in thresholds:
        if type(reps) != list: reps=[reps]
        for rep in reps:
            if rep.__class__.__name__[-4:] == 'alar':
                active_BSM_scalars.append(rep)
            else:
                active_BSM_fermions.append(rep)
        ts.append(t(mu))
        cRGE.append((-rge1LM(BSMfermions=active_BSM_fermions, BSMscalars=active_BSM_scalars), -rge2LM(BSMfermions=active_BSM_fermions, BSMscalars=active_BSM_scalars)/(4*pi)))
    return np.array(ts), cRGE

def rgefig(ts, ys):
    logms = np.array([math.log10(mu(t)) for t in ts])
    fig, ax = plt.subplots(figsize=[4,3])
    ax.plot(logms, ys[:,0], 'b--', label=r'$\alpha_{1}^{-1}$')
    ax.plot(logms, ys[:,1], 'r-', label=r'$\alpha_{2}^{-1}$')
    props = dict(color="green", linestyle="-", linewidth=1)
    ax.axhline(y=0, **props)
    ax.set_ylabel(r'$\alpha_{i}^{-1}$', fontsize='14')
    ax.set_xlabel(r'$\log_{10}(\mu/{\rm GeV})$', fontsize='14')
    ax.legend(loc='upper right').draw_frame(0)
    plt.show()


I am working on the file "LandauPole.ipynb".
The first task is:
One-loop RGE running
Can you write Python code for it?

mu_max = 1e8  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = False
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("One-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


mu_max = 1e33  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = False
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("One-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,0]).argmin()])))


mu_max = 1e6  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = False
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("One-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


I am working on the file "LandauPole.ipynb".
The first task is:
RGE coefficients
Definition of 1- and 2-loop RGE
Can you write Python code for it?

ng = Symbol('n_g')  # number of generations
nh = Symbol('n_h')  # number of SM Higgs doublets = 1
# we count only left-handed particles
SMfermions = [
    Weyl(1, 2, -1, ng),  # left lepton doublet
    Weyl(1, 1,  2, ng),  # anti-right lepton singlet
    Weyl(3, 2, Rational(1,3), ng),  # left quark doublet
    Weyl(3, 1, Rational(-4,3), ng), # anti-right up quark singlet
    Weyl(3, 1, Rational(2,3), ng) # anti-right down quark singlet
    ]
SMscalars = [
    ComplexScalar(1, 2, 1, nh)     # Higgs doublet
    ]

def rge1L(N, BSMfermions=[], BSMscalars=[], num=False):
    """One loop RGE coefficient."""

    a = Rational(-11,3)*CA(N)
    for rep in SMfermions + BSMfermions:
        a += Rational(4,3) * rep.wS2(N)
    for rep in SMscalars + BSMscalars:
        a += Rational(1,3) * rep.wS2(N)
    if num:
        return float(a.subs({ng:3, nh:1}))
    else:
        return a

def rge1LM(BSMfermions=[], BSMscalars=[]):
    """All three one-loop RGE coefficients."""
    return np.array([rge1L(N, BSMfermions=BSMfermions, BSMscalars=BSMscalars, num=True) for N in [1,2,3]])

def rge2L(N, M, BSMfermions=[], BSMscalars=[], num=False):
    """Two-loop RGE coeffs. Same syntax as for one-loop."""

    b = 0
    if M != N:
        for rep in SMfermions + SMscalars + BSMfermions + BSMscalars:
            b += 4 * rep.wS2(N) * rep.C2(M)
    if M == N:
        b += Rational(-34,3)*CA(N)**2
        for rep in SMfermions + BSMfermions:
            b += rep.wS2(N)*(4*rep.C2(N)+Rational(20,3)*CA(N))
        for rep in SMscalars + BSMscalars:
            b += rep.wS2(N)*(4*rep.C2(N)+Rational(2,3)*CA(N))
    if num:
        return float(b.subs({ng:3, nh:1}))
    else:
        return b

def rge2LM(BSMfermions=[], BSMscalars=[]):
    """Whole matrix of two-loop RGE coefficients."""
    return np.array([[rge2L(N, M, BSMfermions=BSMfermions, BSMscalars=BSMscalars, num=True) for M in [1,2,3]] for N in [1,2,3]])


I am working on the file "LandauPole.ipynb".
The first task is:
So we are beyond Planck mass ...
ČKP model
Can you write Python code for it?

threshs, cRGE = precalcRGE(thresholds=[(1e3, ComplexScalar(1,2,1)), (10e3, Weyl(1,5,0,ng)), (5*10e3, [ComplexScalar(1,5,-2),RealScalar(1,7,0)])])

def func(y, t, twoloop=True):
    """Derivative of inverse alpha.
    d inv_alpha/dt = c0 + c1/inv_alpha
    """
    c0, c1 = cRGE[np.searchsorted(threshs, t)-1]
    if twoloop:
        return c0 + c1.dot(1/y)
    else:
        return c0


I am working on the file "LandauPole.ipynb".
The first task is:
Testing two loop RGE coefs with SARAH
SM
Can you write Python code for it?


I am working on the file "LandauPole.ipynb".
The first task is:
This agrees with Aristizabal Sierra et al's 5.e6 GeV.
ČKP model (with low-lying 5-plet scalar)
Can you write Python code for it?

threshs, cRGE = precalcRGE(thresholds=[(400, ComplexScalar(1,5,-2)), (750, ComplexScalar(1,2,1)), (10e3, Weyl(1,5,0,ng)), (5*10e3, RealScalar(1,7,0))])

def func(y, t, twoloop=True):
    """Derivative of inverse alpha.
    d inv_alpha/dt = c0 + c1/inv_alpha
    """
    c0, c1 = cRGE[np.searchsorted(threshs, t)-1]
    if twoloop:
        return c0 + c1.dot(1/y)
    else:
        return c0


I am working on the file "LandauPole.ipynb".
The first task is:
This is factor 2 below Aristizabal Sierra et al's 6e5 GeV from their Fig. 6.
Triplet neutrino mass model
Can you write Python code for it?

threshs, cRGE = precalcRGE(thresholds=[(0.375e3, ComplexScalar(1,1,2)), (0.375e3, Weyl(1,2,1, ng)), (0.375e3, Weyl(1,2,-1, ng)), (0.375e3, RealScalar(1,3,0))])

def func(y, t, twoloop=True):
    """Derivative of inverse alpha.
    d inv_alpha/dt = c0 + c1/inv_alpha
    """
    c0, c1 = cRGE[np.searchsorted(threshs, t)-1]
    if twoloop:
        return c0 + c1.dot(1/y)
    else:
        return c0


I am working on the file "LandauPole.ipynb".
The first task is:
This is in nice agreement with Di Luzio et al's 9e28. arXiv:1504.00359
Can you write Python code for it?

mu_max = 3.5e21  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = True
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


I am working on the file "LandauPole.ipynb".
The first task is:
This is in nice agreement with Di Luzio et al. who have 4e21 GeV
SM + real scalar (1,7,0)
Can you write Python code for it?

threshs, cRGE = precalcRGE(thresholds=[(2.5e4, RealScalar(1,7,0))])

def func(y, t, twoloop=True):
    """Derivative of inverse alpha.
    d inv_alpha/dt = c0 + c1/inv_alpha
    """
    c0, c1 = cRGE[np.searchsorted(threshs, t)-1]
    if twoloop:
        return c0 + c1.dot(1/y)
    else:
        return c0

mu_max = 1e42  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = False
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("One-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,0]).argmin()])))


I am working on the file "LandauPole.ipynb".
The first task is:
Two-loop RGE running
Can you write Python code for it?

mu_max = 2.83e5  # we integrate up to this scale in GeV
npts = 2000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = True
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


mu_max = 4.55e6  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = True
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


mu_max = 2.331e6  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = True
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


mu_max = 7.8e5  # we integrate up to this scale in GeV
npts = 1000     # number of integration points
ts = np.linspace(0, t(mu_max), npts)

twoloop = True
ys = odeint(func, y0, ts, args=(twoloop,))

rgefig(ts, ys)

print("Two-loop pole is at {:.2g} GeV.".format(mu(ts[abs(ys[:,1]).argmin()])))


I am working on the file "LandauPole.ipynb".
The first task is:
What follows is Fig. 6 from http://arXiv.org/pdf/1603.04723
Can you write Python code for it?

mrats = np.linspace(1,10, 5)
Lpoles = []
for mr in mrats:
    pole = find_pole(mchi=mr*1e4, mphi=mr*1e4, msig=10e3, mTHD=1e3, upto=1e18, npts=2000)
    Lpoles.append(pole/1e6)
plt.plot(mrats, Lpoles)
plt.annotate(r"$m_\Sigma = 10\, {\rm TeV}$", xy=(0.65, 0.28), xycoords='axes fraction', fontsize=16)
plt.xlabel(r'$m_{\chi,\phi}/m_\Sigma}$', fontsize=16)
fig = plt.ylabel(r"$\Lambda_{\rm Landau} \; [10^6\,{\rm GeV}]$", fontsize=16)

from math import floor, log10
import matplotlib.ticker as ticker

# Define function for string formatting of scientific notation
def sci_notation(num, decimal_digits=1, precision=None, exponent=None):
    """
    Returns a string representation of the scientific
    notation of the given number formatted for use with
    LaTeX or Mathtext, with specified number of significant
    decimal digits and precision (number of decimal digits
    to show). The exponent to be used can also be specified
    explicitly.
    """
    if not exponent:
        exponent = int(floor(log10(abs(num))))
    coeff = round(num / float(10**exponent), decimal_digits)
    if not precision:
        precision = decimal_digits

    return r"${0:.{2}f}\cdot10^{{{1:d}}}$".format(coeff, exponent, precision)

def fmtGeV(x, pos):
    """Scientific notation formatter."""
    a, b = '{:.1e}'.format(x).split('e')
    b = int(b)
    return r'${} \times 10^{{{}}}\,{{\rm GeV}}$'.format(a, b)

np_find_pole = np.frompyfunc(find_pole, 2, 1)

%%time
resolution = 40  # of calculation grid
xs = np.linspace(375, 2000, resolution)
ys = np.linspace(0.375, 15, resolution)
#levels = [3, 6, 9]
X, Y = np.meshgrid(xs, ys)
Z = np_find_pole(X, Y)
fig, ax = plt.subplots(figsize=(4,4))
CS = plt.contour(X, Y, Z, cmap=plt.cm.viridis)
#CS = plt.contour(X, Y, Z, locator=plt.Locator(), cmap=plt.cm.viridis)
#CS = plt.contour(X, Y, Z, levels, cmap=plt.cm.viridis)
fig = plt.clabel(CS, inline=1, fmt=ticker.FuncFormatter(fmtGeV), fontsize=12, colors='black')
#fig = plt.clabel(CS, inline=1, fmt=r'$%.0g \;{\rm GeV}$', fontsize=12,...


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:

Plot energy absorbed versus segment depth to make sure everything looks good:

Can you write Python code for it?

plt.figure(figsize=(12,6))

plt.plot(np.multiply(deptharray[1],10**6),np.multiply(Eabsorbed_Segments[0],10**3),label='338 keV')
plt.plot(np.multiply(deptharray[1],10**6),np.multiply(Eabsorbed_Segments[1],10**3),label='391 keV')
plt.plot(np.multiply(deptharray[1],10**6),np.multiply(Eabsorbed_Segments[2],10**3),label='440 keV')
plt.plot(np.multiply(deptharray[1],10**6),np.multiply(Eabsorbed_Segments[3],10**3),label='485 keV')

plt.xlabel('Depth (um)')
plt.ylabel('Energy absorbed (keV)')
plt.legend()


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:

and plot the results:

Can you write Python code for it?

mpl.rcParams.update({'font.size': 24, 'font.family': 'serif'})
sns.set(context='poster',font_scale=1.5)
sns.set_style("darkgrid")
sns.set_palette(palette='deep')
sns.set_color_codes(palette='deep')

plt.figure(figsize=(12,6))
plt.plot(np.multiply(KEsim,1.0),MCNPSignalRatio[0],linewidth=2,label='1 '+ u'\u03bcm')
plt.plot(np.multiply(KEsim,1.0),MCNPSignalRatio[1],linewidth=2,label='2 '+ u'\u03bcm')
plt.plot(np.multiply(KEsim,1.0),MCNPSignalRatio[4],linewidth=2,color='y',label='5 '+ u'\u03bcm')
plt.plot(np.multiply(KEsim,1.0),MCNPSignalRatio[9],linewidth=2,color='c',label='10 '+ u'\u03bcm')
#plt.plot(np.multiply(KEsim,1.0),MCNPSignalRatio[14],linewidth=2,color='purple',label='MCNP 15 '+ u'\u03bcm photoactive region')
plt.plot(np.divide(ChosenEnergies,10**6*q),SignalRatio,linewidth=2,color='r',marker='o',label='Experiment')
plt.xlabel('Electron Energy (keV)')
plt.ylabel('Signal Ratio (No Vinyl / Vinyl)')
plt.xlim(0.33,0.50)
plt.legend(title='Photoactive Region Thickness')
plt.subplots_adjust(left=0.14,bottom=0.15) #Adjust spacing to prevent clipping of x and y labels
#plt.savefig('Signal_Ratio.svg')


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Calculate signal ratios
Can you write Python code for it?

yM = 0.5 #magnet edge position in inches
CCDpos = 3.02 #CCD y-position relative to magnet edge in mm

yM = yM*.0254 #Convert to base units
CCDpos = CCDpos*10**-3 #Convert to base units

yf = yM + CCDpos #Screen position

KEcalcGeneralVec = np.vectorize(KEcalcGeneral) #Vectorize espec function for KE

ChosenPixels = [479.37419602, 658.07909036, 805.53608568, 929.62974462, 1079.52432829, 1325.661891,\
                1454.37508809, 1637.97442378, 1801.34325342, 1984.12923212, 2202.34572023, 2412.63089924,\
                2595.3896852]
ChosenPositions = np.multiply(np.add(np.multiply(ChosenPixels,8.0/10**3),9.5),10**-3) #Convert to meters
ChosenEnergies = KEcalcGeneralVec(ChosenPositions,yM,yf) #Energies corresponding to the chosen pixels

SignalRatio = LanexInterpFunc(ChosenPixels)/VinylInterpFunc(ChosenPixels)
print(ChosenEnergies/(1000*q),'\n')
print(SignalRatio)


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Data read functions
Can you write Python code for it?

def getfns(folder,ext=''):
    """Get a list of full path filenames for all files in a folder and subfolders for the given extension"""
    fns = []
    for file in os.listdir(folder):
            if file.endswith(ext):
                fns.append(os.path.join(folder,file))
    return fns

def readcsv(filename):
    """Read in a csv file and load it as an array"""
    return np.loadtxt(open(filename, "rb"), delimiter=",")

def readcsvs(filenames):
    """Read in multiple csv files and load them in to a two-dimensional array"""
    template = readcsv(filenames[0])
    numfiles = len(filenames)
    Data = np.zeros([numfiles,len(template)])
    for i in range(numfiles):
        spectrum = readcsv(filenames[i])
        Data[i,:] = spectrum
    return Data

def DataClean(Data):
    """Read in data and clean it by removing rows that have saturated pixel values"""
    maxes = np.max(Data[:,500:],1)
    includes = maxes<(2**16-1)
    rejects = maxes>(2**16-2)
    CleanData = Data[includes,:]
    return CleanData

def DataAverage(Data):
    """Average input 2d array into a 1d array"""
    return np.mean(Data,0)


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Espec functions
Can you write Python code for it?

B0 = 1710.0/10**4 #Magnetic field strength in Tesla

def KEcalc(z,y):
    """Returns KE in J given z-position in m"""
    return me*c**2*(sqrt((q*B0/(me*c))**2*((z**2+y**2)/(2*y))**2+1)-1)

def Radius(KE):
    """#Radius of electron orbit in m given KE in keV"""
    return me*c/(q*B0)*sqrt((KE*1000*q/(me*c**2)+1)**2-1)

def zfcalc(KE,y):
    """#Returns z-position at screen in inches given KE in keV"""
    R = Radius(KE)
    return sqrt(R**2 - (y-R)**2)

def zfcalcGeneral(KE,yM,y):
    R = Radius(KE)
    zM = zfcalc(KE,yM)
    return zM + (y - yM)*(R - yM)/zM

def KEcalcGeneral(zf,yM,yf):
    """Returns KE in J given z-position of electrons, y-position of magnet edge, and y-position of screen, all in m"""
    a = (yM+yf)**2
    b = -2*yM*(yf*(yM+yf)+zf**2)
    d = yM**2*(zf**2+yf**2)
    f = (me*c)/(q*B0)
    g = (-b+sqrt(b**2-4*a*d))/(2*a)
    return me*c**2*(sqrt(g**2+f**2)/f - 1)

def AngleIncidence(KE,yM):
    R = Radius(KE)
    return asin((R-yM)/R)


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Mesh plot all unsaturated data
Can you write Python code for it?

sns.set(font_scale=1.5)

fig1 = plt.figure(figsize=(8,6))
ax1 = fig1.add_subplot(111)
ax1.set_xlim(500,3648)
ax1.set_ylim(0,len(MagnetInData))
ax1.set_xlabel('Pixel')
ax1.set_ylabel('Instance')

# mesh1 = ax1.pcolormesh(MagnetInData, cmap='hot',vmin=0, vmax=10000)
mesh1 = ax1.pcolormesh(MagnetOutDataClean, cmap='inferno',vmin=0, vmax=65535)


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Plot averaged data
Can you write Python code for it?

MagnetOutDataCleanAvg = DataAverage(MagnetOutDataClean)
MagnetOutDataCleanAvgHat = savitzky_golay(MagnetOutDataCleanAvg,51,3) #Smoothed Data

fig2 = plt.figure(figsize=(12,8))
ax2 = fig2.add_subplot(111)
ax2.set_xlim(0,3648)
ax2.set_ylim(0,65535)
ax2.set_xlabel('Pixel Number')
ax2.set_ylabel('Pixel Value')
#ax2.semilogy()

ax2.plot(MagnetOutDataCleanAvg,linewidth=1)
ax2.plot(MagnetOutDataCleanAvgHat,linewidth=1,color='r')


I am working on the file "Lanex_Strip_Test.ipynb".
The first task is:
Plot interpolated data
Can you write Python code for it?

# sns.set(context='poster',font_scale=1.5)
# sns.set_style("darkgrid")
# sns.set_palette(palette='deep')
# sns.set_color_codes(palette='deep')
plt.figure(figsize=(12,6))

#With Lanex
#plt.plot(LanexPixelArray,LanexData,linestyle='None',marker='.')
plt.plot(LanexPixelArrayInterp,LanexDataInterp, linewidth=2,linestyle='--',label='Without Vinyl',color='b')

#Without Lanex
#plt.plot(VinylPixelArray,VinylData,linestyle='None',marker='.')
plt.plot(VinylPixelArrayInterp,VinylDataInterp,linewidth=2,linestyle='--',label='With Vinyl',color='r')

#All smoothed data
plt.plot(MagnetOutDataCleanAvgHat,linewidth=2,label='All Data',color='g')

plt.xlim(0,2000)
plt.xlabel('Pixel Number')
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
plt.ylabel('Pixel Value')
plt.legend()
plt.subplots_adjust(left=0.12,bottom=0.14) #Adjust spacing to prevent clipping of x and y labels
plt.savefig('strip_test.svg')


I am working on the file "Language Recognition Neural Net- Training and Testing.ipynb".
The first task is:
It really didn't do very well with French at all...
It seemed to mix it up with English a lot. I am not sure why the confusion is so uneven...
Can you write Python code for it?

# Mandarin Words
# Note I am typing in pinyin with no tones
mandarin_words = "xuexi zhongwen zhende hen nan".split()
mandarin_words += "wo hen xihuan pinyin yinwei bangzhu wo kanshu de bijiao rongyi".split()
mandarin_words += "sishisi jiu shi tebie nan shuochulai".split()
mandarin_words += "qilai, bu yuan zuo nuli de renmen!".split()  # Gotta please the censors ;)
predictions_table(mandarin_words, answers=[2]*len(mandarin_words))


I am working on the file "Language Recognition Neural Net- Training and Testing.ipynb".
The first task is:
Make the neural net
Can you write Python code for it?

# Input and output variables
x = tf.placeholder(tf.float32, [None, feature_length])
y_ = tf.placeholder(tf.float32, [None, num_of_languages])

# Define the number of neurons in each layer
layer_lengths = [feature_length, 40, num_of_languages]

# Create each layer
neural_net = []
last_output = x
for i, current_layer_length in enumerate(layer_lengths[1:]):

    # Define the length of the last layer
    last_layer_length = layer_lengths[i]

    # Create the variables for this layer
    W = tf.Variable(tf.truncated_normal([last_layer_length, current_layer_length],
                                        stddev=1 / math.sqrt(last_layer_length)))
    b = tf.Variable(tf.constant(0.1, shape=[current_layer_length]))
    h = tf.sigmoid(tf.matmul(last_output, W) + b)

    # Store the variables for this layer
    neural_net.append((W, b, h))

    # Update the last output
    last_output = h

# Output layer (softmax)
y = tf.nn.softmax(last_output)

# Scoring (use cross-entropy storing)
cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), axis=1))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)


I am working on the file "Language Recognition Neural Net- Training and Testing.ipynb".
The first task is:
Test the neural net with some words I made up
This isn't the most rigorous source of test data admittedly, but oh well
Can you write Python code for it?

# English words
english_words = "hello my dear chap let's have a bit of coffee".split()
english_words += "oh my heavens look at what this neural net can do".split()
english_words += "it looks like english words are often quite similar to french ones".split()
predictions_table(english_words, answers=[0]*len(english_words))


I am working on the file "LanguageDetectionModel.ipynb".
The first task is:
Test the model trained during the last step.
Set the name of the training file in test_file_name below:
Can you write Python code for it?

test_file_name = '/Users/frank/data/LanguageDetectionModel/europarl.test'
lo2 = lbw.LangByWord.load_object_from_file(object_file)
lo2.test_on_test(test_file_name, report_freq=0)


I am working on the file "LanguageDetectionModel.ipynb".
The first task is:
Train the language detection model and save as an object.
Can you write Python code for it?

lo = lbw.LangByWord()
lo.train(base_output_dir, max_words_per_lang=0, report_freq=0)
lo.print_most_prob_words()
object_file = 'LbyW_obj.pck'
lo.save_object_to_file(object_file)


I am working on the file "Language_European_Small-final.ipynb".
The first task is:
As explained earlier each document has a CHAPTER ID and SPEAKER ID within the 'HTML' kind of tags and they pretty much don't add anything to the model and its important to remove them.
There are many punctuation marks, symbols and numbers that are totally not necessary for the ML model to know, So let's remove them.
Can you write Python code for it?

df["Data"]

#with open('C:/Users/Sathvik/Desktop/DS/europarl-test/europarl_test.txt',encoding=" utf-8") as f2:
#    lines = f2.readlines()

df_test=pd.read_csv('C:/Users/Sathvik/Desktop/DS/europarl-test/europarl_test.txt',encoding=" utf-8",sep='\t',header=None)

df_test.columns=["Label_test","Data_test"]

df_test["Data_test"]=df_test["Data_test"].apply(remove_strings)

df_test

import matplotlib.pyplot as plt
import numpy as np
import scipy
import seaborn as sns

from sklearn import ensemble
from sklearn import feature_extraction
from sklearn import linear_model
from sklearn import pipeline
from sklearn import cross_validation
from sklearn import metrics


I am working on the file "Language_European_Small-final.ipynb".
The first task is:
Model building and what worked and what could have worked!
The most interesting part! I first took  linear logistic regression as my model for classification. Scikit learn has a beautiful way of wrapping everything in  a pipeline and building the model.
When it comes to dealing with text, you need to convert to a group of vectors and then input to your model. Here I have chosen the tfidf vectorizer. More about tfidf vectorizer? http://www.markhneedham.com/blog/2015/02/15/pythonscikit-learn-calculating-tfidf-on-how-i-met-your-mother-transcripts/, Nice way to learn about it.
I have used my anlayser to be character based as opposed to word based, the reason being some of these languages are very close to each other and it's best to use character based when using a language detection.
Can you write Python code for it?

vectorizer = feature_extraction.text.TfidfVectorizer(ngram_range=(1, 6),
                             analyzer='char',)

pipe = pipeline.Pipeline([
    ('vectorizer', vectorizer),
    ('clf', linear_model.LogisticRegression())
])

pipe.fit(df["Data"], df["Label"])

y_predicted = pipe.predict(df_test["Data_test"])

cm = metrics.confusion_matrix(df_test["Label_test"], y_predicted)

a=(y_predicted==df_test["Label_test"])

accuracy=sum(a)/len(a)


I am working on the file "Language_European_Small-final.ipynb".
The first task is:
Uncomment the following to execute Xgboost Classifier
Can you write Python code for it?

#import xgboost as xgb

#from xgboost.sklearn import XGBClassifier

#vectorizer = feature_extraction.text.TfidfVectorizer(ngram_range=(1, 6),analyzer='char',)

#pipe_xgb = pipeline.Pipeline([('vectorizer', vectorizer),('clf',  XGBClassifier())])

#pipe_xgb.fit(df["Data"], df["Label"])


I am working on the file "Laplace Transform.ipynb".
The first task is:
A more complicated transformoration
$$
f(t) = 5 t^2 \cos (3 t + \frac{\pi}{4})
$$
Can you write Python code for it?

t = sympy.symbols('t', positive=True)
s = sympy.symbols('s', complex=True)
f = 5 * t**2 * sympy.cos(3*t + sympy.pi/4)
f

F = sympy.laplace_transform(5*t**2*sympy.cos(t), t, s, noconds=True)
F


I am working on the file "Laplace Transform.ipynb".
The first task is:
Now just inverse Laplace to find the solution
Can you write Python code for it?

sol = sympy.inverse_laplace_transform(XofS, s, t)
sol


I am working on the file "Laplace Transform.ipynb".
The first task is:
Now lets define the EOMs
sympy doesn't handle undefined functions in the Laplace transform yet, so we will input the EOMS in the Laplace domain ourselves.
https://github.com/sympy/sympy/issues/7219#issuecomment-154768904
Can you write Python code for it?

eoms = sympy.Eq(3*(s**2 * X(s) - s * x0 - xd0) +
               30 * (s * X(s) - x0) +
               63 * X(s),
               sympy.laplace_transform(4 * sympy.diff(g, t) + 6 * g, t, s)[0])
eoms


I am working on the file "LassoRegression.ipynb".
The first task is:
From the lasso regression, the employ rate and the urban rate have been discarded of the explanatory variables list. Then the most influential variables are in order :

Internet use rate
Residential electric consumption per person
Polity score
Life expectancy
Can you write Python code for it?

# plot coefficient progression
plt.semilogx(model.alphas_, model.coef_path_.T)
plt.legend(explanatory_vars)
plt.axvline(model.alpha_, linestyle='--', color='k',
            label='alpha CV')
plt.ylabel('Regression Coefficients')
plt.xlabel(r'$\alpha$')
plt.title('Regression Coefficients Progression for Lasso Paths');


I am working on the file "LassoRegression.ipynb".
The first task is:
Predictors selections and standardization
Can you write Python code for it?

explanatory_vars = ['employrate', 'urbanrate', 'polityscore', 'lifeexpectancy', 'internetuserate', 'relectricperperson']
response_var = 'incomeperperson'

constructor_dict = dict()
for var in explanatory_vars + [response_var, ]:
    constructor_dict[var] = pd.to_numeric(data[var], errors='coerce')

numeric_data = pd.DataFrame(constructor_dict, index=data.index).dropna()

display(Markdown("Number of countries after discarding countries with missing data: {}".format(len(numeric_data))))


I am working on the file "LassoRegression.ipynb".
The first task is:
The plot above showing the evolution of the regression coefficient proves the order of appearance of the four remaining explanatory variables; the internet use rate starting right from the beginning as it is the most infuential.
Can you write Python code for it?

# plot mean square error for each fold
plt.semilogx(model.cv_alphas_, model.cv_mse_path_, ':')
plt.semilogx(model.cv_alphas_, model.cv_mse_path_.mean(axis=-1), 'k',
         label='Average across the folds', linewidth=2)
plt.axvline(model.alpha_, linestyle='--', color='k',
            label='alpha CV')
plt.legend()
plt.xlabel(r'$\alpha$')
plt.ylabel('Mean squared error')
plt.title('Mean squared error on each fold');


I am working on the file "LassoRegression.ipynb".
The first task is:
Unexpectedly one fold have a rising error with the lasso regression analysis progression. A country may have an unusual combination of parameters explaining it.
Can you write Python code for it?

# MSE from training and test data
from sklearn.metrics import mean_squared_error
train_error = mean_squared_error(tar_train, model.predict(pred_train))
test_error = mean_squared_error(tar_test, model.predict(pred_test))

# R-square from training and test data
rsquared_train=model.score(pred_train,tar_train)
rsquared_test=model.score(pred_test,tar_test)

pd.DataFrame({'MSE' : [train_error, test_error], 'R-square' : [rsquared_train, rsquared_test]},
             index=['Training data', 'Test data'])


I am working on the file "LassoVsRidgeRegression.ipynb".
The first task is:
Checking predictive power using the test set:
Can you write Python code for it?

print(lass.score(X_test, Y_test))

print(lassBig.score(X_test2, Y_test))


I am working on the file "LassoVsRidgeRegression.ipynb".
The first task is:
Regularization parameter: Lasso
The $\lambda$ for lasso can var between 0 (no penalty, acts like OLS) and infinity.  If $\lambda$ is too large, all parameters will be set to zero.
Create a plot below of how $R^2$ varies across different values of $\lambda$ for ridge and lasso regression. Use logic and code similar to the ridge regression demonstration above, and base your plot on the X_train2 feature set.
Do lasso and ridge yield the same $R^2$ for a given lambda value?
Submit your work and discuss the results with your mentor.
Can you write Python code for it?

# Suppress annoying harmless error.
import warnings
warnings.filterwarnings(action="ignore")

# Your code here
# WHY DO WE CALL LAMBDA, ALPHA?! THIS IS WORTH YELLING ABOUT. :-)
from sklearn.model_selection import cross_val_score
from sklearn import linear_model

interval = 0.1
alpha_range = np.arange(0,10,interval)

Y = Y_train
X = X_train2

# search for an optimal correlation value for Linear Models
lasso_scores = []
ridge_scores = []
for alphaVal in alpha_range:
    lassBig = linear_model.Lasso(alpha=alphaVal)
    lasso_score = cross_val_score(lassBig, X, Y, cv=10)
    lasso_scores.append(lasso_score.mean())

    ridgeBig = linear_model.Ridge(alpha=alphaVal, fit_intercept=False)
    ridge_score = cross_val_score(ridgeBig, X, Y, cv=10)
    ridge_scores.append(ridge_score.mean())

# plot the performance of Lasso and Ridge Regression against the value of alpha
plt.plot(alpha_range, lasso_scores, label = "Lasso Regression")
plt.plot(alpha_range, ridge_scores, label = "Rid")
plt.xlabel("Value of Alpha")
plt.ylabel("Cross-Validated Regression Score")
plt.legend()
plt.title("Optimal Alpha Values for Lasso and Ridge Regression")
plt.show()


I am working on the file "Latency.ipynb".
The first task is:
Finally, we make the call that will run our function. We use a module called timeit for this purpose. Again, this follows the Python theme of "don't roll your own, when experts have done it for you". If you're that ambitious, it is better to improve their code if you can than start from scratch, everybody wins. timeit is a core module and, supposedly, avoids a number of common function timing pitfalls.
To use it, you supply a routine to test timings for as the first argument, and the number of repeats as the second. Please keep that number argument in there, as the default is 100,000 and I would rather you all didn't hit my server with 100M of socket traffic at the same time.
Another feature of Python has been slipped in there at the same time. You can see a lambda function. This is a very simple construct of the form:
python
lambda arg1, arg2: statement_using(arg1 + arg2)
It is equivalent to
python
def func(arg1, arg2):
    return statement_using(arg1 + arg2)
Basically, anywhere we would use "func", passing it as a callback or whatever, we can swap our anonymous lambda function. In this particular case, timeit will always call the function in the first argument with no arguments, but we need to pass the socket to our routine. How do we solve this? By creating a function that timeit can call with no arguments, but that forwards the call on to round_trip with the skt variable shoehorned in.
Why don't we just name a function and forget about lambda? It is subjective, but here it is likely to confuse our code - what we need is a single line function that gets called with no arguments and calls round_trip with one. If we add another routine called round_trip_caller or something like that, on first glance we will wonder where it is being used and why, it doubles the number of def blocks in our code, and adds a couple of extra source lines that don't really clarify anything that a good comment wouldn't fix.
Hit F5, or go to Run->Run
If you get a dialog saying "Run Settings", choose "Execute in a new dedicated...
Can you write Python code for it?

# Threads are given identifiers as integers 1-N
# This is easier to handle in numpy
threads = [threading.Thread(target=run) for i in range(thread_count)]

# The items method turns a module into pairs (tuples) of key and value
for thread in threads:
    thread.start()

# Wait for all threads to complete
for thread in threads:
    thread.join()


I am working on the file "Latency.ipynb".
The first task is:
First we introduce the concept of a generator. This is a Python construct that embeds a loop in quite a few possible places. Mostly you will see them used to create lists and dicts. The format is (for basic use):
python
[fn(x) for x in iterable] --- e.g. [s.upper() for s in strings]
This takes each item in an iterable, such as list, and does something to it to provide each entry in a new list. The example given goes through a list of, say, strings and uppercases all of them. The whole thing is another (equal size) list. In our case, we are using a bit of short-hand to say we want "thread_count" threads, each element in the list being a new instance of the threading.Thread class.
We then start each thread, so a new parallel run of the run routine heads off. For completeness, we make sure every thread has finished before we reach the final logging statement. The join method blocks returning until that thread has completed.
Can you write Python code for it?

def run():
    # Get the current thread
    currentThread = threading.currentThread()

    # Send out a message
    logging.info("Hi, my name is {name}".format(
        name=currentThread.getName()
    ))


I am working on the file "Latency.ipynb".
The first task is:
Here we add some simple text to the run routine. threading provides a handy currentThread function that each thread can call to get itself. When it does, we can get a name for it. We use a new method on the string class here: format. This is allows us to name fields in the string and is actually the recommended approach. The values themselves are passed as named arguments to format.
Try running again - you should get a "Hello" from every thread.
Unfortunately, we cannot rely on thread names to be unique and, while all threads get a unique ID, it's horrendously un-user-friendly. As such we will name our threads from 1 up to thread_count
Can you write Python code for it?

# Generate some simple numeric way to refer to these
thread_indices = range(1, thread_count + 1)
threads = {i: threading.Thread(target=run) for i in thread_indices}

# The items method turns a module into pairs (tuples) of key and value
for idx, thread in threads.items():
    thread.index = idx
    thread.start()

# Wait for all threads to complete
for thread in threads.values():
    thread.join()


I am working on the file "Latency.ipynb".
The first task is:
I mentioned earlier that try-except is your friend. Bear in mind, if you highlight several lines in Spyder, Tab will indent them - Shift+Tab deindents.
If we get a socket error from anywhere inside the with (including our test routine from earlier), it will get caught here. In practice this is only being used to inject an extra logging line, but it illustrates the point. Note that we aren't catching the IOError I mentioned earlier. To do so, you can add an extra except clause, or turn "OSError as e" into "(IOError, OSError) as e", if we are happy to use the same one. Note that you do need the tuple parens.
Can you write Python code for it?

        logger.info("Created connection")
        # This is going to add a bit of misleading overhead, but for this
        # purpose we'll use lambda for simplicity
        return_time = timeit.timeit(
            lambda: round_trip(skt),
            number=repeats
        )
        logger.info("Completed trial")


I am working on the file "Latency.ipynb".
The first task is:
This is the approach I used. I told you not to worry about atomic operations (Python, for built-ins, actually looks after this itself). However, to illustrate, I have created a lock object, along with the result list, just before the thread_indices assignment
Can you write Python code for it?

    # sockets over-arching try-catch here

    # Strictly, a lock isn't required for accessing a dict, but this is an
    # opportunity to demonstrate the use of locks
    with lock:
        results.append((currentThread.index, average_return_time))

    logger.info("Average time taken: {delay} s".format(delay=average_return_time))


I am working on the file "Latency.ipynb".
The first task is:
We define this as a function that takes a TCP socket. It creates 1024 bytes of random data (this is the crypto-quality generator - overkill but useful for you to be aware of). We send our payload off down the wire, and expect to see it arrive here. Any exceptions will get thrown straight through for simplicity, but we will catch them later.
We even throw our own exception, a common superclass for most IO exceptions, IOError when there is a problem with the payload. It is maybe helpful to subclass IOError (or a more appropriate error class) to create more specific exceptions - see this link for a tutorial. Remember that the decision in an except statement, whether to handle an exception or pass it on it, is based on the class. Creating your own allows you to grab it and it only in a try-except.
Can you write Python code for it?

# Use a `with` context to make sure the socket automatically
# gets cleaned up
with socket.create_connection(address=(host, port), timeout=timeout) as skt:
    logger.info("Created connection")

    logger.info("Completed trial")


I am working on the file "Latent_Me.ipynb".
The first task is:
Add your image(s)
Upload your image using the Sidebar now.
To open the sidebar select "View" and then "Table of Contents".
The sidebar should now be open. Click the "Files" tab.
Before uploading make sure:

The image(s) you're using can be opened by PIL (jpg, png, etc)
The images are larger than 1024x1024. Preferably significantly larger so the aligner can crop out a high resolution section of the image containing your face.
Your face in the image is well lit and facing the camera (for best results)

Click ''Upload" in the sidebar and select the images you want to upload from your computer.
Note: All files uploaded in this manner end up in the root of the file tree. We'll move them into the correct spot next.
Can you write Python code for it?

# e.g. mv ../me.jpg raw_images/
!mv ../<YOUR UPLOAD NAME> raw_images/


I am working on the file "Latent_Me.ipynb".
The first task is:
Age transformation
Can you write Python code for it?

move_and_show(me, age_direction, [-1, 0, 1])